WEBVTT

99:59:59.999 --> 99:59:59.999
- [Narrator] On his quest 
to master econometrics,

99:59:59.999 --> 99:59:59.999
Grasshopper Kamal has 
made great progress

99:59:59.999 --> 99:59:59.999
stretching his capabilities 
and outsmarting his foes.

99:59:59.999 --> 99:59:59.999
Alas, today he's despondent.

99:59:59.999 --> 99:59:59.999
For one challenge remains unmet.

99:59:59.999 --> 99:59:59.999
Kamal cannot yet decode 
the scriptures of academic research,

99:59:59.999 --> 99:59:59.999
journals like 
"The American Economic Review"

99:59:59.999 --> 99:59:59.999
and "Econometrica."

99:59:59.999 --> 99:59:59.999
These seemed to him to be inscribed 
in an obscure foreign tongue.

99:59:59.999 --> 99:59:59.999
- [Kamal] Ugh, what the... ?

99:59:59.999 --> 99:59:59.999
- [Narrator] These volumes are 
opaque to novice, Kamal,

99:59:59.999 --> 99:59:59.999
but can be deciphered with study.

99:59:59.999 --> 99:59:59.999
Let us learn to read them together.

99:59:59.999 --> 99:59:59.999
Let's dive into the West Point study,

99:59:59.999 --> 99:59:59.999
published in the "Economics 
of Education Review."

99:59:59.999 --> 99:59:59.999
This paper reports 
on a randomized evaluation

99:59:59.999 --> 99:59:59.999
of student electronics use 
in Economics 101 classrooms.

99:59:59.999 --> 99:59:59.999
First, a quick review 
of the research design.

99:59:59.999 --> 99:59:59.999
- Okay.

99:59:59.999 --> 99:59:59.999
- [Josh] 'Metrics masters 
teaching at West Point,

99:59:59.999 --> 99:59:59.999
the military college that trains 
American Army officers

99:59:59.999 --> 99:59:59.999
designed a randomized trial
to answer this question.

99:59:59.999 --> 99:59:59.999
These masters randomly assigned
West Point cadets

99:59:59.999 --> 99:59:59.999
into Economics classes
operating under different rules.

99:59:59.999 --> 99:59:59.999
Unlike most American colleges,

99:59:59.999 --> 99:59:59.999
the West Point default 
is no electronics.

99:59:59.999 --> 99:59:59.999
For purposes of this experiment,
some students were left

99:59:59.999 --> 99:59:59.999
in such traditional 
technology-free classes,

99:59:59.999 --> 99:59:59.999
no lap tops, no tablets 
and no phones!

99:59:59.999 --> 99:59:59.999
[voice echoes]

99:59:59.999 --> 99:59:59.999
This is the control group,
or baseline case.

99:59:59.999 --> 99:59:59.999
Another group was allowed
to use electronics.

99:59:59.999 --> 99:59:59.999
This is the treatment group,
subject to a changed environment.

99:59:59.999 --> 99:59:59.999
The treatment in this case
is the unrestricted use

99:59:59.999 --> 99:59:59.999
of laptops or tablets in class.

99:59:59.999 --> 99:59:59.999
Every causal question 
has a clear outcome,

99:59:59.999 --> 99:59:59.999
the variables we hope to influence
defined in advance of the study.

99:59:59.999 --> 99:59:59.999
The outcomes in the West Point
electronics study

99:59:59.999 --> 99:59:59.999
are final exam scores.

99:59:59.999 --> 99:59:59.999
The study seeks to answer
the following question,

99:59:59.999 --> 99:59:59.999
what is the causal effect 
of classroom electronics on learning

99:59:59.999 --> 99:59:59.999
as measured by exam scores?

99:59:59.999 --> 99:59:59.999
- Economics journal articles 
usually begin with a table

99:59:59.999 --> 99:59:59.999
of descriptive statistics,
giving key facts

99:59:59.999 --> 99:59:59.999
about the study sample.

99:59:59.999 --> 99:59:59.999
- Oh my gosh, I remember this table,
so confusing!

99:59:59.999 --> 99:59:59.999
Columns 1 to 3 report mean,
or average, characteristics.

99:59:59.999 --> 99:59:59.999
These give a sense 
of who we're studying.

99:59:59.999 --> 99:59:59.999
Let's start with column 1 
which describes covariates

99:59:59.999 --> 99:59:59.999
in a control group.

99:59:59.999 --> 99:59:59.999
Covariates are characteristics 
of the control and treatment groups

99:59:59.999 --> 99:59:59.999
measured before 
the experiment begins.

99:59:59.999 --> 99:59:59.999
For example, we see the control group
has an average age a bit over 20.

99:59:59.999 --> 99:59:59.999
Many of these covariates 
are dummy variables.

99:59:59.999 --> 99:59:59.999
A dummy variable can only have 
two values, a zero or a one.

99:59:59.999 --> 99:59:59.999
For example, student gender
is captured by a dummy variable

99:59:59.999 --> 99:59:59.999
that equals one for woman
and zero for man.

99:59:59.999 --> 99:59:59.999
The mean of this variable 
is the proportion female.

99:59:59.999 --> 99:59:59.999
We also see that the control group 
is 13% Hispanic

99:59:59.999 --> 99:59:59.999
and 19% had prior military service.

99:59:59.999 --> 99:59:59.999
The table notes are key.

99:59:59.999 --> 99:59:59.999
Refer to these 
as you scan the table.

99:59:59.999 --> 99:59:59.999
These notes explain what's shown 
in each column and panel.

99:59:59.999 --> 99:59:59.999
The notes tell us, for example,
that standard deviations

99:59:59.999 --> 99:59:59.999
are reported in brackets.

99:59:59.999 --> 99:59:59.999
Standard deviations tell us how
spread out the data are.

99:59:59.999 --> 99:59:59.999
For example, a standard deviation 
of 0.52 tells us that most

99:59:59.999 --> 99:59:59.999
of the control group's GPAs
fall between 2.35,

99:59:59.999 --> 99:59:59.999
which is 0.52 below 
the mean GPA of 2.87,

99:59:59.999 --> 99:59:59.999
and 3.39, which is 0.52 above 2.87.

99:59:59.999 --> 99:59:59.999
A lower standard deviation
would mean the GPAs were

99:59:59.999 --> 99:59:59.999
more tightly clustered 
around the mean.

99:59:59.999 --> 99:59:59.999
- [Kamal] Yeah, but they're missing
from most of the variables.

99:59:59.999 --> 99:59:59.999
- [Narrator] That's right.

99:59:59.999 --> 99:59:59.999
Masters usually omit 
standard deviations for dummies

99:59:59.999 --> 99:59:59.999
because the mean of this variable
determines its standard deviaton.

99:59:59.999 --> 99:59:59.999
This study compares two treatment 
groups with the control group.

99:59:59.999 --> 99:59:59.999
The first was allowed free use 
of laptops and tablets.

99:59:59.999 --> 99:59:59.999
The second treatment
was more restrictive,

99:59:59.999 --> 99:59:59.999
allowing only tablets placed 
flat on the desk.

99:59:59.999 --> 99:59:59.999
The treatment groups
look much like the control group.

99:59:59.999 --> 99:59:59.999
Hmmm...

99:59:59.999 --> 99:59:59.999
This takes us to the next feature 
of this table, columns 4 through 6

99:59:59.999 --> 99:59:59.999
use statistical tests to compare
the characteristics

99:59:59.999 --> 99:59:59.999
of the treatment and control group
before the experiment.

99:59:59.999 --> 99:59:59.999
In column 4, the two treatment 
groups are combined.

99:59:59.999 --> 99:59:59.999
You can see that the difference
in proportion female

99:59:59.999 --> 99:59:59.999
between the treatment 
and control group is only 0.03.

99:59:59.999 --> 99:59:59.999
The difference is not 
statistically significant.

99:59:59.999 --> 99:59:59.999
It is the sort of difference
we can easily put down

99:59:59.999 --> 99:59:59.999
to chance results 
in our sample selection process.

99:59:59.999 --> 99:59:59.999
- [Kamal] Hmm, how do we know that?

99:59:59.999 --> 99:59:59.999
- [Narrator] Remember 
the rule of thumb?

99:59:59.999 --> 99:59:59.999
Statistical estimates 
that exceed the standard

99:59:59.999 --> 99:59:59.999
by a multiple of 2 
in absolute value

99:59:59.999 --> 99:59:59.999
are usually said
to be statistically significant.

99:59:59.999 --> 99:59:59.999
The standard error is 0.03,

99:59:59.999 --> 99:59:59.999
same as the difference 
in proportion female.

99:59:59.999 --> 99:59:59.999
So the ratio of the latter 
to the former is only 1,

99:59:59.999 --> 99:59:59.999
which of course is less than 2.

99:59:59.999 --> 99:59:59.999
- [Kamal] Uh huh. So none
of the treatment/control differences

99:59:59.999 --> 99:59:59.999
in the table are more than twice
their standard errors.

99:59:59.999 --> 99:59:59.999
- [Narrator] Correct.

99:59:59.999 --> 99:59:59.999
The random division of students
appears to have succeeded

99:59:59.999 --> 99:59:59.999
in creating groups 
that are indeed comparable.

99:59:59.999 --> 99:59:59.999
We can be confident therefore
that any later differences

99:59:59.999 --> 99:59:59.999
in classroom achievement
are the result of the experimental

99:59:59.999 --> 99:59:59.999
intervention rather 
than a reflection

99:59:59.999 --> 99:59:59.999
of preexisting differences.

99:59:59.999 --> 99:59:59.999
Ceteris paribus achieved!

99:59:59.999 --> 99:59:59.999
- [Kamal] Cool. Wait, 
what about the bottom,

99:59:59.999 --> 99:59:59.999
the numbers with the stars?

99:59:59.999 --> 99:59:59.999
Those differences are a lot more
than double the standard error.

99:59:59.999 --> 99:59:59.999
- [Narrator] Good eye, Kamal!

99:59:59.999 --> 99:59:59.999
The table has many numbers.

99:59:59.999 --> 99:59:59.999
Those in Panel B are important too.

99:59:59.999 --> 99:59:59.999
This panel measures the extent 
to which students in treatment

99:59:59.999 --> 99:59:59.999
and control groups actually use 
computers in class.