- [Narrator] On his quest to master econometrics,
Grasshopper Kamal has made great progress
stretching his capabilities and outsmarting his foes.
Alas, today he's despondent.
For one challenge remains unmet.
Kamal cannot yet decode the scriptures of academic research,
journals like "The American Economic Review"
and "Econometrica."
These seemed to him to be inscribed in an obscure foreign tongue.
- [Kamal] Ugh, what the... ?
- [Narrator] These volumes are opaque to novice, Kamal,
but can be deciphered with study.
Let us learn to read them together.
Let's dive into the West Point study,
published in the "Economics of Education Review."
This paper reports on a randomized evaluation
of student electronics use in Economics 101 classrooms.
First, a quick review of the research design.
- Okay.
- [Josh] 'Metrics masters teaching at West Point,
the military college that trains American Army officers
designed a randomized trial to answer this question.
These masters randomly assigned West Point cadets
into Economics classes operating under different rules.
Unlike most American colleges,
the West Point default is no electronics.
For purposes of this experiment, some students were left
in such traditional technology-free classes,
no lap tops, no tablets and no phones!
[voice echoes]
This is the control group, or baseline case.
Another group was allowed to use electronics.
This is the treatment group, subject to a changed environment.
The treatment in this case is the unrestricted use
of laptops or tablets in class.
Every causal question has a clear outcome,
the variables we hope to influence defined in advance of the study.
The outcomes in the West Point electronics study
are final exam scores.
The study seeks to answer the following question,
what is the causal effect of classroom electronics on learning
as measured by exam scores?
- Economics journal articles usually begin with a table
of descriptive statistics, giving key facts
about the study sample.
- Oh my gosh, I remember this table, so confusing!
Columns 1 to 3 report mean, or average, characteristics.
These give a sense of who we're studying.
Let's start with column 1 which describes covariates
in a control group.
Covariates are characteristics of the control and treatment groups
measured before the experiment begins.
For example, we see the control group has an average age a bit over 20.
Many of these covariates are dummy variables.
A dummy variable can only have two values, a zero or a one.
For example, student gender is captured by a dummy variable
that equals one for woman and zero for man.
The mean of this variable is the proportion female.
We also see that the control group is 13% Hispanic
and 19% had prior military service.
The table notes are key.
Refer to these as you scan the table.
These notes explain what's shown in each column and panel.
The notes tell us, for example, that standard deviations
are reported in brackets.
Standard deviations tell us how spread out the data are.
For example, a standard deviation of 0.52 tells us that most
of the control group's GPAs fall between 2.35,
which is 0.52 below the mean GPA of 2.87,
and 3.39, which is 0.52 above 2.87.
A lower standard deviation would mean the GPAs were
more tightly clustered around the mean.
- [Kamal] Yeah, but they're missing from most of the variables.
- [Narrator] That's right.
Masters usually omit standard deviations for dummies
because the mean of this variable determines its standard deviaton.
This study compares two treatment groups with the control group.
The first was allowed free use of laptops and tablets.
The second treatment was more restrictive,
allowing only tablets placed flat on the desk.
The treatment groups look much like the control group.
Hmmm...
This takes us to the next feature of this table, columns 4 through 6
use statistical tests to compare the characteristics
of the treatment and control group before the experiment.
In column 4, the two treatment groups are combined.
You can see that the difference in proportion female
between the treatment and control group is only 0.03.
The difference is not statistically significant.
It is the sort of difference we can easily put down
to chance results in our sample selection process.
- [Kamal] Hmm, how do we know that?
- [Narrator] Remember the rule of thumb?
Statistical estimates that exceed the standard
by a multiple of 2 in absolute value
are usually said to be statistically significant.
The standard error is 0.03,
same as the difference in proportion female.
So the ratio of the latter to the former is only 1,
which of course is less than 2.
- [Kamal] Uh huh. So none of the treatment/control differences
in the table are more than twice their standard errors.
- [Narrator] Correct.
The random division of students appears to have succeeded
in creating groups that are indeed comparable.
We can be confident therefore that any later differences
in classroom achievement are the result of the experimental
intervention rather than a reflection
of preexisting differences.
Ceteris paribus achieved!
- [Kamal] Cool. Wait, what about the bottom,
the numbers with the stars?
Those differences are a lot more than double the standard error.
- [Narrator] Good eye, Kamal!
The table has many numbers.
Those in Panel B are important too.
This panel measures the extent to which students in treatment
and control groups actually use computers in class.