-
- [Narrator] On his quest
to master econometrics,
-
Grasshopper Kamal has
made great progress,
-
stretching his capabilities
and outsmarting his foes.
-
Alas, today he's despondent,
-
for one challenge remains unmet.
-
Kamal cannot yet decode
the scriptures of academic research,
-
journals like
"The American Economic Review"
-
and "Econometrica."
-
These seemed to him to be inscribed
in an obscure foreign tongue.
-
- [Kamal] Ugh, what the... ?
-
- These volumes are
opaque to the novice, Kamal,
-
but can be deciphered with study.
-
Let us learn to read them together.
-
Let's dive into the West Point study,
-
published in the "Economics
of Education Review."
-
This paper reports
on a randomized evaluation
-
of student electronics use
in Economics 101 classrooms.
-
First, a quick review
of the research design.
-
- Okay.
-
- [Josh] 'Metrics masters
teaching at West Point,
-
the military college that trains
American Army officers
-
designed a randomized trial
to answer this question.
-
These masters randomly assigned
West Point cadets
-
into Economics classes
operating under different rules.
-
Unlike most American colleges,
-
the West Point default
is no electronics.
-
For purposes of this experiment,
some students were left
-
in such traditional
technology-free classes,
-
no laptops, no tablets
and no phones!
-
[voice echoes]
-
This is the control group,
or baseline case.
-
Another group was allowed
to use electronics.
-
This is the treatment group,
subject to a changed environment.
-
The treatment in this case
is the unrestricted use
-
of laptops or tablets in class.
-
Every causal question
has a clear outcome,
-
the variables we hope to influence
defined in advance of the study.
-
The outcomes in the West Point
electronics study
-
are final exam scores.
-
The study seeks to answer
the following question,
-
what is the causal effect
of classroom electronics on learning
-
as measured by exam scores?
-
- Economics journal articles
usually begin with a table
-
of descriptive statistics,
giving key facts
-
about the study sample.
-
- Oh my gosh, I remember this table,
so confusing!
-
- [Narrator] Columns 1 to 3 report
mean, or average, characteristics.
-
These give a sense
of who we're studying.
-
Let's start with column 1
which describes covariates
-
in the control group.
-
Covariates are characteristics
of the control and treatment groups
-
measured before
the experiment begins.
-
For example, we see the control group
has an average age a bit over 20.
-
Many of these covariates
are dummy variables.
-
A dummy variable can only have
two values, a zero or a one.
-
For example, student gender
is captured by a dummy variable
-
that equals one for women
and zero for men.
-
The mean of this variable
is the proportion female.
-
We also see that the control group
is 13% Hispanic
-
and 19% had prior military service.
-
The table notes are key.
-
Refer to these
as you scan the table.
-
These notes explain what's shown
in each column and panel.
-
The notes tell us, for example,
that standard deviations
-
are reported in brackets.
-
Standard deviations tell us how
spread out the data are.
-
For example, a standard deviation
of 0.52 tells us that most
-
of the control group's GPAs
fall between 2.35,
-
which is 0.52 below
the mean GPA of 2.87,
-
and 3.39, which is 0.52 above 2.87.
-
A lower standard deviation
would mean the GPAs were
-
more tightly clustered
around the mean.
-
- [Kamal] Yeah, but they're missing
for most of the variables.
-
- [Narrator] That's right.
-
Masters usually omit
standard deviations for dummies
-
because the mean of this variable
determines its standard deviation.
-
This study compares two treatment
groups with the control group.
-
The first was allowed free use
of laptops and tablets.
-
The second treatment
was more restrictive,
-
allowing only tablets placed
flat on the desk.
-
The treatment groups
look much like the control group.
-
This takes us to the next feature
of this table, columns 4 through 6
-
use statistical tests to compare
the characteristics
-
of the treatment and control group
before the experiment.
-
In column 4, the two treatment
groups are combined.
-
You can see that the difference
in proportion female
-
between the treatment
and control group is only 0.03.
-
The difference is not
statistically significant.
-
It is the sort of difference
we can easily put down
-
to chance results
in our sample selection process.
-
- [Kamal] Hmm, how do we know that?
-
- [Narrator] Remember
the rule of thumb?
-
Statistical estimates
that exceed the standard error
-
by a multiple of 2
in absolute value
-
are usually said
to be statistically significant.
-
The standard error is 0.03,
-
same as the difference
in proportion female.
-
So the ratio of the latter
to the former is only 1,
-
which of course is less than 2.
-
- [Kamal] Uh huh. So none
of the treatment/control differences
-
in the table are more than twice
their standard errors.
-
- [Narrator] Correct.
-
The random division of students
appears to have succeeded
-
in creating groups
that are indeed comparable.
-
We can be confident therefore
that any later differences
-
in classroom achievement
are the result of the experimental
-
intervention rather
than a reflection
-
of preexisting differences.
-
Ceteris paribus achieved!
-
- [Kamal] Cool. Wait,
what about the bottom,
-
the numbers with the stars?
-
Those differences are a lot more
than double the standard error.
-
- [Narrator] Good eye, Kamal!
-
The table has many numbers.
-
Those in Panel B are important too.
-
This panel measures the extent
to which students in treatment
-
and control groups actually use
computers in class.
-
The treatment here was
to allow computer use.
-
The researchers must show
that students allowed
-
to use computers took advantage
of the opportunity to do so.
-
If they didn't, then there's
really no treatment.
-
Luckily, 81% of those
in the first treatment group
-
used computers compared
with none in the control group.
-
And many in the second
tablet treatment group
-
used computers as well.
-
These differences
in computer use are large
-
and statistically significant.
-
We also get to see
the sample size in each group.
-
- [Kamal] The stars
are just like decoration?
-
- [Narrator] Some academic papers
use stars to indicate differences
-
that are statistically significant.
-
This makes them jump out at you.
-
Here three stars indicate that
the result is statistically different
-
from zero with a p value
less than 1%.
-
In other words, there's less
than a 1 in 100 chance
-
this result is purely
a chance finding.
-
[applause]
-
Two stars indicate a 1 in 20
or 5% chance of a chance finding.
-
And one star denotes results
we might see as often as 10%
-
of the time merely due to chance.
-
Today, stars are seen
as a little old fashioned.
-
Some journals omit them.
-
- [Kamal] What about
those last two columns?
-
- [Narrator] Unlike column 4,
which combines
-
both treatment groups into one,
these last two columns
-
look separately
at treatment/control differences
-
for each treatment group.
-
This provides a more detailed
analysis of balance.
-
Also, for now,
you can ignore this row
-
which provides
another test of significance.
-
Now we get to the article's
punchline, table 4.
-
This table reports
regression estimates
-
of the effects of electronics use
on measures of student learning.
-
- [Kamal Why does the study
report regression estimates?
-
See, that's why I was getting lost.
-
I thought one reason
why we liked randomized trials
-
is that we use them
to obtain causal effects
-
simply by comparing
treatment and control groups.
-
Since these groups are balanced,
no need to use regression.
-
- [Narrator] Well said, Kamal.
-
Not Synced
In practice, it's customary
to report regression estimates
-
Not Synced
for two reasons.
-
Not Synced
First, evidence of balance
not withstanding, an abundance
-
Not Synced
of caution might lead the analyst
to allow for chance differences.
-
Not Synced
Second, regression estimates
are likely to be more precise.
-
Not Synced
That is, they have lower
standard errors than
-
Not Synced
the simple treatment
control comparisons.
-
Not Synced
The dependent variable
in this study
-
Not Synced
is the outcome of interest.
-
Not Synced
Since the question at hand
is how classroom electronics
-
Not Synced
affect learning, a good outcome
is the economics final exam score.
-
Not Synced
Each column reports results
from a different regression model.
-
Not Synced
Models are distinguished
by the control variables
-
Not Synced
or covariates they include
besides treatment status.
-
Not Synced
Estimates with no covariates
are simple comparisons
-
Not Synced
of treatment and control groups.
-
Not Synced
- [Kamal] I thought
they just forgot to fill it out.
-
Not Synced
- [Narrator] Column 1 suggests
electronics use reduced
-
Not Synced
final exam scores
by 0.28 standard deviations.
-
Not Synced
In our last lesson, Master Joshway
explained, we use standard deviation
-
Not Synced
units because these units
are easily compared across studies.
-
Not Synced
Column 2 reports results
from a model
-
Not Synced
that adds demographic controls.
-
Not Synced
Here we're comparing test scores
but holding constant factors
-
Not Synced
such as age and sex.
-
Not Synced
Column 3 reports results
from a model that adds GPA
-
Not Synced
to the list of covariates.
-
Not Synced
Column 4 adds ACT scores.
-
Not Synced
Analysts often report
results this way,
-
Not Synced
starting with models that include
few or no covariates
-
Not Synced
and then reporting estimates
from models that add more
-
Not Synced
and more covariates
as we move across columns.
-
Not Synced
Looking across columns,
what do you notice?
-
Not Synced
- [Kamal] Well, the coefficient
on using a computer is always
-
Not Synced
a pretty big negative number.
-
Not Synced
- [Narrator] That's right!
-
Not Synced
We can also see that
the standard errors are small enough
-
Not Synced
to make these negative results
statistically significant.
-
Not Synced
In other words, the primary
takeaway from this experiment
-
Not Synced
is that electronics in the classroom
reduce student learning.
-
Not Synced
- [Kama] GPA and ACT scores
are also significant.
-
Not Synced
Why is that?
-
Not Synced
- [Narrator] Good observation!
-
Not Synced
That's not surprising.
-
Not Synced
We expect these variables
to predict college performance.
-
Not Synced
- [Kamal] Oh right, of course.
-
Not Synced
Kids who got better grades before
are more likely to get
-
Not Synced
a better grade in this course.
-
Not Synced
- [Narrator] You'll also notice a lot
of other information on this table.
-
Not Synced
Remaining panels in the table
report effects of electronics use
-
Not Synced
on components of the final exam,
-
Not Synced
such as the multiple
choice questions.
-
Not Synced
These results are mostly consistent
with computer use effects
-
Not Synced
on overall scores.
-
Not Synced
- [Kamal] What about the rows
not in English?
-
Not Synced
- [Narrator] These rows give
additional statistical information.
-
Not Synced
R-squared is a measure
of goodness of fit.
-
Not Synced
This isn't too important, though
some readers may want to know it.
-
Not Synced
Other rows report on alternative
tests of statistical significance
-
Not Synced
that you can ignore for now.
-
Not Synced
- [Kamal] Oh my gosh,
these tables aren't that hard.
-
Not Synced
Thank you so much.
-
Not Synced
Next up is regression.
-
Not Synced
See you then!
-
Not Synced
♪ [music] ♪
-
Not Synced
You're on your way
to mastering econometrics.
-
Not Synced
Make sure this video sticks
-
Not Synced
by taking a few
quick practice questions.
-
Not Synced
Or, if you're ready,
click for the next video.
-
Not Synced
You can also check out MRU's
website for more courses,
-
Not Synced
teacher resources and more.