WEBVTT

00:00:01.425 --> 00:00:05.060
- [Narrator] On his quest 
to master econometrics,

00:00:05.223 --> 00:00:08.813
Grasshopper Kamal has 
made great progress,

00:00:08.813 --> 00:00:13.662
stretching his capabilities 
and outsmarting his foes.

00:00:14.223 --> 00:00:16.640
Alas, today he's despondent,

00:00:16.640 --> 00:00:19.336
for one challenge remains unmet.

00:00:19.336 --> 00:00:24.130
Kamal cannot yet decode 
the scriptures of academic research,

00:00:24.130 --> 00:00:27.347
journals like 
"The American Economic Review"

00:00:27.347 --> 00:00:29.080
and "Econometrica."

00:00:29.202 --> 00:00:33.501
These seemed to him to be inscribed 
in an obscure foreign tongue.

00:00:33.501 --> 00:00:35.478
- [Kamal] Ugh, what the... ?

00:00:36.711 --> 00:00:40.018
- These volumes are 
opaque to the novice, Kamal,

00:00:40.018 --> 00:00:42.205
but can be deciphered with study.

00:00:42.467 --> 00:00:45.109
Let us learn to read them together.

00:00:52.485 --> 00:00:55.317
Let's dive into the West Point study,

00:00:55.317 --> 00:00:58.278
published in the "Economics 
of Education Review."

00:00:58.538 --> 00:01:01.688
This paper reports 
on a randomized evaluation

00:01:01.688 --> 00:01:05.859
of student electronics use 
in Economics 101 classrooms.

00:01:06.242 --> 00:01:09.192
First, a quick review 
of the research design.

00:01:09.423 --> 00:01:10.523
- Okay.

00:01:11.553 --> 00:01:13.630
- [Josh] 'Metrics masters 
teaching at West Point,

00:01:13.630 --> 00:01:16.620
the military college that trains 
American Army officers

00:01:16.620 --> 00:01:19.854
designed a randomized trial
to answer this question.

00:01:20.372 --> 00:01:23.233
These masters randomly assigned
West Point cadets

00:01:23.233 --> 00:01:26.383
into Economics classes
operating under different rules.

00:01:26.595 --> 00:01:28.962
Unlike most American colleges,

00:01:28.962 --> 00:01:31.945
the West Point default 
is no electronics.

00:01:32.345 --> 00:01:35.428
For purposes of this experiment,
some students were left

00:01:35.428 --> 00:01:38.679
in such traditional 
technology-free classes,

00:01:38.679 --> 00:01:41.911
no laptops, no tablets 
and no phones!

00:01:41.911 --> 00:01:43.280
[voice echoes]

00:01:43.328 --> 00:01:45.743
This is the control group,
or baseline case.

00:01:46.213 --> 00:01:49.198
Another group was allowed
to use electronics.

00:01:49.269 --> 00:01:52.704
This is the treatment group,
subject to a changed environment.

00:01:53.313 --> 00:01:55.858
The treatment in this case
is the unrestricted use

00:01:55.858 --> 00:01:58.107
of laptops or tablets in class.

00:01:58.844 --> 00:02:01.812
Every causal question 
has a clear outcome,

00:02:01.859 --> 00:02:05.276
the variables we hope to influence
defined in advance of the study.

00:02:05.860 --> 00:02:08.375
The outcomes in the West Point
electronics study

00:02:08.375 --> 00:02:09.994
are final exam scores.

00:02:10.047 --> 00:02:13.364
The study seeks to answer
the following question,

00:02:13.364 --> 00:02:17.299
what is the causal effect 
of classroom electronics on learning

00:02:17.299 --> 00:02:19.765
as measured by exam scores?

00:02:20.632 --> 00:02:24.199
- Economics journal articles 
usually begin with a table

00:02:24.199 --> 00:02:26.933
of descriptive statistics,
giving key facts

00:02:26.933 --> 00:02:28.500
about the study sample.

00:02:28.500 --> 00:02:31.781
- Oh my gosh, I remember this table,
so confusing!

00:02:31.781 --> 00:02:36.666
- [Narrator] Columns 1 to 3 report 
mean, or average, characteristics.

00:02:36.736 --> 00:02:39.688
These give a sense 
of who we're studying.

00:02:39.948 --> 00:02:43.736
Let's start with column 1 
which describes covariates

00:02:43.736 --> 00:02:45.251
in the control group.

00:02:45.408 --> 00:02:48.972
Covariates are characteristics 
of the control and treatment groups

00:02:48.972 --> 00:02:51.621
measured before 
the experiment begins.

00:02:51.621 --> 00:02:56.986
For example, we see the control group
has an average age a bit over 20.

00:02:57.321 --> 00:03:00.339
Many of these covariates 
are dummy variables.

00:03:00.790 --> 00:03:05.670
A dummy variable can only have 
two values, a zero or a one.

00:03:05.981 --> 00:03:10.015
For example, student gender
is captured by a dummy variable

00:03:10.015 --> 00:03:13.148
that equals one for women
and zero for men.

00:03:13.248 --> 00:03:16.580
The mean of this variable 
is the proportion female.

00:03:16.815 --> 00:03:20.651
We also see that the control group 
is 13% Hispanic

00:03:20.651 --> 00:03:23.769
and 19% had prior military service.

00:03:25.035 --> 00:03:26.635
The table notes are key.

00:03:26.635 --> 00:03:28.686
Refer to these 
as you scan the table.

00:03:29.102 --> 00:03:33.369
These notes explain what's shown 
in each column and panel.

00:03:39.485 --> 00:03:43.375
The notes tell us, for example,
that standard deviations

00:03:43.375 --> 00:03:45.175
are reported in brackets.

00:03:45.947 --> 00:03:49.598
Standard deviations tell us how
spread out the data are.

00:03:50.282 --> 00:03:54.887
For example, a standard deviation 
of 0.52 tells us that most

00:03:54.887 --> 00:03:59.233
of the control group's GPAs
fall between 2.35,

00:03:59.233 --> 00:04:03.454
which is 0.52 below 
the mean GPA of 2.87,

00:04:03.454 --> 00:04:08.337
and 3.39, which is 0.52 above 2.87.

00:04:09.001 --> 00:04:12.221
A lower standard deviation
would mean the GPAs were

00:04:12.221 --> 00:04:14.404
more tightly clustered 
around the mean.

00:04:14.549 --> 00:04:17.451
- [Kamal] Yeah, but they're missing
for most of the variables.

00:04:17.499 --> 00:04:18.600
- [Narrator] That's right.

00:04:18.600 --> 00:04:22.497
Masters usually omit 
standard deviations for dummies

00:04:22.497 --> 00:04:26.500
because the mean of this variable
determines its standard deviation.

00:04:26.936 --> 00:04:31.370
This study compares two treatment 
groups with the control group.

00:04:31.370 --> 00:04:35.753
The first was allowed free use 
of laptops and tablets.

00:04:35.753 --> 00:04:38.252
The second treatment
was more restrictive,

00:04:38.252 --> 00:04:41.553
allowing only tablets placed 
flat on the desk.

00:04:42.152 --> 00:04:45.238
The treatment groups
look much like the control group.

00:04:46.310 --> 00:04:51.270
This takes us to the next feature 
of this table, columns 4 through 6

00:04:51.407 --> 00:04:54.558
use statistical tests to compare
the characteristics

00:04:54.558 --> 00:04:57.591
of the treatment and control group
before the experiment.

00:04:58.023 --> 00:05:01.674
In column 4, the two treatment 
groups are combined.

00:05:01.856 --> 00:05:04.840
You can see that the difference
in proportion female

00:05:04.840 --> 00:05:09.690
between the treatment 
and control group is only 0.03.

00:05:09.991 --> 00:05:13.740
The difference is not 
statistically significant.

00:05:14.290 --> 00:05:17.205
It is the sort of difference
we can easily put down

00:05:17.205 --> 00:05:20.497
to chance results 
in our sample selection process.

00:05:20.497 --> 00:05:22.133
- [Kamal] Hmm, how do we know that?

00:05:22.133 --> 00:05:23.790
- [Narrator] Remember 
the rule of thumb?

00:05:23.790 --> 00:05:26.968
Statistical estimates 
that exceed the standard error

00:05:26.968 --> 00:05:29.882
by a multiple of 2 
in absolute value

00:05:30.105 --> 00:05:33.997
are usually said
to be statistically significant.

00:05:35.132 --> 00:05:38.419
The standard error is 0.03,

00:05:38.566 --> 00:05:41.483
same as the difference 
in proportion female.

00:05:42.015 --> 00:05:46.041
So the ratio of the latter 
to the former is only 1,

00:05:46.041 --> 00:05:48.607
which of course is less than 2.

00:05:48.607 --> 00:05:51.191
- [Kamal] Uh huh. So none
of the treatment/control differences

00:05:51.191 --> 00:05:54.333
in the table are more than twice
their standard errors.

00:05:54.333 --> 00:05:55.789
- [Narrator] Correct.

00:05:55.789 --> 00:05:59.000
The random division of students
appears to have succeeded

00:05:59.014 --> 00:06:01.945
in creating groups 
that are indeed comparable.

00:06:02.846 --> 00:06:06.362
We can be confident therefore
that any later differences

00:06:06.362 --> 00:06:09.830
in classroom achievement
are the result of the experimental

00:06:09.830 --> 00:06:12.579
intervention rather 
than a reflection

00:06:12.579 --> 00:06:14.646
of preexisting differences.

00:06:14.646 --> 00:06:17.230
Ceteris paribus achieved!

00:06:17.359 --> 00:06:20.718
- [Kamal] Cool. Wait, 
what about the bottom,

00:06:20.718 --> 00:06:22.522
the numbers with the stars?

00:06:22.714 --> 00:06:25.479
Those differences are a lot more
than double the standard error.

00:06:25.479 --> 00:06:27.402
- [Narrator] Good eye, Kamal!

00:06:27.402 --> 00:06:29.386
The table has many numbers.

00:06:29.386 --> 00:06:32.246
Those in Panel B are important too.

00:06:32.246 --> 00:06:35.715
This panel measures the extent 
to which students in treatment

00:06:35.715 --> 00:06:39.047
and control groups actually use 
computers in class.

00:06:39.497 --> 00:06:42.749
The treatment here was 
to allow computer use.

00:06:42.749 --> 00:06:46.066
The researchers must show
that students allowed

00:06:46.083 --> 00:06:49.448
to use computers took advantage 
of the opportunity to do so.

00:06:49.899 --> 00:06:53.032
If they didn't, then there's 
really no treatment.

00:06:53.248 --> 00:06:57.799
Luckily, 81% of those 
in the first treatment group

00:06:57.799 --> 00:07:01.832
used computers compared 
with none in the control group.

00:07:02.082 --> 00:07:05.032
And many in the second 
tablet treatment group

00:07:05.032 --> 00:07:06.997
used computers as well.

00:07:07.214 --> 00:07:09.731
These differences 
in computer use are large

00:07:09.731 --> 00:07:11.798
and statistically significant.

00:07:12.081 --> 00:07:15.366
We also get to see 
the sample size in each group.

00:07:15.366 --> 00:07:18.098
- [Kamal] The stars 
are just like decoration?

00:07:18.098 --> 00:07:21.748
- [Narrator] Some academic papers
use stars to indicate differences

00:07:21.748 --> 00:07:23.865
that are statistically significant.

00:07:23.865 --> 00:07:26.925
This makes them jump out at you.

00:07:26.925 --> 00:07:31.621
Here three stars indicate that 
the result is statistically different

00:07:31.621 --> 00:07:34.942
from zero with a p value 
less than 1%.

00:07:35.336 --> 00:07:39.436
In other words, there's less 
than a 1 in 100 chance

00:07:39.436 --> 00:07:42.171
this result is purely 
a chance finding.

00:07:42.171 --> 00:07:43.181
[applause]

00:07:43.181 --> 00:07:48.603
Two stars indicate a 1 in 20
or 5% chance of a chance finding.

00:07:48.802 --> 00:07:53.201
And one star denotes results 
we might see as often as 10%

00:07:53.201 --> 00:07:56.036
of the time merely due to chance.

00:07:56.473 --> 00:07:59.741
Today, stars are seen 
as a little old fashioned.

00:07:59.741 --> 00:08:01.440
Some journals omit them.

00:08:01.440 --> 00:08:03.324
- [Kamal] What about 
those last two columns?

00:08:03.324 --> 00:08:06.007
- [Narrator] Unlike column 4,
which combines

00:08:06.007 --> 00:08:09.689
both treatment groups into one, 
these last two columns

00:08:09.689 --> 00:08:12.357
look separately
at treatment/control differences

00:08:12.357 --> 00:08:14.360
for each treatment group.

00:08:14.360 --> 00:08:17.441
This provides a more detailed 
analysis of balance.

00:08:17.865 --> 00:08:21.064
Also, for now, 
you can ignore this row

00:08:21.064 --> 00:08:24.233
which provides 
another test of significance.

00:08:24.233 --> 00:08:28.916
Now we get to the article's 
punchline, table 4.

00:08:29.933 --> 00:08:32.993
This table reports
regression estimates

00:08:32.993 --> 00:08:36.984
of the effects of electronics use
on measures of student learning.

00:08:37.173 --> 00:08:40.026
- [Kamal Why does the study 
report regression estimates?

00:08:40.026 --> 00:08:42.205
See, that's why I was getting lost.

00:08:42.205 --> 00:08:44.806
I thought one reason 
why we liked randomized trials

00:08:44.806 --> 00:08:47.260
is that we use them 
to obtain causal effects

00:08:47.260 --> 00:08:50.138
simply by comparing 
treatment and control groups.

00:08:50.289 --> 00:08:53.489
Since these groups are balanced,
no need to use regression.

00:08:53.489 --> 00:08:55.290
- [Narrator] Well said, Kamal.

00:08:55.290 --> 00:08:59.272
In practice, it's customary 
to report regression estimates

00:08:59.272 --> 00:09:00.839
for two reasons.

00:09:00.975 --> 00:09:05.008
First, evidence of balance 
not withstanding, an abundance

00:09:05.008 --> 00:09:09.058
of caution might lead the analyst
to allow for chance differences.

00:09:09.392 --> 00:09:13.360
Second, regression estimates
are likely to be more precise.

00:09:13.726 --> 00:09:16.509
That is, they have lower 
standard errors than

00:09:16.509 --> 00:09:18.893
the simple treatment 
control comparisons.

00:09:19.905 --> 00:09:22.526
The dependent variable 
in this study

00:09:22.526 --> 00:09:24.305
is the outcome of interest.

00:09:24.652 --> 00:09:27.717
Since the question at hand
is how classroom electronics

00:09:27.717 --> 00:09:32.617
affect learning, a good outcome
is the Economics final exam score.

00:09:32.915 --> 00:09:37.167
Each column reports results
from a different regression model.

00:09:37.304 --> 00:09:40.476
Models are distinguished 
by the control variables

00:09:40.476 --> 00:09:44.712
or covariates they include
besides treatment status.

00:09:44.712 --> 00:09:48.425
Estimates with no covariates 
are simple comparisons

00:09:48.425 --> 00:09:50.502
of treatment and control groups.

00:09:50.619 --> 00:09:52.720
- [Kamal] I thought 
they just forgot to fill it out.

00:09:52.720 --> 00:09:56.668
- [Narrator] Column 1 suggests
electronics use reduced

00:09:56.668 --> 00:10:00.835
final exam scores
by 0.28 standard deviations.

00:10:01.102 --> 00:10:06.552
In our last lesson, Master Joshway
explained, we use standard deviation

00:10:06.552 --> 00:10:10.501
units because these units 
are easily compared across studies.

00:10:11.002 --> 00:10:13.702
Column 2 reports results 
from a model

00:10:13.702 --> 00:10:15.952
that adds demographic controls.

00:10:15.952 --> 00:10:19.502
Here we're comparing test scores
but holding constant factors

00:10:19.502 --> 00:10:21.435
such as age and sex.

00:10:21.886 --> 00:10:25.285
Column 3 reports results
from a model that adds GPA

00:10:25.285 --> 00:10:27.186
to the list of covariates.

00:10:27.603 --> 00:10:30.502
Column 4 adds ACT scores.

00:10:30.502 --> 00:10:33.503
Analysts often report 
results this way,

00:10:33.503 --> 00:10:36.803
starting with models that include 
few or no covariates

00:10:36.803 --> 00:10:40.452
and then reporting estimates 
from models that add more

00:10:40.452 --> 00:10:43.586
and more covariates 
as we move across columns.

00:10:44.035 --> 00:10:46.802
Looking across columns,
what do you notice?

00:10:47.252 --> 00:10:49.919
- [Kamal] Well, the coefficient 
on using a computer is always

00:10:49.919 --> 00:10:51.635
a pretty big negative number.

00:10:51.635 --> 00:10:53.002
- [Narrator] That's right!

00:10:53.002 --> 00:10:56.455
We can also see that 
the standard errors are small enough

00:10:56.455 --> 00:11:00.202
to make these negative results
statistically significant.

00:11:00.316 --> 00:11:04.201
In other words, the primary 
takeaway from this experiment

00:11:04.201 --> 00:11:08.318
is that electronics in the classroom
reduce student learning.

00:11:08.767 --> 00:11:11.884
- [Kama] GPA and ACT scores 
are also significant.

00:11:11.884 --> 00:11:13.600
Why is that?

00:11:13.600 --> 00:11:15.100
- [Narrator] Good observation!

00:11:15.100 --> 00:11:16.866
That's not surprising.

00:11:16.866 --> 00:11:20.267
We expect these variables 
to predict college performance.

00:11:20.267 --> 00:11:21.984
- [Kamal] Oh right, of course.

00:11:21.984 --> 00:11:24.817
Kids who got better grades before
are more likely to get

00:11:24.817 --> 00:11:26.317
a better grade in this course.

00:11:26.317 --> 00:11:29.849
- [Narrator] You'll also notice a lot 
of other information on this table.

00:11:29.849 --> 00:11:34.234
Remaining panels in the table
report effects of electronics use

00:11:34.234 --> 00:11:36.933
on components of the final exam,

00:11:36.933 --> 00:11:39.467
such as the multiple 
choice questions.

00:11:39.467 --> 00:11:43.285
These results are mostly consistent 
with computer use effects

00:11:43.285 --> 00:11:45.216
on overall scores.

00:11:45.216 --> 00:11:47.740
- [Kamal] What about the rows
not in English?

00:11:47.740 --> 00:11:50.561
- [Narrator] These rows give 
additional statistical information.

00:11:50.828 --> 00:11:53.978
R-squared is a measure
of goodness of fit.

00:11:54.311 --> 00:11:58.010
This isn't too important, though 
some readers may want to know it.

00:11:58.660 --> 00:12:02.827
Other rows report on alternative 
tests of statistical significance

00:12:02.827 --> 00:12:05.028
that you can ignore for now.

00:12:05.028 --> 00:12:07.644
- [Kamal] Oh my gosh,
these tables aren't that hard!

00:12:07.644 --> 00:12:09.488
Thank you so much.

00:12:09.488 --> 00:12:11.787
- [Narrator] Next up is regression.

00:12:11.787 --> 00:12:13.179
See you then!

00:12:15.974 --> 00:12:17.263
♪ [music] ♪

00:12:17.263 --> 00:12:20.575
You're on your way 
to mastering econometrics.

00:12:20.834 --> 00:12:22.783
Make sure this video sticks

00:12:22.783 --> 00:12:24.982
by taking a few 
quick practice questions.

00:12:25.153 --> 00:12:28.855
Or, if you're ready,
click for the next video.

00:12:28.855 --> 00:12:32.620
You can also check out MRU's 
website for more courses,

00:12:32.620 --> 00:12:35.298
teacher resources and more.