0:00:01.425,0:00:05.060
- [Narrator] On his quest [br]to master econometrics,

0:00:05.223,0:00:08.813
Grasshopper Kamal has [br]made great progress,

0:00:08.813,0:00:13.662
stretching his capabilities [br]and outsmarting his foes.

0:00:14.223,0:00:16.640
Alas, today he's despondent,

0:00:16.640,0:00:19.336
for one challenge remains unmet.

0:00:19.336,0:00:24.130
Kamal cannot yet decode [br]the scriptures of academic research,

0:00:24.130,0:00:27.347
journals like [br]"The American Economic Review"

0:00:27.347,0:00:29.080
and "Econometrica."

0:00:29.202,0:00:33.501
These seemed to him to be inscribed [br]in an obscure foreign tongue.

0:00:33.501,0:00:35.478
- [Kamal] Ugh, what the... ?

0:00:36.711,0:00:40.018
- These volumes are [br]opaque to the novice, Kamal,

0:00:40.018,0:00:42.205
but can be deciphered with study.

0:00:42.467,0:00:45.109
Let us learn to read them together.

0:00:52.485,0:00:55.317
Let's dive into the West Point study,

0:00:55.317,0:00:58.278
published in the "Economics [br]of Education Review."

0:00:58.538,0:01:01.688
This paper reports [br]on a randomized evaluation

0:01:01.688,0:01:05.859
of student electronics use [br]in Economics 101 classrooms.

0:01:06.242,0:01:09.192
First, a quick review [br]of the research design.

0:01:09.423,0:01:10.523
- Okay.

0:01:11.553,0:01:13.630
- [Josh] 'Metrics masters [br]teaching at West Point,

0:01:13.630,0:01:16.620
the military college that trains [br]American Army officers

0:01:16.620,0:01:19.854
designed a randomized trial[br]to answer this question.

0:01:20.372,0:01:23.233
These masters randomly assigned[br]West Point cadets

0:01:23.233,0:01:26.383
into Economics classes[br]operating under different rules.

0:01:26.595,0:01:28.962
Unlike most American colleges,

0:01:28.962,0:01:31.945
the West Point default [br]is no electronics.

0:01:32.345,0:01:35.428
For purposes of this experiment,[br]some students were left

0:01:35.428,0:01:38.679
in such traditional [br]technology-free classes,

0:01:38.679,0:01:41.911
no laptops, no tablets [br]and no phones!

0:01:41.911,0:01:43.280
[voice echoes]

0:01:43.328,0:01:45.743
This is the control group,[br]or baseline case.

0:01:46.213,0:01:49.198
Another group was allowed[br]to use electronics.

0:01:49.269,0:01:52.704
This is the treatment group,[br]subject to a changed environment.

0:01:53.313,0:01:55.858
The treatment in this case[br]is the unrestricted use

0:01:55.858,0:01:58.107
of laptops or tablets in class.

0:01:58.844,0:02:01.812
Every causal question [br]has a clear outcome,

0:02:01.859,0:02:05.276
the variables we hope to influence[br]defined in advance of the study.

0:02:05.860,0:02:08.375
The outcomes in the West Point[br]electronics study

0:02:08.375,0:02:09.994
are final exam scores.

0:02:10.047,0:02:13.364
The study seeks to answer[br]the following question,

0:02:13.364,0:02:17.299
what is the causal effect [br]of classroom electronics on learning

0:02:17.299,0:02:19.765
as measured by exam scores?

0:02:20.632,0:02:24.199
- Economics journal articles [br]usually begin with a table

0:02:24.199,0:02:26.933
of descriptive statistics,[br]giving key facts

0:02:26.933,0:02:28.500
about the study sample.

0:02:28.500,0:02:31.781
- Oh my gosh, I remember this table,[br]so confusing!

0:02:31.781,0:02:36.666
- [Narrator] Columns 1 to 3 report [br]mean, or average, characteristics.

0:02:36.736,0:02:39.688
These give a sense [br]of who we're studying.

0:02:39.948,0:02:43.736
Let's start with column 1 [br]which describes covariates

0:02:43.736,0:02:45.251
in the control group.

0:02:45.408,0:02:48.972
Covariates are characteristics [br]of the control and treatment groups

0:02:48.972,0:02:51.621
measured before [br]the experiment begins.

0:02:51.621,0:02:56.986
For example, we see the control group[br]has an average age a bit over 20.

0:02:57.321,0:03:00.339
Many of these covariates [br]are dummy variables.

0:03:00.790,0:03:05.670
A dummy variable can only have [br]two values, a zero or a one.

0:03:05.981,0:03:10.015
For example, student gender[br]is captured by a dummy variable

0:03:10.015,0:03:13.148
that equals one for women[br]and zero for men.

0:03:13.248,0:03:16.580
The mean of this variable [br]is the proportion female.

0:03:16.815,0:03:20.651
We also see that the control group [br]is 13% Hispanic

0:03:20.651,0:03:23.769
and 19% had prior military service.

0:03:25.035,0:03:26.635
The table notes are key.

0:03:26.635,0:03:28.686
Refer to these [br]as you scan the table.

0:03:29.102,0:03:33.369
These notes explain what's shown [br]in each column and panel.

0:03:39.485,0:03:43.375
The notes tell us, for example,[br]that standard deviations

0:03:43.375,0:03:45.175
are reported in brackets.

0:03:45.947,0:03:49.598
Standard deviations tell us how[br]spread out the data are.

0:03:50.282,0:03:54.887
For example, a standard deviation [br]of 0.52 tells us that most

0:03:54.887,0:03:59.233
of the control group's GPAs[br]fall between 2.35,

0:03:59.233,0:04:03.454
which is 0.52 below [br]the mean GPA of 2.87,

0:04:03.454,0:04:08.337
and 3.39, which is 0.52 above 2.87.

0:04:09.001,0:04:12.221
A lower standard deviation[br]would mean the GPAs were

0:04:12.221,0:04:14.404
more tightly clustered [br]around the mean.

0:04:14.549,0:04:17.451
- [Kamal] Yeah, but they're missing[br]for most of the variables.

0:04:17.499,0:04:18.600
- [Narrator] That's right.

0:04:18.600,0:04:22.497
Masters usually omit [br]standard deviations for dummies

0:04:22.497,0:04:26.500
because the mean of this variable[br]determines its standard deviation.

0:04:26.936,0:04:31.370
This study compares two treatment [br]groups with the control group.

0:04:31.370,0:04:35.753
The first was allowed free use [br]of laptops and tablets.

0:04:35.753,0:04:38.252
The second treatment[br]was more restrictive,

0:04:38.252,0:04:41.553
allowing only tablets placed [br]flat on the desk.

0:04:42.152,0:04:45.238
The treatment groups[br]look much like the control group.

0:04:46.310,0:04:51.270
This takes us to the next feature [br]of this table, columns 4 through 6

0:04:51.407,0:04:54.558
use statistical tests to compare[br]the characteristics

0:04:54.558,0:04:57.591
of the treatment and control group[br]before the experiment.

0:04:58.023,0:05:01.674
In column 4, the two treatment [br]groups are combined.

0:05:01.856,0:05:04.840
You can see that the difference[br]in proportion female

0:05:04.840,0:05:09.690
between the treatment [br]and control group is only 0.03.

0:05:09.991,0:05:13.740
The difference is not [br]statistically significant.

0:05:14.290,0:05:17.205
It is the sort of difference[br]we can easily put down

0:05:17.205,0:05:20.497
to chance results [br]in our sample selection process.

0:05:20.497,0:05:22.133
- [Kamal] Hmm, how do we know that?

0:05:22.133,0:05:23.790
- [Narrator] Remember [br]the rule of thumb?

0:05:23.790,0:05:26.968
Statistical estimates [br]that exceed the standard error

0:05:26.968,0:05:29.882
by a multiple of 2 [br]in absolute value

0:05:30.105,0:05:33.997
are usually said[br]to be statistically significant.

0:05:35.132,0:05:38.419
The standard error is 0.03,

0:05:38.566,0:05:41.483
same as the difference [br]in proportion female.

0:05:42.015,0:05:46.041
So the ratio of the latter [br]to the former is only 1,

0:05:46.041,0:05:48.607
which of course is less than 2.

0:05:48.607,0:05:51.191
- [Kamal] Uh huh. So none[br]of the treatment/control differences

0:05:51.191,0:05:54.333
in the table are more than twice[br]their standard errors.

0:05:54.333,0:05:55.789
- [Narrator] Correct.

0:05:55.789,0:05:59.000
The random division of students[br]appears to have succeeded

0:05:59.014,0:06:01.945
in creating groups [br]that are indeed comparable.

0:06:02.846,0:06:06.362
We can be confident therefore[br]that any later differences

0:06:06.362,0:06:09.830
in classroom achievement[br]are the result of the experimental

0:06:09.830,0:06:12.579
intervention rather [br]than a reflection

0:06:12.579,0:06:14.646
of preexisting differences.

0:06:14.646,0:06:17.230
Ceteris paribus achieved!

0:06:17.359,0:06:20.718
- [Kamal] Cool. Wait, [br]what about the bottom,

0:06:20.718,0:06:22.522
the numbers with the stars?

0:06:22.714,0:06:25.479
Those differences are a lot more[br]than double the standard error.

0:06:25.479,0:06:27.402
- [Narrator] Good eye, Kamal!

0:06:27.402,0:06:29.386
The table has many numbers.

0:06:29.386,0:06:32.246
Those in Panel B are important too.

0:06:32.246,0:06:35.715
This panel measures the extent [br]to which students in treatment

0:06:35.715,0:06:39.047
and control groups actually use [br]computers in class.

0:06:39.497,0:06:42.749
The treatment here was [br]to allow computer use.

0:06:42.749,0:06:46.066
The researchers must show[br]that students allowed

0:06:46.083,0:06:49.448
to use computers took advantage [br]of the opportunity to do so.

0:06:49.899,0:06:53.032
If they didn't, then there's [br]really no treatment.

0:06:53.248,0:06:57.799
Luckily, 81% of those [br]in the first treatment group

0:06:57.799,0:07:01.832
used computers compared [br]with none in the control group.

0:07:02.082,0:07:05.032
And many in the second [br]tablet treatment group

0:07:05.032,0:07:06.997
used computers as well.

0:07:07.214,0:07:09.731
These differences [br]in computer use are large

0:07:09.731,0:07:11.798
and statistically significant.

0:07:12.081,0:07:15.366
We also get to see [br]the sample size in each group.

0:07:15.366,0:07:18.098
- [Kamal] The stars [br]are just like decoration?

0:07:18.098,0:07:21.748
- [Narrator] Some academic papers[br]use stars to indicate differences

0:07:21.748,0:07:23.865
that are statistically significant.

0:07:23.865,0:07:26.925
This makes them jump out at you.

0:07:26.925,0:07:31.621
Here three stars indicate that [br]the result is statistically different

0:07:31.621,0:07:34.942
from zero with a p value [br]less than 1%.

0:07:35.336,0:07:39.436
In other words, there's less [br]than a 1 in 100 chance

0:07:39.436,0:07:42.171
this result is purely [br]a chance finding.

0:07:42.171,0:07:43.181
[applause]

0:07:43.181,0:07:48.603
Two stars indicate a 1 in 20[br]or 5% chance of a chance finding.

0:07:48.802,0:07:53.201
And one star denotes results [br]we might see as often as 10%

0:07:53.201,0:07:56.036
of the time merely due to chance.

0:07:56.473,0:07:59.741
Today, stars are seen [br]as a little old fashioned.

0:07:59.741,0:08:01.440
Some journals omit them.

0:08:01.440,0:08:03.324
- [Kamal] What about [br]those last two columns?

0:08:03.324,0:08:06.007
- [Narrator] Unlike column 4,[br]which combines

0:08:06.007,0:08:09.689
both treatment groups into one, [br]these last two columns

0:08:09.689,0:08:12.357
look separately[br]at treatment/control differences

0:08:12.357,0:08:14.360
for each treatment group.

0:08:14.360,0:08:17.441
This provides a more detailed [br]analysis of balance.

0:08:17.865,0:08:21.064
Also, for now, [br]you can ignore this row

0:08:21.064,0:08:24.233
which provides [br]another test of significance.

0:08:24.233,0:08:28.916
Now we get to the article's [br]punchline, table 4.

0:08:29.933,0:08:32.993
This table reports[br]regression estimates

0:08:32.993,0:08:36.984
of the effects of electronics use[br]on measures of student learning.

0:08:37.173,0:08:40.026
- [Kamal Why does the study [br]report regression estimates?

0:08:40.026,0:08:42.205
See, that's why I was getting lost.

0:08:42.205,0:08:44.806
I thought one reason [br]why we liked randomized trials

0:08:44.806,0:08:47.260
is that we use them [br]to obtain causal effects

0:08:47.260,0:08:50.138
simply by comparing [br]treatment and control groups.

0:08:50.289,0:08:53.489
Since these groups are balanced,[br]no need to use regression.

0:08:53.489,0:08:55.257
- [Narrator] Well said, Kamal.

9:59:59.000,9:59:59.000
In practice, it's customary [br]to report regression estimates

9:59:59.000,9:59:59.000
for two reasons.

9:59:59.000,9:59:59.000
First, evidence of balance [br]not withstanding, an abundance

9:59:59.000,9:59:59.000
of caution might lead the analyst[br]to allow for chance differences.

9:59:59.000,9:59:59.000
Second, regression estimates[br]are likely to be more precise.

9:59:59.000,9:59:59.000
That is, they have lower [br]standard errors than

9:59:59.000,9:59:59.000
the simple treatment [br]control comparisons.

9:59:59.000,9:59:59.000
The dependent variable [br]in this study

9:59:59.000,9:59:59.000
is the outcome of interest.

9:59:59.000,9:59:59.000
Since the question at hand[br]is how classroom electronics

9:59:59.000,9:59:59.000
affect learning, a good outcome[br]is the economics final exam score.

9:59:59.000,9:59:59.000
Each column reports results[br]from a different regression model.

9:59:59.000,9:59:59.000
Models are distinguished [br]by the control variables

9:59:59.000,9:59:59.000
or covariates they include[br]besides treatment status.

9:59:59.000,9:59:59.000
Estimates with no covariates [br]are simple comparisons

9:59:59.000,9:59:59.000
of treatment and control groups.

9:59:59.000,9:59:59.000
- [Kamal] I thought [br]they just forgot to fill it out.

9:59:59.000,9:59:59.000
- [Narrator] Column 1 suggests[br]electronics use reduced

9:59:59.000,9:59:59.000
final exam scores[br]by 0.28 standard deviations.

9:59:59.000,9:59:59.000
In our last lesson, Master Joshway[br]explained, we use standard deviation

9:59:59.000,9:59:59.000
units because these units [br]are easily compared across studies.

9:59:59.000,9:59:59.000
Column 2 reports results [br]from a model

9:59:59.000,9:59:59.000
that adds demographic controls.

9:59:59.000,9:59:59.000
Here we're comparing test scores[br]but holding constant factors

9:59:59.000,9:59:59.000
such as age and sex.

9:59:59.000,9:59:59.000
Column 3 reports results[br]from a model that adds GPA

9:59:59.000,9:59:59.000
to the list of covariates.

9:59:59.000,9:59:59.000
Column 4 adds ACT scores.

9:59:59.000,9:59:59.000
Analysts often report [br]results this way,

9:59:59.000,9:59:59.000
starting with models that include [br]few or no covariates

9:59:59.000,9:59:59.000
and then reporting estimates [br]from models that add more

9:59:59.000,9:59:59.000
and more covariates [br]as we move across columns.

9:59:59.000,9:59:59.000
Looking across columns,[br]what do you notice?

9:59:59.000,9:59:59.000
- [Kamal] Well, the coefficient [br]on using a computer is always

9:59:59.000,9:59:59.000
a pretty big negative number.

9:59:59.000,9:59:59.000
- [Narrator] That's right!

9:59:59.000,9:59:59.000
We can also see that [br]the standard errors are small enough

9:59:59.000,9:59:59.000
to make these negative results[br]statistically significant.

9:59:59.000,9:59:59.000
In other words, the primary [br]takeaway from this experiment

9:59:59.000,9:59:59.000
is that electronics in the classroom[br]reduce student learning.

9:59:59.000,9:59:59.000
- [Kama] GPA and ACT scores [br]are also significant.

9:59:59.000,9:59:59.000
Why is that?

9:59:59.000,9:59:59.000
- [Narrator] Good observation!

9:59:59.000,9:59:59.000
That's not surprising.

9:59:59.000,9:59:59.000
We expect these variables [br]to predict college performance.

9:59:59.000,9:59:59.000
- [Kamal] Oh right, of course.

9:59:59.000,9:59:59.000
Kids who got better grades before[br]are more likely to get

9:59:59.000,9:59:59.000
a better grade in this course.

9:59:59.000,9:59:59.000
- [Narrator] You'll also notice a lot [br]of other information on this table.

9:59:59.000,9:59:59.000
Remaining panels in the table[br]report effects of electronics use

9:59:59.000,9:59:59.000
on components of the final exam,

9:59:59.000,9:59:59.000
such as the multiple [br]choice questions.

9:59:59.000,9:59:59.000
These results are mostly consistent [br]with computer use effects

9:59:59.000,9:59:59.000
on overall scores.

9:59:59.000,9:59:59.000
- [Kamal] What about the rows[br]not in English?

9:59:59.000,9:59:59.000
- [Narrator] These rows give [br]additional statistical information.

9:59:59.000,9:59:59.000
R-squared is a measure[br]of goodness of fit.

9:59:59.000,9:59:59.000
This isn't too important, though [br]some readers may want to know it.

9:59:59.000,9:59:59.000
Other rows report on alternative [br]tests of statistical significance

9:59:59.000,9:59:59.000
that you can ignore for now.

9:59:59.000,9:59:59.000
- [Kamal] Oh my gosh,[br]these tables aren't that hard.

9:59:59.000,9:59:59.000
Thank you so much.

9:59:59.000,9:59:59.000
Next up is regression.

9:59:59.000,9:59:59.000
See you then!

9:59:59.000,9:59:59.000
♪ [music] ♪

9:59:59.000,9:59:59.000
You're on your way [br]to mastering econometrics.

9:59:59.000,9:59:59.000
Make sure this video sticks

9:59:59.000,9:59:59.000
by taking a few [br]quick practice questions.

9:59:59.000,9:59:59.000
Or, if you're ready,[br]click for the next video.

9:59:59.000,9:59:59.000
You can also check out MRU's [br]website for more courses,

9:59:59.000,9:59:59.000
teacher resources and more.