1
00:00:01,425 --> 00:00:05,015
- [Narrator] On his quest 
to master econometrics,

2
00:00:05,619 --> 00:00:08,813
Grasshopper Kamal
has made great progress,

3
00:00:08,813 --> 00:00:13,662
stretching his capabilities 
and outsmarting his foes.

4
00:00:14,223 --> 00:00:16,640
Alas, today he's despondent,

5
00:00:16,640 --> 00:00:19,614
for one challenge remains unmet.

6
00:00:19,614 --> 00:00:24,230
Kamal cannot yet decode 
the scriptures of academic research --

7
00:00:24,230 --> 00:00:27,347
journals like 
"The American Economic Review"

8
00:00:27,347 --> 00:00:29,317
and "Econometrica."

9
00:00:29,317 --> 00:00:33,501
These seemed to him to be inscribed 
in an obscure foreign tongue.

10
00:00:33,501 --> 00:00:35,478
- [Kamal] Ugh, what the... ?

11
00:00:36,711 --> 00:00:40,069
- [Narrator] These volumes
are opaque to the novice, Kamal,

12
00:00:40,069 --> 00:00:42,205
but can be deciphered with study.

13
00:00:42,467 --> 00:00:45,109
Let us learn to read them together.

14
00:00:52,657 --> 00:00:55,317
Let's dive into the West Point study,

15
00:00:55,317 --> 00:00:58,278
published in the "Economics 
of Education Review."

16
00:00:58,658 --> 00:01:01,688
This paper reports 
on a randomized evaluation

17
00:01:01,688 --> 00:01:05,859
of student electronics use 
in Economics 101 classrooms.

18
00:01:06,485 --> 00:01:09,192
First, a quick review 
of the research design.

19
00:01:09,423 --> 00:01:10,523
- Okay.

20
00:01:11,553 --> 00:01:13,630
- [Josh] Metrics masters 
teaching at West Point,

21
00:01:13,630 --> 00:01:17,029
the military college that trains 
American Army officers

22
00:01:17,029 --> 00:01:19,854
designed a randomized trial
to answer this question.

23
00:01:20,582 --> 00:01:23,233
These masters randomly assigned
West Point cadets

24
00:01:23,233 --> 00:01:26,574
into Economics classes
operating under different rules.

25
00:01:27,116 --> 00:01:28,962
Unlike most American colleges,

26
00:01:28,962 --> 00:01:31,945
the West Point default 
is no electronics.

27
00:01:32,482 --> 00:01:35,784
For purposes of this experiment,
some students were left

28
00:01:35,784 --> 00:01:38,679
in such traditional 
technology-free classes --

29
00:01:38,679 --> 00:01:41,911
no laptops, no tablets 
and no phones!

30
00:01:41,911 --> 00:01:43,324
[voice echoes]

31
00:01:43,324 --> 00:01:45,743
This is the control group,
or baseline case.

32
00:01:46,372 --> 00:01:49,292
Another group was allowed
to use electronics.

33
00:01:49,292 --> 00:01:52,704
This is the treatment group,
subject to a changed environment.

34
00:01:53,313 --> 00:01:56,000
The treatment in this case
is the unrestricted use

35
00:01:56,000 --> 00:01:58,107
of laptops or tablets in class.

36
00:01:59,195 --> 00:02:01,972
Every causal question 
has a clear outcome --

37
00:02:01,972 --> 00:02:05,379
the variables we hope to influence
defined in advance of the study.

38
00:02:06,058 --> 00:02:08,375
The outcomes in the West Point
electronics study

39
00:02:08,375 --> 00:02:10,535
are final exam scores.

40
00:02:10,535 --> 00:02:13,810
The study seeks to answer
the following question:

41
00:02:13,810 --> 00:02:17,629
What is the causal effect 
of classroom electronics on learning

42
00:02:17,629 --> 00:02:19,765
as measured by exam scores?

43
00:02:20,852 --> 00:02:24,199
- Economics journal articles 
usually begin with a table

44
00:02:24,199 --> 00:02:25,994
of descriptive statistics,

45
00:02:25,994 --> 00:02:28,694
giving key facts
about the study sample.

46
00:02:28,694 --> 00:02:32,129
- Oh my gosh, I remember
this table -- so confusing!

47
00:02:32,129 --> 00:02:37,224
- [Narrator] Columns 1 to 3 report 
mean, or average, characteristics.

48
00:02:37,224 --> 00:02:40,089
These give a sense 
of who we're studying.

49
00:02:40,089 --> 00:02:43,736
Let's start with column 1 
which describes covariates

50
00:02:43,736 --> 00:02:45,438
in the control group.

51
00:02:45,438 --> 00:02:49,183
Covariates are characteristics 
of the control and treatment groups

52
00:02:49,183 --> 00:02:52,091
measured before 
the experiment begins.

53
00:02:52,091 --> 00:02:57,514
For example, we see the control group
has an average age a bit over 20.

54
00:02:57,514 --> 00:03:00,339
Many of these covariates 
are dummy variables.

55
00:03:00,997 --> 00:03:06,087
A dummy variable can only have 
two values -- a zero or a one.

56
00:03:06,087 --> 00:03:10,298
For example, student gender
is captured by a dummy variable

57
00:03:10,298 --> 00:03:13,386
that equals one for women
and zero for men.

58
00:03:13,386 --> 00:03:17,104
The mean of this variable 
is the proportion female.

59
00:03:17,104 --> 00:03:20,651
We also see that the control group 
is 13% Hispanic

60
00:03:20,651 --> 00:03:23,905
and 19% had prior military service.

61
00:03:25,035 --> 00:03:26,635
The table notes are key.

62
00:03:26,635 --> 00:03:29,218
Refer to these 
as you scan the table.

63
00:03:29,218 --> 00:03:33,534
These notes explain what's shown 
in each column and panel.

64
00:03:39,485 --> 00:03:41,858
The notes tell us, for example,

65
00:03:41,858 --> 00:03:45,175
that standard deviations
are reported in brackets.

66
00:03:45,947 --> 00:03:49,598
Standard deviations tell us how
spread out the data are.

67
00:03:50,448 --> 00:03:54,887
For example, a standard deviation 
of 0.52 tells us that most

68
00:03:54,887 --> 00:03:59,397
of the control group's GPAs
fall between 2.35,

69
00:03:59,397 --> 00:04:03,454
which is 0.52 below 
the mean GPA of 2.87,

70
00:04:03,454 --> 00:04:08,337
and 3.39, which is 0.52 above 2.87.

71
00:04:09,337 --> 00:04:12,122
A lower standard deviation
would mean the GPAs

72
00:04:12,122 --> 00:04:14,706
were more tightly clustered 
around the mean.

73
00:04:14,706 --> 00:04:17,543
- [Kamal] Yeah, but they're missing
for most of the variables.

74
00:04:17,543 --> 00:04:18,600
- [Narrator] That's right.

75
00:04:18,600 --> 00:04:22,497
Masters usually omit 
standard deviations for dummies

76
00:04:22,497 --> 00:04:26,500
because the mean of this variable
determines its standard deviation.

77
00:04:27,542 --> 00:04:30,102
This study compares
two treatment groups

78
00:04:30,102 --> 00:04:32,078
with the control group.

79
00:04:32,078 --> 00:04:35,886
The first was allowed free use 
of laptops and tablets.

80
00:04:35,886 --> 00:04:38,252
The second treatment
was more restrictive,

81
00:04:38,252 --> 00:04:41,713
allowing only tablets
placed flat on the desk.

82
00:04:42,152 --> 00:04:45,238
The treatment groups
look much like the control group.

83
00:04:46,694 --> 00:04:51,443
This takes us to the next feature 
of this table, columns 4 through 6

84
00:04:51,443 --> 00:04:54,558
use statistical tests to compare
the characteristics

85
00:04:54,558 --> 00:04:57,591
of the treatment and control group
before the experiment.

86
00:04:58,158 --> 00:05:01,991
In column 4, the two treatment 
groups are combined.

87
00:05:01,991 --> 00:05:04,998
You can see that the difference
in proportion female

88
00:05:04,998 --> 00:05:09,690
between the treatment 
and control group is only 0.03.

89
00:05:10,508 --> 00:05:13,740
The difference is not 
statistically significant --

90
00:05:14,290 --> 00:05:17,440
it is the sort of difference
we can easily put down

91
00:05:17,440 --> 00:05:20,623
to chance results 
in our sample selection process.

92
00:05:20,623 --> 00:05:22,133
- [Kamal] Hmm, how do we know that?

93
00:05:22,133 --> 00:05:23,790
- [Narrator] Remember 
the rule of thumb?

94
00:05:23,790 --> 00:05:27,122
Statistical estimates 
that exceed the standard error

95
00:05:27,122 --> 00:05:30,108
by a multiple of 2 
in absolute value

96
00:05:30,108 --> 00:05:33,997
are usually said
to be statistically significant.

97
00:05:35,132 --> 00:05:38,766
The standard error is 0.03,

98
00:05:38,766 --> 00:05:41,483
same as the difference 
in proportion female.

99
00:05:42,244 --> 00:05:46,132
So the ratio of the latter 
to the former is only 1,

100
00:05:46,132 --> 00:05:48,607
which, of course, is less than 2.

101
00:05:48,607 --> 00:05:51,191
- [Kamal] Uh-huh! So none
of the treatment/control differences

102
00:05:51,191 --> 00:05:54,455
in the table are more than twice
their standard errors.

103
00:05:54,455 --> 00:05:55,997
- [Narrator] Correct.

104
00:05:55,997 --> 00:05:59,081
The random division of students
appears to have succeeded

105
00:05:59,081 --> 00:06:01,945
in creating groups 
that are indeed comparable.

106
00:06:02,846 --> 00:06:05,008
We can be confident, therefore,

107
00:06:05,008 --> 00:06:07,774
that any later differences
in classroom achievement

108
00:06:07,774 --> 00:06:11,073
are the result
of the experimental intervention

109
00:06:11,073 --> 00:06:14,754
rather than a reflection
of preexisting differences.

110
00:06:14,754 --> 00:06:17,454
Ceteris paribus achieved!

111
00:06:17,454 --> 00:06:20,934
- [Kamal] Cool. Wait, 
what about the bottom,

112
00:06:20,934 --> 00:06:22,833
the numbers with the stars?

113
00:06:22,833 --> 00:06:25,589
Those differences are a lot more
than double the standard error.

114
00:06:25,589 --> 00:06:27,402
- [Narrator] Good eye, Kamal!

115
00:06:27,402 --> 00:06:29,386
The table has many numbers.

116
00:06:29,386 --> 00:06:32,246
Those in Panel B are important too.

117
00:06:32,246 --> 00:06:35,715
This panel measures the extent 
to which students in treatment

118
00:06:35,715 --> 00:06:39,139
and control groups actually use 
computers in class.

119
00:06:39,754 --> 00:06:42,873
The treatment here
was to allow computer use.

120
00:06:43,278 --> 00:06:44,873
The researchers must show

121
00:06:44,873 --> 00:06:47,079
that students allowed
to use computers

122
00:06:47,079 --> 00:06:49,448
took advantage 
of the opportunity to do so.

123
00:06:50,072 --> 00:06:53,033
If they didn't, then there's 
really no treatment.

124
00:06:53,867 --> 00:06:57,799
Luckily, 81% of those 
in the first treatment group

125
00:06:57,799 --> 00:06:59,472
used computers

126
00:06:59,472 --> 00:07:02,178
compared with none
in the control group.

127
00:07:02,178 --> 00:07:05,216
And many in the second 
tablet treatment group

128
00:07:05,216 --> 00:07:07,264
used computers as well.

129
00:07:07,264 --> 00:07:09,879
These differences 
in computer use are large

130
00:07:09,879 --> 00:07:11,798
and statistically significant.

131
00:07:12,081 --> 00:07:15,428
We also get to see 
the sample size in each group.

132
00:07:15,428 --> 00:07:18,098
- [Kamal] The stars 
are just like decoration?

133
00:07:18,098 --> 00:07:21,748
- [Narrator] Some academic papers
use stars to indicate differences

134
00:07:21,748 --> 00:07:23,983
that are statistically significant.

135
00:07:23,983 --> 00:07:26,925
This makes them jump out at you.

136
00:07:26,925 --> 00:07:31,621
Here, three stars indicate that 
the result is statistically different

137
00:07:31,621 --> 00:07:34,942
from zero with a p-value 
less than 1%.

138
00:07:35,672 --> 00:07:39,436
In other words, there's less 
than a 1 in 100 chance

139
00:07:39,436 --> 00:07:42,171
this result is purely 
a chance finding.

140
00:07:42,171 --> 00:07:43,181
[applause]

141
00:07:43,181 --> 00:07:48,997
Two stars indicate a 1 in 20
or 5% chance of a chance finding.

142
00:07:48,997 --> 00:07:52,469
And one star denotes results 
we might see as often

143
00:07:52,469 --> 00:07:56,036
as 10% of the time
merely due to chance.

144
00:07:56,473 --> 00:07:59,957
Today, stars are seen 
as a little old fashioned.

145
00:07:59,957 --> 00:08:01,606
Some journals omit them.

146
00:08:01,606 --> 00:08:03,894
- [Kamal] What about 
those last two columns?

147
00:08:03,894 --> 00:08:06,007
- [Narrator] Unlike column 4,
which combines

148
00:08:06,007 --> 00:08:09,689
both treatment groups into one, 
these last two columns

149
00:08:09,689 --> 00:08:12,357
look separately
at treatment/control differences

150
00:08:12,357 --> 00:08:14,572
for each treatment group.

151
00:08:14,572 --> 00:08:17,441
This provides a more detailed 
analysis of balance.

152
00:08:18,295 --> 00:08:21,288
Also, for now, 
you can ignore this row

153
00:08:21,288 --> 00:08:24,205
which provides 
another test of significance.

154
00:08:24,755 --> 00:08:29,062
Now we get to the article's 
punchline, table 4.

155
00:08:30,075 --> 00:08:32,993
This table reports
regression estimates

156
00:08:32,993 --> 00:08:37,273
of the effects of electronics use
on measures of student learning.

157
00:08:37,273 --> 00:08:40,258
- [Kamal Why does the study 
report regression estimates?

158
00:08:40,258 --> 00:08:42,529
See, that's why I was getting lost.

159
00:08:42,529 --> 00:08:44,806
I thought one reason 
why we liked randomized trials

160
00:08:44,806 --> 00:08:47,260
is that we use them 
to obtain causal effects

161
00:08:47,260 --> 00:08:50,479
simply by comparing 
treatment and control groups.

162
00:08:50,479 --> 00:08:53,883
Since these groups are balanced,
no need to use regression.

163
00:08:53,883 --> 00:08:55,492
- [Narrator] Well said, Kamal.

164
00:08:55,492 --> 00:08:59,272
In practice, it's customary 
to report regression estimates

165
00:08:59,272 --> 00:09:01,013
for two reasons.

166
00:09:01,013 --> 00:09:04,448
First, evidence of balance 
notwithstanding,

167
00:09:04,448 --> 00:09:07,349
an abundance of caution
might lead the analyst

168
00:09:07,349 --> 00:09:09,678
to allow for chance differences.

169
00:09:09,678 --> 00:09:13,622
Second, regression estimates
are likely to be more precise --

170
00:09:13,622 --> 00:09:16,509
that is, they have lower 
standard errors

171
00:09:16,509 --> 00:09:18,893
than the simple treatment 
control comparisons.

172
00:09:20,129 --> 00:09:22,526
The dependent variable 
in this study

173
00:09:22,526 --> 00:09:24,305
is the outcome of interest.

174
00:09:24,652 --> 00:09:26,192
Since the question at hand

175
00:09:26,192 --> 00:09:29,068
is how classroom electronics
affect learning,

176
00:09:29,068 --> 00:09:32,845
a good outcome is the Economics
final exam score.

177
00:09:33,406 --> 00:09:37,650
Each column reports results
from a different regression model.

178
00:09:37,650 --> 00:09:40,476
Models are distinguished 
by the control variables

179
00:09:40,476 --> 00:09:44,453
or covariates they include
besides treatment status.

180
00:09:44,934 --> 00:09:48,425
Estimates with no covariates 
are simple comparisons

181
00:09:48,425 --> 00:09:50,677
of treatment and control groups.

182
00:09:50,677 --> 00:09:53,169
- [Kamal] I thought 
they just forgot to fill it out.

183
00:09:53,169 --> 00:09:56,228
- [Narrator] Column 1
suggests electronics use

184
00:09:56,228 --> 00:10:00,835
reduced final exam scores
by 0.28 standard deviations.

185
00:10:01,547 --> 00:10:02,940
In our last lesson,

186
00:10:02,940 --> 00:10:07,237
Master Joshway explained
that we use standard deviation units

187
00:10:07,237 --> 00:10:10,672
because these units 
are easily compared across studies.

188
00:10:11,352 --> 00:10:13,193
Column 2 reports results

189
00:10:13,193 --> 00:10:16,082
from a model that adds
demographic controls.

190
00:10:16,082 --> 00:10:19,907
Here, we're comparing test scores
but holding constant factors

191
00:10:19,907 --> 00:10:21,435
such as age and sex.

192
00:10:21,886 --> 00:10:25,602
Column 3 reports results
from a model that adds GPA

193
00:10:25,602 --> 00:10:27,186
to the list of covariates.

194
00:10:27,603 --> 00:10:30,822
Column 4 adds ACT scores.

195
00:10:30,822 --> 00:10:33,503
Analysts often report 
results this way,

196
00:10:33,503 --> 00:10:36,992
starting with models that include 
few or no covariates

197
00:10:36,992 --> 00:10:39,667
and then reporting
estimates from models

198
00:10:39,667 --> 00:10:43,586
that add more and more covariates 
as we move across columns.

199
00:10:44,035 --> 00:10:46,802
Looking across columns,
what do you notice?

200
00:10:47,252 --> 00:10:49,919
- [Kamal] Well, the coefficient 
on using a computer is always

201
00:10:49,919 --> 00:10:51,635
a pretty big negative number.

202
00:10:51,635 --> 00:10:53,002
- [Narrator] That's right!

203
00:10:53,002 --> 00:10:56,455
We can also see that 
the standard errors are small enough

204
00:10:56,455 --> 00:11:00,561
to make these negative results
statistically significant.

205
00:11:00,561 --> 00:11:04,446
In other words, the primary 
takeaway from this experiment

206
00:11:04,446 --> 00:11:08,381
is that electronics in the classroom
reduce student learning.

207
00:11:09,000 --> 00:11:12,283
- [Kamal] GPA and ACT scores 
are also significant.

208
00:11:12,283 --> 00:11:13,750
Why is that?

209
00:11:13,750 --> 00:11:15,423
- [Narrator] Good observation!

210
00:11:15,423 --> 00:11:16,866
That's not surprising.

211
00:11:16,866 --> 00:11:20,473
We expect these variables 
to predict college performance.

212
00:11:20,473 --> 00:11:22,190
- [Kamal] Oh right, of course.

213
00:11:22,190 --> 00:11:24,026
Kids who got better grades before

214
00:11:24,026 --> 00:11:26,317
are more likely to get
a better grade in this course.

215
00:11:26,317 --> 00:11:30,226
- [Narrator] You'll also notice a lot 
of other information on this table.

216
00:11:30,226 --> 00:11:34,515
Remaining panels in the table
report effects of electronics use

217
00:11:34,515 --> 00:11:36,933
on components of the final exam,

218
00:11:36,933 --> 00:11:39,816
such as the multiple 
choice questions.

219
00:11:39,816 --> 00:11:43,371
These results are mostly consistent 
with computer use effects

220
00:11:43,371 --> 00:11:45,360
on overall scores.

221
00:11:45,360 --> 00:11:47,740
- [Kamal] What about the rows
not in English?

222
00:11:47,740 --> 00:11:50,994
- [Narrator] These rows give 
additional statistical information.

223
00:11:50,994 --> 00:11:54,247
R-squared is a measure
of goodness of fit.

224
00:11:54,714 --> 00:11:56,009
This isn't too important,

225
00:11:56,009 --> 00:11:58,010
though some readers
may want to know it.

226
00:11:58,660 --> 00:12:02,950
Other rows report on alternative 
tests of statistical significance

227
00:12:02,950 --> 00:12:05,028
that you can ignore for now.

228
00:12:05,028 --> 00:12:07,934
- [Kamal] Oh my gosh,
these tables aren't that hard!

229
00:12:07,934 --> 00:12:09,488
Thank you so much.

230
00:12:09,488 --> 00:12:11,787
- [Narrator] Next up is regression.

231
00:12:11,787 --> 00:12:13,179
See you then!

232
00:12:15,974 --> 00:12:17,263
♪ [music] ♪

233
00:12:17,263 --> 00:12:20,575
You're on your way 
to mastering econometrics.

234
00:12:20,834 --> 00:12:22,783
Make sure this video sticks

235
00:12:22,783 --> 00:12:25,467
by taking a few 
quick practice questions.

236
00:12:25,467 --> 00:12:29,003
Or, if you're ready,
click for the next video.

237
00:12:29,003 --> 00:12:32,901
You can also check out MRU's 
website for more courses,

238
00:12:32,901 --> 00:12:35,298
teacher resources and more.