1
00:00:01,425 --> 00:00:05,060
- [Narrator] On his quest 
to master econometrics,

2
00:00:05,223 --> 00:00:08,813
Grasshopper Kamal has 
made great progress,

3
00:00:08,813 --> 00:00:13,662
stretching his capabilities 
and outsmarting his foes.

4
00:00:14,223 --> 00:00:16,640
Alas, today he's despondent,

5
00:00:16,640 --> 00:00:19,336
for one challenge remains unmet.

6
00:00:19,336 --> 00:00:24,130
Kamal cannot yet decode 
the scriptures of academic research,

7
00:00:24,130 --> 00:00:27,347
journals like 
"The American Economic Review"

8
00:00:27,347 --> 00:00:29,080
and "Econometrica."

9
00:00:29,202 --> 00:00:33,501
These seemed to him to be inscribed 
in an obscure foreign tongue.

10
00:00:33,501 --> 00:00:35,478
- [Kamal] Ugh, what the... ?

11
00:00:36,711 --> 00:00:40,018
- These volumes are 
opaque to the novice, Kamal,

12
00:00:40,018 --> 00:00:42,205
but can be deciphered with study.

13
00:00:42,467 --> 00:00:45,109
Let us learn to read them together.

14
00:00:52,485 --> 00:00:55,317
Let's dive into the West Point study,

15
00:00:55,317 --> 00:00:58,278
published in the "Economics 
of Education Review."

16
00:00:58,538 --> 00:01:01,688
This paper reports 
on a randomized evaluation

17
00:01:01,688 --> 00:01:05,859
of student electronics use 
in Economics 101 classrooms.

18
00:01:06,242 --> 00:01:09,192
First, a quick review 
of the research design.

19
00:01:09,423 --> 00:01:10,523
- Okay.

20
00:01:11,553 --> 00:01:13,630
- [Josh] 'Metrics masters 
teaching at West Point,

21
00:01:13,630 --> 00:01:16,620
the military college that trains 
American Army officers

22
00:01:16,620 --> 00:01:19,854
designed a randomized trial
to answer this question.

23
00:01:20,372 --> 00:01:23,233
These masters randomly assigned
West Point cadets

24
00:01:23,233 --> 00:01:26,383
into Economics classes
operating under different rules.

25
00:01:26,595 --> 00:01:28,962
Unlike most American colleges,

26
00:01:28,962 --> 00:01:31,945
the West Point default 
is no electronics.

27
00:01:32,345 --> 00:01:35,428
For purposes of this experiment,
some students were left

28
00:01:35,428 --> 00:01:38,679
in such traditional 
technology-free classes,

29
00:01:38,679 --> 00:01:41,911
no laptops, no tablets 
and no phones!

30
00:01:41,911 --> 00:01:43,280
[voice echoes]

31
00:01:43,328 --> 00:01:45,743
This is the control group,
or baseline case.

32
00:01:46,213 --> 00:01:49,198
Another group was allowed
to use electronics.

33
00:01:49,269 --> 00:01:52,704
This is the treatment group,
subject to a changed environment.

34
00:01:53,313 --> 00:01:55,858
The treatment in this case
is the unrestricted use

35
00:01:55,858 --> 00:01:58,107
of laptops or tablets in class.

36
00:01:58,844 --> 00:02:01,812
Every causal question 
has a clear outcome,

37
00:02:01,859 --> 00:02:05,276
the variables we hope to influence
defined in advance of the study.

38
00:02:05,860 --> 00:02:08,375
The outcomes in the West Point
electronics study

39
00:02:08,375 --> 00:02:09,994
are final exam scores.

40
00:02:10,047 --> 00:02:13,364
The study seeks to answer
the following question,

41
00:02:13,364 --> 00:02:17,299
what is the causal effect 
of classroom electronics on learning

42
00:02:17,299 --> 00:02:19,765
as measured by exam scores?

43
00:02:20,632 --> 00:02:24,199
- Economics journal articles 
usually begin with a table

44
00:02:24,199 --> 00:02:26,933
of descriptive statistics,
giving key facts

45
00:02:26,933 --> 00:02:28,500
about the study sample.

46
00:02:28,500 --> 00:02:31,781
- Oh my gosh, I remember this table,
so confusing!

47
00:02:31,781 --> 00:02:36,666
- [Narrator] Columns 1 to 3 report 
mean, or average, characteristics.

48
00:02:36,736 --> 00:02:39,688
These give a sense 
of who we're studying.

49
00:02:39,948 --> 00:02:43,736
Let's start with column 1 
which describes covariates

50
00:02:43,736 --> 00:02:45,251
in the control group.

51
00:02:45,408 --> 00:02:48,972
Covariates are characteristics 
of the control and treatment groups

52
00:02:48,972 --> 00:02:51,621
measured before 
the experiment begins.

53
00:02:51,621 --> 00:02:56,986
For example, we see the control group
has an average age a bit over 20.

54
00:02:57,321 --> 00:03:00,339
Many of these covariates 
are dummy variables.

55
00:03:00,790 --> 00:03:05,670
A dummy variable can only have 
two values, a zero or a one.

56
00:03:05,981 --> 00:03:10,015
For example, student gender
is captured by a dummy variable

57
00:03:10,015 --> 00:03:13,148
that equals one for women
and zero for men.

58
00:03:13,248 --> 00:03:16,580
The mean of this variable 
is the proportion female.

59
00:03:16,815 --> 00:03:20,651
We also see that the control group 
is 13% Hispanic

60
00:03:20,651 --> 00:03:23,769
and 19% had prior military service.

61
00:03:25,035 --> 00:03:26,635
The table notes are key.

62
00:03:26,635 --> 00:03:28,686
Refer to these 
as you scan the table.

63
00:03:29,102 --> 00:03:33,369
These notes explain what's shown 
in each column and panel.

64
00:03:39,485 --> 00:03:43,375
The notes tell us, for example,
that standard deviations

65
00:03:43,375 --> 00:03:45,175
are reported in brackets.

66
00:03:45,947 --> 00:03:49,598
Standard deviations tell us how
spread out the data are.

67
00:03:50,282 --> 00:03:54,887
For example, a standard deviation 
of 0.52 tells us that most

68
00:03:54,887 --> 00:03:59,233
of the control group's GPAs
fall between 2.35,

69
00:03:59,233 --> 00:04:03,454
which is 0.52 below 
the mean GPA of 2.87,

70
00:04:03,454 --> 00:04:08,337
and 3.39, which is 0.52 above 2.87.

71
00:04:09,001 --> 00:04:12,221
A lower standard deviation
would mean the GPAs were

72
00:04:12,221 --> 00:04:14,404
more tightly clustered 
around the mean.

73
00:04:14,549 --> 00:04:17,451
- [Kamal] Yeah, but they're missing
for most of the variables.

74
00:04:17,499 --> 00:04:18,600
- [Narrator] That's right.

75
00:04:18,600 --> 00:04:22,497
Masters usually omit 
standard deviations for dummies

76
00:04:22,497 --> 00:04:26,500
because the mean of this variable
determines its standard deviation.

77
00:04:26,936 --> 00:04:31,370
This study compares two treatment 
groups with the control group.

78
00:04:31,370 --> 00:04:35,753
The first was allowed free use 
of laptops and tablets.

79
00:04:35,753 --> 00:04:38,252
The second treatment
was more restrictive,

80
00:04:38,252 --> 00:04:41,553
allowing only tablets placed 
flat on the desk.

81
00:04:42,152 --> 00:04:45,238
The treatment groups
look much like the control group.

82
00:04:46,310 --> 00:04:51,270
This takes us to the next feature 
of this table, columns 4 through 6

83
00:04:51,407 --> 00:04:54,558
use statistical tests to compare
the characteristics

84
00:04:54,558 --> 00:04:57,591
of the treatment and control group
before the experiment.

85
00:04:58,023 --> 00:05:01,674
In column 4, the two treatment 
groups are combined.

86
00:05:01,856 --> 00:05:04,840
You can see that the difference
in proportion female

87
00:05:04,840 --> 00:05:09,690
between the treatment 
and control group is only 0.03.

88
00:05:09,991 --> 00:05:13,740
The difference is not 
statistically significant.

89
00:05:14,290 --> 00:05:17,205
It is the sort of difference
we can easily put down

90
00:05:17,205 --> 00:05:20,497
to chance results 
in our sample selection process.

91
00:05:20,497 --> 00:05:22,133
- [Kamal] Hmm, how do we know that?

92
00:05:22,133 --> 00:05:23,790
- [Narrator] Remember 
the rule of thumb?

93
00:05:23,790 --> 00:05:26,968
Statistical estimates 
that exceed the standard error

94
00:05:26,968 --> 00:05:29,882
by a multiple of 2 
in absolute value

95
00:05:30,105 --> 00:05:33,997
are usually said
to be statistically significant.

96
00:05:35,132 --> 00:05:38,419
The standard error is 0.03,

97
00:05:38,566 --> 00:05:41,483
same as the difference 
in proportion female.

98
00:05:42,015 --> 00:05:46,041
So the ratio of the latter 
to the former is only 1,

99
00:05:46,041 --> 00:05:48,607
which of course is less than 2.

100
00:05:48,607 --> 00:05:51,191
- [Kamal] Uh huh. So none
of the treatment/control differences

101
00:05:51,191 --> 00:05:54,333
in the table are more than twice
their standard errors.

102
00:05:54,333 --> 00:05:55,789
- [Narrator] Correct.

103
00:05:55,789 --> 00:05:59,000
The random division of students
appears to have succeeded

104
00:05:59,014 --> 00:06:01,945
in creating groups 
that are indeed comparable.

105
00:06:02,846 --> 00:06:06,362
We can be confident therefore
that any later differences

106
00:06:06,362 --> 00:06:09,830
in classroom achievement
are the result of the experimental

107
00:06:09,830 --> 00:06:12,579
intervention rather 
than a reflection

108
00:06:12,579 --> 00:06:14,646
of preexisting differences.

109
00:06:14,646 --> 00:06:17,230
Ceteris paribus achieved!

110
00:06:17,359 --> 00:06:20,718
- [Kamal] Cool. Wait, 
what about the bottom,

111
00:06:20,718 --> 00:06:22,522
the numbers with the stars?

112
00:06:22,714 --> 00:06:25,479
Those differences are a lot more
than double the standard error.

113
00:06:25,479 --> 00:06:27,402
- [Narrator] Good eye, Kamal!

114
00:06:27,402 --> 00:06:29,386
The table has many numbers.

115
00:06:29,386 --> 00:06:32,246
Those in Panel B are important too.

116
00:06:32,246 --> 00:06:35,715
This panel measures the extent 
to which students in treatment

117
00:06:35,715 --> 00:06:39,047
and control groups actually use 
computers in class.

118
00:06:39,497 --> 00:06:42,749
The treatment here was 
to allow computer use.

119
00:06:42,749 --> 00:06:46,066
The researchers must show
that students allowed

120
00:06:46,083 --> 00:06:49,448
to use computers took advantage 
of the opportunity to do so.

121
00:06:49,899 --> 00:06:53,032
If they didn't, then there's 
really no treatment.

122
00:06:53,248 --> 00:06:57,799
Luckily, 81% of those 
in the first treatment group

123
00:06:57,799 --> 00:07:01,832
used computers compared 
with none in the control group.

124
00:07:02,082 --> 00:07:05,032
And many in the second 
tablet treatment group

125
00:07:05,032 --> 00:07:06,997
used computers as well.

126
00:07:07,214 --> 00:07:09,731
These differences 
in computer use are large

127
00:07:09,731 --> 00:07:11,798
and statistically significant.

128
00:07:12,081 --> 00:07:15,366
We also get to see 
the sample size in each group.

129
00:07:15,366 --> 00:07:18,098
- [Kamal] The stars 
are just like decoration?

130
00:07:18,098 --> 00:07:21,748
- [Narrator] Some academic papers
use stars to indicate differences

131
00:07:21,748 --> 00:07:23,865
that are statistically significant.

132
00:07:23,865 --> 00:07:26,925
This makes them jump out at you.

133
00:07:26,925 --> 00:07:31,621
Here three stars indicate that 
the result is statistically different

134
00:07:31,621 --> 00:07:34,942
from zero with a p value 
less than 1%.

135
00:07:35,336 --> 00:07:39,436
In other words, there's less 
than a 1 in 100 chance

136
00:07:39,436 --> 00:07:42,171
this result is purely 
a chance finding.

137
00:07:42,171 --> 00:07:43,181
[applause]

138
00:07:43,181 --> 00:07:48,603
Two stars indicate a 1 in 20
or 5% chance of a chance finding.

139
00:07:48,802 --> 00:07:53,201
And one star denotes results 
we might see as often as 10%

140
00:07:53,201 --> 00:07:56,036
of the time merely due to chance.

141
00:07:56,473 --> 00:07:59,741
Today, stars are seen 
as a little old fashioned.

142
00:07:59,741 --> 00:08:01,440
Some journals omit them.

143
00:08:01,440 --> 00:08:03,324
- [Kamal] What about 
those last two columns?

144
00:08:03,324 --> 00:08:06,007
- [Narrator] Unlike column 4,
which combines

145
00:08:06,007 --> 00:08:09,689
both treatment groups into one, 
these last two columns

146
00:08:09,689 --> 00:08:12,357
look separately
at treatment/control differences

147
00:08:12,357 --> 00:08:14,360
for each treatment group.

148
00:08:14,360 --> 00:08:17,441
This provides a more detailed 
analysis of balance.

149
00:08:17,865 --> 00:08:21,064
Also, for now, 
you can ignore this row

150
00:08:21,064 --> 00:08:24,233
which provides 
another test of significance.

151
00:08:24,233 --> 00:08:28,916
Now we get to the article's 
punchline, table 4.

152
00:08:29,933 --> 00:08:32,993
This table reports
regression estimates

153
00:08:32,993 --> 00:08:36,984
of the effects of electronics use
on measures of student learning.

154
00:08:37,173 --> 00:08:40,026
- [Kamal Why does the study 
report regression estimates?

155
00:08:40,026 --> 00:08:42,205
See, that's why I was getting lost.

156
00:08:42,205 --> 00:08:44,806
I thought one reason 
why we liked randomized trials

157
00:08:44,806 --> 00:08:47,260
is that we use them 
to obtain causal effects

158
00:08:47,260 --> 00:08:50,138
simply by comparing 
treatment and control groups.

159
00:08:50,289 --> 00:08:53,489
Since these groups are balanced,
no need to use regression.

160
00:08:53,489 --> 00:08:55,290
- [Narrator] Well said, Kamal.

161
00:08:55,290 --> 00:08:59,272
In practice, it's customary 
to report regression estimates

162
00:08:59,272 --> 00:09:00,839
for two reasons.

163
00:09:00,975 --> 00:09:05,008
First, evidence of balance 
not withstanding, an abundance

164
00:09:05,008 --> 00:09:09,058
of caution might lead the analyst
to allow for chance differences.

165
00:09:09,392 --> 00:09:13,360
Second, regression estimates
are likely to be more precise.

166
00:09:13,726 --> 00:09:16,509
That is, they have lower 
standard errors than

167
00:09:16,509 --> 00:09:18,893
the simple treatment 
control comparisons.

168
00:09:19,905 --> 00:09:22,526
The dependent variable 
in this study

169
00:09:22,526 --> 00:09:24,305
is the outcome of interest.

170
00:09:24,652 --> 00:09:27,717
Since the question at hand
is how classroom electronics

171
00:09:27,717 --> 00:09:32,617
affect learning, a good outcome
is the Economics final exam score.

172
00:09:32,915 --> 00:09:37,167
Each column reports results
from a different regression model.

173
00:09:37,304 --> 00:09:40,476
Models are distinguished 
by the control variables

174
00:09:40,476 --> 00:09:44,712
or covariates they include
besides treatment status.

175
00:09:44,712 --> 00:09:48,425
Estimates with no covariates 
are simple comparisons

176
00:09:48,425 --> 00:09:50,502
of treatment and control groups.

177
00:09:50,619 --> 00:09:52,720
- [Kamal] I thought 
they just forgot to fill it out.

178
00:09:52,720 --> 00:09:56,668
- [Narrator] Column 1 suggests
electronics use reduced

179
00:09:56,668 --> 00:10:00,835
final exam scores
by 0.28 standard deviations.

180
00:10:01,102 --> 00:10:06,552
In our last lesson, Master Joshway
explained, we use standard deviation

181
00:10:06,552 --> 00:10:10,501
units because these units 
are easily compared across studies.

182
00:10:11,002 --> 00:10:13,702
Column 2 reports results 
from a model

183
00:10:13,702 --> 00:10:15,952
that adds demographic controls.

184
00:10:15,952 --> 00:10:19,502
Here we're comparing test scores
but holding constant factors

185
00:10:19,502 --> 00:10:21,435
such as age and sex.

186
00:10:21,886 --> 00:10:25,285
Column 3 reports results
from a model that adds GPA

187
00:10:25,285 --> 00:10:27,186
to the list of covariates.

188
00:10:27,603 --> 00:10:30,502
Column 4 adds ACT scores.

189
00:10:30,502 --> 00:10:33,503
Analysts often report 
results this way,

190
00:10:33,503 --> 00:10:36,803
starting with models that include 
few or no covariates

191
00:10:36,803 --> 00:10:40,452
and then reporting estimates 
from models that add more

192
00:10:40,452 --> 00:10:43,586
and more covariates 
as we move across columns.

193
00:10:44,035 --> 00:10:46,802
Looking across columns,
what do you notice?

194
00:10:47,252 --> 00:10:49,919
- [Kamal] Well, the coefficient 
on using a computer is always

195
00:10:49,919 --> 00:10:51,635
a pretty big negative number.

196
00:10:51,635 --> 00:10:53,002
- [Narrator] That's right!

197
00:10:53,002 --> 00:10:56,455
We can also see that 
the standard errors are small enough

198
00:10:56,455 --> 00:11:00,202
to make these negative results
statistically significant.

199
00:11:00,316 --> 00:11:04,201
In other words, the primary 
takeaway from this experiment

200
00:11:04,201 --> 00:11:08,318
is that electronics in the classroom
reduce student learning.

201
00:11:08,767 --> 00:11:11,884
- [Kama] GPA and ACT scores 
are also significant.

202
00:11:11,884 --> 00:11:13,600
Why is that?

203
00:11:13,600 --> 00:11:15,100
- [Narrator] Good observation!

204
00:11:15,100 --> 00:11:16,866
That's not surprising.

205
00:11:16,866 --> 00:11:20,267
We expect these variables 
to predict college performance.

206
00:11:20,267 --> 00:11:21,984
- [Kamal] Oh right, of course.

207
00:11:21,984 --> 00:11:24,817
Kids who got better grades before
are more likely to get

208
00:11:24,817 --> 00:11:26,317
a better grade in this course.

209
00:11:26,317 --> 00:11:29,849
- [Narrator] You'll also notice a lot 
of other information on this table.

210
00:11:29,849 --> 00:11:34,234
Remaining panels in the table
report effects of electronics use

211
00:11:34,234 --> 00:11:36,933
on components of the final exam,

212
00:11:36,933 --> 00:11:39,467
such as the multiple 
choice questions.

213
00:11:39,467 --> 00:11:43,285
These results are mostly consistent 
with computer use effects

214
00:11:43,285 --> 00:11:45,216
on overall scores.

215
00:11:45,216 --> 00:11:47,740
- [Kamal] What about the rows
not in English?

216
00:11:47,740 --> 00:11:50,561
- [Narrator] These rows give 
additional statistical information.

217
00:11:50,828 --> 00:11:53,978
R-squared is a measure
of goodness of fit.

218
00:11:54,311 --> 00:11:58,010
This isn't too important, though 
some readers may want to know it.

219
00:11:58,660 --> 00:12:02,827
Other rows report on alternative 
tests of statistical significance

220
00:12:02,827 --> 00:12:05,028
that you can ignore for now.

221
00:12:05,028 --> 00:12:07,644
- [Kamal] Oh my gosh,
these tables aren't that hard!

222
00:12:07,644 --> 00:12:09,488
Thank you so much.

223
00:12:09,488 --> 00:12:11,787
- [Narrator] Next up is regression.

224
00:12:11,787 --> 00:12:13,179
See you then!

225
00:12:15,974 --> 00:12:17,263
♪ [music] ♪

226
00:12:17,263 --> 00:12:20,575
You're on your way 
to mastering econometrics.

227
00:12:20,834 --> 00:12:22,783
Make sure this video sticks

228
00:12:22,783 --> 00:12:24,982
by taking a few 
quick practice questions.

229
00:12:25,153 --> 00:12:28,855
Or, if you're ready,
click for the next video.

230
00:12:28,855 --> 00:12:32,620
You can also check out MRU's 
website for more courses,

231
00:12:32,620 --> 00:12:35,298
teacher resources and more.