1
00:00:01,425 --> 00:00:05,060
- [Narrator] On his quest 
to master econometrics,

2
00:00:05,223 --> 00:00:08,813
Grasshopper Kamal has 
made great progress,

3
00:00:08,813 --> 00:00:13,662
stretching his capabilities 
and outsmarting his foes.

4
00:00:14,223 --> 00:00:16,640
Alas, today he's despondent,

5
00:00:16,640 --> 00:00:19,336
for one challenge remains unmet.

6
00:00:19,336 --> 00:00:24,130
Kamal cannot yet decode 
the scriptures of academic research,

7
00:00:24,130 --> 00:00:27,347
journals like 
"The American Economic Review"

8
00:00:27,347 --> 00:00:29,080
and "Econometrica."

9
00:00:29,202 --> 00:00:33,501
These seemed to him to be inscribed 
in an obscure foreign tongue.

10
00:00:33,501 --> 00:00:35,478
- [Kamal] Ugh, what the... ?

11
00:00:36,711 --> 00:00:40,018
- These volumes are 
opaque to the novice, Kamal,

12
00:00:40,018 --> 00:00:42,205
but can be deciphered with study.

13
00:00:42,467 --> 00:00:45,109
Let us learn to read them together.

14
00:00:52,485 --> 00:00:55,317
Let's dive into the West Point study,

15
00:00:55,317 --> 00:00:58,278
published in the "Economics 
of Education Review."

16
99:59:59,999 --> 99:59:59,999
This paper reports 
on a randomized evaluation

17
99:59:59,999 --> 99:59:59,999
of student electronics use 
in Economics 101 classrooms.

18
99:59:59,999 --> 99:59:59,999
First, a quick review 
of the research design.

19
99:59:59,999 --> 99:59:59,999
- Okay.

20
99:59:59,999 --> 99:59:59,999
- [Josh] 'Metrics masters 
teaching at West Point,

21
99:59:59,999 --> 99:59:59,999
the military college that trains 
American Army officers

22
99:59:59,999 --> 99:59:59,999
designed a randomized trial
to answer this question.

23
99:59:59,999 --> 99:59:59,999
These masters randomly assigned
West Point cadets

24
99:59:59,999 --> 99:59:59,999
into Economics classes
operating under different rules.

25
99:59:59,999 --> 99:59:59,999
Unlike most American colleges,

26
99:59:59,999 --> 99:59:59,999
the West Point default 
is no electronics.

27
99:59:59,999 --> 99:59:59,999
For purposes of this experiment,
some students were left

28
99:59:59,999 --> 99:59:59,999
in such traditional 
technology-free classes,

29
99:59:59,999 --> 99:59:59,999
no lap tops, no tablets 
and no phones!

30
99:59:59,999 --> 99:59:59,999
[voice echoes]

31
99:59:59,999 --> 99:59:59,999
This is the control group,
or baseline case.

32
99:59:59,999 --> 99:59:59,999
Another group was allowed
to use electronics.

33
99:59:59,999 --> 99:59:59,999
This is the treatment group,
subject to a changed environment.

34
99:59:59,999 --> 99:59:59,999
The treatment in this case
is the unrestricted use

35
99:59:59,999 --> 99:59:59,999
of laptops or tablets in class.

36
99:59:59,999 --> 99:59:59,999
Every causal question 
has a clear outcome,

37
99:59:59,999 --> 99:59:59,999
the variables we hope to influence
defined in advance of the study.

38
99:59:59,999 --> 99:59:59,999
The outcomes in the West Point
electronics study

39
99:59:59,999 --> 99:59:59,999
are final exam scores.

40
99:59:59,999 --> 99:59:59,999
The study seeks to answer
the following question,

41
99:59:59,999 --> 99:59:59,999
what is the causal effect 
of classroom electronics on learning

42
99:59:59,999 --> 99:59:59,999
as measured by exam scores?

43
99:59:59,999 --> 99:59:59,999
- Economics journal articles 
usually begin with a table

44
99:59:59,999 --> 99:59:59,999
of descriptive statistics,
giving key facts

45
99:59:59,999 --> 99:59:59,999
about the study sample.

46
99:59:59,999 --> 99:59:59,999
- Oh my gosh, I remember this table,
so confusing!

47
99:59:59,999 --> 99:59:59,999
Columns 1 to 3 report mean,
or average, characteristics.

48
99:59:59,999 --> 99:59:59,999
These give a sense 
of who we're studying.

49
99:59:59,999 --> 99:59:59,999
Let's start with column 1 
which describes covariates

50
99:59:59,999 --> 99:59:59,999
in a control group.

51
99:59:59,999 --> 99:59:59,999
Covariates are characteristics 
of the control and treatment groups

52
99:59:59,999 --> 99:59:59,999
measured before 
the experiment begins.

53
99:59:59,999 --> 99:59:59,999
For example, we see the control group
has an average age a bit over 20.

54
99:59:59,999 --> 99:59:59,999
Many of these covariates 
are dummy variables.

55
99:59:59,999 --> 99:59:59,999
A dummy variable can only have 
two values, a zero or a one.

56
99:59:59,999 --> 99:59:59,999
For example, student gender
is captured by a dummy variable

57
99:59:59,999 --> 99:59:59,999
that equals one for woman
and zero for man.

58
99:59:59,999 --> 99:59:59,999
The mean of this variable 
is the proportion female.

59
99:59:59,999 --> 99:59:59,999
We also see that the control group 
is 13% Hispanic

60
99:59:59,999 --> 99:59:59,999
and 19% had prior military service.

61
99:59:59,999 --> 99:59:59,999
The table notes are key.

62
99:59:59,999 --> 99:59:59,999
Refer to these 
as you scan the table.

63
99:59:59,999 --> 99:59:59,999
These notes explain what's shown 
in each column and panel.

64
99:59:59,999 --> 99:59:59,999
The notes tell us, for example,
that standard deviations

65
99:59:59,999 --> 99:59:59,999
are reported in brackets.

66
99:59:59,999 --> 99:59:59,999
Standard deviations tell us how
spread out the data are.

67
99:59:59,999 --> 99:59:59,999
For example, a standard deviation 
of 0.52 tells us that most

68
99:59:59,999 --> 99:59:59,999
of the control group's GPAs
fall between 2.35,

69
99:59:59,999 --> 99:59:59,999
which is 0.52 below 
the mean GPA of 2.87,

70
99:59:59,999 --> 99:59:59,999
and 3.39, which is 0.52 above 2.87.

71
99:59:59,999 --> 99:59:59,999
A lower standard deviation
would mean the GPAs were

72
99:59:59,999 --> 99:59:59,999
more tightly clustered 
around the mean.

73
99:59:59,999 --> 99:59:59,999
- [Kamal] Yeah, but they're missing
from most of the variables.

74
99:59:59,999 --> 99:59:59,999
- [Narrator] That's right.

75
99:59:59,999 --> 99:59:59,999
Masters usually omit 
standard deviations for dummies

76
99:59:59,999 --> 99:59:59,999
because the mean of this variable
determines its standard deviaton.

77
99:59:59,999 --> 99:59:59,999
This study compares two treatment 
groups with the control group.

78
99:59:59,999 --> 99:59:59,999
The first was allowed free use 
of laptops and tablets.

79
99:59:59,999 --> 99:59:59,999
The second treatment
was more restrictive,

80
99:59:59,999 --> 99:59:59,999
allowing only tablets placed 
flat on the desk.

81
99:59:59,999 --> 99:59:59,999
The treatment groups
look much like the control group.

82
99:59:59,999 --> 99:59:59,999
Hmmm...

83
99:59:59,999 --> 99:59:59,999
This takes us to the next feature 
of this table, columns 4 through 6

84
99:59:59,999 --> 99:59:59,999
use statistical tests to compare
the characteristics

85
99:59:59,999 --> 99:59:59,999
of the treatment and control group
before the experiment.

86
99:59:59,999 --> 99:59:59,999
In column 4, the two treatment 
groups are combined.

87
99:59:59,999 --> 99:59:59,999
You can see that the difference
in proportion female

88
99:59:59,999 --> 99:59:59,999
between the treatment 
and control group is only 0.03.

89
99:59:59,999 --> 99:59:59,999
The difference is not 
statistically significant.

90
99:59:59,999 --> 99:59:59,999
It is the sort of difference
we can easily put down

91
99:59:59,999 --> 99:59:59,999
to chance results 
in our sample selection process.

92
99:59:59,999 --> 99:59:59,999
- [Kamal] Hmm, how do we know that?

93
99:59:59,999 --> 99:59:59,999
- [Narrator] Remember 
the rule of thumb?

94
99:59:59,999 --> 99:59:59,999
Statistical estimates 
that exceed the standard

95
99:59:59,999 --> 99:59:59,999
by a multiple of 2 
in absolute value

96
99:59:59,999 --> 99:59:59,999
are usually said
to be statistically significant.

97
99:59:59,999 --> 99:59:59,999
The standard error is 0.03,

98
99:59:59,999 --> 99:59:59,999
same as the difference 
in proportion female.

99
99:59:59,999 --> 99:59:59,999
So the ratio of the latter 
to the former is only 1,

100
99:59:59,999 --> 99:59:59,999
which of course is less than 2.

101
99:59:59,999 --> 99:59:59,999
- [Kamal] Uh huh. So none
of the treatment/control differences

102
99:59:59,999 --> 99:59:59,999
in the table are more than twice
their standard errors.

103
99:59:59,999 --> 99:59:59,999
- [Narrator] Correct.

104
99:59:59,999 --> 99:59:59,999
The random division of students
appears to have succeeded

105
99:59:59,999 --> 99:59:59,999
in creating groups 
that are indeed comparable.

106
99:59:59,999 --> 99:59:59,999
We can be confident therefore
that any later differences

107
99:59:59,999 --> 99:59:59,999
in classroom achievement
are the result of the experimental

108
99:59:59,999 --> 99:59:59,999
intervention rather 
than a reflection

109
99:59:59,999 --> 99:59:59,999
of preexisting differences.

110
99:59:59,999 --> 99:59:59,999
Ceteris paribus achieved!

111
99:59:59,999 --> 99:59:59,999
- [Kamal] Cool. Wait, 
what about the bottom,

112
99:59:59,999 --> 99:59:59,999
the numbers with the stars?

113
99:59:59,999 --> 99:59:59,999
Those differences are a lot more
than double the standard error.

114
99:59:59,999 --> 99:59:59,999
- [Narrator] Good eye, Kamal!

115
99:59:59,999 --> 99:59:59,999
The table has many numbers.

116
99:59:59,999 --> 99:59:59,999
Those in Panel B are important too.

117
99:59:59,999 --> 99:59:59,999
This panel measures the extent 
to which students in treatment

118
99:59:59,999 --> 99:59:59,999
and control groups actually use 
computers in class.

119
99:59:59,999 --> 99:59:59,999
The treatment here was 
to allow computer use.

120
99:59:59,999 --> 99:59:59,999
The researchers must show
that students allowed

121
99:59:59,999 --> 99:59:59,999
to use computers took advantage 
of the opportunity to do so.

122
99:59:59,999 --> 99:59:59,999
If they didn't, then there's 
really no treatment.

123
99:59:59,999 --> 99:59:59,999
Luckily, 81% of those 
in the first treatment group

124
99:59:59,999 --> 99:59:59,999
used computers compared 
with none in the control group.

125
99:59:59,999 --> 99:59:59,999
And many in the second 
tablet treatment group

126
99:59:59,999 --> 99:59:59,999
used computers as well.

127
99:59:59,999 --> 99:59:59,999
These differences 
in computer use are large

128
99:59:59,999 --> 99:59:59,999
and statistically significant.

129
99:59:59,999 --> 99:59:59,999
We also get to see 
the sample size in each group.

130
99:59:59,999 --> 99:59:59,999
- [Kamal] The stars 
are just like decoration?

131
99:59:59,999 --> 99:59:59,999
- [Narrator] Some academic papers
use stars to indicate differences

132
99:59:59,999 --> 99:59:59,999
that are statistically significant.

133
99:59:59,999 --> 99:59:59,999
This makes them jump out at you.

134
99:59:59,999 --> 99:59:59,999
Here three stars indicate that 
the result is statistically different

135
99:59:59,999 --> 99:59:59,999
from zero with a p value 
less than 1%.

136
99:59:59,999 --> 99:59:59,999
In other words, there's less 
than a 1 in 100 chance

137
99:59:59,999 --> 99:59:59,999
this result is purely 
a chance finding.

138
99:59:59,999 --> 99:59:59,999
[applause]

139
99:59:59,999 --> 99:59:59,999
Two stars indicate a 1 in 20
or 5% chance of a chance finding.

140
99:59:59,999 --> 99:59:59,999
And one star denotes results 
we might see as often as 10%

141
99:59:59,999 --> 99:59:59,999
of the time merely due to chance.

142
99:59:59,999 --> 99:59:59,999
Today, stars are seen 
as a little old fashioned.

143
99:59:59,999 --> 99:59:59,999
Some journals omit them.

144
99:59:59,999 --> 99:59:59,999
- [Kamal] What about 
those last two columns?

145
99:59:59,999 --> 99:59:59,999
- [Narrator] Unlike column 4,
which combines

146
99:59:59,999 --> 99:59:59,999
both treatment groups into one, 
these last two columns

147
99:59:59,999 --> 99:59:59,999
look separately
at treatment/control differences

148
99:59:59,999 --> 99:59:59,999
for each treatment group.

149
99:59:59,999 --> 99:59:59,999
This provides a more detailed 
analysis of balance.

150
99:59:59,999 --> 99:59:59,999
Also, for now, 
you can ignore this row

151
99:59:59,999 --> 99:59:59,999
which provides 
another test of significance.

152
99:59:59,999 --> 99:59:59,999
Now we get to the article's 
punchline, table 4.

153
99:59:59,999 --> 99:59:59,999
This table reports
regression estimates

154
99:59:59,999 --> 99:59:59,999
of the effects of electronics use
on measures of student learning.

155
99:59:59,999 --> 99:59:59,999
- [Kamal Why does the study 
report regression estimates?

156
99:59:59,999 --> 99:59:59,999
See, that's why I was getting lost.

157
99:59:59,999 --> 99:59:59,999
I thought one reason 
why we liked randomized trials

158
99:59:59,999 --> 99:59:59,999
is that we use them 
to obtain causal effects

159
99:59:59,999 --> 99:59:59,999
simply by comparing 
treatment and control groups.

160
99:59:59,999 --> 99:59:59,999
Since these groups are balanced,
no need to use regression.

161
99:59:59,999 --> 99:59:59,999
- [Narrator] Well said, Kamal.

162
99:59:59,999 --> 99:59:59,999
In practice, it's customary 
to report regression estimates

163
99:59:59,999 --> 99:59:59,999
for two reasons.

164
99:59:59,999 --> 99:59:59,999
First, evidence of balance 
not withstanding, an abundance

165
99:59:59,999 --> 99:59:59,999
of caution might lead the analyst
to allow for chance differences.

166
99:59:59,999 --> 99:59:59,999
Second, regression estimates
are likely to be more precise.

167
99:59:59,999 --> 99:59:59,999
That is, they have lower 
standard errors than

168
99:59:59,999 --> 99:59:59,999
the simple treatment 
control comparisons.

169
99:59:59,999 --> 99:59:59,999
The dependent variable 
in this study

170
99:59:59,999 --> 99:59:59,999
is the outcome of interest.

171
99:59:59,999 --> 99:59:59,999
Since the question at hand
is how classroom electronics

172
99:59:59,999 --> 99:59:59,999
affect learning, a good outcome
is the economics final exam score.

173
99:59:59,999 --> 99:59:59,999
Each column reports results
from a different regression model.

174
99:59:59,999 --> 99:59:59,999
Models are distinguished 
by the control variables

175
99:59:59,999 --> 99:59:59,999
or covariates they include
besides treatment status.

176
99:59:59,999 --> 99:59:59,999
Estimates with no covariates 
are simple comparisons

177
99:59:59,999 --> 99:59:59,999
of treatment and control groups.

178
99:59:59,999 --> 99:59:59,999
- [Kamal] I thought 
they just forgot to fill it out.

179
99:59:59,999 --> 99:59:59,999
- [Narrator] Column 1 suggests
electronics use reduced

180
99:59:59,999 --> 99:59:59,999
final exam scores
by 0.28 standard deviations.

181
99:59:59,999 --> 99:59:59,999
In our last lesson, Master Joshway
explained, we use standard deviation

182
99:59:59,999 --> 99:59:59,999
units because these units 
are easily compared across studies.

183
99:59:59,999 --> 99:59:59,999
Column 2 reports results 
from a model

184
99:59:59,999 --> 99:59:59,999
that adds demographic controls.

185
99:59:59,999 --> 99:59:59,999
Here we're comparing test scores
but holding constant factors

186
99:59:59,999 --> 99:59:59,999
such as age and sex.

187
99:59:59,999 --> 99:59:59,999
Column 3 reports results
from a model that adds GPA

188
99:59:59,999 --> 99:59:59,999
to the list of covariates.

189
99:59:59,999 --> 99:59:59,999
Column 4 adds ACT scores.

190
99:59:59,999 --> 99:59:59,999
Analysts often report 
results this way,

191
99:59:59,999 --> 99:59:59,999
starting with models that include 
few or no covariates

192
99:59:59,999 --> 99:59:59,999
and then reporting estimates 
from models that add more

193
99:59:59,999 --> 99:59:59,999
and more covariates 
as we move across columns.

194
99:59:59,999 --> 99:59:59,999
Looking across columns,
what do you notice?

195
99:59:59,999 --> 99:59:59,999
- [Kamal] Well, the coefficient 
on using a computer is always

196
99:59:59,999 --> 99:59:59,999
a pretty big negative number.

197
99:59:59,999 --> 99:59:59,999
- [Narrator] That's right!

198
99:59:59,999 --> 99:59:59,999
We can also see that 
the standard errors are small enough

199
99:59:59,999 --> 99:59:59,999
to make these negative results
statistically significant.

200
99:59:59,999 --> 99:59:59,999
In other words, the primary 
takeaway from this experiment

201
99:59:59,999 --> 99:59:59,999
is that electronics in the classroom
reduce student learning.

202
99:59:59,999 --> 99:59:59,999
- [Kama] GPA and ACT scores 
are also significant.

203
99:59:59,999 --> 99:59:59,999
Why is that?

204
99:59:59,999 --> 99:59:59,999
- [Narrator] Good observation!

205
99:59:59,999 --> 99:59:59,999
That's not surprising.

206
99:59:59,999 --> 99:59:59,999
We expect these variables 
to predict college performance.

207
99:59:59,999 --> 99:59:59,999
- [Kamal] Oh right, of course.

208
99:59:59,999 --> 99:59:59,999
Kids who got better grades before
are more likely to get

209
99:59:59,999 --> 99:59:59,999
a better grade in this course.

210
99:59:59,999 --> 99:59:59,999
- [Narrator] You'll also notice a lot 
of other information on this table.

211
99:59:59,999 --> 99:59:59,999
Remaining panels in the table
report effects of electronics use

212
99:59:59,999 --> 99:59:59,999
on components of the final exam,

213
99:59:59,999 --> 99:59:59,999
such as the multiple 
choice questions.

214
99:59:59,999 --> 99:59:59,999
These results are mostly consistent 
with computer use effects

215
99:59:59,999 --> 99:59:59,999
on overall scores.

216
99:59:59,999 --> 99:59:59,999
- [Kamal] What about the rows
not in English?

217
99:59:59,999 --> 99:59:59,999
- [Narrator] These rows give 
additional statistical information.

218
99:59:59,999 --> 99:59:59,999
R-squared is a measure
of goodness of fit.

219
99:59:59,999 --> 99:59:59,999
This isn't too important, though 
some readers may want to know it.

220
99:59:59,999 --> 99:59:59,999
Other rows report on alternative 
tests of statistical significance

221
99:59:59,999 --> 99:59:59,999
that you can ignore for now.

222
99:59:59,999 --> 99:59:59,999
- [Kamal] Oh my gosh,
these tables aren't that hard.

223
99:59:59,999 --> 99:59:59,999
Thank you so much.

224
99:59:59,999 --> 99:59:59,999
Next up is regression.

225
99:59:59,999 --> 99:59:59,999
See you then!

226
99:59:59,999 --> 99:59:59,999
♪ [music] ♪

227
99:59:59,999 --> 99:59:59,999
You're on your way 
to mastering econometrics.

228
99:59:59,999 --> 99:59:59,999
Make sure this video sticks

229
99:59:59,999 --> 99:59:59,999
by taking a few 
quick practice questions.

230
99:59:59,999 --> 99:59:59,999
Or, if you're ready,
click for the next video.

231
99:59:59,999 --> 99:59:59,999
You can also check out MRU's 
website for more courses,

232
99:59:59,999 --> 99:59:59,999
teacher resources and more.