How to Read Economics Research Papers: Randomized Controlled Trials (RCTs)
-
0:01 - 0:05- [Narrator] On his quest
to master econometrics, -
0:06 - 0:09Grasshopper Kamal
has made great progress, -
0:09 - 0:14stretching his capabilities
and outsmarting his foes. -
0:14 - 0:17Alas, today he's despondent,
-
0:17 - 0:20for one challenge remains unmet.
-
0:20 - 0:24Kamal cannot yet decode
the scriptures of academic research, -
0:24 - 0:27journals like
"The American Economic Review" -
0:27 - 0:29and "Econometrica."
-
0:29 - 0:34These seemed to him to be inscribed
in an obscure foreign tongue. -
0:34 - 0:35- [Kamal] Ugh, what the... ?
-
0:37 - 0:40- These volumes are opaque
to the novice, Kamal, -
0:40 - 0:42but can be deciphered with study.
-
0:42 - 0:45Let us learn to read them together.
-
0:53 - 0:55Let's dive into the West Point study,
-
0:55 - 0:58published in the "Economics
of Education Review." -
0:59 - 1:02This paper reports
on a randomized evaluation -
1:02 - 1:06of student electronics use
in Economics 101 classrooms. -
1:06 - 1:09First, a quick review
of the research design. -
1:09 - 1:11- Okay.
-
1:12 - 1:14- [Josh] 'Metrics masters
teaching at West Point, -
1:14 - 1:17the military college that trains
American Army officers -
1:17 - 1:20designed a randomized trial
to answer this question. -
1:21 - 1:23These masters randomly assigned
West Point cadets -
1:23 - 1:27into Economics classes
operating under different rules. -
1:27 - 1:29Unlike most American colleges,
-
1:29 - 1:32the West Point default
is no electronics. -
1:32 - 1:36For purposes of this experiment,
some students were left -
1:36 - 1:39in such traditional
technology-free classes -- -
1:39 - 1:42no laptops, no tablets
and no phones! -
1:42 - 1:43[voice echoes]
-
1:43 - 1:46This is the control group,
or baseline case. -
1:46 - 1:49Another group was allowed
to use electronics. -
1:49 - 1:53This is the treatment group,
subject to a changed environment. -
1:53 - 1:56The treatment in this case
is the unrestricted use -
1:56 - 1:58of laptops or tablets in class.
-
1:59 - 2:02Every causal question
has a clear outcome -- -
2:02 - 2:05the variables we hope to influence
defined in advance of the study. -
2:06 - 2:08The outcomes in the West Point
electronics study -
2:08 - 2:11are final exam scores.
-
2:11 - 2:14The study seeks to answer
the following question: -
2:14 - 2:18What is the causal effect
of classroom electronics on learning -
2:18 - 2:20as measured by exam scores?
-
2:21 - 2:24- Economics journal articles
usually begin with a table -
2:24 - 2:26of descriptive statistics,
-
2:26 - 2:29giving key facts
about the study sample. -
2:29 - 2:32- Oh my gosh, I remember
this table, so confusing! -
2:32 - 2:37- [Narrator] Columns 1 to 3 report
mean, or average, characteristics. -
2:37 - 2:40These give a sense
of who we're studying. -
2:40 - 2:44Let's start with column 1
which describes covariates -
2:44 - 2:45in the control group.
-
2:45 - 2:49Covariates are characteristics
of the control and treatment groups -
2:49 - 2:52measured before
the experiment begins. -
2:52 - 2:58For example, we see the control group
has an average age a bit over 20. -
2:58 - 3:00Many of these covariates
are dummy variables. -
3:01 - 3:06A dummy variable can only have
two values -- a zero or a one. -
3:06 - 3:10For example, student gender
is captured by a dummy variable -
3:10 - 3:13that equals one for women
and zero for men. -
3:13 - 3:17The mean of this variable
is the proportion female. -
3:17 - 3:21We also see that the control group
is 13% Hispanic -
3:21 - 3:24and 19% had prior military service.
-
3:25 - 3:27The table notes are key.
-
3:27 - 3:29Refer to these
as you scan the table. -
3:29 - 3:34These notes explain what's shown
in each column and panel. -
3:39 - 3:42The notes tell us, for example,
-
3:42 - 3:45that standard deviations
are reported in brackets. -
3:46 - 3:50Standard deviations tell us how
spread out the data are. -
3:50 - 3:55For example, a standard deviation
of 0.52 tells us that most -
3:55 - 3:59of the control group's GPAs
fall between 2.35, -
3:59 - 4:03which is 0.52 below
the mean GPA of 2.87, -
4:03 - 4:08and 3.39, which is 0.52 above 2.87.
-
4:09 - 4:12A lower standard deviation
would mean the GPAs -
4:12 - 4:15were more tightly clustered
around the mean. -
4:15 - 4:18- [Kamal] Yeah, but they're missing
for most of the variables. -
4:18 - 4:19- [Narrator] That's right.
-
4:19 - 4:22Masters usually omit
standard deviations for dummies -
4:22 - 4:26because the mean of this variable
determines its standard deviation. -
4:28 - 4:32This study compares two treatment
groups with the control group. -
4:32 - 4:36The first was allowed free use
of laptops and tablets. -
4:36 - 4:38The second treatment
was more restrictive, -
4:38 - 4:42allowing only tablets
placed flat on the desk. -
4:42 - 4:45The treatment groups
look much like the control group. -
4:47 - 4:51This takes us to the next feature
of this table, columns 4 through 6 -
4:51 - 4:55use statistical tests to compare
the characteristics -
4:55 - 4:58of the treatment and control group
before the experiment. -
4:58 - 5:02In column 4, the two treatment
groups are combined. -
5:02 - 5:05You can see that the difference
in proportion female -
5:05 - 5:10between the treatment
and control group is only 0.03. -
5:11 - 5:14The difference is not
statistically significant. -
5:14 - 5:17It is the sort of difference
we can easily put down -
5:17 - 5:21to chance results
in our sample selection process. -
5:21 - 5:22- [Kamal] Hmm, how do we know that?
-
5:22 - 5:24- [Narrator] Remember
the rule of thumb? -
5:24 - 5:27Statistical estimates
that exceed the standard error -
5:27 - 5:30by a multiple of 2
in absolute value -
5:30 - 5:34are usually said
to be statistically significant. -
5:35 - 5:39The standard error is 0.03,
-
5:39 - 5:41same as the difference
in proportion female. -
5:42 - 5:46So the ratio of the latter
to the former is only 1, -
5:46 - 5:49which of course is less than 2.
-
5:49 - 5:51- [Kamal] Uh huh. So none
of the treatment/control differences -
5:51 - 5:54in the table are more than twice
their standard errors. -
5:54 - 5:56- [Narrator] Correct.
-
5:56 - 5:59The random division of students
appears to have succeeded -
5:59 - 6:02in creating groups
that are indeed comparable. -
6:03 - 6:05We can be confident, therefore,
-
6:05 - 6:08that any later differences
in classroom achievement -
6:08 - 6:11are the result
of the experimental intervention -
6:11 - 6:15rather than a reflection
of preexisting differences. -
6:15 - 6:17Ceteris paribus achieved!
-
6:17 - 6:21- [Kamal] Cool. Wait,
what about the bottom, -
6:21 - 6:23the numbers with the stars?
-
6:23 - 6:26Those differences are a lot more
than double the standard error. -
6:26 - 6:27- [Narrator] Good eye, Kamal!
-
6:27 - 6:29The table has many numbers.
-
6:29 - 6:32Those in Panel B are important too.
-
6:32 - 6:36This panel measures the extent
to which students in treatment -
6:36 - 6:39and control groups actually use
computers in class. -
6:40 - 6:43The treatment here
was to allow computer use. -
6:43 - 6:45The researchers must show
-
6:45 - 6:47that students allowed
to use computers -
6:47 - 6:49took advantage
of the opportunity to do so. -
6:50 - 6:54If they didn't, then there's
really no treatment. -
6:54 - 6:58Luckily, 81% of those
in the first treatment group -
6:58 - 6:59used computers
-
6:59 - 7:02compared with none
in the control group. -
7:02 - 7:05And many in the second
tablet treatment group -
7:05 - 7:07used computers as well.
-
7:07 - 7:10These differences
in computer use are large -
7:10 - 7:12and statistically significant.
-
7:12 - 7:15We also get to see
the sample size in each group. -
7:15 - 7:18- [Kamal] The stars
are just like decoration? -
7:18 - 7:22- [Narrator] Some academic papers
use stars to indicate differences -
7:22 - 7:24that are statistically significant.
-
7:24 - 7:27This makes them jump out at you.
-
7:27 - 7:32Here, three stars indicate that
the result is statistically different -
7:32 - 7:35from zero with a p value
less than 1%. -
7:36 - 7:39In other words, there's less
than a 1 in 100 chance -
7:39 - 7:42this result is purely
a chance finding. -
7:42 - 7:43[applause]
-
7:43 - 7:49Two stars indicate a 1 in 20
or 5% chance of a chance finding. -
7:49 - 7:52And one star denotes results
we might see as often -
7:52 - 7:56as 10% of the time
merely due to chance. -
7:56 - 8:00Today, stars are seen
as a little old fashioned. -
8:00 - 8:02Some journals omit them.
-
8:02 - 8:04- [Kamal] What about
those last two columns? -
8:04 - 8:06- [Narrator] Unlike column 4,
which combines -
8:06 - 8:10both treatment groups into one,
these last two columns -
8:10 - 8:12look separately
at treatment/control differences -
8:12 - 8:15for each treatment group.
-
8:15 - 8:17This provides a more detailed
analysis of balance. -
8:18 - 8:21Also, for now,
you can ignore this row -
8:21 - 8:25which provides
another test of significance. -
8:25 - 8:29Now we get to the article's
punchline, table 4. -
8:30 - 8:33This table reports
regression estimates -
8:33 - 8:37of the effects of electronics use
on measures of student learning. -
8:37 - 8:40- [Kamal Why does the study
report regression estimates? -
8:40 - 8:43See, that's why I was getting lost.
-
8:43 - 8:45I thought one reason
why we liked randomized trials -
8:45 - 8:47is that we use them
to obtain causal effects -
8:47 - 8:50simply by comparing
treatment and control groups. -
8:50 - 8:54Since these groups are balanced,
no need to use regression. -
8:54 - 8:55- [Narrator] Well said, Kamal.
-
8:55 - 8:59In practice, it's customary
to report regression estimates -
8:59 - 9:01for two reasons.
-
9:01 - 9:04First, evidence of balance
notwithstanding, -
9:04 - 9:07an abundance of caution
might lead the analyst -
9:07 - 9:10to allow for chance differences.
-
9:10 - 9:14Second, regression estimates
are likely to be more precise. -
9:14 - 9:17That is, they have lower
standard errors -
9:17 - 9:19than the simple treatment
control comparisons. -
9:20 - 9:23The dependent variable
in this study -
9:23 - 9:24is the outcome of interest.
-
9:25 - 9:28Since the question at hand
is how classroom electronics -
9:28 - 9:29affect learning,
-
9:29 - 9:33a good outcome
is the Economics final exam score. -
9:33 - 9:38Each column reports results
from a different regression model. -
9:38 - 9:40Models are distinguished
by the control variables -
9:40 - 9:45or covariates they include
besides treatment status. -
9:45 - 9:48Estimates with no covariates
are simple comparisons -
9:48 - 9:51of treatment and control groups.
-
9:51 - 9:53- [Kamal] I thought
they just forgot to fill it out. -
9:53 - 9:56- [Narrator] Column 1
suggests electronics use -
9:56 - 10:01reduced final exam scores
by 0.28 standard deviations. -
10:02 - 10:03In our last lesson,
-
10:03 - 10:07Master Joshway explained
that we use standard deviation units -
10:07 - 10:11because these units
are easily compared across studies. -
10:11 - 10:14Column 2 reports results
from a model -
10:14 - 10:16that adds demographic controls.
-
10:16 - 10:20Here, we're comparing test scores
but holding constant factors -
10:20 - 10:21such as age and sex.
-
10:22 - 10:26Column 3 reports results
from a model that adds GPA -
10:26 - 10:27to the list of covariates.
-
10:28 - 10:31Column 4 adds ACT scores.
-
10:31 - 10:34Analysts often report
results this way, -
10:34 - 10:37starting with models that include
few or no covariates -
10:37 - 10:40and then reporting
estimates from models -
10:40 - 10:44that add more and more covariates
as we move across columns. -
10:44 - 10:47Looking across columns,
what do you notice? -
10:47 - 10:50- [Kamal] Well, the coefficient
on using a computer is always -
10:50 - 10:52a pretty big negative number.
-
10:52 - 10:53- [Narrator] That's right!
-
10:53 - 10:56We can also see that
the standard errors are small enough -
10:56 - 11:01to make these negative results
statistically significant. -
11:01 - 11:04In other words, the primary
takeaway from this experiment -
11:04 - 11:08is that electronics in the classroom
reduce student learning. -
11:09 - 11:12- [Kamal] GPA and ACT scores
are also significant. -
11:12 - 11:14Why is that?
-
11:14 - 11:15- [Narrator] Good observation!
-
11:15 - 11:17That's not surprising.
-
11:17 - 11:20We expect these variables
to predict college performance. -
11:20 - 11:22- [Kamal] Oh right, of course.
-
11:22 - 11:24Kids who got better grades before
-
11:24 - 11:26are more likely to get
a better grade in this course. -
11:26 - 11:30- [Narrator] You'll also notice a lot
of other information on this table. -
11:30 - 11:35Remaining panels in the table
report effects of electronics use -
11:35 - 11:37on components of the final exam,
-
11:37 - 11:40such as the multiple
choice questions. -
11:40 - 11:43These results are mostly consistent
with computer use effects -
11:43 - 11:45on overall scores.
-
11:45 - 11:48- [Kamal] What about the rows
not in English? -
11:48 - 11:51- [Narrator] These rows give
additional statistical information. -
11:51 - 11:54R-squared is a measure
of goodness of fit. -
11:55 - 11:58This isn't too important, though
some readers may want to know it. -
11:59 - 12:03Other rows report on alternative
tests of statistical significance -
12:03 - 12:05that you can ignore for now.
-
12:05 - 12:08- [Kamal] Oh my gosh,
these tables aren't that hard! -
12:08 - 12:09Thank you so much.
-
12:09 - 12:12- [Narrator] Next up is regression.
-
12:12 - 12:13See you then!
-
12:16 - 12:17♪ [music] ♪
-
12:17 - 12:21You're on your way
to mastering econometrics. -
12:21 - 12:23Make sure this video sticks
-
12:23 - 12:25by taking a few
quick practice questions. -
12:25 - 12:29Or, if you're ready,
click for the next video. -
12:29 - 12:33You can also check out MRU's
website for more courses, -
12:33 - 12:35teacher resources and more.
- Title:
- How to Read Economics Research Papers: Randomized Controlled Trials (RCTs)
- Description:
-
This video walks you through how to read economics research papers that use randomized trials (sometimes called randomized controlled trials or randomized clinical trials or RCTs).
First, we’ll learn how to read descriptive statistics and check for balance between control and treatment groups. Then we’ll move onto reading study results, including an explanation of why regression is used with randomized controlled trials.
This video builds off of Josh Angrist’s introduction to randomized trials (link below) that featured an economics research study from the Economics of Education Review. This research paper covered a randomized trial conducted at West Point that measured the impact of classroom electronics on learning.
***INSTRUCTOR RESOURCES***
High school teacher resources: https://mru.io/pat
Professor resources: https://mru.io/h1b
EconInbox: https://mru.io/vbd***MORE LEARNING***
Try out our practice questions: https://mru.io/xjr
Introduction to Randomized Trials video: https://mru.io/nx9
See the full course: https://mru.io/rka
Receive updates when we release new videos: https://mru.io/dif
More from Marginal Revolution University: https://mru.io/agp - Video Language:
- English
- Team:
- Marginal Revolution University
- Project:
- Mastering Econometrics
- Duration:
- 12:40
Theresa Ranft edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Theresa Ranft edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Theresa Ranft edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Kirstin Cosper edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Kirstin Cosper edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Kirstin Cosper edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Kirstin Cosper edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) | ||
Kirstin Cosper edited English subtitles for How to Read Economics Research Papers: Randomized Controlled Trials (RCTs) |