Return to Video

How to Read Economics Research Papers: Randomized Controlled Trials (RCTs)

  • 0:01 - 0:05
    - [Narrator] On his quest
    to master econometrics,
  • 0:06 - 0:09
    Grasshopper Kamal
    has made great progress,
  • 0:09 - 0:14
    stretching his capabilities
    and outsmarting his foes.
  • 0:14 - 0:17
    Alas, today he's despondent,
  • 0:17 - 0:20
    for one challenge remains unmet.
  • 0:20 - 0:24
    Kamal cannot yet decode
    the scriptures of academic research --
  • 0:24 - 0:27
    journals like
    "The American Economic Review"
  • 0:27 - 0:29
    and "Econometrica."
  • 0:29 - 0:34
    These seemed to him to be inscribed
    in an obscure foreign tongue.
  • 0:34 - 0:35
    - [Kamal] Ugh, what the... ?
  • 0:37 - 0:40
    - [Narrator] These volumes
    are opaque to the novice, Kamal,
  • 0:40 - 0:42
    but can be deciphered with study.
  • 0:42 - 0:45
    Let us learn to read them together.
  • 0:53 - 0:55
    Let's dive into the West Point study,
  • 0:55 - 0:58
    published in the "Economics
    of Education Review."
  • 0:59 - 1:02
    This paper reports
    on a randomized evaluation
  • 1:02 - 1:06
    of student electronics use
    in Economics 101 classrooms.
  • 1:06 - 1:09
    First, a quick review
    of the research design.
  • 1:09 - 1:11
    - Okay.
  • 1:12 - 1:14
    - [Josh] Metrics masters
    teaching at West Point,
  • 1:14 - 1:17
    the military college that trains
    American Army officers
  • 1:17 - 1:20
    designed a randomized trial
    to answer this question.
  • 1:21 - 1:23
    These masters randomly assigned
    West Point cadets
  • 1:23 - 1:27
    into Economics classes
    operating under different rules.
  • 1:27 - 1:29
    Unlike most American colleges,
  • 1:29 - 1:32
    the West Point default
    is no electronics.
  • 1:32 - 1:36
    For purposes of this experiment,
    some students were left
  • 1:36 - 1:39
    in such traditional
    technology-free classes --
  • 1:39 - 1:42
    no laptops, no tablets
    and no phones!
  • 1:42 - 1:43
    [voice echoes]
  • 1:43 - 1:46
    This is the control group,
    or baseline case.
  • 1:46 - 1:49
    Another group was allowed
    to use electronics.
  • 1:49 - 1:53
    This is the treatment group,
    subject to a changed environment.
  • 1:53 - 1:56
    The treatment in this case
    is the unrestricted use
  • 1:56 - 1:58
    of laptops or tablets in class.
  • 1:59 - 2:02
    Every causal question
    has a clear outcome --
  • 2:02 - 2:05
    the variables we hope to influence
    defined in advance of the study.
  • 2:06 - 2:08
    The outcomes in the West Point
    electronics study
  • 2:08 - 2:11
    are final exam scores.
  • 2:11 - 2:14
    The study seeks to answer
    the following question:
  • 2:14 - 2:18
    What is the causal effect
    of classroom electronics on learning
  • 2:18 - 2:20
    as measured by exam scores?
  • 2:21 - 2:24
    - Economics journal articles
    usually begin with a table
  • 2:24 - 2:26
    of descriptive statistics,
  • 2:26 - 2:29
    giving key facts
    about the study sample.
  • 2:29 - 2:32
    - Oh my gosh, I remember
    this table -- so confusing!
  • 2:32 - 2:37
    - [Narrator] Columns 1 to 3 report
    mean, or average, characteristics.
  • 2:37 - 2:40
    These give a sense
    of who we're studying.
  • 2:40 - 2:44
    Let's start with column 1
    which describes covariates
  • 2:44 - 2:45
    in the control group.
  • 2:45 - 2:49
    Covariates are characteristics
    of the control and treatment groups
  • 2:49 - 2:52
    measured before
    the experiment begins.
  • 2:52 - 2:58
    For example, we see the control group
    has an average age a bit over 20.
  • 2:58 - 3:00
    Many of these covariates
    are dummy variables.
  • 3:01 - 3:06
    A dummy variable can only have
    two values -- a zero or a one.
  • 3:06 - 3:10
    For example, student gender
    is captured by a dummy variable
  • 3:10 - 3:13
    that equals one for women
    and zero for men.
  • 3:13 - 3:17
    The mean of this variable
    is the proportion female.
  • 3:17 - 3:21
    We also see that the control group
    is 13% Hispanic
  • 3:21 - 3:24
    and 19% had prior military service.
  • 3:25 - 3:27
    The table notes are key.
  • 3:27 - 3:29
    Refer to these
    as you scan the table.
  • 3:29 - 3:34
    These notes explain what's shown
    in each column and panel.
  • 3:39 - 3:42
    The notes tell us, for example,
  • 3:42 - 3:45
    that standard deviations
    are reported in brackets.
  • 3:46 - 3:50
    Standard deviations tell us how
    spread out the data are.
  • 3:50 - 3:55
    For example, a standard deviation
    of 0.52 tells us that most
  • 3:55 - 3:59
    of the control group's GPAs
    fall between 2.35,
  • 3:59 - 4:03
    which is 0.52 below
    the mean GPA of 2.87,
  • 4:03 - 4:08
    and 3.39, which is 0.52 above 2.87.
  • 4:09 - 4:12
    A lower standard deviation
    would mean the GPAs
  • 4:12 - 4:15
    were more tightly clustered
    around the mean.
  • 4:15 - 4:18
    - [Kamal] Yeah, but they're missing
    for most of the variables.
  • 4:18 - 4:19
    - [Narrator] That's right.
  • 4:19 - 4:22
    Masters usually omit
    standard deviations for dummies
  • 4:22 - 4:26
    because the mean of this variable
    determines its standard deviation.
  • 4:28 - 4:30
    This study compares
    two treatment groups
  • 4:30 - 4:32
    with the control group.
  • 4:32 - 4:36
    The first was allowed free use
    of laptops and tablets.
  • 4:36 - 4:38
    The second treatment
    was more restrictive,
  • 4:38 - 4:42
    allowing only tablets
    placed flat on the desk.
  • 4:42 - 4:45
    The treatment groups
    look much like the control group.
  • 4:47 - 4:51
    This takes us to the next feature
    of this table, columns 4 through 6
  • 4:51 - 4:55
    use statistical tests to compare
    the characteristics
  • 4:55 - 4:58
    of the treatment and control group
    before the experiment.
  • 4:58 - 5:02
    In column 4, the two treatment
    groups are combined.
  • 5:02 - 5:05
    You can see that the difference
    in proportion female
  • 5:05 - 5:10
    between the treatment
    and control group is only 0.03.
  • 5:11 - 5:14
    The difference is not
    statistically significant --
  • 5:14 - 5:17
    it is the sort of difference
    we can easily put down
  • 5:17 - 5:21
    to chance results
    in our sample selection process.
  • 5:21 - 5:22
    - [Kamal] Hmm, how do we know that?
  • 5:22 - 5:24
    - [Narrator] Remember
    the rule of thumb?
  • 5:24 - 5:27
    Statistical estimates
    that exceed the standard error
  • 5:27 - 5:30
    by a multiple of 2
    in absolute value
  • 5:30 - 5:34
    are usually said
    to be statistically significant.
  • 5:35 - 5:39
    The standard error is 0.03,
  • 5:39 - 5:41
    same as the difference
    in proportion female.
  • 5:42 - 5:46
    So the ratio of the latter
    to the former is only 1,
  • 5:46 - 5:49
    which, of course, is less than 2.
  • 5:49 - 5:51
    - [Kamal] Uh-huh! So none
    of the treatment/control differences
  • 5:51 - 5:54
    in the table are more than twice
    their standard errors.
  • 5:54 - 5:56
    - [Narrator] Correct.
  • 5:56 - 5:59
    The random division of students
    appears to have succeeded
  • 5:59 - 6:02
    in creating groups
    that are indeed comparable.
  • 6:03 - 6:05
    We can be confident, therefore,
  • 6:05 - 6:08
    that any later differences
    in classroom achievement
  • 6:08 - 6:11
    are the result
    of the experimental intervention
  • 6:11 - 6:15
    rather than a reflection
    of preexisting differences.
  • 6:15 - 6:17
    Ceteris paribus achieved!
  • 6:17 - 6:21
    - [Kamal] Cool. Wait,
    what about the bottom,
  • 6:21 - 6:23
    the numbers with the stars?
  • 6:23 - 6:26
    Those differences are a lot more
    than double the standard error.
  • 6:26 - 6:27
    - [Narrator] Good eye, Kamal!
  • 6:27 - 6:29
    The table has many numbers.
  • 6:29 - 6:32
    Those in Panel B are important too.
  • 6:32 - 6:36
    This panel measures the extent
    to which students in treatment
  • 6:36 - 6:39
    and control groups actually use
    computers in class.
  • 6:40 - 6:43
    The treatment here
    was to allow computer use.
  • 6:43 - 6:45
    The researchers must show
  • 6:45 - 6:47
    that students allowed
    to use computers
  • 6:47 - 6:49
    took advantage
    of the opportunity to do so.
  • 6:50 - 6:53
    If they didn't, then there's
    really no treatment.
  • 6:54 - 6:58
    Luckily, 81% of those
    in the first treatment group
  • 6:58 - 6:59
    used computers
  • 6:59 - 7:02
    compared with none
    in the control group.
  • 7:02 - 7:05
    And many in the second
    tablet treatment group
  • 7:05 - 7:07
    used computers as well.
  • 7:07 - 7:10
    These differences
    in computer use are large
  • 7:10 - 7:12
    and statistically significant.
  • 7:12 - 7:15
    We also get to see
    the sample size in each group.
  • 7:15 - 7:18
    - [Kamal] The stars
    are just like decoration?
  • 7:18 - 7:22
    - [Narrator] Some academic papers
    use stars to indicate differences
  • 7:22 - 7:24
    that are statistically significant.
  • 7:24 - 7:27
    This makes them jump out at you.
  • 7:27 - 7:32
    Here, three stars indicate that
    the result is statistically different
  • 7:32 - 7:35
    from zero with a p-value
    less than 1%.
  • 7:36 - 7:39
    In other words, there's less
    than a 1 in 100 chance
  • 7:39 - 7:42
    this result is purely
    a chance finding.
  • 7:42 - 7:43
    [applause]
  • 7:43 - 7:49
    Two stars indicate a 1 in 20
    or 5% chance of a chance finding.
  • 7:49 - 7:52
    And one star denotes results
    we might see as often
  • 7:52 - 7:56
    as 10% of the time
    merely due to chance.
  • 7:56 - 8:00
    Today, stars are seen
    as a little old fashioned.
  • 8:00 - 8:02
    Some journals omit them.
  • 8:02 - 8:04
    - [Kamal] What about
    those last two columns?
  • 8:04 - 8:06
    - [Narrator] Unlike column 4,
    which combines
  • 8:06 - 8:10
    both treatment groups into one,
    these last two columns
  • 8:10 - 8:12
    look separately
    at treatment/control differences
  • 8:12 - 8:15
    for each treatment group.
  • 8:15 - 8:17
    This provides a more detailed
    analysis of balance.
  • 8:18 - 8:21
    Also, for now,
    you can ignore this row
  • 8:21 - 8:24
    which provides
    another test of significance.
  • 8:25 - 8:29
    Now we get to the article's
    punchline, table 4.
  • 8:30 - 8:33
    This table reports
    regression estimates
  • 8:33 - 8:37
    of the effects of electronics use
    on measures of student learning.
  • 8:37 - 8:40
    - [Kamal Why does the study
    report regression estimates?
  • 8:40 - 8:43
    See, that's why I was getting lost.
  • 8:43 - 8:45
    I thought one reason
    why we liked randomized trials
  • 8:45 - 8:47
    is that we use them
    to obtain causal effects
  • 8:47 - 8:50
    simply by comparing
    treatment and control groups.
  • 8:50 - 8:54
    Since these groups are balanced,
    no need to use regression.
  • 8:54 - 8:55
    - [Narrator] Well said, Kamal.
  • 8:55 - 8:59
    In practice, it's customary
    to report regression estimates
  • 8:59 - 9:01
    for two reasons.
  • 9:01 - 9:04
    First, evidence of balance
    notwithstanding,
  • 9:04 - 9:07
    an abundance of caution
    might lead the analyst
  • 9:07 - 9:10
    to allow for chance differences.
  • 9:10 - 9:14
    Second, regression estimates
    are likely to be more precise --
  • 9:14 - 9:17
    that is, they have lower
    standard errors
  • 9:17 - 9:19
    than the simple treatment
    control comparisons.
  • 9:20 - 9:23
    The dependent variable
    in this study
  • 9:23 - 9:24
    is the outcome of interest.
  • 9:25 - 9:26
    Since the question at hand
  • 9:26 - 9:29
    is how classroom electronics
    affect learning,
  • 9:29 - 9:33
    a good outcome is the Economics
    final exam score.
  • 9:33 - 9:38
    Each column reports results
    from a different regression model.
  • 9:38 - 9:40
    Models are distinguished
    by the control variables
  • 9:40 - 9:44
    or covariates they include
    besides treatment status.
  • 9:45 - 9:48
    Estimates with no covariates
    are simple comparisons
  • 9:48 - 9:51
    of treatment and control groups.
  • 9:51 - 9:53
    - [Kamal] I thought
    they just forgot to fill it out.
  • 9:53 - 9:56
    - [Narrator] Column 1
    suggests electronics use
  • 9:56 - 10:01
    reduced final exam scores
    by 0.28 standard deviations.
  • 10:02 - 10:03
    In our last lesson,
  • 10:03 - 10:07
    Master Joshway explained
    that we use standard deviation units
  • 10:07 - 10:11
    because these units
    are easily compared across studies.
  • 10:11 - 10:13
    Column 2 reports results
  • 10:13 - 10:16
    from a model that adds
    demographic controls.
  • 10:16 - 10:20
    Here, we're comparing test scores
    but holding constant factors
  • 10:20 - 10:21
    such as age and sex.
  • 10:22 - 10:26
    Column 3 reports results
    from a model that adds GPA
  • 10:26 - 10:27
    to the list of covariates.
  • 10:28 - 10:31
    Column 4 adds ACT scores.
  • 10:31 - 10:34
    Analysts often report
    results this way,
  • 10:34 - 10:37
    starting with models that include
    few or no covariates
  • 10:37 - 10:40
    and then reporting
    estimates from models
  • 10:40 - 10:44
    that add more and more covariates
    as we move across columns.
  • 10:44 - 10:47
    Looking across columns,
    what do you notice?
  • 10:47 - 10:50
    - [Kamal] Well, the coefficient
    on using a computer is always
  • 10:50 - 10:52
    a pretty big negative number.
  • 10:52 - 10:53
    - [Narrator] That's right!
  • 10:53 - 10:56
    We can also see that
    the standard errors are small enough
  • 10:56 - 11:01
    to make these negative results
    statistically significant.
  • 11:01 - 11:04
    In other words, the primary
    takeaway from this experiment
  • 11:04 - 11:08
    is that electronics in the classroom
    reduce student learning.
  • 11:09 - 11:12
    - [Kamal] GPA and ACT scores
    are also significant.
  • 11:12 - 11:14
    Why is that?
  • 11:14 - 11:15
    - [Narrator] Good observation!
  • 11:15 - 11:17
    That's not surprising.
  • 11:17 - 11:20
    We expect these variables
    to predict college performance.
  • 11:20 - 11:22
    - [Kamal] Oh right, of course.
  • 11:22 - 11:24
    Kids who got better grades before
  • 11:24 - 11:26
    are more likely to get
    a better grade in this course.
  • 11:26 - 11:30
    - [Narrator] You'll also notice a lot
    of other information on this table.
  • 11:30 - 11:35
    Remaining panels in the table
    report effects of electronics use
  • 11:35 - 11:37
    on components of the final exam,
  • 11:37 - 11:40
    such as the multiple
    choice questions.
  • 11:40 - 11:43
    These results are mostly consistent
    with computer use effects
  • 11:43 - 11:45
    on overall scores.
  • 11:45 - 11:48
    - [Kamal] What about the rows
    not in English?
  • 11:48 - 11:51
    - [Narrator] These rows give
    additional statistical information.
  • 11:51 - 11:54
    R-squared is a measure
    of goodness of fit.
  • 11:55 - 11:56
    This isn't too important,
  • 11:56 - 11:58
    though some readers
    may want to know it.
  • 11:59 - 12:03
    Other rows report on alternative
    tests of statistical significance
  • 12:03 - 12:05
    that you can ignore for now.
  • 12:05 - 12:08
    - [Kamal] Oh my gosh,
    these tables aren't that hard!
  • 12:08 - 12:09
    Thank you so much.
  • 12:09 - 12:12
    - [Narrator] Next up is regression.
  • 12:12 - 12:13
    See you then!
  • 12:16 - 12:17
    ♪ [music] ♪
  • 12:17 - 12:21
    You're on your way
    to mastering econometrics.
  • 12:21 - 12:23
    Make sure this video sticks
  • 12:23 - 12:25
    by taking a few
    quick practice questions.
  • 12:25 - 12:29
    Or, if you're ready,
    click for the next video.
  • 12:29 - 12:33
    You can also check out MRU's
    website for more courses,
  • 12:33 - 12:35
    teacher resources and more.
Titolo:
How to Read Economics Research Papers: Randomized Controlled Trials (RCTs)
Descrizione:

This video walks you through how to read economics research papers that use randomized trials (sometimes called randomized controlled trials or randomized clinical trials or RCTs).

First, we’ll learn how to read descriptive statistics and check for balance between control and treatment groups. Then we’ll move onto reading study results, including an explanation of why regression is used with randomized controlled trials.

This video builds off of Josh Angrist’s introduction to randomized trials (link below) that featured an economics research study from the Economics of Education Review. This research paper covered a randomized trial conducted at West Point that measured the impact of classroom electronics on learning.

***INSTRUCTOR RESOURCES***
High school teacher resources: https://mru.io/pat
Professor resources: https://mru.io/h1b
EconInbox: https://mru.io/vbd

***MORE LEARNING***
Try out our practice questions: https://mru.io/xjr
Introduction to Randomized Trials video: https://mru.io/nx9
See the full course: https://mru.io/rka
Receive updates when we release new videos: https://mru.io/dif
More from Marginal Revolution University: https://mru.io/agp

more » « less
Video Language:
English
Team:
Marginal Revolution University
Progetto:
Mastering Econometrics
Duration:
12:40

English subtitles

Revisioni Compare revisions