Randomized Trials: The Ideal Weapon

MIT’s Josh Angrist—aka Master Joshway—introduces us to our most powerful weapon: randomized trials!

Randomized trials originate in medical research, where they’re called “randomized clinical trials” or “RCTs.” That’s why randomized trials are said to measure “treatment effects”.

Josh covers a fascinating study from West Point that seeks to answer a common question using randomized trials: Are our devices more distracting than useful when it comes to learning?

In this video, we cover the following:
-What's the difference between control and treatment groups?
-How to "check for balance"
-The Law of Large Numbers
-Treatment effects
-Standard errors and statistical significance

***INSTRUCTOR RESOURCES***
High school teacher resources: https://bit.ly/2RcCsta
Professor resources: https://bit.ly/2XgdHQG
EconInbox: https://bit.ly/2yzTjzz

***MORE LEARNING***
Try out our practice questions: https://bit.ly/2xVuvSg
See the full course: https://bit.ly/2Xc88CQ
Receive updates when we release new videos: https://bit.ly/2VaLTdH
More from Marginal Revolution University: https://bit.ly/39QeWZo

- [Narrator] The path from cause
to effect is dark and dangerous,

but the weapons
of Econometrics are strong.

Behold the most powerful,
the sword of random assignment,

which cuts to the heart
of causal questions.

We begin with our most powerful
and costliest weapon --

randomized trials.

- [Student] Awesome.

- Every metric's mission begins
with a causal question.

Clear questions lead
to clear answers.

The clearest answers come
from randomized trials.

Let's see how and why
randomized trials provide

especially convincing answers
to causal questions.

- [Josh] Like a finely honed sword,
randomized trials cut

to the heart of a causal problem,

creating convincing apples-
to-apples comparisons.

Yet like any finely made weapon,

randomized trials are expensive
and cannot be done quickly.

- Randomized trials originate
in medical research

where they're called
randomized clinical trials or RCTs.

The U.S. Food and Drug Administration
requires drug manufacturers

to establish the safety
and efficacy of new drugs

or medical treatments.

They do this through a series
of RCTs.

That's why randomized trials are
said to measure treatment effects.

You may have contributed
to another kind

of randomized trial --

the A/B tests
Silicon Valley companies use

to compare marketing strategies.

For example, Amazon randomizes
search results in a constant stream

of hidden experiments.

- [Woman] Oh.
- [Man] Interesting.

- Randomized trials are also
important in education research.

They've been used to answer
a causal question near and dear

to my beating teacher's heart --

should laptops
and other electronic devices

be allowed in the classroom?

Many see classroom electronics
as a learning aid.

But others, like me, think they're
a damaging distraction.

Who's right?

Metrics masters teaching
at West Point,

the military college
that trains American Army officers,

designed a randomized trial
to answer this question.

These masters randomly assign
West Point cadets

into economics classes
operating under different rules.

Unlike most American colleges,

the West Point default
is no electronics.

For purposes of this experiment,
some students were left

in such traditional
technology-free classes --

no laptops, no tablets,
and no phones!

This is the control group,
or baseline case.

Another group was allowed
to use electronics.

This is the treatment group,
subject to a changed environment.

The treatment in this case
is the unrestricted use

of laptops or tablets in class.

Every causal question has
a clear outcome --

the variables we hope to influence,
defined in advance of the study.

The outcomes
in the West Point electronics study

are final exam scores.

The study seeks to answer
the following question --

what is the causal effect
of classroom electronics on learning

as measured by exam scores?

- [Narrator] West Point economics
students were randomly assigned

to either the treatment
or control groups.

Random assignment creates
ceteris paribus comparisons,

allowing us to draw causal
conclusions by comparing groups.

The causality revealing power
of a randomized trial comes

from a statistical property
called the law of large numbers.

When statisticians
and mathematicians uncover

something important
and reliably true

about the natural world,
they call it a law.

The law of large numbers says
that when the groups randomized

are large enough, the students
in them are sure to be similar

on average, in every way.

This means that groups of students
randomly divided can be expected

to have similar family background,
motivation, and ability.

Sayonara selection bias,
at least in theory.

In practice, the groups randomized
might not be large enough

for the law
of large numbers to kick in.

Or the researchers might have
messed up the random assignment.

As in any technical endeavor,
even experienced masters

are alert for a possible foul-up.

We therefore check for balance,

comparing student background
variables across groups

to make sure
they indeed look similar.

- [Narrator] Here's
the balance check for West Point.

This table has two columns --

one showing data
from the control group

and one showing data
from the treatment group.

The rows list a few of the variables
that we hope are balanced --

sex, age, race, and high school GPA,
among others.

The first row indicates
what percentage

of each group is female.

It's 17% for the control group
and 20% for the treatment group.

Kamal, how does balance
look for GPA?

- [Kamal] Controls have
a GPA of 2.87,

while treated have
a GPA of 2.82, pretty close.

- [Narrator] Happily the two groups
look similar all around.

- How big does our sample have to be

for the law
of large numbers to kick in?

- [Narrator] The West Point study,

which involved about 250 students
in each group,

is almost certainly big enough.

There are no hard
and fast rules here.

In another video, you'll learn
how to confirm the hypothesis

of group balance
with formal statistical tests.

- [Man] Exciting.

- The heart of the matter
in this table

is the estimated treatment effect.

Remember the treatment
in this case is permission

to use electronics in class.

Treatment effects compare averages
between control and treatment groups.

The group Allowed Classroom
Electronics had

an average final exam score
0.28 standard deviations

below the score for students
in the control group.

How big an effect is this?

Social scientists measure test scores
in standard deviation units

because these units are easily
compared across studies.

We know from a long history
of research on classroom learning

that 0.28 is huge.

A decline of 0.28 is like taking
the middle student in the class

and pushing him or her down
to the bottom third.

How certain can we be
that these large results

are meaningful?

After all, we're looking
at a single randomized division

between treatment
and control groups.

Other random splits might have
produced something different.

- We therefore quantify
the sampling variance

in estimates of causal effects.

- What is sampling variance?

- [Narrator] Sampling variance
tells us how likely

a particular statistical finding
is to be a chance result,

rather than indicative
of an underlying relationship.

Sampling variance is summarized
by a number,

called the standard error
of the estimated treatment effect.

- [Students muttering] I don't get it.
- What is she even talking... ?

- Don't worry -- we'll cover
this important idea in depth later.

- [Student] Great.

- Just remember the smaller
the standard error,

the more conclusive the findings.

On the other hand,
if the standard error is large

relative to the effect
we're trying to estimate,

there's a pretty good chance
we'd get something different

if we were to rerun the experiment.

You can think of the standard error
as a way to gauge how much

we can trust the result we're seeing.

- [Students] Okay.

- [Narrator] In this study,
the relevant standard error is 0.1.

- For now it's enough to learn
a simple rule of thumb.

When an estimated
treatment effect is more

than double its standard error,

the odds this non-zero result is due
to chance are very low,

roughly 1 in 20.

Because this is so unlikely,
we say that estimates that are two

or more times larger
than the associated standard errors

are statistically significant.

- Camilla, is the treatment effect
in the West Point study

statistically significant?

- The standard error is 0.10
and the treatment effect is 0.28,

so 0.28 is more than two times larger
than 0.10, so yeah.

- Correct.

The lost learning caused
by electronics use in Econ 101

is therefore both large
and statistically significant.

- [Man] Nice.

- Randomized trials usually provide
the most convincing answers

to causal questions.

When this weapon
is in our tool kit, we use it.

Random assignment allows us
to claim ceteris

has indeed been made paribus.

But randomized trials
could be tough to organize.

They may be expensive,
time-consuming,

and, in some cases,
considered unethical.

So masters look
for convincing alternatives.

The alternatives try to mimic
the causality revealing power

of a randomized trial
but without the time, trouble,

and expense
of a purpose-built experiment.

These alternative tools are applied
in real world scenarios

that mimic random assignment.

- [Narrator] You're on your way
to mastering Econometrics.

Make sure this video sticks

by taking a few
quick practice questions.

Or, if you're ready,
click for the next video.

You can also check out MRU's website

for more courses,
teacher resources, and more.