Joshua Angrist Nobel Prize Lecture 2021

Joshua Angrist, winner of The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (2021), delivers his Nobel Prize lecture on "Empirical Strategies in Economics: Illuminating the Path from Cause to Effect".

**LEARN MORE ABOUT JOSH ANGRIST’S RESEARCH**
Blueprint Labs at MIT: https://blueprintlabs.mit.edu/
Josh Angrist’s Research Publications: https://economics.mit.edu/faculty/angrist/publications
Josh Angrist’s Working Papers: https://economics.mit.edu/faculty/angrist/papers

**LEARN MORE ABOUT JOSH ANGRIST’S BOOKS**
Mostly Harmless Econometrics!: http://www.mostlyharmlesseconometrics.com/
Mastering ‘Metrics: http://www.masteringmetrics.com/

**INSTRUCTOR RESOURCES**
Mastering Econometrics course: https://mru.io/nk7
Econometrics test bank: https://mru.io/of6
More professor resources: https://mru.io/9gq

**MORE LEARNING**
Mastering Econometrics course: https://mru.io/nk7
Receive updates when we release new videos: https://mru.io/ri7
More from Marginal Revolution University: https://mru.io/0nf

♪ [music] ♪

- [Joshua] As I stilled
my trembling iPhone

early on October 11th,

my thoughts went to the question
of whether Nobel-level recognition

might change life
for the Angrist family.

Ours is a close-knit family.

We lack for nothing.

So I worried
that stressful Nobel celebrity

might not be a plus.

But with the first cup of coffee,

I began to relax.

It occurred to me

that the matter
of how public recognition

affects a scholar's life

is, after all,
a simple causal question.

The Nobel intervention

is substantial, sudden,
and well-measured.

Outcomes like health and wealth
are easy to record.

Having just been recognized
with my co-laureates,

Guido Imbens and David Card,

for answering causal questions
using observational data,

my thoughts moved
from personal upheaval

to the more familiar demands

of identification and estimation
of causal effects.

I was able to soothe
my worried mind

by imagining a study
of the Nobel Prize treatment effect.

How would such a study
be organized?

In a 1999 essay published in
the "Handbook of Labor Economics,"

Alan Krueger and I embraced
the phrase "empirical strategy."

The handbook volume
in question was edited

by two of my Princeton
Ph.D. thesis advisors,

Orley Ashenfelter and David Card --

among the most successful
and prolific graduate advisors

economics has known.

An empirical strategy
is a research plan

that encompasses data collection,

identification,
and econometric estimation.

Identification is the applied
econometricians term

for research design --

a randomized clinical trial,

an RCT, is the simplest
and most powerful research design.

In RCTs, causal effects
are identified

by the random assignment
of treatment.

Random assignment ensures

that treatment and control groups

are comparable
in the absence of treatment.

So differences
between them afterwards

reflect only the treatment effect.

Nobel Prizes are probably
not randomly assigned.

This challenge notwithstanding,

a compelling empirical strategy
for the Nobel treatment effect

comes to mind,

at least as a flight
of empirical fancy.

Imagine a pool of prize-eligible
Nobel applicants,

the group under consideration
for the prize.

Applicants need not
apply themselves.

They are, I presume, nominated
by their peer scholars.

My fanciful Nobel impact study
looks only at Nobel applicants,

since these are all elite scholars.

But that is only the first step.

Credible applicants, I imagine,
are evaluated by judges

using criteria like publications,
citations, nominating statements.

I imagine this material is reviewed
and assigned a numerical score,

using some kind of scoring rubric.

Top scorers up to three per field,
in any single year

win a prize.

Having identified applicants

and collected data
on their scores,

the next step
in my Nobel impact study

is to record the relevant cutoffs.

The Nobel cutoff
is the lowest score

among those awarded a prize.

Many Nobel hopefuls
just missed the cutoff.

Looking only at near misses,
along with the winners,

differences in scores

between those above
and below the cutoff

begin to look serendipitous,
almost randomly assigned.

After all,

near-Nobels are among
the most eminent of scholars, too.

With one more
high-impact publication,

a little more support
from nominators,

they would have been awarded

Nobel gold.

Some of them, someday,
surely will be.

The empirical strategy
sketched here is called

a Regression Discontinuity Design,

RD.

RD exploits the jumps
in human affairs,

induced by rules, regulations,

and the need to classify people
for various assignment purposes.

When treatment
or intervention is determined

by whether a tiebreaking variable
crosses a threshold,

those just below the threshold
become a natural control group

for those who clear it.

RD does not require
that the variable

whose causes we seek,

switch fully on or fully off
at the cutoff.

We require only

that the average value
of this variable

jump at the cutoff.

RD can allow,
for example, for the fact

that this year's near-Nobel
might be next year's winner.

Allowing for this
leads to the use of jumps

in the rate at which
treatment is assigned

to construct
instrumental variables, IV,

estimates of the effect
of treatment received.

This sort of RD
is said to be "fuzzy,"

But as Steve Pischke and I
wrote in our first book:

"Fuzzy RD is IV."

[kids cheering]

The first RD study I contributed to

was written with my frequent
collaborator, Victor Lavy.

This study is motivated

by the high costs
and uncertain returns

to smaller elementary school classes.

We exploited a rule
used by Israeli elementary schools

to determine class size.

This rule is used to estimate
class-size effects,

as if in a class-size RCT.

In the 1990s,
Israeli classes were large.

Students enrolled
in a grade cohort of 40

were likely to be seated
in a class of 40 --

that's the relevant cutoff.

Add another child
to the cohort, making 41,

and the cohort
was likely to be split

into two much smaller classes.

This leads to the Maimonides'
rule research design,

so named because
the 12th-century Rambam

proposed a maximum
class size of 40.

This figure plots Israeli
4th grade class sizes

as a function
of 4th grade enrollment,

overlaid with the theoretical
class size rule,

Maimonides' rule.

The fit isn't perfect --

that's a feature that makes
this application of RD fuzzy.

But the gist of the thing
is a marked class size drop

at each integer multiple of 40,
the relevant cutoff,

just as predicted by the rule.

As it turns out,
these drops in class size

are reflected in jumps

in 4th and 5th grade math scores.

Would a comparison
of Nobel laureates to near-laureates

really be a good
natural experiment?

The logic behind this sort of claim

seems more compelling
for comparisons of schools

with 40 and 41 fourth graders

than for comparisons
of laureates and near-laureates.

Yet, both scenarios exploit
a feature of the physical world.

Provided the tiebreaking variable,

known to RD mavens
as the "running variable,"

has a continuous distribution,

the probability of crossing
the cutoff approaches one half

when examined in a narrow window

around the cutoff.

In RD empirical work,

the window around such cutoffs
is known as a bandwidth.

Importantly, this limiting
probability is 0.5 for everybody

regardless of
how qualified they look

going into the Nobel competition.

This remarkable fact can be seen
in data on applicants

to one of New York's
highly coveted screen schools.

By way of background,

roughly 40% of New York City's
middle and high schools

select their applicants
on the basis of test scores, grades,

and other exacting criteria.

In other words,

the admissions regime
for screen schools

is a lot like the scheme
I've imagined

for the Nobel Prize.

Screen schools
are but one of a number

of highly selective systems
within a system

in large U.S. school districts.

Boston, Chicago, San Francisco,
and Washington, D.C.

all feature highly selective
institutions,

often known as exam schools.

Exam schools operate

as part of larger
public school systems

that enroll students
without screening.

Motivated by
the enduring controversy

over the equity
of screened admissions,

my Blueprint
Labs collaborators and I

have examined the causal effects
of exam school attendance

in Boston, Chicago, and New York.

This figure shows the probability
of being offered a seat

at New York's storied
Townsend Harris High School,

ranked 12th nationwide.

Bar height in the figure
marks the qualification rate --

that is, the likelihood of earning
a Townsend Harris admission score

above that of the lowest
scoring applicant offered a seat.

Importantly, the bars show
qualification rates conditional

on a measure of pre-application
baseline achievement.

In particular, the bars mark
qualification rates conditional

on whether an applicant

has upper quartile or lower quartile
6th grade math scores.

Townsend Harris applicants
with high baseline scores

are much more likely to qualify

than applicants
with low baseline scores.

This isn't surprising.

But in a shrinking
symmetric bandwidth

around the school's cutoff,

qualification rates
in the two groups converge.

Qualification rates in the last
and smallest groups

are both remarkably close to one half.

This is what we'd expect to see

were Townsend Harris
to admit students

by tossing a coin,

rather than by selecting
only those who scored highly

on the school's entrance exam.

Even when admissions
operates by screening,

the data can be arranged
so as to mimic an RCT.

Few of the questions I've studied
are more controversial

than the question of access
to public exam schools,

like the Boston Latin School,

Chicago's Payton and Northside
selective enrollment high schools,

and New York's legendary
Brooklyn Tech,

Bronx Science,

and Stuyvesant
specialized high schools,

which have graduated
14 Nobel laureates between them.

Townsend Harris, the school
we started with today,

graduated three Nobels,
including economist Ken Arrow.

Exam school proponents

see the opportunities
these schools provide

as democratizing public education.

Wealthy families, they argue,

can access exam school curricula
in the private sector.

Shouldn't ambitious
low-income students

be afforded the same chance
at elite education?

Critics of selective
enrollment schools

argue that rather
than expanding equity,

exam schools are inherently biased

against the Black
and Hispanic students

that make up the bulk
of America's urban districts.

New York's super selective
Stuyvesant, for example,

enrolled only
seven Black students in 2019,

out of an incoming class of 895.

But are exam school seats
really worth fighting for?

My collaborators and I
have repeatedly used

RD empirical strategies
to study the causal effects

of attendance at exam schools

like Townsend Harris
and Boston Latin.

Our first exam school study,

which looks at schools
in Boston and New York,

encapsulates
these findings in its title:

"The Elite Illusion."

The elite illusion
refers to the fact

that while exam school students
undoubtedly have high test scores

and other good outcomes,

this is not a causal effect
of exam school attendance.

Our estimates consistently suggest

that the causal effects
of exam school attendance

on their students learning
and college-going are zero --

maybe even negative.

The good performance
of exam school students

reflect selection bias --

that is, the process by which
these students are chosen,

rather than causal effects.

Data from Chicago's large exam
school sector

illustrate the elite illusion.

This figure plots
peer mean achievement --

that is, the 6th grade test scores
of my 9th grade classmates

against the admissions tiebreaker

for a subset of applicants

to any one of Chicago's
nine exam schools.

Applicants to these schools
rank up to 6,

while the exam schools
prioritize their applicants

using a common composite index,

formed from an admissions test,

GPAs, and grade 7
standardized scores.

This composite tiebreaker
is the running variable

for an RD design
that reveals what happens

when any applicant is offered
an exam school seat.

In Chicago's exam school match,

which is actually an application

of the celebrated Gale and Shapley
matching algorithm,

exam school applicants are sure
to be offered a seat somewhere

when they clear the lowest
in their set of cutoffs

among the schools they rank.

We call this lowest cutoff
the "qualifying cutoff."

The figure shows a sharp jump
in peer mean achievement

for Chicago exam school applicants

who clear their qualifying cutoff.

This jump reflects the fact

that most applicants
offered an exam school seat take it,

and applicants who enroll

at one of Chicago's selective
enrollment high schools

are sure to be seated
in a 9th grade classroom

filled with academically
precocious peers,

because only the relatively
precocious make it in.

The increase in peer achievement
across the qualifying cutoff

amounts to almost
half a standard deviation --

a very large effect.

And yet, precocious peers
notwithstanding,

the offer of an exam school seat

does not appear
to increase learning.

Let's plot applicants ACT scores
against their tiebreaker values.

This plot shows that exam
school applicants

who clear their qualifying cutoff

perform sharply worse on the ACT.

What explains this?

It takes a tale of IV and RD
to untangle the forces

behind this intriguing
and unexpected negative effect.

But first, some IV theory.

Guido Imbens and I
developed theoretical tools

that enhance
economists' understanding

of empirical strategies

involving IV and RD.

The prize we share
is in recognition of this work.

Guido and I overlapped
for only one year at Harvard,

where we had both taken
our first jobs post Ph.D.

I welcomed Guido
to Cambridge, Massachusetts

with a pair of interesting
instrumental variables.

I had used
the draft lottery instrument

in my Ph.D. thesis

to estimate the long-run
economic consequences

of serving in the Armed Forces

for soldiers who were drafted.

The draft lottery instrument
relies on the fact

that lottery numbers
randomly assigned to birthdays

determined Vietnam-era
conscription risk.

Yet, even then, most soldiers
were volunteers,

as they are today.

The quarter birth instrument

is used in my 1991 paper
with Alan Krueger

to estimate the economic
returns to schooling.

This instrument uses the fact

that men who were born
earlier in the year

were allowed to drop out
of high school

on their 16th birthday

with less schooling completed
than those born later.

Guido and I soon began
asking each other,

"What really do we learn
from the draft eligibility

and quarter of birth
natural experiments?"

An early result in our quest
for a new understanding of IV

was a solution to the problem
of selection bias

in an RCT with partial compliance.

Even in a randomized clinical trial,

some of the people assigned
to treatment may opt out.

This fact has long vexed trialists

because decisions to opt out
are not made by random assignment.

Our first manuscript together

shows that in a randomized trial
with partial compliance,

you can use IV

to estimate the effect
of treatment on the treated,

even when some offered treatment

decline it.

This works in spite of the fact

that those who comply
with treatment

may be a very select group.

Unfortunately, for us,
we were late to the party.

Not long after releasing
our first working paper,

we learned of a concise contribution
from Howard Bloom

that includes this theoretical result.

Remarkably, Bloom had derived
this from first principles

without making a connection to IV.

So Guido and I went back
to the drawing board.

And a few months later,
we had LATE --

a theorem showing how to estimate

the Local Average Treatment Effect.

The LATE theorem
generalizes the Bloom theorem

and establishes the connection
between compliance and IV.

Maintaining the clinical
trials analogy,

let "Zi" indicate whether subject "i"
is offered treatment.

This is randomly assigned.

Also, let "D1i" indicate
subject i's treatment status

when assigned to treatment,

and let "D0i" indicate
subject i's treatment status

when assigned to control.

I'll use this formal notation

to give a clear statement
of the LATE result,

and then follow up with examples.

A key piece of the LATE framework,

pioneered by statistician Don Rubin,

is the pair of potential outcomes.

As is customary,

I denote potential outcomes
for subject i

in the treated and untreated states

by "Y1i" and "Y0i", respectively.

The observed outcome
is Y1i for the treated

and Y0i for those not treated.

Y1i minus Y0i

is the causal effect
of treatment on individual i,

but this we can never see.

We try, therefore, to estimate
some kind of average causal effect.

The LATE framework
allows us to do that in an RCT

where some controls are treated.

The theorem says

that the average causal
effect on people

whose treatment status
can be changed

by the offer of treatment

is the ratio of ITT
to the treatment control difference

in compliance rates.

A mathematical statement
of this result appears here,

where Greek letter Delta
symbolizes the ITT effect

and Greek symbols Pi1 and Pi0

are compliance rates
in the group assigned to treatment

and the group assigned
to control, respectively.

The print version of this lecture

delves deeper
into LATE intellectual history,

highlighting key contributions
made with Rubin.

For now, though, I'd like to make
the LATE theorem concrete for you

by sharing one
of my favorite applications of it.

I'll explain the LATE framework

through a research question
that has fascinated me

for almost two decades.

What is the causal effect

of charter school attendance
on learning?

Charter schools are public schools

that operate independently

of traditional American
public school districts.

A charter, the right
to operate a public school

is typically awarded
for a limited period,

subject to renewal, conditional
on good school performance.

Charter schools
are free to structure

their curriculum
and school environment.

The most controversial difference

between charters
and traditional public schools

is the fact that
the teachers and staff

who work at charter schools

rarely belonged to labor unions.

By contrast, most
big city public school teachers

work under union contracts.

The 2010 documentary film
"Waiting for Superman"

features schools belonging to
the Knowledge is Power Program,

KIPP.

KIPP schools are emblematic
of the high expectations,

sometimes also called "no excuses"
approach to public education.

The "no excuses" model features

a long school day
and extended school year,

selective teacher hiring,

and focuses on traditional
reading and math skills.

The American debate
over education reform

often focuses
on the achievement gap --

that's shorthand
for large test score differences

by race and ethnicity.

Because of its focus
on minority students,

KIPP is often central
in this debate

with supporters
pointing to the fact

that non-White KIPP students
have markedly higher test scores

than non-White students
from nearby schools.

KIPP skeptics, on the other hand,

argue that KIPP's apparent success

reflects the fact
that KIPP attracts families

whose children would be
more likely to succeed anyway.

Who's right?

As you've probably guessed by now,

a randomized trial
might prove decisive

in the debate
over schools like KIPP.

Like Nobel Prizes, though,

seats at KIPP
are not randomly assigned.

Well, at least, not entirely.

In fact,

Massachusetts charter schools
with more applicants than seats

must offer their seats by lottery.

Sounds like a good,
natural experiment.

A little over a decade ago,

my collaborators and I
collected data

on KIPP admissions lotteries,

laying the foundation
for two pioneering charter studies,

the first to use lotteries
to study KIPP.

Our KIPP analysis
is a classic IV story

because many students
offered a seat in the KIPP lottery

failed to show up in the fall,

while a few not offered a seat
nevertheless find their way in.

This graphic shows KIPP
middle school applicants math scores

one year after applying to KIPP.

The entries above the line

show that KIPP applicants
who were offered a seat

have standardized
math scores close to zero --

that is, near the state average.

As before, we're working
with standardized score data

that has a mean of 0
and a standard deviation of 1.

Because KIPP applicants
start with 4th grade scores

that are roughly 0.3
standard deviations

below the state mean,

achievement at the level
of the state average is impressive.

By contrast, the average math score
among those not offered a seat

is about -0.36 sigma --

that is, 0.36 standard deviations
below the state mean,

a result typical for urban students
in Massachusetts.

Since lottery offers
are randomly assigned,

we could say with confidence
that the offer of a seat at KIPP

boost math scores
by an average of 0.36 sigma --

a large effect
that's also statistically precise.

We can be confident
this isn't a chance finding.

What does an offer effect
of 0.36 sigma

tell us about the effects
of actually going to KIPP?

IV methods convert
KIPP offer effects

into KIPP attendance effects.

I'll use this brief clip

from my Marginal Revolution
University short course

to quickly review
the key assumptions

behind this conversion.

- [Narrator] IV describes
a chain reaction.

Why do offers affect achievement?

Probably because they affect
charter attendance,

and charter attendance
improves math scores.

The first link in the chain
called the First Stage

is the effect of the lottery
on charter attendance.

The Second Stage is the link

between attending a charter
and an outcome variable --

in this case, math scores.

The instrumental variable,
or instrument for short,

is the variable that initiates
the chain reaction.

The effect of the instrument
on the outcome

is called the Reduced Form.

This chain reaction can be
represented mathematically.

We multiply the First Stage --
the effect of winning on attendance,

by the Second Stage --
the effect of attendance on scores,

and we get the Reduced Form --

the effect of winning
the lottery on scores.

The Reduced Form and First Stage
are observable and easy to compute.

However, the effect
of attendance on achievement

is not directly observed.

This is the causal effect
we're trying to determine.

Given some important assumptions
we'll discuss shortly,

we can find the effect
of KIPP attendance

by dividing the Reduced Form
by the First Stage.

- [Joshua] IV eliminates
selection bias,

but like all of our tools,

the solution builds
on a set of assumptions

not to be taken for granted.

First, there must be
a substantial first stage --

that is, the instrumental variable,

winning or losing the lottery,

must really change the variable
whose effect we're interested in --

here, KIPP attendance.

In this case, the first stage
is not really in doubt.

Winning the lottery

makes KIPP attendance
much more likely.

Not all IV stories are like that.

Second, the instrument must be
as good as randomly assigned,

meaning lottery winners and losers
have similar characteristics.

This is the independence assumption.

Of course, KIPP lottery wins
really are randomly assigned.

Still, we should check
for balance and confirm

that winners and losers

have similar family backgrounds,
similar aptitudes, and so on.

In essence, we're checking
to ensure KIPP lotteries are fair,

with no group of applicants
suspiciously likely to win.

Finally, we require
the instrument change outcomes

solely through
the variable of interest --

in this case, attending KIPP.

This assumption is called
the Exclusion Restriction.

The causal effect
of KIPP attendance

can therefore be written

as the ratio of the effect
of offers on scores

in the numerator

over the effect of offers
on KIPP enrollment

in the denominator.

The numerator in this IV formula --

that is, the direct effect
of the instrument on outcomes

has a special name.

This is called the Reduced Form.

The denominator is the First Stage.

The exclusion restriction
is often the trickiest

or most controversial part
of an IV story.

Here, the exclusion restriction
amounts to the claim

that the 0.36 score differential
between lottery winners and losers

is entirely attributable

to the 0.74 win/loss difference
in attendance rates.

Plugging in the numbers,

the effect of KIPP attendance
works out to be 0.48 sigma,

almost half
a standard deviation gain

in math scores --

that's a remarkably large effect.

Who exactly benefits
so spectacularly from KIPP?

Does everyone who applies
to KIPP see such large gains?

LATE answers this question.

The LATE interpretation
of the KIPP IV empirical strategy

is illuminated
by the biblical story of Passover,

which explains that there are
four types of children,

each with characteristic behaviors.

To keep track of these children
and their behavior,

I'll give them alliterative names.

Applicants like Alvaro
are dying to go to KIPP.

If Alvaro loses the KIPP lottery,

his mother finds a way
to enroll him in KIPP anyway,

perhaps by reapplying.

Applicants like Camila
are happy to go to KIPP

if they win a seat in the lottery,

but stoically accept
the verdict, if they lose.

Finally, applicants like Normando

worry about long days
and lots of homework at KIPP.

Normando doesn't really want to go

and refuses to go to KIPP
when told that he won the lottery.

Normando was called a never-taker

because win or lose,
he doesn't go to KIPP.

At the other end
of KIPP commitment,

Alvaro is called an always-taker.

He'll happily take a seat
when offered

while his mother simply finds a way
to make it happen for him,

even when he loses.

For Alvaro and Normando both,

choice of school, KIPP, traditional,
is unaffected by the lottery.

Camila is the type of applicant
who gives IV its power.

The instrument determines
her treatment status.

IV strategies depend
on applicants like Camilla,

who are called compliers.

This term comes from the world
of randomized trials

introduced earlier.

As we've already discussed,

many randomized trials randomize
only the opportunity to be treated

while the decision
to comply with the treatment

remains voluntary and non-random.

RCT compliers are those
who take treatment

when the offer of treatment is made,

but not otherwise.

With lottery instruments,

LATE is the effect
of KIPP attendance on Camila

and other compliers like her

who enroll at KIPP, take treatment

when offered treatment
through the lottery,

but not otherwise.

IV methods are uninformative

for always-takers like Alvaro
and never-takers like Normando

because the instrument
is unrelated

to their treatment status.

Hey, didn't I say
there are four types of children?

A fourth type of child in IV theory
behaves perversely.

Every family has one!

These defiant children
enroll in KIPP

only when they lose the lottery.

Actually, the LATE theorem

requires us to assume
there are few defiers --

that seems like
a reasonable assumption

for charter lottery instruments,

if not in life.

The LATE theorem is sometimes seen

as limiting the relevance
of econometric estimates

because it focuses attention
on groups of compliers.

Yet, the population of compliers

is a group we'd very much like
to learn about.

In the KIPP example,

compliers are children
likely to be drawn into KIPP

were the school to expand

and offer additional seats
in a lottery.

How relevant is this?

A few years ago,

Massachusetts indeed allowed
thriving charter schools to expand.

A recent study
by some of my lab mates

shows that LATE estimates,

like the one
we just computed for KIPP,

predict learning gains

at the schools created
by charter expansion.

LATE isn't just a theorem --

it's a framework.

The LATE framework can be used
to estimate the entire distribution

of potential outcomes for compliers

as if we really did have
a randomized trial for this group.

Although the theory
behind this fact

is necessarily technical,

the value of the framework
is easily appreciated in practice.

By way of illustration,
recall that the KIPP study

is motivated in part by differences
in test scores by race.

Let's look at the distribution
of 4th grade scores,

separately by race,

for applicants to Boston
charter middle schools.

The two sides of this figure

show distributions for treated
and untreated compliers.

Treated compliers are compliers
offered a charter seat in a lottery,

while untreated compliers
are not offered a seat.

Because these are 4th grade scores,

while middle school begins
in 5th or 6th grade,

the two sides
of the figure are similar.

Both sides show score distributions
for Black applicants

shifted to the left

of the corresponding
score distributions for Whites.

By 8th grade,

treated compliers have completed
middle school at a Boston charter,

while untreated compliers
have remained

in traditional public school.

Remarkably, this next graphic

shows that the 8th grade
score distributions

of Black and White treated compliers

are indistinguishable.

Boston charter middle schools
closed the achievement gap.

But for the untreated,

Black and White score distributions
remained distinct

with Black students
behind White students

as they were in 4th grade.

Boston charters closed
the achievement gap

because those who enter
charter schools

the farthest behind

tend to gain the most
from charter enrollment.

I elaborate on this point
in the print version of this talk.

Remember the puzzle

of negative Chicago
exam school effects?

I'll finish the scientific
part of my talk by using IV and RD

to explain this surprising finding.

The resolution of this puzzle
starts with the fact

that economic reasoning
is about alternatives.

So what's the alternative
to an exam school education?

For most applicants
to Chicago exam schools,

the leading non-exam alternative
is a traditional public school.

But many of Chicago's rejected
exam school applicants

enroll in a charter school.

Exam school offers

therefore reduce the likelihood
of charter school attendance.

Specifically, exam schools
divert applicants

away from high schools

in the Noble Network
of Charter Schools.

Noble, with pedagogy
much like KIPP,

is one of Chicago's most visible
charter providers.

Also like KIPP, convincing evidence
on Noble effectiveness

comes from admissions lotteries.

The x-axis in this graphic

shows lottery offer effects
on years enrolled at Noble.

This is the Noble first stage,

for an IV setup that uses a dummy

indicating Noble lottery offers

as an instrument
for Noble enrollment.

Now this graphic has a feature
that distinguishes it

from the simpler KIPP analysis.

The plot shows first-stage effects

for two groups:

one for Noble applicants

who live in Chicago's lowest income
neighborhoods, Tier 1,

and one for Noble applicants

who live in higher-income areas,

Tier 3.

Remember the IV chain reaction?

Each point in this graphic

has coordinates given
by first-stage reduced form

and therefore implies
an IV estimate.

The effect of Noble enrollment
on ACT scores

is the ratio
of Reduced-Form coordinate

to First-Stage coordinate.

The graphic shows two such ratios.

The relevant results
for Tier 1 are 0.35,

while for Tier 3, we have 0.33 --

not bad.

For Noble applicants
from both tiers,

these First-Stage
and Reduced-Form estimates

imply a yearly
Noble enrollment effect

of about a third
of a standard deviation gain

in ACT math scores.

Notice there's also a line

connecting the two IV estimates
in the figure.

Because this line
passes through the origin,

its slope, "rise over run,"

is about equal
to the two IV estimates --

in this case,
the slope is about 0.34.

The fact that the line
passes through 0,0

is significant for another reason.

By this fact, we've substantiated
the exclusion restriction.

Specifically,
the exclusion restriction

says that given a group

for which Noble offers
are unrelated to Noble enrollment,

we should expect to see

0 reduced-form effect
of these offers

made to applicants in that group.

How consistent is the evidence
for a Noble cause learning gain

on the order
of 0.34 sigma per year?

In this next graphic,

we've added 12 more points
to the original 2.

The red points here
show First-Stage and Reduced-Form

Noble offer effects
for 12 additional groups,

2 more tiers and 12 groups

defined by demographic
characteristics

related to race, sex,
family income,

and baseline scores.

Although not a perfect fit,

these points cluster around a line
with slope 0.36 sigma

much like the line we saw earlier
for applicants from Tiers 1 and 3.

You're likely now wondering

what the Noble IV
estimates in this figure

have to do
with exam school enrollment.

Here's the answer.

The blue line in this new graphic
shows, as we should expect,

that exam school exposure jumps up

for applicants who clear
their qualifying cutoff.

At the same time,

the red line shows
that Noble school enrollment

clearly falls at the same point.

This is the diversion effect
of exam school offers

on Noble enrollment.

Many kids offered
an exam school seat

prefer that exam school seat
to enrollment at Noble.

IV affords us the opportunity
to go out on a limb

with strong claims
about the mechanism

behind the causal effect.

Here's a strong causal claim

regarding why Chicago exam schools
reduce achievement.

The primary force

driving reduced-form
exam school qualification effects

on ACT scores, I claim,

is the effect of exam school offers
on Noble enrollment.

In support of this claim,

consider the points
plotted here in blue,

all well to the left
of 0 on the x-axis.

These points are negative

because they mark the effect
of exam school qualification

on Noble school enrollment

for particular groups of applicants.

Now we've already seen

that Noble applicants
offered a Noble seat

realize large ACT
math gains as a result.

Now consider exam school offers

as an instrument
for Noble enrollment.

As always, IV is a chain reaction.

If exam school qualification
reduces time at Noble

by 0.37 years,

and each year of Noble enrollment

boosts ACT math scores
by about 0.36 sigma,

we should expect
reduced-form effects

of exam school qualification

to reduce ACT scores

by the product
of these two numbers --

that is, by about 0.13 sigma.

The reduced-form
qualification effects

at the left of the figure

are broadly consistent with this.

They cluster closer to -0.16
than to -0.13,

but that difference is well
within the sampling variance

of the underlying estimates.

The causal story told here

postulates diversion
away from charter schools

as the mechanism
by which exam school offers

affect achievement.

In other words,
it's Noble enrollment

that's presumed to satisfy
an exclusion restriction

when we use exam school offers

as an instrumental variable.

Importantly, as we saw before,

the line in this final graphic,
with two sets of 14 points,

runs through the origin.

This fact supports
our new exclusion restriction.

For any applicant group

for which exam school offers

have little or no effect
on Noble school enrollment,

we should also see
ACT scores unchanged.

At the same time,

because the blue and red dots
cluster around the same line,

the IV estimates of Noble school
enrollment effects

generated by both Noble
and exam school offers

are about the same.

I hope this empirical story

convinces you
of the power of IV and RD

to generate new causal knowledge.

For decades,
I've been lucky to work

on many equally engaging
empirical problems.

I computed the draft
lottery IV estimates

in my Princeton Ph.D. thesis

on a big, hairy mainframe monster,

using 9-track tapes and leased space

on a communal hard drive.

Princeton graduate students

learned to mount
and manipulate tape reels

the size of a cheesecake.

Thankfully, empirical work today
is a little less labor-intensive.

What else has improved
in the modern empirical era?

in a 2010 article, Steve Pischke
and I coined the phrase

"Credibility Revolution."

By this, we mean economic shift

towards transparent
empirical strategies

applied to concrete
causal questions,

like the questions David Card
has studied so convincingly.

The econometrics of my school days

focused more on models
than on questions.

The modeling concerns
of that era have mostly faded,

but econometricians have since
found much to contribute.

I'll save my personal lists

of greatest hits
and exciting new artists

for the print version of this lecture.

I'll wrap up here by saying

that I'm proud to be part
of the contemporary

empirical economics enterprise,

and I'm gratified beyond words

to have been recognized
for contributing to it.

Back at Princeton, in the late '80s,

my graduate school classmates
and I chuckled

reading Ed Leamer's lament

that no economist takes another
economist's empirical work seriously.

This is no longer true.

Empirical work today aspires
to tell convincing causal stories.

Not that every effort succeeds --

far from it.

But as any economics job
market candidate will tell you,

empirical work carefully executed
and clearly explained

is taken seriously indeed --

that is a measure
of our enterprise's success.

♪ [music] ♪

- [Narrator] If you'd like
to learn more from Josh,

check out his free course
"Mastering Econometrics."

If you'd like to explore
Josh's research,

check out the links
in the description,

or you can click to watch more
of Josh's videos.

♪ [music] ♪