-
♪ [music] ♪
-
- [Joshua] As I stilled
my trembling iPhone
-
early on October 11th,
-
my thoughts went to the question
of whether Nobel-level recognition
-
might change life
for the Angrist family.
-
Ours is a close-knit family.
-
We lack for nothing.
-
So I worried
that stressful Nobel celebrity
-
might not be a plus.
-
But with the first cup of coffee,
-
I began to relax.
-
It occurred to me
-
that the matter
of how public recognition
-
affects a scholar's life
-
is, after all,
a simple causal question.
-
The Nobel intervention
-
is substantial, sudden,
and well-measured.
-
Outcomes like health and wealth
are easy to record.
-
Having just been recognized
with my co-laureates,
-
Guido Imbens and David Card,
-
for answering causal questions
using observational data,
-
my thoughts moved
from personal upheaval
-
to the more familiar demands
-
of identification and estimation
of causal effects.
-
I was able to soothe
my worried mind
-
by imagining a study
of the Nobel Prize treatment effect.
-
How would such a study
be organized?
-
In a 1999 essay
-
published in the "Handbook
of Labor Economics,"
-
Alan Krueger and I embraced
the phrase "empirical strategy."
-
The handbook volume
in question was edited
-
by two of my Princeton
PhD thesis advisors,
-
Orley Ashenfelter and David Card,
-
among the most successful
and prolific graduate advisors
-
economics has known.
-
An empirical strategy
is a research plan
-
that encompasses data collection,
-
identification,
and econometric estimation.
-
Identification is the applied
econometricians term
-
for research design --
-
a randomized, clinical trial.
-
And RCT is the simplest
and most powerful research design.
-
In RCTs, causal effects
are identified
-
by the random assignment
of treatment.
-
Random assignment ensures
-
that treatment and control groups
-
are comparable
in the absence of treatment.
-
So differences
between them afterwards
-
reflect only the treatment effect.
-
Nobel Prizes are probably
not randomly assigned.
-
This challenge notwithstanding,
-
a compelling empirical strategy
for the Nobel treatment effect
-
comes to mind,
-
at least as a flight of empirical fancy.
-
Imagine a pool of prize-eligible
Nobel applicants,
-
the group under consideration
for the prize.
-
Applicants need not apply themselves.
-
They are, I presume, nominated
by their peer scholars.
-
My fanciful Nobel impact study
looks only at Nobel applicants,
-
since these are all elite scholars.
-
But that is only the first step.
-
Credible applicants, I imagine,
are evaluated by judges
-
using criteria like publications,
citations, nominating statements.
-
I imagine this material is reviewed
and assigned a numerical score,
-
using some kind of scoring rubric.
-
Top scorers up to three per field,
in any single year
-
win a prize.
-
Having identified applicants
-
that collected data
on their scores,
-
the next step
in my Nobel impact study
-
is to record the relevant cutoffs.
-
The Nobel cutoff
is the lowest score
-
among those awarded a prize.
-
Many Nobel hopefuls
just missed the cutoff.
-
Looking only at near misses,
along with the winners,
-
differences in scores
-
between those above
and below the cutoff
-
begin to look serendipitous,
almost randomly assigned.
-
After all,
-
near-Nobels are among
the most eminent of scholars, too.
-
With one more
high-impact publication,
-
a little more support
from nominators,
-
they would have been awarded
-
Nobel gold.
-
Some of them, someday,
surely will be.
-
The empirical strategy
sketched here is called
-
a Regression Discontinuity Design,
-
RD.
-
RD exploits the jumps
in human affairs,
-
induced by rules, regulations,
-
and the need to classify people
for various assignment purposes.
-
When treatment
or intervention is determined
-
by whether a tiebreaking variable
crosses a threshold,
-
those just below the threshold
become a natural control group
-
for those who clear it.
-
RD does not require
that the variable
-
whose causes we seek,
-
switch fully on or fully off
at the cutoff.
-
We require only
-
that the average value
of this variable
-
jump at the cutoff.
-
RD can allow,
for example, for the fact
-
that this year's near-Nobel
might be next year's winner.
-
Allowing for this
leads to the use of jumps
-
and the rate at which
treatment is assigned
-
to construct instrumental variables,
-
IV,
-
estimates of the effect
of treatment received.
-
This sort of RD
is said to be fuzzy,
-
But as Steve Pischke and I
wrote in our first book:
-
"Fuzzy RD is IV."
-
(kids cheering)
-
The first RD study I contributed to
-
was written with my frequent
collaborator, Victor Lavy.
-
This study is motivated
-
by the high costs
and uncertain returns
-
to smaller elementary school classes.
-
We exploited a rule
used by Israeli elementary schools
-
to determine class size.
-
This rule is used to estimate
class size effects,
-
as if in a class size RCT.
-
In the 1990s,
Israeli classes were large.
-
Students enrolled
in a grade cohort of 40
-
were likely to be seated
in a class of 40.
-
That's the relevant cutoff.
-
Add another child
to the cohort, making 41,
-
and the cohort
was likely to be split
-
into two much smaller classes.
-
This leads to the Maimonides'
rule research design,
-
so named because
the 12th-century Rambam
-
proposed a maximum
class size of 40.
-
This figure plots [is really]
4th grade class sizes
-
as a function
of 4th grade enrollment,
-
overlaid with the theoretical
class size rule,
-
Maimonides' rule.
-
The fit isn't perfect --
-
that's a feature that makes
this application of RD fuzzy.
-
But the gist of the thing
is a marked class size drop
-
at each integer multiple of 40,
the relevant cutoff,
-
just as predicted by the rule.
-
As it turns out,
these drops in class size
-
are reflected in jumps
-
in 4th and 5th grade math scores.
-
Would a comparison
of Nobel laureates to near-laureates
-
really be a good natural experiment?
-
The logic behind this sort of claim
-
seems more compelling
for comparisons of schools
-
with 40 and 41 4th graders
-
than for comparisons
of laureates and near-laureates.
-
Yet, both scenarios exploit
a feature of the physical world.
-
Provided the tiebreaking variable,
-
known to RD mavens
as the "running variable,"
-
has a continuous distribution,
-
the probability of crossing
the cutoff approaches one-half
-
when examined in a narrow window
-
around the cutoff.
-
In RD empirical work,
-
the window around such cutoffs
is known as a bandwidth.
-
Importantly, this limiting
probability is 0.5 for everybody
-
regardless of how qualified they look
-
going into the Nobel competition.
-
This remarkable fact can be seen
in data on applicants
-
to one of New York's
highly coveted screen schools.
-
By way of background,
-
roughly 40% of New York City's
middle and high schools
-
select their applicants
on the basis of test scores, grades,
-
and other exacting criteria.
-
In other words, the admissions
regime for screen schools
-
is a lot like the scheme
I've imagined
-
for the Nobel Prize.
-
Screen schools
are but one of a number
-
of highly selective systems
within a system
-
in large US school districts.
-
Boston, Chicago, San Francisco,
and Washington, D.C
-
all feature highly selective
institutions,
-
often known as exam schools.
-
Exam schools operate
-
as part of larger
public school systems
-
that enroll students
without screening.
-
Motivated by
the enduring controversy
-
over the equity
of screened admissions,
-
my blueprint
labs collaborators and I
-
have examined the causal effects
of exam school attendance
-
in Boston, Chicago, and New York.
-
This figure shows the probability
of being offered a seat
-
at New York's storied
Townsend Harris High School,
-
ranked 12th, nationwide.
-
Bar height in the figure
marks the qualification rate --
-
that is, the likelihood of earning
a Townsend Harris admission score
-
above that of the lowest
scoring applicant
-
offered a seat.
-
Importantly, the bars show
qualification rates conditional
-
on a measure of pre-application
baseline achievement.
-
In particular, the bars mark
qualification rates conditional
-
on whether an applicant
-
has upper quartile or lower quartile
6th grade math scores.
-
Townsend Harris applicants
with high baseline scores
-
are much more likely to qualify
-
than applicants
with low baseline scores.
-
This isn't surprising.
-
But in a shrinking
symmetric bandwidth
-
around the school's cutoff,
-
qualification rates
in the two groups converge.
-
Qualification rates in the last
and smallest groups
-
are both remarkably close
to one-half.
-
This is what we'd expect to see
where Townsend Harris
-
to admit students
by tossing a coin,
-
rather than by selecting
only those who scored highly
-
on the school's entrance exam.
-
Even when admissions
operates by screening,
-
the data can be arranged
so as to mimic an RCT.
-
Few of the questions I've studied
are more controversial
-
than the question of access
to public exam schools,
-
like the Boston Latin School,
-
Chicago's Payton and Northside
selective enrollment high schools,
-
and New York's legendary
Brooklyn Tech,
-
Bronx Science,
-
and Stuyvesant
specialized high schools,
-
which have graduated
14 Nobel laureates between them.
-
Townsend Harris, the school
we started with today,
-
graduated three Nobels,
including economist Ken Arrow.
-
Exam school proponents
-
see the opportunities
these schools provide
-
as democratizing public education.
-
Wealthy families, they argue,
-
can access exam school curricula
in the private sector.
-
Shouldn't ambitious
low-income students
-
be afforded the same chance
at elite education?
-
Critics of selective
enrollment schools
-
argue that rather than
expanding equity,
-
exam schools are inherently biased
-
against the Black
and Hispanic students
-
that make up the bulk
of America's urban districts.
-
New York's super selective
Stuyvesant, for example,
-
enrolled only
seven Black students in 2019,
-
out of an incoming class of 895.
-
But are exam school seats
really worth fighting for?
-
My collaborators and I
have repeatedly used
-
RD empirical strategies
to study the causal effects
-
of attendance at exam schools
-
like Townsend Harris
and Boston Latin.
-
Our first exam school study,
-
which looks at schools
in Boston and New York,
-
encapsulates
these findings in its title:
-
"The Elite Illusion."
-
"The Elite Illusion"
refers to the fact
-
that while exam school students
undoubtedly have high test scores
-
and other good outcomes,
-
this is not a causal effect
of exam school attendance.
-
Our estimates consistently suggest
-
that the causal effects
of exam school attendance
-
on their students learning
and college-going are zero.
-
Maybe even negative.
-
The good performance
of exam school students
-
reflect selection bias --
-
that is, the process by which
these students are chosen,
-
rather than causal effects.
-
Data from Chicago's large exam
school sector
-
illustrate the elite illusion.
-
This figure plots
peer mean achievement --
-
that is, the 6th grade test scores
of my 9th grade classmates
-
against the admissions tiebreaker
-
for a subset of applicants
-
to any one of Chicago's
nine exam schools.
-
Applicants to these schools
rank up to 6,
-
while the exam schools
prioritize their applicants
-
using a common composite index,
-
formed from an admissions test,
-
GPAs, and grade 7
standardized scores.
-
This composite tiebreaker
is the running variable
-
for an RD design
that reveals what happens
-
when any applicant is offered
an exam school seat.
-
In Chicago's exam school match,
-
which is actually an application
-
of the celebrated Gale and Shapley
matching algorithm,
-
exam school applicants are sure
to be offered a seat somewhere
-
when they clear the lowest
in their set of cutoffs
-
among the schools they rank.
-
We call this lowest cutoff
the "qualifying cutoff."
-
The figure shows a sharp jump
in peer mean achievement
-
for Chicago exam school applicants
-
who clear their qualifying cutoff.
-
This jump reflects the fact
-
that most applicants
offered an exam school seat take it,
-
and applicants who enroll
-
at one of Chicago's selective
enrollment high schools
-
are sure to be seated
in a 9th grade classroom
-
filled with academically
precocious peers,
-
because only the relatively
precocious make it in.
-
The increase in peer achievement
across the qualifying cutoff
-
amounts to almost
half a standard deviation --
-
a very large effect.
-
And yet, precocious peers,
notwithstanding,
-
the offer of an exam school seat
does not appear to increase learning.
-
Let's plot applicants ACT scores
against their tiebreaker values.
-
This plot shows that exam
school applicants
-
who clear their qualifying cutoff
-
perform sharply worse on the ACT.
-
What explains this?
-
It takes a tale of IV and RD
to untangle the forces
-
behind this intriguing
and unexpected negative effect.
-
But first, some IV theory,
-
Guido Imbens and I
developed theoretical tools
-
that enhance
economists' understanding
-
of empirical strategies
-
involving IV and RD.
-
The prize we share
is in recognition of this work.
-
Guido and I overlapped
for only one year at Harvard,
-
where we had both taken
our first jobs post PhD.
-
I welcomed Guido
to Cambridge, Massachusetts
-
with a pair of interesting
instrumental variables.
-
I had used
the draft lottery instrument
-
in my PhD thesis
-
to estimate the long-run
economic consequences
-
of serving in the Armed Forces
-
for soldiers who were drafted.
-
The draft lottery instrument
relies on the fact
-
that lottery numbers
randomly assigned to birthdays
-
determined Vietnam-era
conscription risk.
-
Yet, even then, most soldiers
were volunteers,
-
as they are today,
-
The quarter birth instrument
-
is used in my 1991 paper
with Alan Krueger
-
to estimate the economic
returns to schooling.
-
This instrument uses the fact
-
that men who were born
earlier in the year
-
were allowed to drop out
of high school
-
on their 16th birthday
-
with less schooling completed
than those born later.
-
Guido and I soon began
asking each other,
-
"What really do we learn
from the draft eligibility
-
and quarter of birth
natural experiments?"
-
An early result in our quest
for a new understanding of IV
-
was a solution to the problem
of selection bias
-
in an RCT with partial compliance.
-
Even in a randomized clinical trial,
-
some of the people assigned
to treatment may opt out.
-
This fact has long vexed trialists
-
because decisions to opt out
are not made by random assignment.
-
Our first manuscript together
-
shows that in a randomized trial
with partial compliance,
-
you can use IV
-
to estimate the effect
of treatment on the treated,
-
even when some offered treatment
-
decline it.
-
This works in spite of the fact
-
that those who comply
with treatment
-
may be a very select group.
-
Unfortunately, for us,
we were late to the party.
-
Not long after releasing
our first working paper,
-
we learned of a concise contribution
from Howard Bloom
-
that includes this theoretical result.
-
Remarkably, Bloom had derived
this from first principles
-
without making a connection to IV.
-
So Guido and I went back
to the drawing board.
-
And a few months later,
we had LATE --
-
a theorem showing how to estimate
-
the Local Average Treatment Effect.
-
The LATE theorem generalizes
the Bloom theorem
-
and establishes the connection
between compliance and IV.
-
Maintaining the clinical trials analogy,
-
let "Zi" indicate whether subject "i"
is offered treatment.
-
This is randomly assigned.
-
Also, let "D1i" indicate
subject "i's" treatment status
-
when assigned to treatment,
-
and let the "0i" indicate
subject "i's" treatment status
-
when assigned to control.
-
I'll use this formal notation
-
to give a clear statement
of the late result,
-
and then follow up with examples.
-
A key piece of the late framework,
-
pioneered by statistician
Don Rubin,
-
is the pair of potential outcomes.
-
As is customary,
-
I denote potential outcomes
for subject "i"
-
in the treated and untreated states
-
by "Y1i" and "Y0i", respectively,
-
The observed outcome
is "Y1i" for the treated
-
and "Y0i" for those not treated.
-
"Y1i - Y0i"
-
is the causal effect
of treatment on individual "i",
-
but we can never see.
-
We try, therefore, to estimate
some kind of average causal effect.
-
The late framework allows us
to do that in an RCT
-
where some controls are treated.
-
The theorem says
-
that the average causal
effect on people
-
whose treatment status
can be changed
-
by the offer of treatment
-
is the ratio of ITT
to the treatment control difference
-
in compliance rates.
-
A mathematical statement
of this result appears here,
-
where Greek letter Delta
symbolizes the ITT effect
-
and Greek symbols pi1 and pi0
-
are compliance rates
in the group assigned to treatment
-
and the group assigned
to control, respectively.
-
The print version of this lecture
-
delves deeper
into LATE intellectual history,
-
highlighting key contributions
made with Rubin.
-
For now, though, I'd like to make
the late theorem concrete for you
-
by sharing one
of my favorite applications of it.
-
I'll explain the late framework
-
through a research question
that has fascinated me
-
for almost two decades.
-
What is the causal effect
-
of charter school attendance
on learning?
-
Charter schools are public schools
-
that operate independently
-
of traditional American
public school districts.
-
A charter, the right
to operate a public school
-
is typically awarded
for a limited period
-
subject to renewal, conditional
on good school performance.
-
Charter schools
are free to structure
-
their curriculum
and school environment.
-
The most controversial difference
-
between charters
and traditional public schools
-
is the fact that the teachers
and staff who work at charter schools
-
rarely belonged to labor unions.
-
By contrast, most
big city public school teachers
-
work under union contracts.
-
The 2010 documentary film
"Waiting for Superman"
-
features schools belonging to
the Knowledge is Power Program,
-
KIPP.
-
KIPP schools are emblematic
of the high expectations,
-
sometimes also called "no excuses"
approach to public education.
-
The "no excuses" model features
-
a long school day
and extended school year,
-
selective teacher hiring
-
and focuses on traditional
reading and math skills.
-
The American debate
over education reform
-
often focuses on the achievement gap --
-
that's shorthand
for large test score differences
-
by race and ethnicity.
-
Because of its focus
on minority students,
-
KIPP is often central
in this debate
-
with supporters pointing to the fact
-
that non-White KIPP students
have markedly higher test scores
-
than non-White students
from nearby schools.
-
KIPP skeptics, on the other hand,
-
argue that KIPP's apparent success
-
reflects the fact
that KIPP attracts families
-
whose children would be
more likely to succeed anyway.
-
Who's right?
-
As you've probably guessed by now,
-
a randomized trial
might prove decisive
-
in the debate
over schools like KIPP.
-
Like Nobel Prize is, though,
-
seats at KIPP
are not randomly assigned.
-
Well, at least, not entirely.
-
In fact,
-
Massachusetts charter schools
with more applicants than seats
-
must offer their seats by lottery.
-
Sounds like a good natural experiment.
-
A little over a decade ago,
-
my collaborators and I
collected data
-
on KIPP admissions lotteries,
-
laying the foundation
for two pioneering charter studies,
-
the first to use lotteries
to study KIPP.
-
Our KIPP analysis
is a classic IV story
-
because many students
offered a seat in the KIPP lottery
-
failed to show up in the fall,
-
while a few not offered a seat
nevertheless find their way in.
-
This graphic shows KIPP
middle school applicants math scores
-
one year after applying to KIPP.
-
The entries above the line
-
show that Kip applicants
who were offered a seat
-
have standardized
math scores close to zero --
-
that is, near the state average.
-
As before, we're working
with standardized score data
-
that has a mean of 0
and a standard deviation of 1.
-
Because KIPP applicants
start with 4th grade scores
-
that are roughly 0.3
standard deviations
-
below the state mean,
-
achievement at the level
of the state average is impressive.
-
By contrast, the average math score
among those not offered a seat
-
is about -0.36 sigma --
-
that is, 0.36 standard deviations
below the state mean,
-
a result typical for urban students
in Massachusetts.
-
Since lottery offers
are randomly assigned,
-
we could say with confidence
that the offer of a seated KIPP
-
boost math scores
by an average of 0.36 sigma --
-
a large effect
that's also statistically precise.
-
We can be confident
this isn't a chance finding.
-
What does an offer effect
0.36 sigma
-
tell us about the effects
of actually going to KIPP?
-
IV methods
convert KIPP offer effects
-
into KIPP attendance effects.
-
I'll use this brief clip
-
from my Marginal Revolution
University short course
-
to quickly review
the key assumptions
-
behind this conversion.
-
- [Narrator] IV describes
a chain reaction.
-
Why do offers affect achievement?
-
Probably because they affect
charter attendance,
-
and charter attendance
improves math scores.
-
The first link in the chain
called the First Stage
-
is the effect of the lottery
on charter attendance.
-
The Second Stage is the link
-
between attending a charter
and an outcome variable --
-
in this case, math scores.
-
The instrumental variable,
or Instrument for short,
-
is the variable that initiates
the chain reaction.
-
The effect of the instrument
on the outcome
-
is called the Reduced Form.
-
This chain reaction can be
represented mathematically.
-
We multiply the first stage --
the effect of winning on attendance,
-
by the second stage --
the effect of attendance on scores,
-
and we get the reduced form --
-
the effect of winning
the lottery on scores.
-
The Reduced Form and First Stage
are observable and easy to compute.
-
However, the effect
of attendance on achievement
-
is not directly observed.
-
This is the causal effect
we're trying to determine.
-
Given some important assumptions
we'll discuss shortly,
-
we can find the effect
of KIPP attendance
-
by dividing the Reduced Form
by the First Stage.
-
- [Joshua] IV eliminates selection bias,
-
but like all of our tools,
-
the solution builds
on a set of assumptions
-
not to be taken for granted.
-
First, there must be
a substantial first stage --
-
that is, the instrumental variable,
-
winning or losing the lottery,
-
must really change the variable
whose effect we're interested in,
-
here, KIPP attendance.
-
In this case, the first stage
is not really in doubt.
-
Winning the lottery
-
makes KIPP attendance
much more likely.
-
Not all IV stories are like that.
-
Second, the instrument must be
as good as randomly assigned,
-
meaning lottery winners and losers
have similar characteristics.
-
This is the independence assumption.
-
Of course, KIPP lottery wins
really are randomly assigned.
-
Still, we should check
for balance and confirm
-
that winners and losers
-
have similar family backgrounds,
similar aptitudes, and so on.
-
In essence, we're checking
to ensure KIPP lotteries are fair,
-
with no group of applicants
suspiciously likely to win.
-
Finally, we require
the instrument change outcomes
-
solely through the variable of interest --
-
in this case, attending KIPP.
-
This assumption is called
the Exclusion Restriction.
-
The causal effect
of KIPP attendance
-
can therefore be written
-
as the ratio of the effect
of offers on scores
-
in the numerator
-
over the effect of offers
on KIPP enrollment
-
in the denominator.
-
The numerator in this IV formula --
-
that is, the direct effect
of the instrument on outcomes
-
has a special name.
-
This is called the Reduced Form.
-
The denominator is the first stage.
-
The exclusion restriction
is often the trickiest
-
or most controversial part
of an IV story.
-
Here, the exclusion restriction
amounts to the claim
-
that the 0.36 score differential
between lottery winners and losers
-
is entirely attributable
-
to the 0.74 win/loss difference
in attendance rates.
-
Plugging in the numbers,
-
the effect of KIPP attendance
works out to be 0.84 sigma,
-
almost half
a standard deviation gain
-
in math scores.
-
That's a remarkably large effect.
-
Who exactly benefits
so spectacularly from KIPP?
-
Does everyone who applies
to KIPP see such large gains?
-
LATE answers this question.
-
The LATE interpretation
of the KIPP IV empirical strategy
-
is illuminated by the
Biblical story of Passover,
-
which explains that there are
four types of children,
-
each with characteristic behaviors.
-
To keep track of these children
and their behavior,
-
I'll give them alliterative names.
-
Applicants like Alvaro
are dying to go to KIPP.
-
If Alvaro loses the KIPP lottery,
-
his mother finds a way
to enroll him in KIPP anyway,
-
perhaps by reapplying.
-
Applicants like Camila
are happy to go to KIPP
-
if they win a seat in the lottery,
-
but stoically accept
the verdict, if they lose.
-
Finally, applicants like Normando
-
worry about long days
and lots of homework at KIPP.
-
Normando doesn't really want to go
-
and refuses to go to KIPP
when told that he won the lottery.
-
Normando was called a never-taker
-
because win or lose,
he doesn't go to KIPP.
-
At the other end
of KIPP commitment,
-
Alvaro is called an always-taker.
-
He'll happily take a seat
when offered
-
while his mother simply finds a way
to make it happen for him,
-
even when he loses.
-
For Alvaro and Normando both,
-
choice of school, KIPP traditional,
is unaffected by the lottery.
-
Camila is the type of applicant
who gives IV its power.
-
The instrument determines
her treatment status.
-
IV strategies depend
on applicants like Camilla,
-
who are called compliers.
-
This term comes from the world
of randomized trials
-
introduced earlier.
-
As we've already discussed,
-
many randomized trials randomize
only the opportunity to be treated
-
while the decision
to comply with the treatment
-
remains voluntary and non-random.
-
RCT compliers are those
who take treatment
-
when the offer of treatment is made,
-
but not otherwise.
-
With lottery instruments,
-
LATE is the effect
of KIPP attendance on Camila
-
and other compliers like her
-
who enroll at KIPP, take treatment
-
when offered treatment
through the lottery,
-
but not otherwise.
-
IV methods are uninformative
-
for always-takers like Alvaro
and never-takers like Normando
-
because the instrument
is unrelated
-
to their treatment status.
-
Hey, didn't I say
there are four types of children?
-
A fourth type of child in IV theory
behaves perversely.
-
Every family has one.
-
These defiant children and enroll in
KIPP only when they lose the lottery.
-
Actually, the late theorem requires
us to assume there are few defiers --
-
that seems like a reasonable assumption
for charter lottery instruments,
-
if not in life.
-
The late theorem is sometimes
seen as limiting the relevance
-
of econometric estimates
-
because it focuses attention
on groups of compliers
-
Yet, the population of compliers is a
group we'd very much like to learn about.
-
In the KIPP example, compliers
our children likely to be drawn into KIPP
-
were the school to expand
and offer additional seats in a lottery.
-
How relevant is this?
-
A few years ago,
-
Massachusetts indeed allowed
thriving charter schools to expand.
-
A recent study by some of my lab mates
-
shows that late estimates like
the one we just computed for KIPP
-
predict learning gains at the
schools created by charter expansion.
-
LATE isn't just a theorem.
-
It's a framework.
-
The late framework can be used to
estimate the entire distribution
-
of potential outcomes for compliers
-
as if we really did have a
randomized trial for this group.
-
Although the theory behind this
fact is necessarily technical,
-
the value of the framework is easily
appreciated in practice.
-
By way of illustration, recall that the KIPP study
-
is motivated in part by
differences in test scores by race.
-
Let's look at the distribution of
4th grade scores separately by race,
-
for applicants to Boston
charter middle schools.
-
The two sides of this figure
-
show distributions for treated
and untreated compliers.
-
Treated compliers are compliers offered a charter seat in a lottery,
-
while untreated compliers are not offered a seat.
-
Because these are 4th grade scores,
-
while middle school begins in 5th or 6th grade.
-
The two sides
of the figure are similar.
-
Both sides show score distributions for
Black applicants
-
shifted to the left of the corresponding score distributions
-
for Whites.
-
By 8th grade, treated compliers have completed
middle school at a Boston charter,
-
while untreated compliers have
remained in traditional public school.
-
Remarkably, this next graphic shows that the
8th grade score distributions
-
of Black and White treated compliers
-
are indistinguishable.
-
Boston charter middle schools,
closed the achievement gap.
-
But for the untreated, Black and White
score distributions remained distinct
-
with Black students behind White students
-
as they were in 4th grade,
-
Boston charters closed the achievement gap
-
because those who enter charter schools the farthest behind
-
tend to gain
the most from Charter enrollment.
-
I elaborate on this point in
the print version of this talk.
-
Remember the puzzle of
negative Chicago exam school effects?
-
I'll finish the scientific part of my
talk by using IV and RD
-
to explain this this surprising finding.
-
The resolution of this puzzle starts
with the fact
-
that economic reasoning is about alternatives.
-
So what's the alternative
to an exam school education?
-
For most applicants to
Chicago exam schools,
-
The leading non exam alternative
is a traditional public school.
-
But many of Chicago's rejected exam school
applicants
-
enroll in a charter school
-
Exam school offers
-
therefore reduce the
likelihood of charter school attendance.
-
Specifically, exam schools divert
applicants away from high schools
-
in the Noble Network of Charter Schools.
-
Noble, with pedagogy much like KIPP,
-
is one of
Chicago's most visible charter providers.
-
Also like KIPP convincing evidence on
Noble effectiveness comes from admissions lotteries
-
The x-axis in this graphic shows lottery
offer effects on years enrolled at Noble.
-
This is the Noble first stage.
-
For an IV setup that uses a dummy,
-
indicating Noble lottery offers as
an instrument for Noble enrollment.
-
Now, this graphic has a
feature that distinguishes it
-
from the simpler KIPP analysis.
-
The plot shows first-stage effects
-
for two groups:
-
one for Noble applicants
-
who live in Chicago's lowest income
neighborhoods, Tier 1,
-
and one for Noble applicants
-
who live in higher income
areas, tier 3.
-
Remember the IV chain reaction?
-
Each point in this graphic has coordinates
given by first-stage reduced form
-
and therefore implies an IV estimate.
-
The effect of Noble enrollment on ACT scores
-
is the ratio of reduced form coordinate
-
to first stage coordinate.
-
The graphic shows two such ratios.
-
The relevant results for Tier 1 are .35,
-
while for tier 3, we have .33 --
-
not bad.
-
For Noble applicants from both tiers,
-
these first stage and reduced form estimates
-
imply a yearly Noble and enrollment effect
-
of about a third of a standard deviation gain,
-
in ACT math scores.
-
Notice there's also a line connecting
the two IV estimates in the figure.
-
Because this line passes
through the origin,
-
its slope rise over run is about equal
to the two IV estimates.
-
In this case, the slope is about .34.
-
The fact that the line passes
through 0,0
-
is significant for another reason
-
By this fact, we've
substantiated the exclusion restriction.
-
Specifically, the exclusion restriction
-
says that given a group
-
for which Noble
offers are unrelated to Noble enrollment,
-
we should expect to see 0 reduced form
effect of these offers
-
made to applicants in that group.
-
How consistent is the evidence for a
Noble cause learning gain
-
on the order of .34 sigma per year?
-
In this next graphic,
-
we've added 12 more points to the original 2.
-
The red points here show first-stage
in reduced form Noble offer effects
-
for 12 additional groups
-
2 more tiers and 12 groups,
defined by demographic characteristics
-
related to race sex, family income, and baseline scores.
-
Although not a perfect fit,
-
these points cluster around a line
with slope .36 sigma
-
much like the line we saw earlier
for applicants from Tiers 1 and 3.
-
You're likely now wondering
-
what the Noble IV estimates
in this figure
-
have to do with exam school enrollment.
-
Here's the answer.
-
The blue line in this new graphic
shows, ss we should expect,
-
that exam school exposure jumps up
-
for applicants who clear their
qualifying cutoff.
-
At the same time,
-
the red line shows that Noble school
enrollment
-
clearly falls at the same point.
-
This is the diversion effect of exam
school offers
-
on Noble enrollment.
-
Many kids offered an exam school
seat
-
prefer that exam school seat
to enrollment at Noble
-
IV affords us
the opportunity to go out on a limb
-
with strong claims about the
mechanism behind the causal effect.
-
Here's a strong causal claim
-
regarding why
Chicago exam schools reduce achievement.
-
The primary force
-
driving reduced form
exam school qualification effects
-
on ACT scores.
-
Qualification effects on ACT scores. I claim,
-
is the effect of exam
school offers on Noble enrollment.
-
In support of this claim,
-
consider the points plotted here in blue,
-
all well to the left of 0 on the x-axis.
-
These points are negative
-
because they mark
the effect of exam school qualification
-
on Noble School enrollment
-
for
particular groups of applicants.
-
Now we've already seen
-
that Noble applicants offered a Noble seat
-
realize large ACT math gains as a result.
-
Now consider exam school offers
-
as an instrument for Noble enrollment.
-
As always, IV is a chain reaction.
-
If exam school qualification reduces
time at Noble, by .37 years,
-
and each year of Noble enrollment
-
boosts ACT math scores by
about .36 sigma,
-
we should expect reduced form
effects of exam school qualification
-
to reduce ACT scores by the
product of these two numbers --
-
that is, by about .13 sigma.
-
The reduced form qualification
effects at the left of the figure
-
are broadly consistent with this.
-
They cluster closer to -0.16
than to -0.13,
-
but that difference is well
within the sampling variance
-
of the underlying estimates.
-
The causal story told here
-
postulates diversion
away from charter schools
-
as the mechanism by which exam
school offers effect achievement.
-
In other words, its Noble enrollment
-
that's presumed to satisfy
an exclusion restriction
-
when we use exam school offers
-
as an instrumental variable.
-
Importantly, as we saw before,
-
the line in this final graphic,
with two sets of 14 points,
-
runs through the origin.
-
This fact supports our
new exclusion restriction.
-
For any applicant group
-
for which exam school offers
-
have little or no effect
on Noble School enrollment,
-
we should also see
ACT scores unchanged.
-
At the same time,
-
because the blue and
red dots cluster around the same line,
-
the IV estimates of noble school
enrollment effects
-
generated by both Noble and exam school
offers
-
are about the same.
-
I hope this empirical story convinces
you
-
of the power of IV and RD
-
to generate new causal knowledge.
-
For decades, I've been lucky to work
-
on many
equally engaging empirical problems.
-
I computed the draft
lottery IV estimates
-
in my Princeton PhD thesis
-
on a big, hairy mainframe monster,
-
using nine track tapes and lease space
-
on a communal hard drive,
-
Princeton graduate students
-
learn
to mount and manipulate tape reels
-
the size of a cheesecake.
-
Thankfully, empirical work today
is a little less labor-intensive.
-
What else has improved
in the modern empirical era?
-
in a 2010 article, Steve Pischke
and I coined the phrase
-
"Credibility Revolution."
-
By this, we mean economic shift
-
towards
transparent empirical strategies
-
applied to concrete, causal questions,
-
like the questions David Card
has studied so convincingly.
-
The econometrics of my schooldays
-
focused more on models than on questions.
-
The modeling concerns of
that era have mostly faded,
-
but econometricians have since
found much to contribute.
-
I'll save my personal lists
-
of greatest
hits and exciting new artists
-
for the print version of this lecture.
-
I'll wrap up here by saying
-
that I'm
proud to be part of the contemporary.
-
empirical economics enterprise,
-
and I'm gratified beyond words
-
to have been recognized for contributing
to it.
-
Back at Princeton, in the late '80s,
-
my graduate school classmates and I
chuckled
-
reading Ed Leamer's lament
-
that no economist takes another
economists empirical work seriously.
-
This is no longer true.
-
Empirical work today aspires
to tell convincing causal stories.
-
Not that every
effort succeeds -- far from it.
-
But as any economic's job
market candidate will tell you,
-
empirical work carefully executed and
clearly explained
-
is taken seriously indeed.
-
That is a measure of
our enterprise's success.
-
- [Narrator] If you'd like
to learn more from Josh,
-
check out his free course
"Mastering Econometrics."
-
If you'd like to explore Josh's research,
-
check out the links in the description,
-
or you can click to watch more
of Josh's videos.
-
♪ [music] ♪