WEBVTT 00:00:01.263 --> 00:00:03.908 ♪ [music] ♪ 00:00:10.800 --> 00:00:14.664 - [Joshua] As I stilled my trembling iPhone early on October 11th, 00:00:14.664 --> 00:00:18.100 my thoughts went to the question of whether Nobel-level recognition 00:00:18.300 --> 00:00:20.600 might change life for the Angrist family. 00:00:20.800 --> 00:00:22.599 Ours is a close-knit family. 00:00:22.599 --> 00:00:24.114 We lack for nothing. 00:00:24.114 --> 00:00:26.513 So I worried that stressful Nobel celebrity 00:00:26.513 --> 00:00:27.900 might not be a plus. 00:00:28.200 --> 00:00:30.000 But with the first cup of coffee, 00:00:30.000 --> 00:00:31.600 I began to relax. 00:00:32.100 --> 00:00:33.270 It occurred to me 00:00:33.270 --> 00:00:35.359 that the matter of how public recognition 00:00:35.359 --> 00:00:36.720 affects a scholars life 00:00:36.720 --> 00:00:40.100 is, after all, a simple causal question. 00:00:40.461 --> 00:00:41.812 The Nobel intervention 00:00:41.812 --> 00:00:45.316 is substantial, sudden, and well-measured. 00:00:45.316 --> 00:00:48.500 Outcomes like health and wealth are easy to record. 00:00:49.000 --> 00:00:51.662 Having just been recognized with my co-laureates, 00:00:51.662 --> 00:00:53.503 Guido Imbens and David Card, 00:00:53.503 --> 00:00:57.100 for answering causal questions using observational data, 00:00:57.400 --> 00:00:59.800 my thoughts moved from personal upheaval 00:01:00.216 --> 00:01:02.193 to the more familiar demands 00:01:02.193 --> 00:01:05.700 of identification and estimation of causal effects. 00:01:05.900 --> 00:01:08.200 I was able to soothe my worried mind 00:01:08.400 --> 00:01:12.700 by imagining a study of the Nobel Prize treatment effect. 00:01:13.100 --> 00:01:15.300 How would such a study be organized? 00:01:15.700 --> 00:01:19.600 in a 1999 essay published in the "Handbook of Labor Economics," 00:01:19.800 --> 00:01:23.500 Alan Krueger and I embraced the phrase "empirical strategy." 00:01:23.900 --> 00:01:26.061 The handbook volume in question was edited 00:01:26.061 --> 00:01:29.313 by two of my Princeton PhD thesis advisors, 00:01:29.313 --> 00:01:31.570 Orley Ashenfelter and David Card, 00:01:31.570 --> 00:01:34.940 among the most successful and prolific graduate advisors 00:01:34.940 --> 00:01:36.690 economics has known 00:01:36.690 --> 00:01:39.100 An empirical strategy is a research plan 00:01:39.100 --> 00:01:41.207 that encompasses data collection, 00:01:41.207 --> 00:01:44.905 identification, and econometric estimation. 00:01:44.905 --> 00:01:48.113 Identification is the applied econometricians term 00:01:48.113 --> 00:01:49.910 for research design -- 00:01:49.910 --> 00:01:51.868 a randomized, clinical trial. 00:01:51.868 --> 00:01:56.592 And RCT is the simplest and most powerful research design. 00:01:56.592 --> 00:01:59.378 In RCTs, causal effects are identified 00:01:59.378 --> 00:02:01.870 by the random assignment of treatment. 00:02:01.870 --> 00:02:03.300 Random assignment ensures 00:02:03.300 --> 00:02:04.700 that treatment and control groups 00:02:04.700 --> 00:02:07.100 are comparable in the absence of treatment. 00:02:07.100 --> 00:02:09.300 So differences between them afterwards 00:02:09.300 --> 00:02:11.500 reflect only the treatment effect. 00:02:11.800 --> 00:02:14.700 Nobel prizes are probably not randomly assigned. 00:02:15.300 --> 00:02:16.900 This challenge notwithstanding, 00:02:17.100 --> 00:02:20.004 a compelling empirical strategy for the Nobel treatment effect 00:02:20.004 --> 00:02:21.276 comes to mind, 00:02:21.276 --> 00:02:23.500 at least as a flight of empirical fancy. 00:02:24.200 --> 00:02:28.027 Imagine a pool of prize-eligible Nobel applicants, 00:02:28.027 --> 00:02:30.825 the group under consideration for the prize. 00:02:30.825 --> 00:02:33.345 Applicants need not apply themselves. 00:02:33.345 --> 00:02:36.500 They are, I presume, nominated by their peer scholars. 00:02:36.800 --> 00:02:40.600 My fanciful Nobel impact study looks only at Nobel applicants 00:02:40.800 --> 00:02:43.048 since these are all elite scholars. 00:02:43.048 --> 00:02:44.700 But that is only the first step. 00:02:45.000 --> 00:02:48.112 Credible applicants, I imagine, are evaluated by judges 00:02:48.112 --> 00:02:53.800 using criteria, like publications, citations, nominating statements. 00:02:53.800 --> 00:02:58.000 I imagine this material is reviewed and assigned a numerical score, 00:02:58.000 --> 00:03:00.674 using some kind of scoring rubric. 00:03:00.674 --> 00:03:04.282 Top scorers up to three per field, in any single year, 00:03:04.282 --> 00:03:05.508 win a prize. 00:03:06.200 --> 00:03:07.652 Having identified applicants 00:03:07.652 --> 00:03:09.700 that collected data on their scores, 00:03:09.700 --> 00:03:12.200 the next step in my Nobel impact study 00:03:12.200 --> 00:03:14.700 is to record the relevant cutoffs. 00:03:15.100 --> 00:03:17.376 The Nobel cutoff is the lowest score 00:03:17.376 --> 00:03:19.089 among those awarded a prize, 00:03:19.500 --> 00:03:23.291 Many Nobel hopefuls just missed the cutoff. 00:03:23.291 --> 00:03:26.717 Looking only at near-misses, along with the winners, 00:03:26.717 --> 00:03:27.903 differences in scores 00:03:27.903 --> 00:03:30.475 between those above and below the cutoff 00:03:30.475 --> 00:03:34.795 begin to look serendipitous, almost randomly assigned. 00:03:34.795 --> 00:03:35.911 After all, 00:03:35.911 --> 00:03:39.400 near-Nobels are among the most eminent of scholars, too. 00:03:40.000 --> 00:03:42.674 With one more high-impact publication, 00:03:42.674 --> 00:03:44.983 a little more support from nominators. 00:03:44.983 --> 00:03:47.291 they would have been awarded Nobel gold -- 00:03:47.291 --> 00:03:49.662 some of them, someday, surely will be. 00:03:50.100 --> 00:03:52.246 The empirical strategy sketched here is called 00:03:52.246 --> 00:03:54.291 a Regression Discontinuity Design, 00:03:54.291 --> 00:03:55.593 RD. 00:03:55.593 --> 00:03:57.696 RD exploits the jumps in human affairs, 00:03:57.696 --> 00:04:00.978 induced by rules, regulations, 00:04:00.978 --> 00:04:04.600 and the need to classify people for various assignment purposes. 00:04:05.000 --> 00:04:07.100 When treatment or intervention is determined 00:04:07.100 --> 00:04:10.052 by whether a tiebreaking variable crosses a threshold, 00:04:10.052 --> 00:04:13.317 those just below the threshold become a natural control group 00:04:13.317 --> 00:04:14.602 for those who clear it. 00:04:15.000 --> 00:04:17.303 RD does not require that the variable, 00:04:17.303 --> 00:04:18.906 whose causes we seek, 00:04:18.906 --> 00:04:22.100 switch fully on or fully off at the cutoff. 00:04:22.100 --> 00:04:23.148 We require only 00:04:23.148 --> 00:04:25.582 that the average value of this variable 00:04:25.582 --> 00:04:27.000 jump at the cutoff 00:04:27.200 --> 00:04:29.520 RD can allow, for example, for the fact 00:04:29.520 --> 00:04:33.103 that this year's near-Nobel might be next year's winner. 00:04:33.103 --> 00:04:35.581 Allowing for this leads to the use of jumps 00:04:35.581 --> 00:04:37.518 and the rate at which treatment is assigned 00:04:37.518 --> 00:04:39.710 to construct instrumental variables, 00:04:39.710 --> 00:04:40.863 IV, 00:04:40.863 --> 00:04:43.551 estimates of the effect of treatment received. 00:04:43.551 --> 00:04:47.086 This sort of RD is said to be fuzzy, 00:04:47.086 --> 00:04:50.155 But as Steve Pischke and I wrote in our first book: 00:04:50.155 --> 00:04:51.600 "fuzzy RD is IV." 00:04:51.600 --> 00:04:52.600 (cheering) 00:04:52.600 --> 00:04:54.849 The first RD study I contributed to 00:04:54.849 --> 00:04:58.016 was written with my frequent collaborator, Victor Lavy. 00:04:58.016 --> 00:04:59.145 This study is motivated 00:04:59.145 --> 00:05:01.509 by the high costs and uncertain returns 00:05:01.509 --> 00:05:04.380 to smaller elementary school classes. 00:05:04.380 --> 00:05:07.344 We exploited a rule used by Israeli elementary schools 00:05:07.344 --> 00:05:08.898 to determine class size. 00:05:09.400 --> 00:05:12.400 This rule is used to estimate class size effects, 00:05:12.400 --> 00:05:14.970 as if in a class size RCT. 00:05:15.500 --> 00:05:19.090 In the 1990s, Israeli classes were large. 00:05:19.090 --> 00:05:22.200 Students enrolled in a grade cohort of 40. 00:05:22.400 --> 00:05:24.700 We're likely to be seated in a class of 40. 00:05:25.200 --> 00:05:27.092 That's the relevant cutoff. 00:05:27.092 --> 00:05:30.254 Add another child to the cohort, making 41, 00:05:30.254 --> 00:05:32.527 and the cohort was likely to be split 00:05:32.527 --> 00:05:34.800 into two much smaller classes. 00:05:35.100 --> 00:05:38.539 This leads to the Maimonide's rule research design, 00:05:38.539 --> 00:05:41.069 so named because the 12th-century Rambam 00:05:41.069 --> 00:05:43.600 proposed a maximum class size of 40. 00:05:44.100 --> 00:05:47.208 This figure plots is rarely fourth grade class sizes 00:05:47.208 --> 00:05:49.500 as a function of fourth grade enrollment, 00:05:49.500 --> 00:05:52.500 overlaid with the theoretical class size rule, 00:05:52.800 --> 00:05:54.300 Maimonides rule. 00:05:54.300 --> 00:05:55.921 The fit isn't perfect. 00:05:55.921 --> 00:05:59.155 That's a feature that makes this application of RD fuzzy. 00:05:59.155 --> 00:06:02.476 But the gist of the thing is a marked class size drop, 00:06:02.476 --> 00:06:05.964 at each integer multiple of 40, the relevant cutoff, 00:06:05.964 --> 00:06:08.103 just as predicted by the rule. 00:06:08.103 --> 00:06:11.000 As it turns out, these drops in class size 00:06:11.000 --> 00:06:12.219 are reflected in jumps 00:06:12.219 --> 00:06:14.600 in 4th and 5th grade math scores. 00:06:15.610 --> 00:06:17.700 ♪ [music] ♪ 00:06:19.800 --> 00:06:23.000 Would a comparison of Nobel laureates to near laureates 00:06:23.000 --> 00:06:25.400 really be a good natural experiment? 00:06:25.700 --> 00:06:27.700 The logic behind this sort of claim 00:06:27.700 --> 00:06:30.050 seems more compelling for comparisons of schools 00:06:30.050 --> 00:06:32.400 with 40 and 41 4th graders 00:06:32.700 --> 00:06:35.500 than for comparisons of laureates and near laureates. 00:06:36.000 --> 00:06:40.000 Yet, both scenarios exploit a feature of the physical world. 00:06:40.300 --> 00:06:42.352 Provided the tiebreaking variable 00:06:42.352 --> 00:06:45.000 known to RD mavens as the running variable, 00:06:45.200 --> 00:06:47.300 has a continuous distribution, 00:06:47.300 --> 00:06:50.698 the probability of crossing the cutoff approaches one-half 00:06:50.698 --> 00:06:52.538 when examined in a narrow window 00:06:52.538 --> 00:06:53.696 around the cutoff. 00:06:54.300 --> 00:06:55.900 In RD empirical work, 00:06:56.000 --> 00:06:59.300 the window around such cutoffs is known as a bandwidth. 00:06:59.600 --> 00:07:03.600 Importantly, this limiting probability is 0.5 for everybody 00:07:03.800 --> 00:07:06.050 regardless of how qualified they look 00:07:06.050 --> 00:07:08.300 going into the Nobel competition. 00:07:08.600 --> 00:07:11.768 This remarkable fact can be seen in data on applicants 00:07:11.768 --> 00:07:15.146 to one of New York's highly coveted screen schools. 00:07:15.146 --> 00:07:16.496 By way of background, 00:07:16.496 --> 00:07:20.068 roughly 40% of New York City's middle and high schools 00:07:20.068 --> 00:07:23.654 select their applicants on the basis of test scores, grades, 00:07:23.654 --> 00:07:25.700 and other exacting criteria. 00:07:26.000 --> 00:07:28.900 In other words, the admissions regime for screen schools 00:07:29.100 --> 00:07:31.006 is a lot like the scheme I've imagined 00:07:31.006 --> 00:07:32.400 for the Nobel Prize. 00:07:32.910 --> 00:07:34.917 Screen schools are but one of a number 00:07:34.917 --> 00:07:37.626 of highly selective systems within a system 00:07:37.626 --> 00:07:39.558 in large US school districts. 00:07:39.558 --> 00:07:43.000 Boston, Chicago, San Francisco, and Washington, D.C 00:07:43.300 --> 00:07:46.084 all feature highly selective institutions, 00:07:46.084 --> 00:07:47.800 often known as exam schools. 00:07:48.000 --> 00:07:49.278 Exam schools operate 00:07:49.278 --> 00:07:51.448 as part of larger public school systems 00:07:51.448 --> 00:07:53.500 that enroll students without screening. 00:07:53.700 --> 00:07:55.750 Motivated by the enduring controversy 00:07:55.750 --> 00:07:58.017 over the equity of screened admissions, 00:07:58.017 --> 00:08:00.049 my blueprint labs collaborators and I 00:08:00.049 --> 00:08:03.500 have examined the causal effects of exam school attendance 00:08:03.500 --> 00:08:06.000 in Boston, Chicago, and New York. 00:08:06.000 --> 00:08:08.889 This figure shows the probability of being offered a seat 00:08:08.889 --> 00:08:12.538 at New York's storied Townsend Harris High School, 00:08:12.538 --> 00:08:14.673 ranked 12th, nationwide. 00:08:14.673 --> 00:08:17.769 Bar height in the figure marks the qualification rate -- 00:08:17.769 --> 00:08:22.200 that is, the likelihood of earning a Townsend Harris admission score, 00:08:22.200 --> 00:08:24.858 above that of the lowest scoring applicant 00:08:24.858 --> 00:08:26.238 offered a seat. 00:08:26.238 --> 00:08:30.075 Importantly, the bars show qualification rates conditional 00:08:30.075 --> 00:08:33.800 on a measure of pre-application baseline achievement. 00:08:34.029 --> 00:08:37.080 In particular, the bars mark qualification rates conditional 00:08:37.080 --> 00:08:38.269 on whether an applicant 00:08:38.269 --> 00:08:43.398 has upper quartile or lower quartile 6th grade math scores. 00:08:43.398 --> 00:08:46.362 Townsend Harris applicants with high baseline scores 00:08:46.362 --> 00:08:47.881 are much more likely to qualify 00:08:47.881 --> 00:08:50.431 than applicants with low baseline scores. 00:08:50.431 --> 00:08:52.000 This isn't surprising. 00:08:52.400 --> 00:08:54.879 But in a shrinking symmetric bandwidth 00:08:54.879 --> 00:08:56.700 around the schools cutoff, 00:08:56.700 --> 00:08:59.800 qualification rates in the two groups converge 00:09:00.100 --> 00:09:03.400 Qualification rate in the last and smallest groups 00:09:03.600 --> 00:09:06.300 are both remarkably close to one-half 00:09:06.800 --> 00:09:08.550 This is what we'd expect to see 00:09:08.550 --> 00:09:10.300 where Townsend Harris to admit students, 00:09:10.300 --> 00:09:11.857 by tossing a coin, 00:09:11.857 --> 00:09:15.030 rather than by selecting only those who scored highly 00:09:15.030 --> 00:09:17.200 on the school's entrance exam. 00:09:17.200 --> 00:09:19.891 Even when admissions operates by screening, 00:09:19.891 --> 00:09:23.200 the data can be arranged so as to mimic an RCT. 00:09:28.187 --> 00:09:31.128 A few of the questions I've studied are more controversial 00:09:31.128 --> 00:09:34.317 than the question of access to public exam schools, 00:09:34.317 --> 00:09:36.100 like the Boston Latin School, 00:09:36.100 --> 00:09:39.500 Chicago's Payton and Northside selective enrollment high schools, 00:09:39.500 --> 00:09:42.217 and New York's legendary Brooklyn Tech, 00:09:42.217 --> 00:09:43.383 Bronx Science, 00:09:43.383 --> 00:09:45.500 and Stuyvesant specialized high schools, 00:09:45.800 --> 00:09:49.383 which have graduated 14 Nobel laureates between them. 00:09:49.383 --> 00:09:51.908 Townsend Harris, the school we started with today, 00:09:51.908 --> 00:09:55.985 graduated three Nobels, including economist Ken Arrow. 00:09:55.985 --> 00:09:57.137 Exam school proponents 00:09:57.137 --> 00:09:59.357 see the opportunities these schools provide 00:09:59.357 --> 00:10:01.800 as democratizing public education. 00:10:02.300 --> 00:10:04.102 "Wealthy families," they argue, 00:10:04.102 --> 00:10:07.100 can access exam school curricula in the private sector. 00:10:07.500 --> 00:10:09.512 Shouldn't ambitious low-income students 00:10:09.512 --> 00:10:12.300 be afforded the same chance at elite education? 00:10:13.000 --> 00:10:14.819 Critics of selective enrollment schools 00:10:14.819 --> 00:10:17.778 argue that rather than expanding equity, 00:10:17.778 --> 00:10:20.023 exam schools are inherently biased 00:10:20.023 --> 00:10:22.268 against the Black and Hispanic students 00:10:22.268 --> 00:10:25.045 that make up the bulk of America's urban districts, 00:10:25.045 --> 00:10:28.253 New York's super selective Stuyvesant, for example, 00:10:28.253 --> 00:10:31.477 enrolled only seven Black students in 2019, 00:10:31.477 --> 00:10:33.800 out of an incoming class of 895. 00:10:34.500 --> 00:10:38.200 But are exam school seats really worth fighting for? 00:10:39.000 --> 00:10:41.532 My collaborators and I have repeatedly used 00:10:41.532 --> 00:10:44.962 RD empirical strategies to study the causal effects 00:10:44.962 --> 00:10:46.545 of attendance at exam schools 00:10:46.545 --> 00:10:48.808 like Townsend Harris and Boston Latin. 00:10:49.200 --> 00:10:51.028 Our first exam school study, 00:10:51.028 --> 00:10:53.643 which looks at schools in Boston and New York 00:10:53.643 --> 00:10:55.998 encapsulates these findings in its title: 00:10:55.998 --> 00:10:57.385 "The Elite Illusion." 00:10:57.800 --> 00:10:59.544 "The Elite Illusion" refers to the fact 00:10:59.544 --> 00:11:03.521 that while exam school students undoubtedly have high test scores 00:11:03.521 --> 00:11:05.207 and other good outcomes, 00:11:05.207 --> 00:11:08.400 this is sot a causal effect of exam School attendance. 00:11:08.900 --> 00:11:11.350 Our estimates consistently suggest 00:11:11.350 --> 00:11:13.800 that the causal effects of exam school attendance 00:11:13.800 --> 00:11:17.000 on their students learning and college-going are 0. 00:11:17.200 --> 00:11:19.500 Maybe even negative. 00:11:19.900 --> 00:11:21.992 The good performance of exam school students 00:11:21.992 --> 00:11:23.975 reflect selection bias -- 00:11:23.975 --> 00:11:26.800 that is, the process by which these students are chosen, 00:11:27.200 --> 00:11:29.100 rather than causal effects. 00:11:29.600 --> 00:11:32.316 Data from Chicago's large exam school sector 00:11:32.316 --> 00:11:33.900 illustrate the elite illusion. 00:11:34.300 --> 00:11:37.124 This figure plots peer mean achievement -- 00:11:37.124 --> 00:11:42.164 that is, the 6th grade test scores of my 9th grade classmates 00:11:42.164 --> 00:11:44.200 against the admissions tiebreaker 00:11:44.500 --> 00:11:46.600 for a subset of applicants 00:11:46.600 --> 00:11:48.700 to any one of Chicago's nine exam schools. 00:11:49.200 --> 00:11:51.700 Applicants to these schools rank up to 6, 00:11:51.900 --> 00:11:54.500 while the exam schools prioritize their applicants 00:11:54.500 --> 00:11:57.350 using a common composite index, 00:11:57.350 --> 00:11:59.100 formed from an admissions test, 00:11:59.200 --> 00:12:02.400 GPAs, and grade 7 standardized scores. 00:12:02.900 --> 00:12:05.947 This composite tiebreaker is the running variable 00:12:05.947 --> 00:12:08.794 for an RD design that reveals what happens 00:12:08.794 --> 00:12:10.400 when any applicant is offered 00:12:10.400 --> 00:12:12.350 an exam school seat. 00:12:12.350 --> 00:12:14.537 In Chicago's exam school match, 00:12:14.537 --> 00:12:15.803 which is actually an application 00:12:15.803 --> 00:12:18.960 of the celebrated Gale and Shapley matching algorithm 00:12:18.960 --> 00:12:22.500 exam school applicants are sure to be offered a seat somewhere 00:12:22.700 --> 00:12:25.698 when they clear the lowest in their set of cutoffs 00:12:25.698 --> 00:12:27.697 among the schools they rank. 00:12:27.697 --> 00:12:31.100 We call this lowest cutoff the "qualifying cutoff." 00:12:31.600 --> 00:12:34.783 The figure shows a sharp jump in peer mean achievement 00:12:34.783 --> 00:12:37.006 for Chicago exam school applicants 00:12:37.006 --> 00:12:39.595 who clear their qualifying cutoff. 00:12:39.595 --> 00:12:41.052 This jump reflects the fact 00:12:41.052 --> 00:12:44.100 that most applicants offered an exam school seat take it, 00:12:44.400 --> 00:12:45.560 and applicants who enroll 00:12:45.560 --> 00:12:48.418 at one of Chicago's selective enrollment high schools 00:12:48.418 --> 00:12:50.805 are sure to be seated in a 9th grade classroom 00:12:50.805 --> 00:12:53.484 filled with academically precocious peers, 00:12:53.484 --> 00:12:56.798 because only the relatively precocious make it in. 00:12:56.798 --> 00:13:00.126 The increase in peer achievement across the qualifying cutoff 00:13:00.126 --> 00:13:02.776 amounts to almost half a standard deviation -- 00:13:02.776 --> 00:13:04.200 a very large effect. 00:13:04.200 --> 00:13:07.100 And yet, precocious peers, notwithstanding, 00:13:07.400 --> 00:13:11.300 the offer of an exam school seat does not appear to increase learning. 00:13:11.700 --> 00:13:15.700 Let's plot applicants ACT scores against their tiebreaker values. 00:13:16.100 --> 00:13:18.748 This plot shows that exam school applicants 00:13:18.748 --> 00:13:20.835 who clear their qualifying cutoff 00:13:20.835 --> 00:13:23.700 perform sharply worse on the ACT. 00:13:24.100 --> 00:13:25.400 What explains this? 00:13:25.700 --> 00:13:29.486 It takes a tale of IV and RD to untangle the forces 00:13:29.486 --> 00:13:33.301 behind this intriguing and unexpected negative effect. 00:13:33.301 --> 00:13:35.400 But first, some IV theory, 00:13:40.016 --> 00:13:42.682 Guido Imbens and I developed theoretical tools 00:13:42.682 --> 00:13:46.337 that enhance economists' understanding of empirical strategies 00:13:46.337 --> 00:13:48.400 involving IV and RD. 00:13:49.100 --> 00:13:51.800 The prize we share is in recognition of this work. 00:13:52.300 --> 00:13:55.146 Guido and I overlapped for only one year at Harvard, 00:13:55.146 --> 00:13:58.286 where we had both taken our first jobs post PhD. 00:13:58.286 --> 00:14:00.806 I welcomed Guido to Cambridge, Massachusetts 00:14:00.806 --> 00:14:03.700 with a pair of interesting instrumental variables. 00:14:04.200 --> 00:14:05.982 I had used the draft lottery instrument 00:14:05.982 --> 00:14:07.500 in my PhD thesis 00:14:07.800 --> 00:14:11.025 to estimate the long-run economic consequences 00:14:11.025 --> 00:14:12.911 of serving in the Armed Forces 00:14:12.911 --> 00:14:14.600 for soldiers who were drafted. 00:14:14.800 --> 00:14:17.125 The draft lottery instrument relies on the fact 00:14:17.125 --> 00:14:19.900 that lottery numbers randomly assigned to birthdays 00:14:20.200 --> 00:14:23.200 determined Vietnam-era conscription risk. 00:14:23.500 --> 00:14:26.534 Yet, even then, most soldiers were volunteers, 00:14:26.534 --> 00:14:27.600 as they are today, 00:14:28.000 --> 00:14:29.243 The quarter birth instrument 00:14:29.243 --> 00:14:32.107 is used in my 1991 paper with Alan Krueger 00:14:32.107 --> 00:14:34.800 to estimate the economic returns to schooling. 00:14:34.800 --> 00:14:36.446 This instrument uses the fact 00:14:36.446 --> 00:14:38.554 that men who are born earlier in the year 00:14:38.554 --> 00:14:40.388 are allowed to drop out of high school 00:14:40.388 --> 00:14:42.045 on their 16th birthday 00:14:42.045 --> 00:14:45.000 with less schooling completed than those born later. 00:14:45.300 --> 00:14:47.800 Guido and I soon began asking each other, 00:14:48.100 --> 00:14:50.800 "What really do we learn from the draft eligibility 00:14:50.800 --> 00:14:53.100 and quarter of birth natural experiments?" 00:14:53.500 --> 00:14:56.800 An early result in our quest for a new understanding of IV 00:14:57.200 --> 00:14:59.650 was a solution to the problem of selection bias 00:14:59.650 --> 00:15:02.100 in an RCT with partial compliance. 00:15:02.700 --> 00:15:04.800 Even in a randomized clinical trial, 00:15:05.100 --> 00:15:07.900 some of the people assigned to treatment may opt out. 00:15:08.100 --> 00:15:10.637 This fact has long vexed trialists 00:15:10.637 --> 00:15:14.546 because decisions to opt out are not made by random assignment. 00:15:15.200 --> 00:15:16.800 Our first manuscript together 00:15:17.000 --> 00:15:20.200 shows that in a randomized trial with partial compliance, 00:15:20.400 --> 00:15:21.958 you can use IV 00:15:21.958 --> 00:15:24.464 to estimate the effect of treatment on the treated, 00:15:24.464 --> 00:15:27.000 even when some offered treatment decline it. 00:15:27.400 --> 00:15:28.400 This works in spite of the fact 00:15:28.400 --> 00:15:30.600 that those who comply with treatment 00:15:30.600 --> 00:15:32.800 may be a very select group. 00:15:33.100 --> 00:15:35.500 Unfortunately, for us, we were late to the party. 00:15:36.000 --> 00:15:38.600 Not long after releasing our first working paper, 00:15:38.800 --> 00:15:41.600 we learned of a concise contribution from Howard Bloom 00:15:41.600 --> 00:15:44.100 that includes this theoretical result. 00:15:44.200 --> 00:15:47.400 Remarkably, Bloom had derived this from first principles 00:15:47.600 --> 00:15:49.700 without making a connection to IV. 00:15:50.200 --> 00:15:52.400 So Guido and I went back to the drawing board. 00:15:52.400 --> 00:15:54.600 And a few months later, we had LATE -- 00:15:54.800 --> 00:15:56.224 a theorem showing how to estimate 00:15:56.224 --> 00:15:58.882 the local average treatment effect. 00:15:58.882 --> 00:16:01.600 The LATE theorem generalizes the Bloom theorem 00:16:01.600 --> 00:16:05.400 and establishes the connection between compliance and IV. 00:16:06.100 --> 00:16:08.300 Maintaining the clinical trials analogy, 00:16:08.300 --> 00:16:11.697 let "Zi" indicate whether subject "i" is offered treatment. 00:16:11.697 --> 00:16:13.586 This is randomly assigned. 00:16:13.586 --> 00:16:16.043 Also, let "D1i" indicate subject "i's" treatment status 00:16:16.043 --> 00:16:18.500 when assigned to treatment, 00:16:18.500 --> 00:16:20.650 and let the "0i" indicate subject "i's" treatment status 00:16:20.650 --> 00:16:22.800 when assigned to control. 00:16:23.300 --> 00:16:24.878 I'll use this formal notation 00:16:24.878 --> 00:16:27.000 to give a clear statement of the late result, 00:16:27.300 --> 00:16:29.100 and then follow up with examples. 00:16:29.600 --> 00:16:31.250 A key piece of the late framework 00:16:31.250 --> 00:16:33.669 pioneered by statistician Don Rubin, 00:16:33.669 --> 00:16:36.443 is the pair of potential outcomes. 00:16:36.443 --> 00:16:37.841 As is customary, 00:16:37.841 --> 00:16:39.870 I denote potential outcomes for subject "i" 00:16:39.870 --> 00:16:41.900 in the treated and untreated states 00:16:42.100 --> 00:16:45.400 by "Y1i" and "Y0i" respectively, 00:16:45.900 --> 00:16:48.550 The observed outcome is "Y1i" for the treated 00:16:48.550 --> 00:16:51.200 and "Y0i" for those not treated. 00:16:51.500 --> 00:16:53.963 "Y1i" minus "Y0i" 00:16:53.963 --> 00:16:56.856 is the causal effect of treatment on individual "i", 00:16:56.856 --> 00:16:58.744 but we can never see. 00:16:58.744 --> 00:17:02.774 We try, therefore, to estimate some kind of average causal effect. 00:17:02.774 --> 00:17:05.645 The late framework allows us to do that in an RCT 00:17:05.645 --> 00:17:07.503 where some controls are treated. 00:17:07.503 --> 00:17:08.629 The theorem says 00:17:08.629 --> 00:17:10.338 that the average causal effect on people, 00:17:10.338 --> 00:17:12.355 whose treatment status can be changed 00:17:12.355 --> 00:17:14.242 by the offer of treatment 00:17:14.242 --> 00:17:17.235 is the ratio of ITT to the treatment control difference 00:17:17.235 --> 00:17:18.400 in compliance rates. 00:17:18.700 --> 00:17:21.400 A mathematical statement of this result appears here, 00:17:21.800 --> 00:17:24.350 where Greek letter Delta symbolizes the ITT effect 00:17:24.350 --> 00:17:28.533 and Greek symbols pi1 and pi0 00:17:28.533 --> 00:17:31.456 are compliance rates in the group assigned to treatment 00:17:31.456 --> 00:17:34.000 and the group assigned to control, respectively, 00:17:34.600 --> 00:17:36.241 The print version of this lecture 00:17:36.241 --> 00:17:38.600 delves deeper into LATE intellectual history, 00:17:38.800 --> 00:17:41.400 highlighting key contributions made with Rubin. 00:17:41.700 --> 00:17:45.100 For now, though, I'd like to make the late theorem concrete for you 00:17:45.300 --> 00:17:48.100 by sharing one of my favorite applications of it. 00:17:52.700 --> 00:17:54.400 I'll explain the late framework 00:17:54.400 --> 00:17:56.948 through a research question that has fascinated me 00:17:56.948 --> 00:17:58.400 for almost two decades. 00:17:58.800 --> 00:18:00.450 What is the causal effect 00:18:00.450 --> 00:18:02.100 of charter school attendance on learning? 00:18:02.500 --> 00:18:04.500 Charter schools are public schools 00:18:04.500 --> 00:18:07.050 that operate independently 00:18:07.050 --> 00:18:09.600 of traditional American public school districts. 00:18:09.900 --> 00:18:12.150 A charter, the right to operate a public school 00:18:12.150 --> 00:18:14.400 is typically awarded for a limited period 00:18:14.600 --> 00:18:17.800 subject to renewal, conditional on good school performance. 00:18:18.600 --> 00:18:20.450 Charter schools are free 00:18:20.450 --> 00:18:22.300 to structure their curriculum and school environment. 00:18:22.300 --> 00:18:24.350 The most controversial difference 00:18:24.350 --> 00:18:26.400 between charters and traditional public schools 00:18:27.100 --> 00:18:28.950 is the fact that the teachers and staff who work at charter schools 00:18:28.950 --> 00:18:31.966 rarely belonged to labor unions. 00:18:31.966 --> 00:18:35.114 By contrast, most big city public school teachers 00:18:35.114 --> 00:18:36.800 work under union contracts. 00:18:37.500 --> 00:18:40.900 The 2010 documentary film "Waiting for Superman" 00:18:41.000 --> 00:18:42.600 feature schools belonging to the Knowledge is Power Program, 00:18:42.600 --> 00:18:44.200 KIPP. 00:18:44.400 --> 00:18:48.600 KIPP schools are emblematic of the high expectations, 00:18:48.800 --> 00:18:53.000 sometimes also called "no excuses" approach to public education. 00:18:53.400 --> 00:18:56.900 The "no excuses" model features 00:18:57.000 --> 00:18:59.100 a long school day and extended school year, 00:18:59.100 --> 00:19:00.150 selective teacher hiring 00:19:00.150 --> 00:19:01.200 and focuses on traditional reading and math skills. 00:19:01.200 --> 00:19:05.400 The American debate over education reform 00:19:05.400 --> 00:19:07.900 often focuses on the achievement gap -- 00:19:07.900 --> 00:19:08.750 that's shorthand for large test score differences 00:19:08.750 --> 00:19:09.600 by race and ethnicity. 00:19:09.600 --> 00:19:14.500 Because of its focus on minority students, 00:19:14.500 --> 00:19:16.850 KIPP is often central in this debate 00:19:16.850 --> 00:19:19.200 with supporters pointing to the fact 00:19:19.200 --> 00:19:23.800 that non-White KIPP students have markedly higher test scores 00:19:23.800 --> 00:19:25.400 than non-White students from nearby schools. 00:19:25.400 --> 00:19:27.000 KIPP skeptics on the other hand, 00:19:27.000 --> 00:19:29.700 argue that KIPP's apparent success, 00:19:29.900 --> 00:19:32.800 reflects the fact that KIPP attracts families 00:19:33.000 --> 00:19:34.750 whose children would be more likely to succeed, anyway. 00:19:34.750 --> 00:19:36.500 Who's right? 00:19:37.300 --> 00:19:38.900 As you've probably guessed by now, 00:19:39.000 --> 00:19:41.050 a randomized trial might prove decisive 00:19:41.050 --> 00:19:43.100 in the debate over schools like KIPP. 00:19:43.800 --> 00:19:45.750 Like Nobel Prize is, though, 00:19:45.750 --> 00:19:47.700 seats at KIPP are not randomly assigned. 00:19:48.100 --> 00:19:50.300 Well, at least, not entirely. 00:19:50.600 --> 00:19:51.500 In fact, 00:19:51.600 --> 00:19:54.700 Massachusetts charter schools with more applicants than seats 00:19:54.900 --> 00:19:56.900 must offer their seats by lottery. 00:19:57.300 --> 00:19:59.800 Sounds like a good natural experiment. 00:20:00.200 --> 00:20:01.950 A little over a decade ago, 00:20:01.950 --> 00:20:03.700 my collaborators and I collected data 00:20:03.700 --> 00:20:06.500 on KIPP admissions lotteries 00:20:06.500 --> 00:20:09.300 laying the foundation for two pioneering charter studies, 00:20:09.300 --> 00:20:11.800 the first to use lotteries to study KIPP. 00:20:12.300 --> 00:20:15.300 Our KIPP analysis is a classic IV story 00:20:15.600 --> 00:20:18.300 because many students offered a seat in the KIPP lottery 00:20:18.600 --> 00:20:20.200 failed to show up in the fall, 00:20:20.500 --> 00:20:23.700 while a few not offered a seat, nevertheless, find their way in. 00:20:24.300 --> 00:20:26.900 This graphic shows KIPP middle school applicants math scores 00:20:27.000 --> 00:20:30.000 one year after applying to KIPP. 00:20:30.200 --> 00:20:31.700 The entries above the line 00:20:31.800 --> 00:20:33.800 show that Kip applicants who were offered a seat 00:20:33.800 --> 00:20:35.800 have standardized math scores close to zero -- 00:20:36.000 --> 00:20:39.100 that is near the state average. 00:20:39.300 --> 00:20:42.100 As before, we're working with standardized score data 00:20:42.300 --> 00:20:45.200 that has a mean of 0 and a standard deviation of 1. 00:20:45.500 --> 00:20:48.400 because KIPP applicants start with 4th grade scores 00:20:48.400 --> 00:20:52.800 that are roughly .3 standard deviations below the state mean, 00:20:53.000 --> 00:20:56.900 achievement at the level of the state average is impressive. 00:20:57.100 --> 00:21:00.500 By contrast, the average math score among those not offered a seat 00:21:00.600 --> 00:21:03.300 is about minus .36 sigma, 00:21:03.400 --> 00:21:06.900 that is, .36 standard deviations below the state mean -- 00:21:07.100 --> 00:21:10.200 a result typical for urban students in Massachusetts. 00:21:10.700 --> 00:21:13.300 Since lottery offers are randomly assigned, 00:21:13.400 --> 00:21:15.750 we could say with confidence that the offer of a seated KIPP 00:21:15.750 --> 00:21:18.100 boost math scores by 00:21:18.100 --> 00:21:20.200 an average of .36 sigma, 00:21:20.500 --> 00:21:23.600 a large effect that's also statistically precise. 00:21:23.900 --> 00:21:26.500 We can be confident this is.n't a chance finding 00:21:27.000 --> 00:21:29.700 What does an offer effect .36 sigma 00:21:29.800 --> 00:21:33.000 tell us about the effects of actually going to KIPP? 00:21:33.600 --> 00:21:37.700 IV methods convert KIPP offer effects into KIPP attendance effects. 00:21:38.300 --> 00:21:42.700 I'll use this brief clip from my Marginal Revolution University short course 00:21:42.900 --> 00:21:46.200 to quickly review the key assumptions behind this conversion. 00:21:46.800 --> 00:21:49.000 - [Narrator] IV describes a chain reaction. 00:21:49.500 --> 00:21:52.300 Why do offers affect achievement? 00:21:52.300 --> 00:21:55.100 Probably because they affect charter attendance 00:21:55.200 --> 00:21:56.900 and charter attendance improves math scores. 00:21:58.500 --> 00:22:00.900 The first link in the chain called the "First Stage" 00:22:00.900 --> 00:22:05.800 is the effect of the lottery on charter attendance. 00:22:06.200 --> 00:22:09.050 The "Second Stage" is the link 00:22:09.050 --> 00:22:11.900 between attending a charter and an outcome variable -- 00:22:12.000 --> 00:22:14.300 in this case, math scores. 00:22:14.300 --> 00:22:16.600 The instrumental variable or "Instrument," for short, 00:22:16.700 --> 00:22:21.800 is the variable that initiates the chain reaction. 00:22:22.900 --> 00:22:25.550 The effect of the instrument on the outcome 00:22:25.550 --> 00:22:28.200 is called the "Reduced Form." 00:22:29.800 --> 00:22:33.200 This chain reaction can be represented mathematically. 00:22:33.700 --> 00:22:38.000 We multiply the first stage -- the effect of winning on attendance, 00:22:38.100 --> 00:22:42.100 by the second stage -- the effect of attendance on scores, 00:22:42.300 --> 00:22:44.750 and we get the reduced form -- 00:22:44.750 --> 00:22:47.200 the effect of winning the lottery on scores. 00:22:48.500 --> 00:22:53.200 The Reduced Form and First Stage are observable and easy to compute. 00:22:53.700 --> 00:22:56.150 However, the effect of attendance on achievement 00:22:56.150 --> 00:22:58.600 is not directly observed. 00:22:59.000 --> 00:23:02.100 This is the causal effect. we're trying to determine 00:23:02.800 --> 00:23:05.600 Given some important assumptions will discuss shortly, 00:23:05.600 --> 00:23:07.650 we can find the effect of KIPP attendance 00:23:07.650 --> 00:23:09.700 by dividing the reduced form by the first stage. 00:23:09.700 --> 00:23:15.100 - [Joshua] IV eliminates selection bias, 00:23:15.300 --> 00:23:16.750 but like all of our tools, 00:23:16.750 --> 00:23:18.200 the solution builds on a set of assumptions 00:23:18.400 --> 00:23:21.100 not to be taken for granted 00:23:21.600 --> 00:23:24.800 first. There must be a substantial first stage. 00:23:25.100 --> 00:23:27.000 That is the instrumental variable, 00:23:27.300 --> 00:23:30.300 winning or losing the lottery must really change. 00:23:30.300 --> 00:23:34.400 The variable, whose effect we're interested in here, KIPP attendance. 00:23:34.900 --> 00:23:36.700 In this case. The first stage is 00:23:36.900 --> 00:23:41.400 not really in doubt, winning the lottery, make skip attendance, much more likely, 00:23:42.100 --> 00:23:44.200 not all IV, stories are like that. 00:23:45.000 --> 00:23:47.900 S the instrument must be as good as randomly. 00:23:48.300 --> 00:23:52.100 Signed meaning lottery winners and losers have similar characteristics. 00:23:52.500 --> 00:23:55.000 This is the independence Assumption. 00:23:55.400 --> 00:23:56.000 Of course, 00:23:56.300 --> 00:23:59.100 KIPP lottery wins really are randomly assigned 00:23:59.300 --> 00:24:03.100 still, we should check for balance and confirm that winners and losers 00:24:03.200 --> 00:24:06.600 have similar family, backgrounds, similar, aptitudes, and so on, 00:24:07.300 --> 00:24:10.300 in essence, we're checking to ensure KIPP lotteries are fair 00:24:10.600 --> 00:24:13.500 with no group of applicants, suspiciously, likely to win. 00:24:14.800 --> 00:24:18.100 Finally, we require the instrument change outcomes soul. 00:24:18.300 --> 00:24:21.300 Through the variable of interest. In this case, attending camp, 00:24:21.900 --> 00:24:24.800 this assumption is called the exclusion restriction. 00:24:27.200 --> 00:24:31.200 The causal effect of KIPP attendance can therefore be written as the ratio of 00:24:31.200 --> 00:24:33.800 the effect of offers on scores in the numerator, 00:24:33.900 --> 00:24:37.400 over the effect of offers on KIPP and Roman in the denominator. 00:24:37.500 --> 00:24:39.700 The numerator in this IV formula, 00:24:39.700 --> 00:24:43.900 that is the direct effect of the instrument on outcomes. Has a special name. 00:24:44.000 --> 00:24:48.200 This is called the reduced form. The denominator is the first step. 00:24:48.300 --> 00:24:48.800 Stage 00:24:49.100 --> 00:24:53.400 the exclusion restriction is often the trickiest or most controversial part of 00:24:53.400 --> 00:24:57.800 an IV story. Here, the exclusion restriction amounts to the claim 00:24:58.000 --> 00:25:02.100 that the .36 score, differential between lottery. Winners and losers 00:25:02.300 --> 00:25:07.500 is entirely attributable to the .74, win-loss difference in attendance rates. 00:25:07.800 --> 00:25:09.000 Plugging in the numbers. 00:25:09.200 --> 00:25:13.100 The effect of KIPP attendance works out to be point four eight Sigma, 00:25:13.300 --> 00:25:18.100 almost half a standard deviation gain in math scores. That's a remarkably large. 00:25:18.300 --> 00:25:23.700 Perfect, who exactly benefits. So spectacularly from KIPP, 00:25:24.000 --> 00:25:27.200 does everyone who applies to KIPP, see such large gains, 00:25:27.600 --> 00:25:29.400 late answers. This question. 00:25:29.900 --> 00:25:33.300 The late interpretation of the KIPP IV, empirical strategy, 00:25:33.400 --> 00:25:36.400 is illuminated by the biblical story of Passover, 00:25:36.600 --> 00:25:39.500 which explains that there are four types of children 00:25:39.800 --> 00:25:41.900 each with characteristic behaviors 00:25:42.300 --> 00:25:44.700 to keep track of these children and their behavior. 00:25:44.900 --> 00:25:48.100 I'll give them a literate of names. Applicants like 00:25:48.200 --> 00:25:53.000 Alvaro are dying to go to KIPP. If Alvaro loses, the KIPP lottery. 00:25:53.100 --> 00:25:56.100 His mother finds a way to enroll him in KIPP. Anyway, 00:25:56.500 --> 00:26:01.800 perhaps by reapplying applicants like Camilla are happy to go to Camp if they win 00:26:01.800 --> 00:26:06.700 a seat in the lottery, but stoically accept the verdict, if they lose finally, 00:26:06.800 --> 00:26:12.100 applicants, like normando worried about long days and lots of homework at KIPP 00:26:12.300 --> 00:26:16.600 normando doesn't really want to go and refuses to go to KIPP when told that he won 00:26:16.600 --> 00:26:17.300 the lottery. 00:26:18.300 --> 00:26:22.100 That was called a never taker because win or lose. He doesn't go to KIPP 00:26:22.400 --> 00:26:26.800 at the other end of KIPP commitment. Alvaro is called an always taker. 00:26:27.100 --> 00:26:31.100 He'll happily take a seat. Went offered while his mother simply finds a way 00:26:31.100 --> 00:26:32.400 to make it happen for him. 00:26:32.700 --> 00:26:37.300 Even when he loses for Alvaro and normando. Both choice of school. 00:26:37.500 --> 00:26:40.900 KIPP traditional is unaffected by the lottery. 00:26:41.200 --> 00:26:44.900 Camilla is the type of applicant who gives IV its power 00:26:45.200 --> 00:26:48.100 the instrument determines her treatment status. 00:26:48.200 --> 00:26:53.700 I IV strategies depend on applicants, like, Camilla who are called compliers. 00:26:54.100 --> 00:26:57.900 This term comes from the world of randomized, Trials, introduced earlier 00:26:58.500 --> 00:27:00.100 as we've already discussed, 00:27:00.200 --> 00:27:04.700 many randomized, trials, randomize only the opportunity to be treated 00:27:04.900 --> 00:27:10.100 while the decision to comply with the treatment remains voluntary and non-random. 00:27:10.700 --> 00:27:15.400 RCT. Compliers are those who take treatment when the offer of treatment is made? 00:27:15.400 --> 00:27:18.100 But not otherwise with Lottery instruments. 00:27:18.400 --> 00:27:23.200 Late is the effect of KIPP attendance on Camilla and other compliers like her. 00:27:23.200 --> 00:27:27.700 Who enroll at KIPP take treatment when offered treatment through the lottery. 00:27:27.900 --> 00:27:29.100 But not otherwise 00:27:29.500 --> 00:27:34.200 IV methods are uninformed of for always takers like Alvaro and never takers 00:27:34.200 --> 00:27:39.000 like normando because the instrument is unrelated to their treatment status. 00:27:39.400 --> 00:27:42.000 Hey, didn't I say there are four types of children. 00:27:42.500 --> 00:27:48.000 A fourth type of child in IV Theory behaves perversely every family has one. 00:27:48.400 --> 00:27:52.400 These defiant children and Roland KIPP only when they lose the lottery. 00:27:52.900 --> 00:27:56.700 Actually, the late theorem requires us to assume there are few defiers, 00:27:57.000 --> 00:28:00.400 that seems like a reasonable assumption for Charter Lottery instruments. 00:28:00.500 --> 00:28:01.700 If not in life. 00:28:02.100 --> 00:28:05.300 The late theorem is sometimes seen as limiting the relevance 00:28:05.300 --> 00:28:06.800 of econometric estimates 00:28:07.000 --> 00:28:09.900 because it focuses attention on groups of compliers 00:28:10.700 --> 00:28:15.000 yet. The population of compliers is a group. We'd very much like to learn about 00:28:15.200 --> 00:28:18.100 in the KIPP example, compliers our children, likely to be, 00:28:18.300 --> 00:28:23.500 Drawn into KIPP where the school to expand and offer additional seats in a lottery. 00:28:24.100 --> 00:28:26.900 How relevant is this a few years ago, 00:28:27.000 --> 00:28:30.800 Massachusetts indeed allowed thriving charter schools to expand 00:28:31.200 --> 00:28:33.400 a recent study by some of my lab mates 00:28:33.600 --> 00:28:36.900 shows that late estimates like the one we just computed for KIPP 00:28:37.000 --> 00:28:41.000 predict learning gains at the schools created by Charter expansion. 00:28:45.800 --> 00:28:47.500 Late isn't just a theorem. 00:28:47.700 --> 00:28:48.000 It's a 00:28:48.200 --> 00:28:48.800 framework. 00:28:49.000 --> 00:28:52.800 The late framework can be used to estimate the entire distribution 00:28:52.800 --> 00:28:54.900 of potential outcomes for compliers 00:28:55.300 --> 00:28:59.000 as if we really did have a randomized trial for this group. 00:28:59.300 --> 00:29:02.800 Although the theory behind this fact is necessarily technical. 00:29:03.100 --> 00:29:08.400 The value of the framework is easily appreciated in practice by way of illustration. 00:29:08.600 --> 00:29:10.200 Recall that the KIPP study 00:29:10.300 --> 00:29:13.700 is motivated in part by differences in test scores by race. 00:29:14.300 --> 00:29:18.000 Let's look at the distribution of 4th grade scores separately by. 00:29:18.100 --> 00:29:21.200 Race for applicants to Boston Charter, Middle Schools, 00:29:21.200 --> 00:29:25.300 the two sides of this figure show distributions for treated 00:29:25.300 --> 00:29:30.000 and untreated compliers treated. Compliers are compliers offered. 00:29:30.000 --> 00:29:34.300 A charter seat in a lottery. While untreated compliers are not offered a seat 00:29:34.300 --> 00:29:38.700 because these are 4th grade scores while middle school begins in 5th 00:29:38.700 --> 00:29:41.700 or 6th grade. The two sides of the figure are similar. 00:29:41.700 --> 00:29:46.600 Both sides show score distributions for black applicants shifted to the left 00:29:46.600 --> 00:29:48.000 of the corresponding. 00:29:48.100 --> 00:29:51.700 Gorgeous tribulations for Whites by 8th grade. 00:29:52.100 --> 00:29:55.700 Treated compliers have completed Middle School at a Boston Charter. 00:29:56.000 --> 00:29:59.500 Well, I'm treated compliers have remained in traditional Public School. 00:30:00.000 --> 00:30:00.700 Remarkably. 00:30:00.700 --> 00:30:04.700 This next graphic shows that the eighth grade score distributions 00:30:04.700 --> 00:30:08.500 of black-and-white treated. Compliers are indistinguishable. 00:30:08.700 --> 00:30:12.400 Boston charter middle schools, close the achievement Gap, 00:30:12.800 --> 00:30:17.100 but for the untreated black and white score distributions remained distinct 00:30:17.100 --> 00:30:18.000 with black students. 00:30:18.200 --> 00:30:21.700 Hind white students as they were in 4th grade, 00:30:22.100 --> 00:30:22.900 Boston Charters, 00:30:22.900 --> 00:30:25.900 close the achievement Gap because those who enter Charter Schools, 00:30:25.900 --> 00:30:30.300 the farthest behind tend to gain the most from Charter. Enrollment 00:30:30.600 --> 00:30:34.000 I elaborate on this point in the print version of this talk. 00:30:39.100 --> 00:30:42.600 Remember the puzzle of - Chicago exam School effects. 00:30:43.000 --> 00:30:48.000 I'll finish the scientific part of my talk by using IV and RD to explain this. 00:30:48.100 --> 00:30:49.800 This surprising finding 00:30:50.300 --> 00:30:54.500 the resolution of this puzzle starts with the fact that economic reasoning 00:30:54.700 --> 00:30:56.300 is about Alternatives. 00:30:56.900 --> 00:31:00.000 So what's the alternative to an exam school education 00:31:00.500 --> 00:31:03.300 for most applicants to Chicago exam schools. 00:31:03.400 --> 00:31:07.400 The leading non exam. Alternative is a traditional public school, 00:31:08.100 --> 00:31:12.900 but many of Chicago's rejected exam school applicants enroll in a charter school 00:31:13.500 --> 00:31:18.000 exam school offers. Therefore reduce the likelihood of Charter School attendance. 00:31:18.500 --> 00:31:23.800 Specifically exam schools divert applicants away from high schools 00:31:23.900 --> 00:31:26.100 in the noble network of charter schools. 00:31:26.600 --> 00:31:27.200 Noble 00:31:27.400 --> 00:31:32.700 with pedagogy much, like KIPP is one of Chicago's most visible Charter providers. 00:31:33.200 --> 00:31:38.000 Also like KIPP convincing evidence on Noble Effectiveness comes from admissions, 00:31:38.000 --> 00:31:38.700 lotteries. 00:31:39.300 --> 00:31:45.300 The x-axis in this graphic shows Lottery offer effects on years enrolled at Noble. 00:31:45.900 --> 00:31:48.000 This is the noble first stage. 00:31:48.100 --> 00:31:50.700 For an IV setup that uses a dummy. 00:31:50.700 --> 00:31:55.400 Indicating Noble Lottery offers as an instrument for Noble enrollment. 00:31:55.400 --> 00:31:57.900 Now, this graphic has a feature that distinguishes. 00:31:57.900 --> 00:32:03.600 It from the simpler KIPP analysis, the plot shows. First stage effects. 00:32:03.600 --> 00:32:04.700 For two groups. 00:32:04.700 --> 00:32:07.100 One for Noble applicants who live 00:32:07.100 --> 00:32:12.300 in Chicago's lowest income neighborhoods Tier 1 and 1/4 Noble, 00:32:12.300 --> 00:32:17.600 applicants who live in higher, income areas, tier 3, remember the IV chain. 00:32:18.900 --> 00:32:23.600 Each point in this graphic has coordinates given by first stage reduced form 00:32:23.800 --> 00:32:26.600 and therefore implies an IV estimate. 00:32:26.900 --> 00:32:32.400 The effect of noble enrollment on ACT scores is the ratio of reduced form coordinate 00:32:32.500 --> 00:32:34.100 to First Stage coordinate. 00:32:34.300 --> 00:32:36.900 The graphic shows to such ratios. 00:32:37.000 --> 00:32:40.600 The relevant results for Tier 1 are .35. 00:32:40.700 --> 00:32:47.300 While for tier 3. We have .33 not bad for Noble applicants from both tears. 00:32:47.500 --> 00:32:47.800 These 00:32:48.100 --> 00:32:50.000 stage in reduced form estimates 00:32:50.100 --> 00:32:54.900 imply a yearly Noble and Roman effective about a third of a standard deviation gain, 00:32:55.100 --> 00:32:56.800 in act math scores. 00:32:57.500 --> 00:33:02.000 Notice. There's also a line connecting the two Ivy estimates in the figure 00:33:02.500 --> 00:33:04.900 because this line passes through the origin, 00:33:05.000 --> 00:33:10.700 it's slope rise. Over run is about equal to the to IV estimates. In this case. 00:33:10.700 --> 00:33:12.700 The slope is about Point 3 for 00:33:13.600 --> 00:33:18.000 the fact that the line passes through 0 0 is significant for 00:33:18.100 --> 00:33:22.700 Reason by this fact, we've substantiated the exclusion restriction, 00:33:22.700 --> 00:33:25.600 specifically, the exclusion restriction, 00:33:25.600 --> 00:33:31.300 says that given a group for which Noble offers are unrelated to Noble enrollment. 00:33:31.300 --> 00:33:37.000 We should expect to see Zero reduced form effect of these offers made to applicants 00:33:37.000 --> 00:33:37.800 in that group. 00:33:37.800 --> 00:33:43.400 How consistent is the evidence for a noble cause learning gain on the order 00:33:43.400 --> 00:33:47.700 of point, three, four Sigma per year in this next graphic? We've added 00:33:48.100 --> 00:33:50.000 L've more points to the original to 00:33:50.500 --> 00:33:54.700 the red points here, show, first stage in reduced form, Noble offer effects 00:33:54.900 --> 00:33:58.200 for 12, additional groups to more tears, 00:33:58.300 --> 00:34:03.600 and 12 groups, defined by demographic, characteristics related to race sex, 00:34:03.700 --> 00:34:06.000 family, income, and Baseline scores. 00:34:06.300 --> 00:34:07.900 Although not a perfect fit 00:34:08.100 --> 00:34:12.100 these points cluster around a line with slope Point Three, Six Sigma 00:34:12.400 --> 00:34:16.400 much like the line. We saw earlier for applicants from tiers 1 and 3. 00:34:16.900 --> 00:34:18.000 You're likely now. 00:34:18.100 --> 00:34:21.700 Ring. What the noble IV estimates in this figure have to do 00:34:21.700 --> 00:34:24.000 with exam School enrollment. 00:34:24.300 --> 00:34:25.300 Here's the answer, 00:34:25.900 --> 00:34:29.500 the Blue Line in this new graphic shows. As we should expect 00:34:29.700 --> 00:34:32.100 that exam. School exposure jumps up 00:34:32.300 --> 00:34:36.400 for applicants who clear their qualifying cutoff. At the same time. 00:34:36.600 --> 00:34:42.100 The Red Line shows that Noble School enrollment clearly Falls at the same point. 00:34:42.500 --> 00:34:47.700 This is the diversion effect of exam school offers on Noble enrollment. 00:34:48.100 --> 00:34:52.700 Many kids offered an exam School seat, prefer that exam School seat, 00:34:52.700 --> 00:34:57.700 to enrollment at Noble IV, affords us, the opportunity to go out on a limb 00:34:57.800 --> 00:35:01.800 with strong claims about the mechanism behind the causal effect. 00:35:02.000 --> 00:35:07.400 Here's a strong causal, claim Regarding why Chicago exam schools reduce achievement. 00:35:07.700 --> 00:35:10.700 The primary force driving reduced form exam School. 00:35:10.700 --> 00:35:13.300 Qualification effects on ACT scores. 00:35:13.600 --> 00:35:17.900 I claim is the effect of exam school offers on Noble and 00:35:18.100 --> 00:35:20.300 Vomit in support of this claim, 00:35:20.500 --> 00:35:26.600 consider the points plotted here in blue all well to the left of 0 on the x-axis. 00:35:27.000 --> 00:35:32.000 These points are negative because they Mark the effect of exam School qualification 00:35:32.100 --> 00:35:36.200 on Noble School, enrollment for particular, groups of applicants. 00:35:36.900 --> 00:35:38.100 Now, we've already seen 00:35:38.300 --> 00:35:40.800 that Noble applicants offered a noble seen 00:35:41.100 --> 00:35:44.400 realize large act math, gains as a result. 00:35:45.000 --> 00:35:47.900 Now consider exam school offers as 00:35:48.000 --> 00:35:49.900 An instrument for Noble enrollment, 00:35:50.600 --> 00:35:53.200 as always, Ivy is a chain reaction. 00:35:53.500 --> 00:35:58.200 If exam School qualification reduces time at Noble, by Point 37 years, 00:35:58.300 --> 00:36:00.300 and each year of noble, enrollment 00:36:00.400 --> 00:36:04.000 boosts act math scores by about Point Three, Six Sigma. 00:36:04.100 --> 00:36:07.900 We should expect reduced form effects of exam School. Qualification 00:36:08.200 --> 00:36:11.700 to reduce ACT scores by the product of these two numbers. 00:36:12.000 --> 00:36:14.600 That is by about Point 1, 3 Sigma 00:36:15.100 --> 00:36:17.900 the reduced form qualification effects at the left of the 00:36:18.100 --> 00:36:20.300 Here are broadly consistent with this. 00:36:20.900 --> 00:36:24.800 They cluster closer to - .16. Then 2 minus Point 1 3, 00:36:25.000 --> 00:36:27.800 but that difference is well within the sampling variance 00:36:27.800 --> 00:36:29.300 of the underlying estimates. 00:36:29.700 --> 00:36:31.400 The causal story told here, 00:36:31.700 --> 00:36:35.100 postulates diversion away from Charter Schools 00:36:35.400 --> 00:36:39.400 as the mechanism by which exam school offers effect achievement. 00:36:39.600 --> 00:36:42.000 In other words, it's Noble enrollment, 00:36:42.000 --> 00:36:44.700 that's presumed to satisfy an exclusion restriction. 00:36:44.900 --> 00:36:47.900 When we use exam school offers as an 00:36:48.000 --> 00:36:49.900 Variable importantly, 00:36:50.000 --> 00:36:55.100 as we saw before the line in this final graphic, with two sets of 14 points, 00:36:55.300 --> 00:36:56.900 runs through the origin. 00:36:57.300 --> 00:37:00.200 This fact supports our new exclusion restriction 00:37:00.800 --> 00:37:02.000 for any applicant group 00:37:02.000 --> 00:37:06.200 for which exam school offers have little or no effect on Noble School enrollment. 00:37:06.400 --> 00:37:09.800 We should also see ACT scores unchanged 00:37:10.000 --> 00:37:14.500 at the same time because the blue and red dots cluster around the same line, 00:37:14.500 --> 00:37:17.900 the IV estimates of noble School, enrollment effects generated. 00:37:18.100 --> 00:37:22.600 I both Noble and exam school offers are about the same. 00:37:23.200 --> 00:37:27.900 I hope this empirical story convinces, you of the power of IV and RD 00:37:28.100 --> 00:37:31.200 to generate new causal, knowledge for decades. 00:37:31.200 --> 00:37:35.300 I've been lucky to work on many equally engaging empirical problems. 00:37:40.000 --> 00:37:42.900 I computed the draft lottery IV estimates in 00:37:42.900 --> 00:37:47.200 my Princeton PhD thesis on a big hairy Mainframe monster. 00:37:47.400 --> 00:37:51.400 Using nine track tapes, and Lease space on a communal. Hard drive, 00:37:51.800 --> 00:37:55.000 Princeton graduate students learn to mount and manipulate tape. 00:37:55.000 --> 00:37:57.300 Reels the size of a cheesecake. 00:37:57.800 --> 00:38:01.700 Thankfully empirical work today is a little less labor-intensive. 00:38:02.300 --> 00:38:05.100 What else is improved in the modern empirical era 00:38:05.700 --> 00:38:09.000 in a 2010 article, Steve Pischke, and I coined the phrase. 00:38:09.800 --> 00:38:10.800 He Revolution 00:38:11.100 --> 00:38:15.800 by this, we mean economic shift towards transparent empirical strategies. 00:38:16.100 --> 00:38:18.500 Applied to concrete, causal questions, 00:38:18.700 --> 00:38:22.200 like the questions. David Carter studied. So convincingly. 00:38:22.900 --> 00:38:27.500 The econometrics of my schooldays focused more on models than on questions. 00:38:28.000 --> 00:38:31.300 The modeling concerns of that era have mostly faded 00:38:31.500 --> 00:38:34.800 but econometricians have since found much to contribute. 00:38:35.100 --> 00:38:39.700 I'll save my personal lists of greatest hits and exciting new artists for the 00:38:39.800 --> 00:38:41.300 The print version of this lecture. 00:38:41.500 --> 00:38:45.800 I'll wrap up here by saying that I'm proud to be part of the Contemporary. 00:38:45.800 --> 00:38:48.100 Empirical economics, Enterprise 00:38:48.500 --> 00:38:50.700 and I'm gratified beyond words 00:38:50.900 --> 00:38:55.900 to have been recognized for contributing to it back at Princeton in the late 80s. 00:38:56.000 --> 00:39:00.000 My graduate school classmates and I chuckled leading Ed lemurs. Lament, 00:39:00.400 --> 00:39:04.400 that no Economist takes another economists. Empirical work. Seriously. 00:39:05.000 --> 00:39:09.600 This is no longer, true, empirical work today aspires to tell convincing. 00:39:09.800 --> 00:39:14.000 Also stories, not that every effort succeeds far from it. 00:39:14.400 --> 00:39:17.700 But as any economics job market candidate will tell you, 00:39:18.100 --> 00:39:23.300 empirical work carefully executed and clearly explained is taken seriously. Indeed. 00:39:23.800 --> 00:39:26.800 That is a measure of our Enterprises success. 00:39:34.200 --> 00:39:35.800 If you'd like to learn more from Josh, 00:39:35.800 --> 00:39:39.500 check out his free course mastering econometrics if you'd like to 00:39:39.700 --> 00:39:41.000 Spore, Josh's research, 00:39:41.000 --> 00:39:43.900 check out the links in the description or you can click to watch more 00:39:43.900 --> 00:39:45.000 of Josh's videos.