WEBVTT 99:59:59.999 --> 99:59:59.999 - [Instructor] The path from cause to effect is dark and dangerous. 99:59:59.999 --> 99:59:59.999 But the weapons of Econometrics are strong. 99:59:59.999 --> 99:59:59.999 [Attack] with fierce and flexible instrumental variables 99:59:59.999 --> 99:59:59.999 when nature blesses you with fortuitous random assignment. 99:59:59.999 --> 99:59:59.999 [gong rings] 99:59:59.999 --> 99:59:59.999 Randomized trials are the surest path to ceteris parabus comparisons. 99:59:59.999 --> 99:59:59.999 Alas, this powerful tool is often unavailable. 99:59:59.999 --> 99:59:59.999 But sometimes, randomization happens by accident. 99:59:59.999 --> 99:59:59.999 That's when we turn to instrumental variables, 99:59:59.999 --> 99:59:59.999 IV for short. 99:59:59.999 --> 99:59:59.999 - [Voice whispers] Instrumental variables. 99:59:59.999 --> 99:59:59.999 - [Instructor] Today's lesson is the first of two on IV. 99:59:59.999 --> 99:59:59.999 Our first IV lesson begins with a story of schools. 99:59:59.999 --> 99:59:59.999 - [Josh] Charter schools are public schools 99:59:59.999 --> 99:59:59.999 freed from daily district oversight and teacher union contracts. 99:59:59.999 --> 99:59:59.999 The question of whether charters boost achievement 99:59:59.999 --> 99:59:59.999 is one of the most important 99:59:59.999 --> 99:59:59.999 in the history of American education reform. 99:59:59.999 --> 99:59:59.999 - The most popular charter schools have more applicants 99:59:59.999 --> 99:59:59.999 so the luck of the lottery draw decides who's offered a seat. 99:59:59.999 --> 99:59:59.999 A lot is at stake for the students vying for their chance. 99:59:59.999 --> 99:59:59.999 Waiting for the lottery results brings up lots of emotions 99:59:59.999 --> 99:59:59.999 as was captured in the award-winning documentary 99:59:59.999 --> 99:59:59.999 "Waiting For Superman." 99:59:59.999 --> 99:59:59.999 - [Mother] Don't cry. You're gonna make Mommy cry. 99:59:59.999 --> 99:59:59.999 Okay? 99:59:59.999 --> 99:59:59.999 - Do charters really provide a better education? 99:59:59.999 --> 99:59:59.999 Critics most definitely say no, 99:59:59.999 --> 99:59:59.999 arguing that charters enroll better students to begin with, 99:59:59.999 --> 99:59:59.999 smarter or more motivated so differences in later outcomes 99:59:59.999 --> 99:59:59.999 reflects selection bias. 99:59:59.999 --> 99:59:59.999 - [Kamal] Wait, this one seems easy. 99:59:59.999 --> 99:59:59.999 In a lottery, winners are chosen randomly, 99:59:59.999 --> 99:59:59.999 so just compare winners and losers. 99:59:59.999 --> 99:59:59.999 - [Student] Obviously. 99:59:59.999 --> 99:59:59.999 - On the right track, Kamal, 99:59:59.999 --> 99:59:59.999 but charter lotteries don't force kids into 99:59:59.999 --> 99:59:59.999 or out of a particular school. 99:59:59.999 --> 99:59:59.999 They randomize offers of a charter seat. 99:59:59.999 --> 99:59:59.999 Some kids get lucky. 99:59:59.999 --> 99:59:59.999 Some kids don't. 99:59:59.999 --> 99:59:59.999 If we just wanted to know the effect of charter school offers, 99:59:59.999 --> 99:59:59.999 we could treat this as a randomized trial. 99:59:59.999 --> 99:59:59.999 But we we're interested in the effects 99:59:59.999 --> 99:59:59.999 of charter school attendance, not offers. 99:59:59.999 --> 99:59:59.999 And not everyone who is offered, accepts. 99:59:59.999 --> 99:59:59.999 IV turns the effect of being offered a charter seat into the effect 99:59:59.999 --> 99:59:59.999 of actually attending a charter school. 99:59:59.999 --> 99:59:59.999 - [Student] Cool. 99:59:59.999 --> 99:59:59.999 - Oh nice. 99:59:59.999 --> 99:59:59.999 - Let's look at an example, a charter school from 99:59:59.999 --> 99:59:59.999 the Knowledge Is Power Program, or KIPP for short. 99:59:59.999 --> 99:59:59.999 This KIPP school is in Lynn, 99:59:59.999 --> 99:59:59.999 a faded industrial town on the coast of Massachusetts. 99:59:59.999 --> 99:59:59.999 The school has more applicants than seats 99:59:59.999 --> 99:59:59.999 and therefore picks its students using a lottery. 99:59:59.999 --> 99:59:59.999 From 2005 to 2008, 371 fourth and fifth graders 99:59:59.999 --> 99:59:59.999 put their names in the KIPP/Lynn lottery, 99:59:59.999 --> 99:59:59.999 253 students won a seat at KIPP, 99:59:59.999 --> 99:59:59.999 118 students lost. 99:59:59.999 --> 99:59:59.999 A year later, lottery winners had much higher match scores 99:59:59.999 --> 99:59:59.999 than lottery losers. 99:59:59.999 --> 99:59:59.999 But remember, we're not trying to figure out 99:59:59.999 --> 99:59:59.999 whether winning a lottery makes you better at math. 99:59:59.999 --> 99:59:59.999 We want to know if attending KIPP makes you better at math. 99:59:59.999 --> 99:59:59.999 Of the 253 lottery winners, only 199 actually went to KIPP. 99:59:59.999 --> 99:59:59.999 The others chose a traditional public school. 99:59:59.999 --> 99:59:59.999 Similarly of the 118 lottery losers, a few actually ended up at KIPP. 99:59:59.999 --> 99:59:59.999 They got an offer later. 99:59:59.999 --> 99:59:59.999 So what was the effect of test scores of actually attending KIPP? 99:59:59.999 --> 99:59:59.999 - [Student] Why can't we just measure their math scores? 99:59:59.999 --> 99:59:59.999 - [Instructor] Great question. 99:59:59.999 --> 99:59:59.999 Who would you compare them to? 99:59:59.999 --> 99:59:59.999 - [Student] Those who didn't attend. 99:59:59.999 --> 99:59:59.999 - [Instructor] Is attendance random? 99:59:59.999 --> 99:59:59.999 - [Camilla] No. 99:59:59.999 --> 99:59:59.999 - Selection bias. 99:59:59.999 --> 99:59:59.999 - [Instructor] Correct. 99:59:59.999 --> 99:59:59.999 - [Otto] What? 99:59:59.999 --> 99:59:59.999 - [Instructor] The KIPP offers are random so we can be confident 99:59:59.999 --> 99:59:59.999 of ceteris parabus, but attendance is not random. 99:59:59.999 --> 99:59:59.999 The choice to accept the offer might be due to characteristics 99:59:59.999 --> 99:59:59.999 that are related to math performance. 99:59:59.999 --> 99:59:59.999 Say, for example, that dedicated parents 99:59:59.999 --> 99:59:59.999 are more likely to accept the offer. 99:59:59.999 --> 99:59:59.999 Their kids are also more likely to do better in math, 99:59:59.999 --> 99:59:59.999 regardless of school. 99:59:59.999 --> 99:59:59.999 - [Student] Right. 99:59:59.999 --> 99:59:59.999 - [Instructor] IV converts the offer effect 99:59:59.999 --> 99:59:59.999 into the effect of KIPP attendance, 99:59:59.999 --> 99:59:59.999 adjusting for the fact that some winners go elsewhere 99:59:59.999 --> 99:59:59.999 and some losers manage to attend KIPP anyway. 99:59:59.999 --> 99:59:59.999 Essentially, IV takes an incomplete randomization 99:59:59.999 --> 99:59:59.999 and makes the appropriate adjustments. 99:59:59.999 --> 99:59:59.999 How? IV describes a chain reaction. 99:59:59.999 --> 99:59:59.999 Why do offers affect achievement? 99:59:59.999 --> 99:59:59.999 Probably because they affect charter attendance 99:59:59.999 --> 99:59:59.999 and charter attendance improves math scores, 99:59:59.999 --> 99:59:59.999 the first link in the chain called the first stage 99:59:59.999 --> 99:59:59.999 is the effect of the lottery on charter attendance. 99:59:59.999 --> 99:59:59.999 The second stage is the length between attending a charter 99:59:59.999 --> 99:59:59.999 and an outcome variable, 99:59:59.999 --> 99:59:59.999 in this case, math scores. 99:59:59.999 --> 99:59:59.999 The instrumental variable or instrument for short 99:59:59.999 --> 99:59:59.999 is the variable that initiates the chain reaction. 99:59:59.999 --> 99:59:59.999 The effect of the instrument on the outcome is called 99:59:59.999 --> 99:59:59.999 the reduced form. 99:59:59.999 --> 99:59:59.999 This chain reaction can be represented mathematically. 99:59:59.999 --> 99:59:59.999 We multiply the first stage, the effect of winning 99:59:59.999 --> 99:59:59.999 on attendance, by the second stage, 99:59:59.999 --> 99:59:59.999 the affect of attendance on scores. 99:59:59.999 --> 99:59:59.999 And we get the reduced form, 99:59:59.999 --> 99:59:59.999 the effect of winning the lottery on scores. 99:59:59.999 --> 99:59:59.999 Reduced form and first stage are observable and easy to compute. 99:59:59.999 --> 99:59:59.999 However, the effect of attendance on achievement 99:59:59.999 --> 99:59:59.999 is not directly observed. 99:59:59.999 --> 99:59:59.999 This is the causal effect we're trying to determine. 99:59:59.999 --> 99:59:59.999 Given some important assumptions we'll discuss shortly, 99:59:59.999 --> 99:59:59.999 we can find the effect of KIPP attendance 99:59:59.999 --> 99:59:59.999 by dividing the reduced form by the first stage. 99:59:59.999 --> 99:59:59.999 This will become more clear as we work through an example. 99:59:59.999 --> 99:59:59.999 - [Student] Let's do this. 99:59:59.999 --> 99:59:59.999 - A quick note on measurement. 99:59:59.999 --> 99:59:59.999 We measure achievement using standard deviations, 99:59:59.999 --> 99:59:59.999 often denoted by the Greek letter sigma (σ). 99:59:59.999 --> 99:59:59.999 One σ is a huge move from around the bottom 15% 99:59:59.999 --> 99:59:59.999 to the middle of most achievement distributions. 99:59:59.999 --> 99:59:59.999 Even a 1/4 or 1/2 σ difference is big. 99:59:59.999 --> 99:59:59.999 - [Instructor] Now we're ready to plug some numbers 99:59:59.999 --> 99:59:59.999 into the equation we introduced earlier. 99:59:59.999 --> 99:59:59.999 First up, what's the effect 99:59:59.999 --> 99:59:59.999 of winning the lottery on math scores? 99:59:59.999 --> 99:59:59.999 KIPP applicant math scores are a third of a standard deviation 99:59:59.999 --> 99:59:59.999 below the state average in the year before they apply to KIPP. 99:59:59.999 --> 99:59:59.999 But a year later, lottery winners score right at the state average 99:59:59.999 --> 99:59:59.999 while the lottery losers are still well behind. 99:59:59.999 --> 99:59:59.999 With an average score around - 0.36 σ. 99:59:59.999 --> 99:59:59.999 The effect of winning the lottery on scores is the difference 99:59:59.999 --> 99:59:59.999 between the winners' scores and the losers' scores. 99:59:59.999 --> 99:59:59.999 Take the winners' average math scores, 99:59:59.999 --> 99:59:59.999 subtract the losers' average math scores, 99:59:59.999 --> 99:59:59.999 and you will have 0.36 σ . 99:59:59.999 --> 99:59:59.999 Next stop: what's the effect of winning the lottery on attendance? 99:59:59.999 --> 99:59:59.999 In other words, if you win the lottery, 99:59:59.999 --> 99:59:59.999 how much more likely are you to attend KIPP than if you lose? 99:59:59.999 --> 99:59:59.999 First, what percentage of lottery winners attend KIPP? 99:59:59.999 --> 99:59:59.999 Divide the number of winners who attended KIPP 99:59:59.999 --> 99:59:59.999 by the total number of lottery winners -- that's 78%. 99:59:59.999 --> 99:59:59.999 To find the percentage of lottery losers who attended KIPP, 99:59:59.999 --> 99:59:59.999 we divide the number of losers who attended KIPP 99:59:59.999 --> 99:59:59.999 by the total number of lottery losers -- that's 4%. 99:59:59.999 --> 99:59:59.999 Subtract 4 from 78, and we find that winning the lottery 99:59:59.999 --> 99:59:59.999 makes you 74% more likely to attend KIPP. 99:59:59.999 --> 99:59:59.999 Now we can find what we're really after, 99:59:59.999 --> 99:59:59.999 the effect of attendance on scores, by dividing 0.36 by 0.74. 99:59:59.999 --> 99:59:59.999 Attending KIPP raises math scores 99:59:59.999 --> 99:59:59.999 by 0.48 standard deviations on average. 99:59:59.999 --> 99:59:59.999 That's an awesome achievement game, 99:59:59.999 --> 99:59:59.999 equal to moving from about the bottom 99:59:59.999 --> 99:59:59.999 to the middle of the achievement distribution. 99:59:59.999 --> 99:59:59.999 - [Student] Whoa, half a sig. 99:59:59.999 --> 99:59:59.999 - [Instructor] These estimates are for kids opting in 99:59:59.999 --> 99:59:59.999 to the KIPP lottery, whose enrollment status 99:59:59.999 --> 99:59:59.999 is changed by winning. 99:59:59.999 --> 99:59:59.999 That's not necessarily a random sample 99:59:59.999 --> 99:59:59.999 of all children in Lynn. 99:59:59.999 --> 99:59:59.999 So we can't assume we'd see the same effect 99:59:59.999 --> 99:59:59.999 for other types of students. 99:59:59.999 --> 99:59:59.999 - [Students] Huh. 99:59:59.999 --> 99:59:59.999 - But this effect on keen for KIPP kids 99:59:59.999 --> 99:59:59.999 is likely to be a good indicator of the consequences 99:59:59.999 --> 99:59:59.999 of adding additional charter seats. 99:59:59.999 --> 99:59:59.999 - [Student] Cool. - [Student] Got it. 99:59:59.999 --> 99:59:59.999 - IV eliminates selection bias but like all of our tools, 99:59:59.999 --> 99:59:59.999 the solution builds on a set of assumptions 99:59:59.999 --> 99:59:59.999 not to be taken for granted. 99:59:59.999 --> 99:59:59.999 First, there must be a substantial first stage -- 99:59:59.999 --> 99:59:59.999 that is the instrumental variable, winning or losing the lottery, 99:59:59.999 --> 99:59:59.999 must really change the variable whose effect we're interested in -- 99:59:59.999 --> 99:59:59.999 here, KIPP attendance. 99:59:59.999 --> 99:59:59.999 In this case, the first stage is not really in doubt. 99:59:59.999 --> 99:59:59.999 Winning the lottery makes KIPP attendance much more likely. 99:59:59.999 --> 99:59:59.999 Not all IV stories are like that.