1 99:59:59,999 --> 99:59:59,999 - [Instructor] The path from cause to effect is dark and dangerous. 2 99:59:59,999 --> 99:59:59,999 But the weapons of Econometrics are strong. 3 99:59:59,999 --> 99:59:59,999 [Attack] with fierce and flexible instrumental variables 4 99:59:59,999 --> 99:59:59,999 when nature blesses you with fortuitous random assignment. 5 99:59:59,999 --> 99:59:59,999 [gong rings] 6 99:59:59,999 --> 99:59:59,999 Randomized trials are the surest path to ceteris parabus comparisons. 7 99:59:59,999 --> 99:59:59,999 Alas, this powerful tool is often unavailable. 8 99:59:59,999 --> 99:59:59,999 But sometimes, randomization happens by accident. 9 99:59:59,999 --> 99:59:59,999 That's when we turn to instrumental variables, 10 99:59:59,999 --> 99:59:59,999 IV for short. 11 99:59:59,999 --> 99:59:59,999 - [Voice whispers] Instrumental variables. 12 99:59:59,999 --> 99:59:59,999 - [Instructor] Today's lesson is the first of two on IV. 13 99:59:59,999 --> 99:59:59,999 Our first IV lesson begins with a story of schools. 14 99:59:59,999 --> 99:59:59,999 - [Josh] Charter schools are public schools 15 99:59:59,999 --> 99:59:59,999 freed from daily district oversight and teacher union contracts. 16 99:59:59,999 --> 99:59:59,999 The question of whether charters boost achievement 17 99:59:59,999 --> 99:59:59,999 is one of the most important 18 99:59:59,999 --> 99:59:59,999 in the history of American education reform. 19 99:59:59,999 --> 99:59:59,999 - The most popular charter schools have more applicants 20 99:59:59,999 --> 99:59:59,999 so the luck of the lottery draw decides who's offered a seat. 21 99:59:59,999 --> 99:59:59,999 A lot is at stake for the students vying for their chance. 22 99:59:59,999 --> 99:59:59,999 Waiting for the lottery results brings up lots of emotions 23 99:59:59,999 --> 99:59:59,999 as was captured in the award-winning documentary 24 99:59:59,999 --> 99:59:59,999 "Waiting For Superman." 25 99:59:59,999 --> 99:59:59,999 - [Mother] Don't cry. You're gonna make Mommy cry. 26 99:59:59,999 --> 99:59:59,999 Okay? 27 99:59:59,999 --> 99:59:59,999 - Do charters really provide a better education? 28 99:59:59,999 --> 99:59:59,999 Critics most definitely say no, 29 99:59:59,999 --> 99:59:59,999 arguing that charters enroll better students to begin with, 30 99:59:59,999 --> 99:59:59,999 smarter or more motivated so differences in later outcomes 31 99:59:59,999 --> 99:59:59,999 reflects selection bias. 32 99:59:59,999 --> 99:59:59,999 - [Kamal] Wait, this one seems easy. 33 99:59:59,999 --> 99:59:59,999 In a lottery, winners are chosen randomly, 34 99:59:59,999 --> 99:59:59,999 so just compare winners and losers. 35 99:59:59,999 --> 99:59:59,999 - [Student] Obviously. 36 99:59:59,999 --> 99:59:59,999 - On the right track, Kamal, 37 99:59:59,999 --> 99:59:59,999 but charter lotteries don't force kids into 38 99:59:59,999 --> 99:59:59,999 or out of a particular school. 39 99:59:59,999 --> 99:59:59,999 They randomize offers of a charter seat. 40 99:59:59,999 --> 99:59:59,999 Some kids get lucky. 41 99:59:59,999 --> 99:59:59,999 Some kids don't. 42 99:59:59,999 --> 99:59:59,999 If we just wanted to know the effect of charter school offers, 43 99:59:59,999 --> 99:59:59,999 we could treat this as a randomized trial. 44 99:59:59,999 --> 99:59:59,999 But we we're interested in the effects 45 99:59:59,999 --> 99:59:59,999 of charter school attendance, not offers. 46 99:59:59,999 --> 99:59:59,999 And not everyone who is offered, accepts. 47 99:59:59,999 --> 99:59:59,999 IV turns the effect of being offered a charter seat into the effect 48 99:59:59,999 --> 99:59:59,999 of actually attending a charter school. 49 99:59:59,999 --> 99:59:59,999 - [Student] Cool. 50 99:59:59,999 --> 99:59:59,999 - Oh nice. 51 99:59:59,999 --> 99:59:59,999 - Let's look at an example, a charter school from 52 99:59:59,999 --> 99:59:59,999 the Knowledge Is Power Program, or KIPP for short. 53 99:59:59,999 --> 99:59:59,999 This KIPP school is in Lynn, 54 99:59:59,999 --> 99:59:59,999 a faded industrial town on the coast of Massachusetts. 55 99:59:59,999 --> 99:59:59,999 The school has more applicants than seats 56 99:59:59,999 --> 99:59:59,999 and therefore picks its students using a lottery. 57 99:59:59,999 --> 99:59:59,999 From 2005 to 2008, 371 fourth and fifth graders 58 99:59:59,999 --> 99:59:59,999 put their names in the KIPP/Lynn lottery, 59 99:59:59,999 --> 99:59:59,999 253 students won a seat at KIPP, 60 99:59:59,999 --> 99:59:59,999 118 students lost. 61 99:59:59,999 --> 99:59:59,999 A year later, lottery winners had much higher match scores 62 99:59:59,999 --> 99:59:59,999 than lottery losers. 63 99:59:59,999 --> 99:59:59,999 But remember, we're not trying to figure out 64 99:59:59,999 --> 99:59:59,999 whether winning a lottery makes you better at math. 65 99:59:59,999 --> 99:59:59,999 We want to know if attending KIPP makes you better at math. 66 99:59:59,999 --> 99:59:59,999 Of the 253 lottery winners, only 199 actually went to KIPP. 67 99:59:59,999 --> 99:59:59,999 The others chose a traditional public school. 68 99:59:59,999 --> 99:59:59,999 Similarly of the 118 lottery losers, a few actually ended up at KIPP. 69 99:59:59,999 --> 99:59:59,999 They got an offer later. 70 99:59:59,999 --> 99:59:59,999 So what was the effect of test scores of actually attending KIPP? 71 99:59:59,999 --> 99:59:59,999 - [Student] Why can't we just measure their math scores? 72 99:59:59,999 --> 99:59:59,999 - [Instructor] Great question. 73 99:59:59,999 --> 99:59:59,999 Who would you compare them to? 74 99:59:59,999 --> 99:59:59,999 - [Student] Those who didn't attend. 75 99:59:59,999 --> 99:59:59,999 - [Instructor] Is attendance random? 76 99:59:59,999 --> 99:59:59,999 - [Camilla] No. 77 99:59:59,999 --> 99:59:59,999 - Selection bias. 78 99:59:59,999 --> 99:59:59,999 - [Instructor] Correct. 79 99:59:59,999 --> 99:59:59,999 - [Otto] What? 80 99:59:59,999 --> 99:59:59,999 - [Instructor] The KIPP offers are random so we can be confident 81 99:59:59,999 --> 99:59:59,999 of ceteris parabus, but attendance is not random. 82 99:59:59,999 --> 99:59:59,999 The choice to accept the offer might be due to characteristics 83 99:59:59,999 --> 99:59:59,999 that are related to math performance. 84 99:59:59,999 --> 99:59:59,999 Say, for example, that dedicated parents 85 99:59:59,999 --> 99:59:59,999 are more likely to accept the offer. 86 99:59:59,999 --> 99:59:59,999 Their kids are also more likely to do better in math, 87 99:59:59,999 --> 99:59:59,999 regardless of school. 88 99:59:59,999 --> 99:59:59,999 - [Student] Right. 89 99:59:59,999 --> 99:59:59,999 - [Instructor] IV converts the offer effect 90 99:59:59,999 --> 99:59:59,999 into the effect of KIPP attendance, 91 99:59:59,999 --> 99:59:59,999 adjusting for the fact that some winners go elsewhere 92 99:59:59,999 --> 99:59:59,999 and some losers manage to attend KIPP anyway. 93 99:59:59,999 --> 99:59:59,999 Essentially, IV takes an incomplete randomization 94 99:59:59,999 --> 99:59:59,999 and makes the appropriate adjustments. 95 99:59:59,999 --> 99:59:59,999 How? IV describes a chain reaction. 96 99:59:59,999 --> 99:59:59,999 Why do offers affect achievement? 97 99:59:59,999 --> 99:59:59,999 Probably because they affect charter attendance 98 99:59:59,999 --> 99:59:59,999 and charter attendance improves math scores, 99 99:59:59,999 --> 99:59:59,999 the first link in the chain called the first stage 100 99:59:59,999 --> 99:59:59,999 is the effect of the lottery on charter attendance. 101 99:59:59,999 --> 99:59:59,999 The second stage is the length between attending a charter 102 99:59:59,999 --> 99:59:59,999 and an outcome variable, 103 99:59:59,999 --> 99:59:59,999 in this case, math scores. 104 99:59:59,999 --> 99:59:59,999 The instrumental variable or instrument for short 105 99:59:59,999 --> 99:59:59,999 is the variable that initiates the chain reaction. 106 99:59:59,999 --> 99:59:59,999 The effect of the instrument on the outcome is called 107 99:59:59,999 --> 99:59:59,999 the reduced form. 108 99:59:59,999 --> 99:59:59,999 This chain reaction can be represented mathematically. 109 99:59:59,999 --> 99:59:59,999 We multiply the first stage, the effect of winning 110 99:59:59,999 --> 99:59:59,999 on attendance, by the second stage, 111 99:59:59,999 --> 99:59:59,999 the affect of attendance on scores. 112 99:59:59,999 --> 99:59:59,999 And we get the reduced form, 113 99:59:59,999 --> 99:59:59,999 the effect of winning the lottery on scores. 114 99:59:59,999 --> 99:59:59,999 Reduced form and first stage are observable and easy to compute. 115 99:59:59,999 --> 99:59:59,999 However, the effect of attendance on achievement 116 99:59:59,999 --> 99:59:59,999 is not directly observed. 117 99:59:59,999 --> 99:59:59,999 This is the causal effect we're trying to determine. 118 99:59:59,999 --> 99:59:59,999 Given some important assumptions we'll discuss shortly, 119 99:59:59,999 --> 99:59:59,999 we can find the effect of KIPP attendance 120 99:59:59,999 --> 99:59:59,999 by dividing the reduced form by the first stage. 121 99:59:59,999 --> 99:59:59,999 This will become more clear as we work through an example. 122 99:59:59,999 --> 99:59:59,999 - [Student] Let's do this. 123 99:59:59,999 --> 99:59:59,999 - A quick note on measurement. 124 99:59:59,999 --> 99:59:59,999 We measure achievement using standard deviations, 125 99:59:59,999 --> 99:59:59,999 often denoted by the Greek letter sigma (σ). 126 99:59:59,999 --> 99:59:59,999 One σ is a huge move from around the bottom 15% 127 99:59:59,999 --> 99:59:59,999 to the middle of most achievement distributions. 128 99:59:59,999 --> 99:59:59,999 Even a 1/4 or 1/2 σ difference is big. 129 99:59:59,999 --> 99:59:59,999 - [Instructor] Now we're ready to plug some numbers 130 99:59:59,999 --> 99:59:59,999 into the equation we introduced earlier. 131 99:59:59,999 --> 99:59:59,999 First up, what's the effect 132 99:59:59,999 --> 99:59:59,999 of winning the lottery on math scores? 133 99:59:59,999 --> 99:59:59,999 KIPP applicant math scores are a third of a standard deviation 134 99:59:59,999 --> 99:59:59,999 below the state average in the year before they apply to KIPP. 135 99:59:59,999 --> 99:59:59,999 But a year later, lottery winners score right at the state average 136 99:59:59,999 --> 99:59:59,999 while the lottery losers are still well behind. 137 99:59:59,999 --> 99:59:59,999 With an average score around - 0.36 σ. 138 99:59:59,999 --> 99:59:59,999 The effect of winning the lottery on scores is the difference 139 99:59:59,999 --> 99:59:59,999 between the winners' scores and the losers' scores. 140 99:59:59,999 --> 99:59:59,999 Take the winners' average math scores, 141 99:59:59,999 --> 99:59:59,999 subtract the losers' average math scores, 142 99:59:59,999 --> 99:59:59,999 and you will have 0.36 σ . 143 99:59:59,999 --> 99:59:59,999 Next stop: what's the effect of winning the lottery on attendance? 144 99:59:59,999 --> 99:59:59,999 In other words, if you win the lottery, 145 99:59:59,999 --> 99:59:59,999 how much more likely are you to attend KIPP than if you lose? 146 99:59:59,999 --> 99:59:59,999 First, what percentage of lottery winners attend KIPP? 147 99:59:59,999 --> 99:59:59,999 Divide the number of winners who attended KIPP 148 99:59:59,999 --> 99:59:59,999 by the total number of lottery winners -- that's 78%. 149 99:59:59,999 --> 99:59:59,999 To find the percentage of lottery losers who attended KIPP, 150 99:59:59,999 --> 99:59:59,999 we divide the number of losers who attended KIPP 151 99:59:59,999 --> 99:59:59,999 by the total number of lottery losers -- that's 4%. 152 99:59:59,999 --> 99:59:59,999 Subtract 4 from 78, and we find that winning the lottery 153 99:59:59,999 --> 99:59:59,999 makes you 74% more likely to attend KIPP. 154 99:59:59,999 --> 99:59:59,999 Now we can find what we're really after, 155 99:59:59,999 --> 99:59:59,999 the effect of attendance on scores, by dividing 0.36 by 0.74. 156 99:59:59,999 --> 99:59:59,999 Attending KIPP raises math scores 157 99:59:59,999 --> 99:59:59,999 by 0.48 standard deviations on average. 158 99:59:59,999 --> 99:59:59,999 That's an awesome achievement game, 159 99:59:59,999 --> 99:59:59,999 equal to moving from about the bottom 160 99:59:59,999 --> 99:59:59,999 to the middle of the achievement distribution. 161 99:59:59,999 --> 99:59:59,999 - [Student] Whoa, half a sig. 162 99:59:59,999 --> 99:59:59,999 - [Instructor] These estimates are for kids opting in 163 99:59:59,999 --> 99:59:59,999 to the KIPP lottery, whose enrollment status 164 99:59:59,999 --> 99:59:59,999 is changed by winning. 165 99:59:59,999 --> 99:59:59,999 That's not necessarily a random sample 166 99:59:59,999 --> 99:59:59,999 of all children in Lynn. 167 99:59:59,999 --> 99:59:59,999 So we can't assume we'd see the same effect 168 99:59:59,999 --> 99:59:59,999 for other types of students. 169 99:59:59,999 --> 99:59:59,999 - [Students] Huh. 170 99:59:59,999 --> 99:59:59,999 - But this effect on keen for KIPP kids 171 99:59:59,999 --> 99:59:59,999 is likely to be a good indicator of the consequences 172 99:59:59,999 --> 99:59:59,999 of adding additional charter seats. 173 99:59:59,999 --> 99:59:59,999 - [Student] Cool. - [Student] Got it. 174 99:59:59,999 --> 99:59:59,999 - IV eliminates selection bias but like all of our tools, 175 99:59:59,999 --> 99:59:59,999 the solution builds on a set of assumptions 176 99:59:59,999 --> 99:59:59,999 not to be taken for granted. 177 99:59:59,999 --> 99:59:59,999 First, there must be a substantial first stage -- 178 99:59:59,999 --> 99:59:59,999 that is the instrumental variable, winning or losing the lottery, 179 99:59:59,999 --> 99:59:59,999 must really change the variable whose effect we're interested in -- 180 99:59:59,999 --> 99:59:59,999 here, KIPP attendance. 181 99:59:59,999 --> 99:59:59,999 In this case, the first stage is not really in doubt. 182 99:59:59,999 --> 99:59:59,999 Winning the lottery makes KIPP attendance much more likely. 183 99:59:59,999 --> 99:59:59,999 Not all IV stories are like that. 184 99:59:59,999 --> 99:59:59,999 Second, the instrument must be as good 185 99:59:59,999 --> 99:59:59,999 as randomly assigned meaning lottery winners and losers 186 99:59:59,999 --> 99:59:59,999 have similar characteristics. 187 99:59:59,999 --> 99:59:59,999 This is the independence assumption. 188 99:59:59,999 --> 99:59:59,999 Of course, KIPP lottery wins really are randomly assigned. 189 99:59:59,999 --> 99:59:59,999 Still, we should check for balance and confirm that winners and losers 190 99:59:59,999 --> 99:59:59,999 have similar family backgrounds, 191 99:59:59,999 --> 99:59:59,999 similar aptitudes and so on. 192 99:59:59,999 --> 99:59:59,999 In essence, we're checking to ensure KIPP lotteries are fair 193 99:59:59,999 --> 99:59:59,999 with no group of applicants suspiciously likely to win. 194 99:59:59,999 --> 99:59:59,999 Finally, we require the instrument change outcomes 195 99:59:59,999 --> 99:59:59,999 solely through the variable of interest, 196 99:59:59,999 --> 99:59:59,999 in this case, attending KIPP. 197 99:59:59,999 --> 99:59:59,999 This assumption is called the exclusion restriction. 198 99:59:59,999 --> 99:59:59,999 - IV only works if you can satisfy these three assumptions. 199 99:59:59,999 --> 99:59:59,999 - I don't understand the exclusion restriction. 200 99:59:59,999 --> 99:59:59,999 How could winning the lottery affect math scores 201 99:59:59,999 --> 99:59:59,999 other than by attending KIPP? 202 99:59:59,999 --> 99:59:59,999 - [Student] Yeah. 203 99:59:59,999 --> 99:59:59,999 - [Instructor] Great question. 204 99:59:59,999 --> 99:59:59,999 Suppose lottery winners are just thrilled to win, 205 99:59:59,999 --> 99:59:59,999 and this happiness motivates them to study more and learn more math 206 99:59:59,999 --> 99:59:59,999 regardless of where they go to school. 207 99:59:59,999 --> 99:59:59,999 This would violate the exclusion restriction 208 99:59:59,999 --> 99:59:59,999 because the motivational effect of winning is a second channel 209 99:59:59,999 --> 99:59:59,999 whereby lotteries might affect test scores. 210 99:59:59,999 --> 99:59:59,999 While it's hard to rule this out entirely, 211 99:59:59,999 --> 99:59:59,999 there's no evidence of any alternative channels 212 99:59:59,999 --> 99:59:59,999 in the KIPP study. 213 99:59:59,999 --> 99:59:59,999 - IV solves the problem of selection bias 214 99:59:59,999 --> 99:59:59,999 in scenarios like the KIPP lottery where treatment offers are random 215 99:59:59,999 --> 99:59:59,999 but some of those offered opt out. 216 99:59:59,999 --> 99:59:59,999 This sort of intentional yet incomplete random assignment 217 99:59:59,999 --> 99:59:59,999 is surprisingly common. 218 99:59:59,999 --> 99:59:59,999 Even randomized clinical trials have this feature. 219 99:59:59,999 --> 99:59:59,999 IV solves the problem of non-random take up 220 99:59:59,999 --> 99:59:59,999 in lotteries or clinical research. 221 99:59:59,999 --> 99:59:59,999 But lotteries are not the only source of compelling instruments. 222 99:59:59,999 --> 99:59:59,999 Many causal questions can be addressed by naturally occurring 223 99:59:59,999 --> 99:59:59,999 as good as randomly assigned variation. 224 99:59:59,999 --> 99:59:59,999 Here's a causal question for you -- 225 99:59:59,999 --> 99:59:59,999 do women who have children early in their careers suffer 226 99:59:59,999 --> 99:59:59,999 a substantial earnings penalty as a result? 227 99:59:59,999 --> 99:59:59,999 After all, women earn less than men. 228 99:59:59,999 --> 99:59:59,999 We could, of course, simply compare the earnings of women 229 99:59:59,999 --> 99:59:59,999 with more and fewer children. 230 99:59:59,999 --> 99:59:59,999 But such comparisons are fraught with selection bias. 231 99:59:59,999 --> 99:59:59,999 If only we could randomly assign babies to different households. 232 99:59:59,999 --> 99:59:59,999 Yeah, right, sounds pretty fanciful. 233 99:59:59,999 --> 99:59:59,999 Our next IV story -- fantastic and not fanciful -- 234 99:59:59,999 --> 99:59:59,999 illustrates an amazing, naturally-occurring instrument 235 99:59:59,999 --> 99:59:59,999 for family size. 236 99:59:59,999 --> 99:59:59,999 - [Instructor] You're on your way to mastering Econometrics. 237 99:59:59,999 --> 99:59:59,999 Make sure this video sticks 238 99:59:59,999 --> 99:59:59,999 by taking a few quick practice questions. 239 99:59:59,999 --> 99:59:59,999 Or, if you're ready, click for the next video. 240 99:59:59,999 --> 99:59:59,999 You can also check out MRU's website for more courses, 241 99:59:59,999 --> 99:59:59,999 teacher resources, and more. 242 99:59:59,999 --> 99:59:59,999 ♪ [music] ♪