0:00:01.425,0:00:05.060 - [Narrator] On his quest [br]to master econometrics, 0:00:05.223,0:00:08.813 Grasshopper Kamal has [br]made great progress, 0:00:08.813,0:00:13.662 stretching his capabilities [br]and outsmarting his foes. 0:00:14.223,0:00:16.640 Alas, today he's despondent, 0:00:16.640,0:00:19.336 for one challenge remains unmet. 0:00:19.336,0:00:24.130 Kamal cannot yet decode [br]the scriptures of academic research, 0:00:24.130,0:00:27.347 journals like [br]"The American Economic Review" 0:00:27.347,0:00:29.080 and "Econometrica." 0:00:29.202,0:00:33.501 These seemed to him to be inscribed [br]in an obscure foreign tongue. 0:00:33.501,0:00:35.478 - [Kamal] Ugh, what the... ? 0:00:36.711,0:00:40.018 - These volumes are [br]opaque to the novice, Kamal, 0:00:40.018,0:00:42.205 but can be deciphered with study. 0:00:42.467,0:00:45.109 Let us learn to read them together. 0:00:52.485,0:00:55.317 Let's dive into the West Point study, 0:00:55.317,0:00:58.278 published in the "Economics [br]of Education Review." 9:59:59.000,9:59:59.000 This paper reports [br]on a randomized evaluation 9:59:59.000,9:59:59.000 of student electronics use [br]in Economics 101 classrooms. 9:59:59.000,9:59:59.000 First, a quick review [br]of the research design. 9:59:59.000,9:59:59.000 - Okay. 9:59:59.000,9:59:59.000 - [Josh] 'Metrics masters [br]teaching at West Point, 9:59:59.000,9:59:59.000 the military college that trains [br]American Army officers 9:59:59.000,9:59:59.000 designed a randomized trial[br]to answer this question. 9:59:59.000,9:59:59.000 These masters randomly assigned[br]West Point cadets 9:59:59.000,9:59:59.000 into Economics classes[br]operating under different rules. 9:59:59.000,9:59:59.000 Unlike most American colleges, 9:59:59.000,9:59:59.000 the West Point default [br]is no electronics. 9:59:59.000,9:59:59.000 For purposes of this experiment,[br]some students were left 9:59:59.000,9:59:59.000 in such traditional [br]technology-free classes, 9:59:59.000,9:59:59.000 no lap tops, no tablets [br]and no phones! 9:59:59.000,9:59:59.000 [voice echoes] 9:59:59.000,9:59:59.000 This is the control group,[br]or baseline case. 9:59:59.000,9:59:59.000 Another group was allowed[br]to use electronics. 9:59:59.000,9:59:59.000 This is the treatment group,[br]subject to a changed environment. 9:59:59.000,9:59:59.000 The treatment in this case[br]is the unrestricted use 9:59:59.000,9:59:59.000 of laptops or tablets in class. 9:59:59.000,9:59:59.000 Every causal question [br]has a clear outcome, 9:59:59.000,9:59:59.000 the variables we hope to influence[br]defined in advance of the study. 9:59:59.000,9:59:59.000 The outcomes in the West Point[br]electronics study 9:59:59.000,9:59:59.000 are final exam scores. 9:59:59.000,9:59:59.000 The study seeks to answer[br]the following question, 9:59:59.000,9:59:59.000 what is the causal effect [br]of classroom electronics on learning 9:59:59.000,9:59:59.000 as measured by exam scores? 9:59:59.000,9:59:59.000 - Economics journal articles [br]usually begin with a table 9:59:59.000,9:59:59.000 of descriptive statistics,[br]giving key facts 9:59:59.000,9:59:59.000 about the study sample. 9:59:59.000,9:59:59.000 - Oh my gosh, I remember this table,[br]so confusing! 9:59:59.000,9:59:59.000 Columns 1 to 3 report mean,[br]or average, characteristics. 9:59:59.000,9:59:59.000 These give a sense [br]of who we're studying. 9:59:59.000,9:59:59.000 Let's start with column 1 [br]which describes covariates 9:59:59.000,9:59:59.000 in a control group. 9:59:59.000,9:59:59.000 Covariates are characteristics [br]of the control and treatment groups 9:59:59.000,9:59:59.000 measured before [br]the experiment begins. 9:59:59.000,9:59:59.000 For example, we see the control group[br]has an average age a bit over 20. 9:59:59.000,9:59:59.000 Many of these covariates [br]are dummy variables. 9:59:59.000,9:59:59.000 A dummy variable can only have [br]two values, a zero or a one. 9:59:59.000,9:59:59.000 For example, student gender[br]is captured by a dummy variable 9:59:59.000,9:59:59.000 that equals one for woman[br]and zero for man. 9:59:59.000,9:59:59.000 The mean of this variable [br]is the proportion female. 9:59:59.000,9:59:59.000 We also see that the control group [br]is 13% Hispanic 9:59:59.000,9:59:59.000 and 19% had prior military service. 9:59:59.000,9:59:59.000 The table notes are key. 9:59:59.000,9:59:59.000 Refer to these [br]as you scan the table. 9:59:59.000,9:59:59.000 These notes explain what's shown [br]in each column and panel. 9:59:59.000,9:59:59.000 The notes tell us, for example,[br]that standard deviations 9:59:59.000,9:59:59.000 are reported in brackets. 9:59:59.000,9:59:59.000 Standard deviations tell us how[br]spread out the data are. 9:59:59.000,9:59:59.000 For example, a standard deviation [br]of 0.52 tells us that most 9:59:59.000,9:59:59.000 of the control group's GPAs[br]fall between 2.35, 9:59:59.000,9:59:59.000 which is 0.52 below [br]the mean GPA of 2.87, 9:59:59.000,9:59:59.000 and 3.39, which is 0.52 above 2.87. 9:59:59.000,9:59:59.000 A lower standard deviation[br]would mean the GPAs were 9:59:59.000,9:59:59.000 more tightly clustered [br]around the mean. 9:59:59.000,9:59:59.000 - [Kamal] Yeah, but they're missing[br]from most of the variables. 9:59:59.000,9:59:59.000 - [Narrator] That's right. 9:59:59.000,9:59:59.000 Masters usually omit [br]standard deviations for dummies 9:59:59.000,9:59:59.000 because the mean of this variable[br]determines its standard deviaton. 9:59:59.000,9:59:59.000 This study compares two treatment [br]groups with the control group. 9:59:59.000,9:59:59.000 The first was allowed free use [br]of laptops and tablets. 9:59:59.000,9:59:59.000 The second treatment[br]was more restrictive, 9:59:59.000,9:59:59.000 allowing only tablets placed [br]flat on the desk. 9:59:59.000,9:59:59.000 The treatment groups[br]look much like the control group. 9:59:59.000,9:59:59.000 Hmmm... 9:59:59.000,9:59:59.000 This takes us to the next feature [br]of this table, columns 4 through 6 9:59:59.000,9:59:59.000 use statistical tests to compare[br]the characteristics 9:59:59.000,9:59:59.000 of the treatment and control group[br]before the experiment. 9:59:59.000,9:59:59.000 In column 4, the two treatment [br]groups are combined. 9:59:59.000,9:59:59.000 You can see that the difference[br]in proportion female 9:59:59.000,9:59:59.000 between the treatment [br]and control group is only 0.03. 9:59:59.000,9:59:59.000 The difference is not [br]statistically significant. 9:59:59.000,9:59:59.000 It is the sort of difference[br]we can easily put down 9:59:59.000,9:59:59.000 to chance results [br]in our sample selection process. 9:59:59.000,9:59:59.000 - [Kamal] Hmm, how do we know that? 9:59:59.000,9:59:59.000 - [Narrator] Remember [br]the rule of thumb? 9:59:59.000,9:59:59.000 Statistical estimates [br]that exceed the standard 9:59:59.000,9:59:59.000 by a multiple of 2 [br]in absolute value 9:59:59.000,9:59:59.000 are usually said[br]to be statistically significant. 9:59:59.000,9:59:59.000 The standard error is 0.03, 9:59:59.000,9:59:59.000 same as the difference [br]in proportion female. 9:59:59.000,9:59:59.000 So the ratio of the latter [br]to the former is only 1, 9:59:59.000,9:59:59.000 which of course is less than 2. 9:59:59.000,9:59:59.000 - [Kamal] Uh huh. So none[br]of the treatment/control differences 9:59:59.000,9:59:59.000 in the table are more than twice[br]their standard errors. 9:59:59.000,9:59:59.000 - [Narrator] Correct. 9:59:59.000,9:59:59.000 The random division of students[br]appears to have succeeded 9:59:59.000,9:59:59.000 in creating groups [br]that are indeed comparable. 9:59:59.000,9:59:59.000 We can be confident therefore[br]that any later differences 9:59:59.000,9:59:59.000 in classroom achievement[br]are the result of the experimental 9:59:59.000,9:59:59.000 intervention rather [br]than a reflection 9:59:59.000,9:59:59.000 of preexisting differences. 9:59:59.000,9:59:59.000 Ceteris paribus achieved! 9:59:59.000,9:59:59.000 - [Kamal] Cool. Wait, [br]what about the bottom, 9:59:59.000,9:59:59.000 the numbers with the stars? 9:59:59.000,9:59:59.000 Those differences are a lot more[br]than double the standard error. 9:59:59.000,9:59:59.000 - [Narrator] Good eye, Kamal! 9:59:59.000,9:59:59.000 The table has many numbers. 9:59:59.000,9:59:59.000 Those in Panel B are important too. 9:59:59.000,9:59:59.000 This panel measures the extent [br]to which students in treatment 9:59:59.000,9:59:59.000 and control groups actually use [br]computers in class. 9:59:59.000,9:59:59.000 The treatment here was [br]to allow computer use. 9:59:59.000,9:59:59.000 The researchers must show[br]that students allowed 9:59:59.000,9:59:59.000 to use computers took advantage [br]of the opportunity to do so. 9:59:59.000,9:59:59.000 If they didn't, then there's [br]really no treatment. 9:59:59.000,9:59:59.000 Luckily, 81% of those [br]in the first treatment group 9:59:59.000,9:59:59.000 used computers compared [br]with none in the control group. 9:59:59.000,9:59:59.000 And many in the second [br]tablet treatment group 9:59:59.000,9:59:59.000 used computers as well. 9:59:59.000,9:59:59.000 These differences [br]in computer use are large 9:59:59.000,9:59:59.000 and statistically significant. 9:59:59.000,9:59:59.000 We also get to see [br]the sample size in each group. 9:59:59.000,9:59:59.000 - [Kamal] The stars [br]are just like decoration? 9:59:59.000,9:59:59.000 - [Narrator] Some academic papers[br]use stars to indicate differences 9:59:59.000,9:59:59.000 that are statistically significant. 9:59:59.000,9:59:59.000 This makes them jump out at you. 9:59:59.000,9:59:59.000 Here three stars indicate that [br]the result is statistically different 9:59:59.000,9:59:59.000 from zero with a p value [br]less than 1%. 9:59:59.000,9:59:59.000 In other words, there's less [br]than a 1 in 100 chance 9:59:59.000,9:59:59.000 this result is purely [br]a chance finding. 9:59:59.000,9:59:59.000 [applause] 9:59:59.000,9:59:59.000 Two stars indicate a 1 in 20[br]or 5% chance of a chance finding. 9:59:59.000,9:59:59.000 And one star denotes results [br]we might see as often as 10% 9:59:59.000,9:59:59.000 of the time merely due to chance. 9:59:59.000,9:59:59.000 Today, stars are seen [br]as a little old fashioned. 9:59:59.000,9:59:59.000 Some journals omit them. 9:59:59.000,9:59:59.000 - [Kamal] What about [br]those last two columns? 9:59:59.000,9:59:59.000 - [Narrator] Unlike column 4,[br]which combines 9:59:59.000,9:59:59.000 both treatment groups into one, [br]these last two columns 9:59:59.000,9:59:59.000 look separately[br]at treatment/control differences 9:59:59.000,9:59:59.000 for each treatment group. 9:59:59.000,9:59:59.000 This provides a more detailed [br]analysis of balance. 9:59:59.000,9:59:59.000 Also, for now, [br]you can ignore this row 9:59:59.000,9:59:59.000 which provides [br]another test of significance. 9:59:59.000,9:59:59.000 Now we get to the article's [br]punchline, table 4. 9:59:59.000,9:59:59.000 This table reports[br]regression estimates 9:59:59.000,9:59:59.000 of the effects of electronics use[br]on measures of student learning. 9:59:59.000,9:59:59.000 - [Kamal Why does the study [br]report regression estimates? 9:59:59.000,9:59:59.000 See, that's why I was getting lost. 9:59:59.000,9:59:59.000 I thought one reason [br]why we liked randomized trials 9:59:59.000,9:59:59.000 is that we use them [br]to obtain causal effects 9:59:59.000,9:59:59.000 simply by comparing [br]treatment and control groups. 9:59:59.000,9:59:59.000 Since these groups are balanced,[br]no need to use regression. 9:59:59.000,9:59:59.000 - [Narrator] Well said, Kamal. 9:59:59.000,9:59:59.000 In practice, it's customary [br]to report regression estimates 9:59:59.000,9:59:59.000 for two reasons. 9:59:59.000,9:59:59.000 First, evidence of balance [br]not withstanding, an abundance 9:59:59.000,9:59:59.000 of caution might lead the analyst[br]to allow for chance differences. 9:59:59.000,9:59:59.000 Second, regression estimates[br]are likely to be more precise. 9:59:59.000,9:59:59.000 That is, they have lower [br]standard errors than 9:59:59.000,9:59:59.000 the simple treatment [br]control comparisons. 9:59:59.000,9:59:59.000 The dependent variable [br]in this study 9:59:59.000,9:59:59.000 is the outcome of interest. 9:59:59.000,9:59:59.000 Since the question at hand[br]is how classroom electronics 9:59:59.000,9:59:59.000 affect learning, a good outcome[br]is the economics final exam score. 9:59:59.000,9:59:59.000 Each column reports results[br]from a different regression model. 9:59:59.000,9:59:59.000 Models are distinguished [br]by the control variables 9:59:59.000,9:59:59.000 or covariates they include[br]besides treatment status. 9:59:59.000,9:59:59.000 Estimates with no covariates [br]are simple comparisons 9:59:59.000,9:59:59.000 of treatment and control groups. 9:59:59.000,9:59:59.000 - [Kamal] I thought [br]they just forgot to fill it out. 9:59:59.000,9:59:59.000 - [Narrator] Column 1 suggests[br]electronics use reduced 9:59:59.000,9:59:59.000 final exam scores[br]by 0.28 standard deviations. 9:59:59.000,9:59:59.000 In our last lesson, Master Joshway[br]explained, we use standard deviation 9:59:59.000,9:59:59.000 units because these units [br]are easily compared across studies. 9:59:59.000,9:59:59.000 Column 2 reports results [br]from a model 9:59:59.000,9:59:59.000 that adds demographic controls. 9:59:59.000,9:59:59.000 Here we're comparing test scores[br]but holding constant factors 9:59:59.000,9:59:59.000 such as age and sex. 9:59:59.000,9:59:59.000 Column 3 reports results[br]from a model that adds GPA 9:59:59.000,9:59:59.000 to the list of covariates. 9:59:59.000,9:59:59.000 Column 4 adds ACT scores. 9:59:59.000,9:59:59.000 Analysts often report [br]results this way, 9:59:59.000,9:59:59.000 starting with models that include [br]few or no covariates 9:59:59.000,9:59:59.000 and then reporting estimates [br]from models that add more 9:59:59.000,9:59:59.000 and more covariates [br]as we move across columns. 9:59:59.000,9:59:59.000 Looking across columns,[br]what do you notice? 9:59:59.000,9:59:59.000 - [Kamal] Well, the coefficient [br]on using a computer is always 9:59:59.000,9:59:59.000 a pretty big negative number. 9:59:59.000,9:59:59.000 - [Narrator] That's right! 9:59:59.000,9:59:59.000 We can also see that [br]the standard errors are small enough 9:59:59.000,9:59:59.000 to make these negative results[br]statistically significant. 9:59:59.000,9:59:59.000 In other words, the primary [br]takeaway from this experiment 9:59:59.000,9:59:59.000 is that electronics in the classroom[br]reduce student learning. 9:59:59.000,9:59:59.000 - [Kama] GPA and ACT scores [br]are also significant. 9:59:59.000,9:59:59.000 Why is that? 9:59:59.000,9:59:59.000 - [Narrator] Good observation! 9:59:59.000,9:59:59.000 That's not surprising. 9:59:59.000,9:59:59.000 We expect these variables [br]to predict college performance. 9:59:59.000,9:59:59.000 - [Kamal] Oh right, of course. 9:59:59.000,9:59:59.000 Kids who got better grades before[br]are more likely to get 9:59:59.000,9:59:59.000 a better grade in this course. 9:59:59.000,9:59:59.000 - [Narrator] You'll also notice a lot [br]of other information on this table. 9:59:59.000,9:59:59.000 Remaining panels in the table[br]report effects of electronics use 9:59:59.000,9:59:59.000 on components of the final exam, 9:59:59.000,9:59:59.000 such as the multiple [br]choice questions. 9:59:59.000,9:59:59.000 These results are mostly consistent [br]with computer use effects 9:59:59.000,9:59:59.000 on overall scores. 9:59:59.000,9:59:59.000 - [Kamal] What about the rows[br]not in English? 9:59:59.000,9:59:59.000 - [Narrator] These rows give [br]additional statistical information. 9:59:59.000,9:59:59.000 R-squared is a measure[br]of goodness of fit. 9:59:59.000,9:59:59.000 This isn't too important, though [br]some readers may want to know it. 9:59:59.000,9:59:59.000 Other rows report on alternative [br]tests of statistical significance 9:59:59.000,9:59:59.000 that you can ignore for now. 9:59:59.000,9:59:59.000 - [Kamal] Oh my gosh,[br]these tables aren't that hard. 9:59:59.000,9:59:59.000 Thank you so much. 9:59:59.000,9:59:59.000 Next up is regression. 9:59:59.000,9:59:59.000 See you then! 9:59:59.000,9:59:59.000 ♪ [music] ♪ 9:59:59.000,9:59:59.000 You're on your way [br]to mastering econometrics. 9:59:59.000,9:59:59.000 Make sure this video sticks 9:59:59.000,9:59:59.000 by taking a few [br]quick practice questions. 9:59:59.000,9:59:59.000 Or, if you're ready,[br]click for the next video. 9:59:59.000,9:59:59.000 You can also check out MRU's [br]website for more courses, 9:59:59.000,9:59:59.000 teacher resources and more.