0:00:01.425,0:00:05.015 - [Narrator] On his quest [br]to master econometrics, 0:00:05.619,0:00:08.813 Grasshopper Kamal[br]has made great progress, 0:00:08.813,0:00:13.662 stretching his capabilities [br]and outsmarting his foes. 0:00:14.223,0:00:16.640 Alas, today he's despondent, 0:00:16.640,0:00:19.614 for one challenge remains unmet. 0:00:19.614,0:00:24.130 Kamal cannot yet decode [br]the scriptures of academic research, 0:00:24.130,0:00:27.347 journals like [br]"The American Economic Review" 0:00:27.347,0:00:29.317 and "Econometrica." 0:00:29.317,0:00:33.501 These seemed to him to be inscribed [br]in an obscure foreign tongue. 0:00:33.501,0:00:35.478 - [Kamal] Ugh, what the... ? 0:00:36.711,0:00:40.069 - These volumes are opaque[br]to the novice, Kamal, 0:00:40.069,0:00:42.205 but can be deciphered with study. 0:00:42.467,0:00:45.109 Let us learn to read them together. 0:00:52.657,0:00:55.317 Let's dive into the West Point study, 0:00:55.317,0:00:58.278 published in the "Economics [br]of Education Review." 0:00:58.658,0:01:01.688 This paper reports [br]on a randomized evaluation 0:01:01.688,0:01:05.859 of student electronics use [br]in Economics 101 classrooms. 0:01:06.485,0:01:09.192 First, a quick review [br]of the research design. 0:01:09.423,0:01:10.523 - Okay. 0:01:11.553,0:01:13.630 - [Josh] 'Metrics masters [br]teaching at West Point, 0:01:13.630,0:01:17.029 the military college that trains [br]American Army officers 0:01:17.029,0:01:19.854 designed a randomized trial[br]to answer this question. 0:01:20.582,0:01:23.233 These masters randomly assigned[br]West Point cadets 0:01:23.233,0:01:27.116 into Economics classes[br]operating under different rules. 0:01:27.116,0:01:28.962 Unlike most American colleges, 0:01:28.962,0:01:31.945 the West Point default [br]is no electronics. 0:01:32.345,0:01:35.784 For purposes of this experiment,[br]some students were left 0:01:35.784,0:01:38.679 in such traditional [br]technology-free classes -- 0:01:38.679,0:01:41.911 no laptops, no tablets [br]and no phones! 0:01:41.911,0:01:43.324 [voice echoes] 0:01:43.324,0:01:45.743 This is the control group,[br]or baseline case. 0:01:46.372,0:01:49.292 Another group was allowed[br]to use electronics. 0:01:49.292,0:01:52.704 This is the treatment group,[br]subject to a changed environment. 0:01:53.313,0:01:55.858 The treatment in this case[br]is the unrestricted use 0:01:55.858,0:01:58.107 of laptops or tablets in class. 0:01:59.195,0:02:01.972 Every causal question [br]has a clear outcome -- 0:02:01.972,0:02:05.379 the variables we hope to influence[br]defined in advance of the study. 0:02:05.860,0:02:08.375 The outcomes in the West Point[br]electronics study 0:02:08.375,0:02:10.535 are final exam scores. 0:02:10.535,0:02:13.810 The study seeks to answer[br]the following question: 0:02:13.810,0:02:17.629 What is the causal effect [br]of classroom electronics on learning 0:02:17.629,0:02:19.765 as measured by exam scores? 0:02:20.852,0:02:24.199 - Economics journal articles [br]usually begin with a table 0:02:24.199,0:02:25.994 of descriptive statistics, 0:02:25.994,0:02:28.694 giving key facts[br]about the study sample. 0:02:28.694,0:02:32.129 - Oh my gosh, I remember[br]this table, so confusing! 0:02:32.129,0:02:37.224 - [Narrator] Columns 1 to 3 report [br]mean, or average, characteristics. 0:02:37.224,0:02:40.089 These give a sense [br]of who we're studying. 0:02:40.089,0:02:43.736 Let's start with column 1 [br]which describes covariates 0:02:43.736,0:02:45.438 in the control group. 0:02:45.438,0:02:49.183 Covariates are characteristics [br]of the control and treatment groups 0:02:49.183,0:02:52.091 measured before [br]the experiment begins. 0:02:52.091,0:02:57.514 For example, we see the control group[br]has an average age a bit over 20. 0:02:57.514,0:03:00.339 Many of these covariates [br]are dummy variables. 0:03:00.997,0:03:06.087 A dummy variable can only have [br]two values -- a zero or a one. 0:03:06.087,0:03:10.298 For example, student gender[br]is captured by a dummy variable 0:03:10.298,0:03:13.386 that equals one for women[br]and zero for men. 0:03:13.386,0:03:17.104 The mean of this variable [br]is the proportion female. 0:03:17.104,0:03:20.651 We also see that the control group [br]is 13% Hispanic 0:03:20.651,0:03:23.905 and 19% had prior military service. 0:03:25.035,0:03:26.635 The table notes are key. 0:03:26.635,0:03:29.218 Refer to these [br]as you scan the table. 0:03:29.218,0:03:33.534 These notes explain what's shown [br]in each column and panel. 0:03:39.485,0:03:41.858 The notes tell us, for example, 0:03:41.858,0:03:45.175 that standard deviations[br]are reported in brackets. 0:03:45.947,0:03:49.598 Standard deviations tell us how[br]spread out the data are. 0:03:50.448,0:03:54.887 For example, a standard deviation [br]of 0.52 tells us that most 0:03:54.887,0:03:59.397 of the control group's GPAs[br]fall between 2.35, 0:03:59.397,0:04:03.454 which is 0.52 below [br]the mean GPA of 2.87, 0:04:03.454,0:04:08.337 and 3.39, which is 0.52 above 2.87. 0:04:09.337,0:04:12.122 A lower standard deviation[br]would mean the GPAs 0:04:12.122,0:04:14.706 were more tightly clustered [br]around the mean. 0:04:14.706,0:04:17.543 - [Kamal] Yeah, but they're missing[br]for most of the variables. 0:04:17.543,0:04:18.600 - [Narrator] That's right. 0:04:18.600,0:04:22.497 Masters usually omit [br]standard deviations for dummies 0:04:22.497,0:04:26.500 because the mean of this variable[br]determines its standard deviation. 0:04:27.542,0:04:32.078 This study compares two treatment [br]groups with the control group. 0:04:32.078,0:04:35.886 The first was allowed free use [br]of laptops and tablets. 0:04:35.886,0:04:38.252 The second treatment[br]was more restrictive, 0:04:38.252,0:04:41.553 allowing only tablets[br]placed flat on the desk. 0:04:42.152,0:04:45.238 The treatment groups[br]look much like the control group. 0:04:46.694,0:04:51.443 This takes us to the next feature [br]of this table, columns 4 through 6 0:04:51.443,0:04:54.558 use statistical tests to compare[br]the characteristics 0:04:54.558,0:04:57.591 of the treatment and control group[br]before the experiment. 0:04:58.158,0:05:01.991 In column 4, the two treatment [br]groups are combined. 0:05:01.991,0:05:04.998 You can see that the difference[br]in proportion female 0:05:04.998,0:05:09.690 between the treatment [br]and control group is only 0.03. 0:05:10.508,0:05:13.740 The difference is not [br]statistically significant. 0:05:14.290,0:05:17.440 It is the sort of difference[br]we can easily put down 0:05:17.440,0:05:20.623 to chance results [br]in our sample selection process. 0:05:20.623,0:05:22.133 - [Kamal] Hmm, how do we know that? 0:05:22.133,0:05:23.790 - [Narrator] Remember [br]the rule of thumb? 0:05:23.790,0:05:27.122 Statistical estimates [br]that exceed the standard error 0:05:27.122,0:05:30.108 by a multiple of 2 [br]in absolute value 0:05:30.108,0:05:33.997 are usually said[br]to be statistically significant. 0:05:35.132,0:05:38.766 The standard error is 0.03, 0:05:38.766,0:05:41.483 same as the difference [br]in proportion female. 0:05:42.244,0:05:46.132 So the ratio of the latter [br]to the former is only 1, 0:05:46.132,0:05:48.607 which of course is less than 2. 0:05:48.607,0:05:51.191 - [Kamal] Uh huh. So none[br]of the treatment/control differences 0:05:51.191,0:05:54.455 in the table are more than twice[br]their standard errors. 0:05:54.455,0:05:55.997 - [Narrator] Correct. 0:05:55.997,0:05:59.081 The random division of students[br]appears to have succeeded 0:05:59.081,0:06:01.945 in creating groups [br]that are indeed comparable. 0:06:02.846,0:06:05.008 We can be confident, therefore, 0:06:05.008,0:06:07.774 that any later differences[br]in classroom achievement 0:06:07.774,0:06:11.073 are the result[br]of the experimental intervention 0:06:11.073,0:06:14.754 rather than a reflection[br]of preexisting differences. 0:06:14.754,0:06:17.454 Ceteris paribus achieved! 0:06:17.454,0:06:20.934 - [Kamal] Cool. Wait, [br]what about the bottom, 0:06:20.934,0:06:22.833 the numbers with the stars? 0:06:22.833,0:06:25.589 Those differences are a lot more[br]than double the standard error. 0:06:25.589,0:06:27.402 - [Narrator] Good eye, Kamal! 0:06:27.402,0:06:29.386 The table has many numbers. 0:06:29.386,0:06:32.246 Those in Panel B are important too. 0:06:32.246,0:06:35.715 This panel measures the extent [br]to which students in treatment 0:06:35.715,0:06:39.139 and control groups actually use [br]computers in class. 0:06:39.754,0:06:43.278 The treatment here[br]was to allow computer use. 0:06:43.278,0:06:44.873 The researchers must show 0:06:44.873,0:06:47.079 that students allowed[br]to use computers 0:06:47.079,0:06:49.448 took advantage [br]of the opportunity to do so. 0:06:50.072,0:06:53.867 If they didn't, then there's [br]really no treatment. 0:06:53.867,0:06:57.799 Luckily, 81% of those [br]in the first treatment group 0:06:57.799,0:06:59.472 used computers 0:06:59.472,0:07:02.178 compared with none[br]in the control group. 0:07:02.178,0:07:05.216 And many in the second [br]tablet treatment group 0:07:05.216,0:07:07.264 used computers as well. 0:07:07.264,0:07:09.879 These differences [br]in computer use are large 0:07:09.879,0:07:11.798 and statistically significant. 0:07:12.081,0:07:15.428 We also get to see [br]the sample size in each group. 0:07:15.428,0:07:18.098 - [Kamal] The stars [br]are just like decoration? 0:07:18.098,0:07:21.748 - [Narrator] Some academic papers[br]use stars to indicate differences 0:07:21.748,0:07:23.983 that are statistically significant. 0:07:23.983,0:07:26.925 This makes them jump out at you. 0:07:26.925,0:07:31.621 Here, three stars indicate that [br]the result is statistically different 0:07:31.621,0:07:34.942 from zero with a p value [br]less than 1%. 0:07:35.672,0:07:39.436 In other words, there's less [br]than a 1 in 100 chance 0:07:39.436,0:07:42.171 this result is purely [br]a chance finding. 0:07:42.171,0:07:43.181 [applause] 0:07:43.181,0:07:48.997 Two stars indicate a 1 in 20[br]or 5% chance of a chance finding. 0:07:48.997,0:07:52.469 And one star denotes results [br]we might see as often 0:07:52.469,0:07:56.036 as 10% of the time[br]merely due to chance. 0:07:56.473,0:07:59.957 Today, stars are seen [br]as a little old fashioned. 0:07:59.957,0:08:01.606 Some journals omit them. 0:08:01.606,0:08:03.894 - [Kamal] What about [br]those last two columns? 0:08:03.894,0:08:06.007 - [Narrator] Unlike column 4,[br]which combines 0:08:06.007,0:08:09.689 both treatment groups into one, [br]these last two columns 0:08:09.689,0:08:12.357 look separately[br]at treatment/control differences 0:08:12.357,0:08:14.572 for each treatment group. 0:08:14.572,0:08:17.441 This provides a more detailed [br]analysis of balance. 0:08:18.295,0:08:21.288 Also, for now, [br]you can ignore this row 0:08:21.288,0:08:24.755 which provides [br]another test of significance. 0:08:24.755,0:08:29.062 Now we get to the article's [br]punchline, table 4. 0:08:30.075,0:08:32.993 This table reports[br]regression estimates 0:08:32.993,0:08:37.273 of the effects of electronics use[br]on measures of student learning. 0:08:37.273,0:08:40.258 - [Kamal Why does the study [br]report regression estimates? 0:08:40.258,0:08:42.529 See, that's why I was getting lost. 0:08:42.529,0:08:44.806 I thought one reason [br]why we liked randomized trials 0:08:44.806,0:08:47.260 is that we use them [br]to obtain causal effects 0:08:47.260,0:08:50.479 simply by comparing [br]treatment and control groups. 0:08:50.479,0:08:53.883 Since these groups are balanced,[br]no need to use regression. 0:08:53.883,0:08:55.492 - [Narrator] Well said, Kamal. 0:08:55.492,0:08:59.272 In practice, it's customary [br]to report regression estimates 0:08:59.272,0:09:01.013 for two reasons. 0:09:01.013,0:09:04.448 First, evidence of balance [br]notwithstanding, 0:09:04.448,0:09:07.349 an abundance of caution[br]might lead the analyst 0:09:07.349,0:09:09.678 to allow for chance differences. 0:09:09.678,0:09:13.622 Second, regression estimates[br]are likely to be more precise. 0:09:13.622,0:09:16.509 That is, they have lower [br]standard errors 0:09:16.509,0:09:18.893 than the simple treatment [br]control comparisons. 0:09:20.129,0:09:22.526 The dependent variable [br]in this study 0:09:22.526,0:09:24.305 is the outcome of interest. 0:09:24.652,0:09:27.717 Since the question at hand[br]is how classroom electronics 0:09:27.717,0:09:29.068 affect learning, 0:09:29.068,0:09:32.845 a good outcome[br]is the Economics final exam score. 0:09:33.406,0:09:37.650 Each column reports results[br]from a different regression model. 0:09:37.650,0:09:40.476 Models are distinguished [br]by the control variables 0:09:40.476,0:09:44.934 or covariates they include[br]besides treatment status. 0:09:44.934,0:09:48.425 Estimates with no covariates [br]are simple comparisons 0:09:48.425,0:09:50.677 of treatment and control groups. 0:09:50.677,0:09:53.169 - [Kamal] I thought [br]they just forgot to fill it out. 0:09:53.169,0:09:56.228 - [Narrator] Column 1[br]suggests electronics use 0:09:56.228,0:10:00.835 reduced final exam scores[br]by 0.28 standard deviations. 0:10:01.547,0:10:02.940 In our last lesson, 0:10:02.940,0:10:07.237 Master Joshway explained[br]that we use standard deviation units 0:10:07.237,0:10:10.672 because these units [br]are easily compared across studies. 0:10:11.352,0:10:13.702 Column 2 reports results [br]from a model 0:10:13.702,0:10:15.952 that adds demographic controls. 0:10:15.952,0:10:19.907 Here, we're comparing test scores[br]but holding constant factors 0:10:19.907,0:10:21.435 such as age and sex. 0:10:21.886,0:10:25.602 Column 3 reports results[br]from a model that adds GPA 0:10:25.602,0:10:27.186 to the list of covariates. 0:10:27.603,0:10:30.822 Column 4 adds ACT scores. 0:10:30.822,0:10:33.503 Analysts often report [br]results this way, 0:10:33.503,0:10:36.992 starting with models that include [br]few or no covariates 0:10:36.992,0:10:39.667 and then reporting[br]estimates from models 0:10:39.667,0:10:43.586 that add more and more covariates [br]as we move across columns. 0:10:44.035,0:10:46.802 Looking across columns,[br]what do you notice? 0:10:47.252,0:10:49.919 - [Kamal] Well, the coefficient [br]on using a computer is always 0:10:49.919,0:10:51.635 a pretty big negative number. 0:10:51.635,0:10:53.002 - [Narrator] That's right! 0:10:53.002,0:10:56.455 We can also see that [br]the standard errors are small enough 0:10:56.455,0:11:00.561 to make these negative results[br]statistically significant. 0:11:00.561,0:11:04.446 In other words, the primary [br]takeaway from this experiment 0:11:04.446,0:11:08.381 is that electronics in the classroom[br]reduce student learning. 0:11:09.000,0:11:12.283 - [Kamal] GPA and ACT scores [br]are also significant. 0:11:12.283,0:11:13.600 Why is that? 0:11:13.600,0:11:15.423 - [Narrator] Good observation! 0:11:15.423,0:11:16.866 That's not surprising. 0:11:16.866,0:11:20.473 We expect these variables [br]to predict college performance. 0:11:20.473,0:11:22.190 - [Kamal] Oh right, of course. 0:11:22.190,0:11:24.026 Kids who got better grades before 0:11:24.026,0:11:26.317 are more likely to get[br]a better grade in this course. 0:11:26.317,0:11:30.226 - [Narrator] You'll also notice a lot [br]of other information on this table. 0:11:30.226,0:11:34.515 Remaining panels in the table[br]report effects of electronics use 0:11:34.515,0:11:36.933 on components of the final exam, 0:11:36.933,0:11:39.816 such as the multiple [br]choice questions. 0:11:39.816,0:11:43.285 These results are mostly consistent [br]with computer use effects 0:11:43.285,0:11:45.360 on overall scores. 0:11:45.360,0:11:47.740 - [Kamal] What about the rows[br]not in English? 0:11:47.740,0:11:50.994 - [Narrator] These rows give [br]additional statistical information. 0:11:50.994,0:11:54.247 R-squared is a measure[br]of goodness of fit. 0:11:54.714,0:11:58.010 This isn't too important, though [br]some readers may want to know it. 0:11:58.660,0:12:02.950 Other rows report on alternative [br]tests of statistical significance 0:12:02.950,0:12:05.028 that you can ignore for now. 0:12:05.028,0:12:07.934 - [Kamal] Oh my gosh,[br]these tables aren't that hard! 0:12:07.934,0:12:09.488 Thank you so much. 0:12:09.488,0:12:11.787 - [Narrator] Next up is regression. 0:12:11.787,0:12:13.179 See you then! 0:12:15.974,0:12:17.263 ♪ [music] ♪ 0:12:17.263,0:12:20.575 You're on your way [br]to mastering econometrics. 0:12:20.834,0:12:22.783 Make sure this video sticks 0:12:22.783,0:12:25.467 by taking a few [br]quick practice questions. 0:12:25.467,0:12:29.003 Or, if you're ready,[br]click for the next video. 0:12:29.003,0:12:32.901 You can also check out MRU's [br]website for more courses, 0:12:32.901,0:12:35.298 teacher resources and more.