WEBVTT 00:00:00.000 --> 00:00:02.550 ♪ [music] ♪ 00:00:03.800 --> 00:00:05.800 - [Narrator] Welcome to Nobel Conversations. 00:00:07.100 --> 00:00:08.100 In this episode, 00:00:08.100 --> 00:00:11.570 Josh Angrist and Guido Imbens sit down with Isaiah Andrews 00:00:11.570 --> 00:00:14.600 to discuss how the field of econometrics is evolving. 00:00:16.100 --> 00:00:18.750 - [Isaiah] So Guido and Josh, you're both pioneers 00:00:18.750 --> 00:00:21.500 in developing tools for empirical research in economics. 00:00:21.500 --> 00:00:22.930 And so I'd like to explore 00:00:22.930 --> 00:00:25.300 sort of where you feel like the field is heading, 00:00:25.300 --> 00:00:28.079 sort of economics, econometrics, the whole thing. 00:00:28.510 --> 00:00:31.290 To start, I'd be interested to hear 00:00:32.200 --> 00:00:35.200 about whether you feel like sort of the way in which 00:00:35.200 --> 00:00:38.510 the local average treatment effects framework sort of took hold 00:00:38.800 --> 00:00:42.100 has any lessons for how new empirical methods in economics 00:00:42.100 --> 00:00:44.300 develop and spread or how they should. 00:00:44.560 --> 00:00:45.960 - [Josh] That's a good question. 00:00:46.610 --> 00:00:47.790 You go first. 00:00:47.790 --> 00:00:49.460 (laughter) 00:00:49.700 --> 00:00:53.180 Yeah, so I think the important thing 00:00:53.180 --> 00:00:58.550 is to come up with good convincing cases 00:00:58.550 --> 00:01:02.207 where the questions are clear 00:01:02.400 --> 00:01:05.720 and where kind of the methods apply in general. 00:01:05.720 --> 00:01:07.560 So one thing I-- 00:01:08.070 --> 00:01:12.000 Kind of looking back at the subsequent literature, 00:01:12.200 --> 00:01:16.700 so I really like the regression discontinuity literature 00:01:16.700 --> 00:01:19.670 [where there were] clearly a bunch of really convincing examples 00:01:19.670 --> 00:01:21.319 and that allowed people to kind of 00:01:22.300 --> 00:01:27.200 think more clearly, look harder at the methodological questions. 00:01:27.400 --> 00:01:28.800 Kind of do clear applications 00:01:28.800 --> 00:01:30.600 that then allow you to kind of think about, 00:01:30.600 --> 00:01:33.600 "Wow, do this type of assumption seem reasonable here? 00:01:33.600 --> 00:01:38.000 What kind of things do we not like in the early papers? 00:01:38.500 --> 00:01:39.802 How can we improve things?" 00:01:39.802 --> 00:01:42.300 So having clear applications motivating, 00:01:43.400 --> 00:01:46.400 these literatures, I think it's very helpful. 00:01:46.800 --> 00:01:49.300 I'm glad you mentioned the regression discontinuity, Guido. 00:01:49.300 --> 00:01:53.300 I think there's a lot of complementarity between IV and RD, 00:01:54.700 --> 00:01:57.060 Instrumental Variables and Regression Discontinuity. 00:02:00.350 --> 00:02:03.260 And a lot of the econometric applications 00:02:03.260 --> 00:02:04.520 of regression discontinuity 00:02:04.520 --> 00:02:07.230 are what used to be called "fuzzy" RD, 00:02:07.230 --> 00:02:11.620 where, you know, it's not discrete or deterministic at the cutoff, 00:02:11.620 --> 00:02:14.900 but just the change in rates or intensity. 00:02:14.900 --> 00:02:18.740 And and the late framework helps us understand those applications 00:02:18.740 --> 00:02:21.140 and gives us a clear interpretation 00:02:21.140 --> 00:02:25.000 for say, something like, in my paper with Victor Lavy, 00:02:25.000 --> 00:02:28.100 where we use Maimonides' rule, the class size cutoffs. 00:02:28.600 --> 00:02:30.200 What are you getting there? 00:02:30.400 --> 00:02:31.970 Of course, you can answer that question 00:02:31.970 --> 00:02:33.900 with a linear constant effects model, 00:02:34.200 --> 00:02:36.310 but it turns out we're not limited to that, 00:02:36.310 --> 00:02:39.889 and an RD is still very powerful and illuminating, 00:02:40.630 --> 00:02:42.100 even when, you know, 00:02:42.100 --> 00:02:46.000 the correlation between the cutoff and the variable of interest, 00:02:46.000 --> 00:02:48.980 in this case class size, is partial, 00:02:48.980 --> 00:02:51.000 maybe even not that strong. 00:02:52.000 --> 00:02:54.999 So there was definitely a kind of, a parallel development. 00:02:54.999 --> 00:02:56.400 It's also interesting, 00:02:56.600 --> 00:02:59.780 you know, nobody talked about regression discontinuity designs 00:02:59.780 --> 00:03:01.220 when we were in graduate school, 00:03:01.220 --> 00:03:05.300 it was something that other social scientists were interested in, 00:03:05.800 --> 00:03:09.507 and that kind of grew up alongside the late framework 00:03:09.507 --> 00:03:14.787 and we've both done work on both applications and methods there 00:03:14.787 --> 00:03:18.377 and it's been very exciting to see that kind of develop 00:03:18.377 --> 00:03:19.800 and become so important. 00:03:20.000 --> 00:03:23.700 It's part of a general evolution, I think, towards, you know, 00:03:24.000 --> 00:03:27.700 credible identification strategies causal effects... 00:03:28.630 --> 00:03:30.200 less, you know, making econometrics 00:03:30.400 --> 00:03:33.300 more about causal questions than about models. 00:03:33.640 --> 00:03:34.650 In terms of the future, 00:03:34.650 --> 00:03:37.860 I think one thing that LATE has helped facilitate 00:03:37.860 --> 00:03:42.700 is a move towards more creative, randomized trials, where, 00:03:42.700 --> 00:03:44.400 you know, there's something of interest, 00:03:45.500 --> 00:03:50.700 it's not possible or straightforward to simply turn it off or on, 00:03:51.000 --> 00:03:54.584 but you can encourage it or discourage it. 00:03:54.584 --> 00:03:58.200 So you subsidize schooling with financial aid, for example. 00:03:59.000 --> 00:04:02.080 So now we have a whole framework for interpreting that, 00:04:03.600 --> 00:04:06.900 and it kind of opens the doors to randomized trials 00:04:06.900 --> 00:04:10.300 of things that that maybe would, you know, 00:04:10.300 --> 00:04:12.471 not have seem possible before. 00:04:14.500 --> 00:04:17.700 We've used that a lot in the work we do on schools in our-- 00:04:17.700 --> 00:04:21.160 in the Blueprint Lab at MIT, 00:04:22.360 --> 00:04:26.600 we're exploiting random assignment and in very creative ways, I think. 00:04:28.100 --> 00:04:32.300 - [Isaiah] Related to that, do you see sort of particular factors 00:04:32.400 --> 00:04:34.300 that make for useful research in econometrics? 00:04:34.400 --> 00:04:38.290 You've alluded to it having a clear connection 00:04:38.290 --> 00:04:40.300 to problems that are actually coming up 00:04:40.300 --> 00:04:42.650 and empirical practice is often a good idea. 00:04:43.290 --> 00:04:45.000 - [Josh] Isn't it always a good idea? 00:04:45.700 --> 00:04:50.100 I often find myself sitting in an econometrics theory seminar, 00:04:50.700 --> 00:04:52.500 say the Harvard MIT seminar, 00:04:53.400 --> 00:04:55.940 and I'm thinking, "What problem is this guy solving? 00:04:55.940 --> 00:04:57.960 Who has this problem?" 00:04:57.960 --> 00:04:59.800 And, you know, 00:05:01.600 --> 00:05:04.700 sometimes there's an embarrassing silence if I ask 00:05:04.900 --> 00:05:08.300 or there might be a fairly contrived scenario. 00:05:08.800 --> 00:05:11.600 I want to see where the tool is useful. 00:05:12.500 --> 00:05:14.900 There are some purely foundational tools, 00:05:14.900 --> 00:05:17.600 I do take the point, you know, there are people who are 00:05:18.200 --> 00:05:22.500 working on conceptual foundations of, you know, 00:05:22.600 --> 00:05:25.300 it's more-- becomes more like mathematical statistics. 00:05:25.800 --> 00:05:28.200 I mean, I remember an early example of that that I, 00:05:28.200 --> 00:05:30.350 you know, I struggled to understand 00:05:30.350 --> 00:05:32.500 was the idea of stochastic equicontinuity, 00:05:32.500 --> 00:05:35.070 which one of my thesis advisors, Whitney Newey, 00:05:35.070 --> 00:05:36.900 was using to great effect 00:05:37.500 --> 00:05:39.900 and I was trying to understand that and there isn't really-- 00:05:40.600 --> 00:05:45.200 It's really foundational, it's not an application that's driving that, 00:05:45.890 --> 00:05:47.300 at least not immediately 00:05:48.600 --> 00:05:53.200 but most things are not like that and so there should be a problem. 00:05:53.800 --> 00:05:59.100 And I think it's on the seller of that sort of thing, 00:06:00.100 --> 00:06:02.250 you know, because there's opportunity cost, 00:06:02.250 --> 00:06:05.170 the time and attention, and effort to understand things 00:06:05.170 --> 00:06:07.200 to, you know, it's on the seller to say, 00:06:07.400 --> 00:06:08.900 "Hey, I'm solving this problem 00:06:09.400 --> 00:06:12.900 and here's a set of results that show that it's useful, 00:06:12.900 --> 00:06:15.200 and here's some insight that I get." 00:06:16.200 --> 00:06:18.280 - [Isaiah] As you said, Josh, great, sort of there's been a move 00:06:18.280 --> 00:06:20.700 in the direction of thinking more about causality 00:06:20.700 --> 00:06:22.800 in economics and empirical work in economics, 00:06:22.900 --> 00:06:24.800 any consequences of sort of the-- 00:06:24.800 --> 00:06:26.570 the spread of that view that surprised you 00:06:26.570 --> 00:06:28.855 or anything that you view as downsides 00:06:28.855 --> 00:06:31.400 of sort of the way that empirical economics has gone? 00:06:31.500 --> 00:06:34.190 - [Josh] Sometimes I see, somebody does IV 00:06:34.190 --> 00:06:38.500 and they get a result which seems implausibly large. 00:06:38.800 --> 00:06:40.200 That's the usual case. 00:06:42.500 --> 00:06:45.220 So it might be, you know, an extraordinarily large 00:06:45.220 --> 00:06:48.600 causal effect of some relatively minor intervention, 00:06:49.100 --> 00:06:52.800 which was randomized or for which you could make a case 00:06:52.900 --> 00:06:54.900 that there's a good design. 00:06:54.900 --> 00:06:58.330 And then when I see that, and, you know, I think 00:06:58.900 --> 00:07:00.465 it's very hard for me to believe 00:07:00.465 --> 00:07:02.030 that this relatively minor intervention 00:07:02.030 --> 00:07:04.000 has such a large effect. 00:07:04.100 --> 00:07:06.110 The author will sometimes resort 00:07:06.110 --> 00:07:08.690 to the local average treatment effects theorem 00:07:08.690 --> 00:07:12.700 and say, "Well, these compliers, you know, they're special in some way." 00:07:13.300 --> 00:07:17.600 And, you know, they just benefit extraordinarily from this intervention. 00:07:18.100 --> 00:07:22.100 And I'm reluctant to take that at face value. I think, you know, 00:07:22.100 --> 00:07:24.100 often when effects are too big, 00:07:24.300 --> 00:07:26.900 it's because the exclusion restriction is failing. So 00:07:27.100 --> 00:07:31.700 Don't really have the right endogenous variable to scale that result. 00:07:32.000 --> 00:07:35.700 And so I'm not too happy to see 00:07:35.800 --> 00:07:38.800 you know, just sort of a generic heterogeneity 00:07:38.900 --> 00:07:43.800 argument being used to excuse something that I think might be a deeper problem. 00:07:45.300 --> 00:07:47.400 I think it played somewhat of an unfortunate roll pin. 00:07:47.400 --> 00:07:52.300 The discussions kind of between reduced form and structural approaches where 00:07:52.600 --> 00:07:54.200 I feel that wasn't quite 00:07:55.000 --> 00:07:59.300 right. The instrumental variables assumptions are 00:08:00.400 --> 00:08:05.200 at the core structural assumptions about Behavior. They were coming from economic 00:08:07.100 --> 00:08:09.900 thinking about the economic behavior of agents, 00:08:10.300 --> 00:08:15.000 and it's somehow it got pushed in a Direction. 00:08:15.100 --> 00:08:17.600 That I think wasn't really very helpful. If 00:08:18.800 --> 00:08:21.700 the way I think, initially the 00:08:22.800 --> 00:08:27.300 we wrote things up. It was it was describing what was happening, there was set of 00:08:27.500 --> 00:08:32.200 methods. People were using be clarified what those methods were doing 00:08:32.900 --> 00:08:38.500 and in a way that I think contain a fair amount of insight, 00:08:39.100 --> 00:08:45.000 but it somehow it got pushed into a corner that I think was not necessarily very 00:08:45.100 --> 00:08:48.600 or even just the language of reduced form versus structural. 00:08:48.600 --> 00:08:51.100 I find kind of funny in the sense that the right 00:08:51.100 --> 00:08:53.100 the local average treatment effect model, right? 00:08:53.100 --> 00:08:55.300 The potential outcomes model is a nonparametric. 00:08:55.300 --> 00:08:56.200 Structural model, 00:08:56.200 --> 00:08:58.600 if you want to think about it, as you sort of suggested, he does. 00:08:58.600 --> 00:08:59.400 So, there's something, 00:09:00.100 --> 00:09:03.700 there's something a little funny about putting these two things in a position when 00:09:03.800 --> 00:09:06.600 yes, well, that language, of course, comes from the area, the 00:09:06.900 --> 00:09:09.800 70s equations framework that we inherited. 00:09:10.400 --> 00:09:12.400 It has the advantage that people seem 00:09:12.400 --> 00:09:15.000 to know what you mean when you use it, but might 00:09:15.100 --> 00:09:18.200 That people are hearing different. Different people are hearing different things. 00:09:18.300 --> 00:09:20.900 Yeah. I think I think veggies Farmers had become use 00:09:20.900 --> 00:09:23.200 a little bit of the pejoratives. Okay? Yeah. 00:09:23.300 --> 00:09:28.300 The word, which is not really quite what it was originally intended for. 00:09:30.100 --> 00:09:34.100 I guess something else that strikes me in thinking about the effects of 00:09:34.100 --> 00:09:38.200 the local average treatment effect framework is that often folks will appeal to 00:09:38.200 --> 00:09:41.800 a local average, treatment effects intuition for settings. Well, beyond 00:09:42.000 --> 00:09:44.900 ones, where any sort of formal results has actually been 00:09:45.000 --> 00:09:49.700 Shhhhht. And I'm curious given all the work that you guys did to, 00:09:49.900 --> 00:09:53.200 you know, establish late results in different in different settings. I'm curious 00:09:53.300 --> 00:09:57.800 any thoughts on that. I think there's going to be a lot of cases where 00:09:57.900 --> 00:10:02.200 the intuition does get. You get you some distance, 00:10:02.800 --> 00:10:07.600 but it's going to be somewhat limited and establishing formal results. There 00:10:08.400 --> 00:10:12.700 may be a little tricky and there may be only work in special circumstances, 00:10:13.100 --> 00:10:13.500 you need. 00:10:14.600 --> 00:10:19.500 And you end up with a lot of formality that may not quite capture the intuition 00:10:19.900 --> 00:10:23.200 sometimes I'm somewhat uneasy with them and they are not necessarily the papers. 00:10:23.200 --> 00:10:25.000 I would want to ride that the 00:10:25.100 --> 00:10:30.000 but I do think something do intuition orphaned US capture part of the 00:10:30.200 --> 00:10:31.100 of the problem. 00:10:33.100 --> 00:10:36.300 I think, in some sense we were kind of very fortunate there 00:10:36.900 --> 00:10:40.500 in the way. The late paper go handle. It. Don't know if that, actually the editor, 00:10:40.600 --> 00:10:41.700 made it much shorter 00:10:42.100 --> 00:10:46.300 and that then allowed us to kind of focus on very clear, crisp results 00:10:47.100 --> 00:10:49.800 where if, you know, this, 00:10:50.000 --> 00:10:54.200 this is somewhat unfortunate tendency in the commercialization of having the papers. 00:10:54.600 --> 00:10:58.800 Well, you should be able to fix that, man. I'm trying to take some time to fix that. 00:10:59.400 --> 00:11:02.700 I think this is an example where it's sort of very clear that having it. Be sure. 00:11:02.900 --> 00:11:08.000 It's actually impose that no paper can be longer than the late paper that wow. 00:11:08.800 --> 00:11:14.300 Great. At least no Theory. No Theory Pig. Yeah, and I think, I think they're well, 00:11:14.500 --> 00:11:16.800 I'm trying very hard to get the papers to be shorter. 00:11:16.800 --> 00:11:18.700 And I think there's a lot of value 00:11:19.200 --> 00:11:22.600 today because it's often the second part of the paper that doesn't actually 00:11:23.700 --> 00:11:26.500 Get you much further and understanding things 00:11:27.000 --> 00:11:31.700 but and it does make things much harder to read and, you know, 00:11:32.400 --> 00:11:33.700 it sort of goes back to 00:11:34.200 --> 00:11:38.500 how I think he kind of a trick should be done to you should focus on the see. 00:11:38.700 --> 00:11:41.300 It should be reasonably close to empirical problems. 00:11:41.500 --> 00:11:43.900 They should be very clear problems. 00:11:44.800 --> 00:11:48.900 But then often the the theory doesn't need to be quite so long. 00:11:49.000 --> 00:11:49.300 Yeah, 00:11:51.100 --> 00:11:53.400 I think they had things have 00:11:53.600 --> 00:11:54.700 On a little off track. 00:11:56.400 --> 00:11:58.400 The relatively recent change has been a 00:11:58.500 --> 00:12:02.200 seeming big increase in demand for people with sort of econometrics. 00:12:02.200 --> 00:12:04.800 Causal effect, estimation skills in the tech sector. 00:12:05.000 --> 00:12:09.000 I'm interested either of you have thoughts on sort of how that's gonna 00:12:09.200 --> 00:12:11.600 how that's going to interact with the development of empirical methods, 00:12:11.600 --> 00:12:14.300 or Empirical research, and economics. Going forward, sort of 00:12:14.600 --> 00:12:21.000 whether sort of a meta point, which is there's this new kind of employer 00:12:21.800 --> 00:12:26.000 the Amazons and the Uber and, you know, 00:12:26.200 --> 00:12:27.600 Riser world 00:12:28.000 --> 00:12:29.300 and I think that's great. 00:12:29.300 --> 00:12:33.200 And I'd like to tell my students about that, you know, especially at MIT. 00:12:33.200 --> 00:12:37.000 We have a lot of computer science Majors. That's our biggest major 00:12:37.400 --> 00:12:42.800 and I try to seduce some of those folks into economics by saying, you know, 00:12:43.200 --> 00:12:45.700 you can go work for these, 00:12:45.800 --> 00:12:48.400 you know companies that people are very keen to 00:12:48.700 --> 00:12:50.800 work for because the work seems exciting, 00:12:52.000 --> 00:12:56.000 you know that the skills that you get in econometrics are are as good or better. 00:12:56.100 --> 00:13:01.100 Better than than any competing discipline has to offer. So you should at least 00:13:01.400 --> 00:13:04.200 take some econ, take some econometrics. And some econ. 00:13:04.800 --> 00:13:07.000 I did a fun project with a uber 00:13:07.600 --> 00:13:12.900 on labor supply of Uber drivers and was very, very exciting to be part of that. 00:13:13.100 --> 00:13:15.400 Plus. I got to drive for Uber for a while 00:13:15.900 --> 00:13:20.700 and I thought that was fun tonight. I did not make enough that I was attempted to 00:13:21.100 --> 00:13:25.100 give up by a mighty job, but I enjoyed the experience. 00:13:25.300 --> 00:13:26.000 I see a 00:13:26.200 --> 00:13:30.900 Cho challenge to our model of graduate education here, 00:13:31.700 --> 00:13:37.400 which is if we're trading people to go work at Amazon, you know, 00:13:37.900 --> 00:13:42.900 it's not clear. Why? You know, we should be paying graduate stipends for that. 00:13:43.200 --> 00:13:45.400 Why should the taxpayer effectively 00:13:46.100 --> 00:13:51.400 be subsidizing? That our graduate education in the u.s. Is generously subsidized? 00:13:51.400 --> 00:13:56.000 Even in private universities. It's ultimately there's a lot of public money. 00:13:56.100 --> 00:13:59.300 Me there. And I think the traditional rationale for that is, 00:13:59.500 --> 00:14:03.900 you know, we were training, Educators and Scholars, and there's a great externality 00:14:04.300 --> 00:14:05.700 from the work that we do. 00:14:05.700 --> 00:14:09.600 It's either the research externality, or a teaching externality. 00:14:10.100 --> 00:14:14.600 But, you know, if many of our students are going to work in the private sector, 00:14:16.300 --> 00:14:21.700 that's fine, but that maybe their employers should pay for that. 00:14:22.300 --> 00:14:25.100 He says, so different from people working for a Consulting. 00:14:26.300 --> 00:14:26.900 Trust me. 00:14:27.200 --> 00:14:33.000 It's not clear to me that the number of jobs in academics has changed. 00:14:33.100 --> 00:14:37.600 It's just, I feel like this is a growing sector whereas Consulting, 00:14:37.700 --> 00:14:42.100 your right to raise that, it might be the same for for Consulting. 00:14:43.300 --> 00:14:44.400 But this, 00:14:44.500 --> 00:14:47.500 you know, I'm placing more and more students in these businesses. 00:14:47.500 --> 00:14:50.400 So, it's on my mind in a way that I've sort of, 00:14:50.800 --> 00:14:55.500 you know, not been attentive to consulting jobs, you know, Consulting was always, 00:14:55.600 --> 00:15:00.400 It's important and I think they'll so there's some movement from Consulting back 00:15:00.400 --> 00:15:02.600 into research. It's a little more fluid. 00:15:02.900 --> 00:15:03.500 The, 00:15:03.900 --> 00:15:05.400 a lot of the work in the 00:15:06.400 --> 00:15:09.800 in both domains. I have to say, it's not really different but 00:15:10.100 --> 00:15:13.700 you know, people who are working in the tech sector are doing things 00:15:13.700 --> 00:15:16.800 that are potentially of scientific interest, but mostly it's hidden. 00:15:17.100 --> 00:15:20.900 Then you really I have to say, you know, why, why is the government paying for this? 00:15:21.800 --> 00:15:25.500 Yeah, although yeah, I mean taquitos point, I guess it. There's a, there's a data. 00:15:25.600 --> 00:15:30.000 Question here of it has the sort of total nanak. It sort of say 00:15:30.500 --> 00:15:30.900 private 00:15:31.300 --> 00:15:34.100 for-profit sector employment of econ Ph.D. 00:15:34.100 --> 00:15:37.700 Program graduates increased or has it just been a substitution from 00:15:37.900 --> 00:15:40.200 finance and Consulting towards tack. 00:15:40.300 --> 00:15:44.300 I may be a reaction to something that's not really happening 00:15:44.400 --> 00:15:48.200 so bad. I've actually done some work with some of these tech companies. 00:15:49.100 --> 00:15:52.300 So I don't disagree with Justice point that we need to think 00:15:52.300 --> 00:15:55.100 a little bit about the funding model whose it was in the end paying for the 00:15:55.600 --> 00:15:59.400 It education. But from a scientific perspective. 00:16:00.100 --> 00:16:03.500 The only do these places have have great data and nowadays. 00:16:03.500 --> 00:16:07.100 They tend to be very careful with that for privacy reasons, 00:16:07.500 --> 00:16:08.800 but also have great questions. 00:16:10.200 --> 00:16:11.200 I find it very 00:16:11.600 --> 00:16:13.300 inspiring kind of to listen to 00:16:13.300 --> 00:16:15.800 the people there and kind of see what kind of questions they have 00:16:15.900 --> 00:16:17.300 and often their questions. 00:16:18.200 --> 00:16:20.600 That also come up outside of these. 00:16:20.700 --> 00:16:25.400 These companies have a couple of papers with the rights in the chat. 00:16:25.800 --> 00:16:26.300 And then 00:16:26.500 --> 00:16:31.600 as soon as an atheist kind of where we look at ways of combining experimental data 00:16:31.700 --> 00:16:34.100 and observational data, and can it there. 00:16:35.500 --> 00:16:38.600 Rights Chetty was interested in what is the effect 00:16:38.700 --> 00:16:44.600 of Early Childhood programs on outcomes later in life? Not just kind of test scores, 00:16:44.600 --> 00:16:48.300 but on earnings and stuff, and we cannot be developed methods 00:16:48.600 --> 00:16:51.500 that would help you shed light on that, on the some, 00:16:52.700 --> 00:16:55.000 in some settings and the same problems. 00:16:56.300 --> 00:17:00.500 Came up kind of in this tech company settings. 00:17:00.800 --> 00:17:03.700 And so for my perspective, it's 00:17:04.400 --> 00:17:07.500 the same kind of a stocking two people doing a protocol work. 00:17:07.600 --> 00:17:11.800 I tried to kind of look at these specific problems and then try to come up 00:17:11.900 --> 00:17:18.300 with more General problems that we formulating the problems at a higher level. 00:17:18.500 --> 00:17:22.900 So that I can think about solutions that work in a range of settings. 00:17:23.400 --> 00:17:25.500 And so from that perspective, the 00:17:25.700 --> 00:17:30.300 His with the the tech companies I just very valuable and very useful. 00:17:30.900 --> 00:17:31.400 It's know. 00:17:31.700 --> 00:17:33.700 We do have students. Now spent 00:17:33.800 --> 00:17:37.200 doing internships there and then coming back and writing 00:17:37.400 --> 00:17:43.400 more interesting thesis, as a result of their experiences there. 00:17:44.600 --> 00:17:47.800 If you'd like to watch more Nobel conversations, click here, 00:17:48.200 --> 00:17:50.500 or if you'd like to learn more about econometrics, 00:17:50.600 --> 00:17:53.200 check out Josh's mastering econometrics series. 00:17:53.700 --> 00:17:55.500 If you'd like to learn more about he do. 00:17:55.600 --> 00:17:58.300 Josh and Isaiah check out the links in the description.