WEBVTT 00:00:00.570 --> 00:00:04.777 I work with a bunch of mathematicians, philosophers and computer scientists 00:00:04.777 --> 00:00:09.986 and we sit around and think about the future of machine intelligence, 00:00:09.986 --> 00:00:12.200 among other things. 00:00:12.200 --> 00:00:16.755 Some people think that some of these people are science fiction-y 00:00:16.755 --> 00:00:19.856 far out there, crazy. 00:00:19.856 --> 00:00:21.326 But I like to say, 00:00:21.326 --> 00:00:24.930 "Okay, let's look at the modern human condition." 00:00:24.930 --> 00:00:26.622 (Laughter) 00:00:26.622 --> 00:00:29.024 This is the normal way for things to be. NOTE Paragraph 00:00:29.024 --> 00:00:31.309 But, if we think about it, 00:00:31.309 --> 00:00:34.602 we are actually recently arrived guests on this planet. 00:00:34.602 --> 00:00:36.214 The human species -- 00:00:36.214 --> 00:00:41.430 think of if earth was created one year ago, 00:00:41.430 --> 00:00:44.978 the human species, then, would be 10-minutes-old. 00:00:44.978 --> 00:00:48.146 The industrial era started two seconds ago. 00:00:48.146 --> 00:00:50.551 Another way to think of this, 00:00:50.551 --> 00:00:54.443 if you think of world GDP over the last 10,000 years, 00:00:54.443 --> 00:00:57.530 I've actually taken the trouble to plot this for you in a graph. 00:00:57.530 --> 00:00:59.304 It looks like this. 00:00:59.304 --> 00:01:00.667 (Laughter) 00:01:00.667 --> 00:01:02.818 It's a curious shape for a normal condition. 00:01:02.818 --> 00:01:04.516 I sure wouldn't want to sit on it. 00:01:04.516 --> 00:01:07.067 (Laughter) NOTE Paragraph 00:01:07.067 --> 00:01:08.681 Let's ask ourselves, 00:01:08.681 --> 00:01:12.122 what is the cost of this current anomaly? 00:01:12.122 --> 00:01:14.393 Some people would say it's technology. 00:01:14.393 --> 00:01:15.851 Now it's true, 00:01:15.851 --> 00:01:19.484 technology has accumulated through human history, 00:01:19.484 --> 00:01:23.713 and right now, technology advances extremely rapidly, 00:01:23.713 --> 00:01:25.278 that is the proximate cause, 00:01:25.278 --> 00:01:28.473 that's why we are currently so very productive. 00:01:28.473 --> 00:01:32.874 But I like to think back further to the ultimate cause. NOTE Paragraph 00:01:32.874 --> 00:01:36.880 Look at these two highly distinguished gentlemen: 00:01:36.880 --> 00:01:38.623 We have Kanzi, 00:01:38.623 --> 00:01:41.260 he's mastered 200 lexical tokens, 00:01:41.260 --> 00:01:43.123 an incredible feat. 00:01:43.123 --> 00:01:46.817 And Ed Witten unleashed the second super string revolution. 00:01:46.817 --> 00:01:49.141 If we look under the hood, this is what we find: 00:01:49.141 --> 00:01:50.711 basically the same thing. 00:01:50.711 --> 00:01:52.524 One is a little larger, 00:01:52.524 --> 00:01:55.282 it maybe also has a few tricks in the exact way it's wired. 00:01:55.282 --> 00:01:59.094 These invisible differences cannot be too complicated, however, 00:01:59.094 --> 00:02:03.379 because they've only been 250,000 generations since 00:02:03.379 --> 00:02:05.111 our last common ancestor. 00:02:05.111 --> 00:02:10.000 We know that complicated mechanisms that a long time to evolve. 00:02:10.000 --> 00:02:12.499 So a bunch of relatively minor changes 00:02:12.499 --> 00:02:15.566 take us from Kanzi to Witten. 00:02:15.566 --> 00:02:17.377 From broken-off tree branches, 00:02:17.377 --> 00:02:20.829 to intercontinental balistic missles. NOTE Paragraph 00:02:20.829 --> 00:02:22.724 So this then seems pretty obvious that 00:02:22.724 --> 00:02:24.954 everything we've achieved, pretty much, 00:02:24.954 --> 00:02:27.212 and everything we care about depends crucially 00:02:27.212 --> 00:02:32.650 on relatively some minor changes that made the human mind. 00:02:32.650 --> 00:02:36.312 And the collaraly, of course, is that any further changes 00:02:36.312 --> 00:02:39.789 that could significantly change the substrate of thinking 00:02:39.789 --> 00:02:43.651 could have potentially enormous consequences. NOTE Paragraph 00:02:43.651 --> 00:02:46.576 Some of my colleagues think we're on the verge 00:02:46.576 --> 00:02:49.756 of something that could cause a profound change 00:02:49.756 --> 00:02:51.384 in that substrate, 00:02:51.384 --> 00:02:54.347 and that is machine super intelligence. 00:02:54.347 --> 00:02:59.086 Artificial intelligence used to be about putting commands in a box. 00:02:59.086 --> 00:03:04.295 You would have human programmers that would painstakingly handcraft items, 00:03:04.295 --> 00:03:06.262 You build up these expert systems, 00:03:06.262 --> 00:03:07.916 and they were kind of useful for some purposes, 00:03:07.916 --> 00:03:09.857 but they were very brittle, 00:03:09.867 --> 00:03:11.023 you couldn't scale them. 00:03:11.023 --> 00:03:14.410 Basically, you got out only what you put in. 00:03:14.410 --> 00:03:16.967 But since then, a paradigm shift has taken place 00:03:16.967 --> 00:03:18.874 in the field of artificial intelligence. NOTE Paragraph 00:03:18.874 --> 00:03:21.894 Today, the action is really around machine learning. 00:03:21.894 --> 00:03:28.061 So rather than handcrafting knowledge representations and features, 00:03:28.061 --> 00:03:31.709 we create algorithms that learn, 00:03:31.709 --> 00:03:34.275 often from raw perceptual data. 00:03:34.275 --> 00:03:39.063 Basically the same thing that the human infant does. 00:03:39.063 --> 00:03:43.270 The result is AI that is not limited to one domain, 00:03:43.270 --> 00:03:47.901 the same system can learn to translate between any pairs of languages, 00:03:47.901 --> 00:03:53.068 or learn to play any computer game at the Atari console. 00:03:53.068 --> 00:03:56.757 Now of course, AI is still no where near having 00:03:56.757 --> 00:04:00.701 the same powerful, cross-domain ability to learn and plan 00:04:00.701 --> 00:04:02.335 as a human being has. 00:04:02.335 --> 00:04:04.461 The cortex still has some algorithmic tricks 00:04:04.461 --> 00:04:07.886 that we don't yet know how to match in machines. NOTE Paragraph 00:04:07.886 --> 00:04:09.785 But so the question is, 00:04:09.785 --> 00:04:13.885 how far are we from being able to match those tricks? 00:04:13.885 --> 00:04:15.798 A couple of years ago, we did a survey 00:04:15.798 --> 00:04:17.927 of some of the world's leading AI experts 00:04:17.927 --> 00:04:19.136 to see what they think 00:04:19.136 --> 00:04:21.440 and one of the questions we asked was, 00:04:21.440 --> 00:04:24.793 "By which year do you think there is a 50 percent probability 00:04:24.793 --> 00:04:28.785 that we will have achieved human-level machine intelligence?" 00:04:28.785 --> 00:04:31.898 We defined human-level here as the ability to perform 00:04:31.898 --> 00:04:35.839 almost any job at least as well as an adult human, 00:04:35.839 --> 00:04:39.844 so real human-level, not just within some limited domain. 00:04:39.844 --> 00:04:43.494 And the median answer was 2040 or 2050, 00:04:43.494 --> 00:04:46.300 depending on precisely which group of experts we asked. 00:04:46.300 --> 00:04:49.229 Now, it could happen much, much later, 00:04:49.229 --> 00:04:52.279 or sooner, the truth is nobody really knows. NOTE Paragraph 00:04:52.279 --> 00:04:55.941 What we do know is that the ultimate limit 00:04:55.941 --> 00:04:58.802 to information processing in machine substrate, 00:04:58.802 --> 00:05:03.241 lie far outside the limits in biological tissue. 00:05:03.241 --> 00:05:05.619 This comes down to physics. 00:05:05.619 --> 00:05:10.337 A biological neuron fires, maybe, at 200 Hertz, 200 times a second. 00:05:10.337 --> 00:05:13.931 But even a present-day transistor operates at a gigahert. 00:05:13.931 --> 00:05:16.640 Neurons propagate slowly in axons, 00:05:16.640 --> 00:05:19.568 100 meters per second, tops. 00:05:19.568 --> 00:05:23.079 But in computers, signals can travel at the speed of light. 00:05:23.079 --> 00:05:24.948 There's also size limitations, 00:05:24.948 --> 00:05:27.975 a human brain has to fit inside a cranium, 00:05:27.975 --> 00:05:32.736 but a computer can be the size of a warehouse or larger. 00:05:32.736 --> 00:05:38.335 So the potential of super intelligence lies dormant in matter, 00:05:38.335 --> 00:05:44.047 much like the power of the atom lay dormant throughout human history, 00:05:44.047 --> 00:05:48.452 patiently waiting there until 1945. 00:05:48.452 --> 00:05:50.920 In this century, scientists may learn to awaken 00:05:50.920 --> 00:05:53.818 the power of artificial intelligence. 00:05:53.818 --> 00:05:58.406 And I think we might then see an intelligence explosion. NOTE Paragraph 00:05:58.406 --> 00:06:02.363 Now most people, when they think about what is smart and what is dumb, 00:06:02.363 --> 00:06:05.386 I think I have in mind a picture roughly like this. 00:06:05.386 --> 00:06:07.984 So at one hand, we have the village idiot, 00:06:07.984 --> 00:06:10.448 and then far over at the other side, 00:06:10.448 --> 00:06:11.857 we have Ed Witten, 00:06:11.857 --> 00:06:15.573 or Albert Einsten or whoever your favorite guru is. 00:06:15.573 --> 00:06:19.057 But I think that from the point of view of artificial intelligence, 00:06:19.057 --> 00:06:23.258 the true picture is actually probably more like this: 00:06:23.258 --> 00:06:26.636 AI starts out at this point here, at zero intelligence, 00:06:26.636 --> 00:06:29.647 and then, after many, many years of really hard work, 00:06:29.647 --> 00:06:33.491 maybe eventually we get to mouse-level artificial intelligence, 00:06:33.491 --> 00:06:35.921 something that can navigate cluttered environments 00:06:35.921 --> 00:06:37.908 as well as a mouse can. 00:06:37.908 --> 00:06:42.221 And then, after many, many more years of really hard work, lots of investment, 00:06:42.221 --> 00:06:46.860 maybe eventually we get to chimpanzee-level artificial intelligence. 00:06:46.860 --> 00:06:50.070 And then, after even more years of really, really hard work, 00:06:50.070 --> 00:06:52.983 we get village idiot artificial intelligence. 00:06:52.983 --> 00:06:56.255 And a few moments later, we are beyond Ed Witten. 00:06:56.255 --> 00:06:59.225 The train doesn't stop at Human-ville Station. 00:06:59.225 --> 00:07:02.247 It's likely, rather, to swoosh right by. NOTE Paragraph 00:07:02.247 --> 00:07:04.231 Now this has profound implications, 00:07:04.231 --> 00:07:08.093 particularly when it comes to questions of power. 00:07:08.093 --> 00:07:09.992 For example, chimpanzees are strong, 00:07:09.992 --> 00:07:15.214 pound for pound, a chimpanzee is about twice as strong as a fit human male. 00:07:15.214 --> 00:07:19.828 And yet, the fate of Kanzi and his pals depends a lot more 00:07:19.828 --> 00:07:24.258 on what we humans do than on what the chimpanzees do themselves. 00:07:24.258 --> 00:07:27.542 Once there is super intelligence, 00:07:27.542 --> 00:07:32.041 the fate of humanity may depend on what the super intelligence does. 00:07:32.041 --> 00:07:36.688 Think about it: machine intelligence is the last invention 00:07:36.688 --> 00:07:38.552 that humanity will ever need to make. 00:07:38.552 --> 00:07:41.525 Machines will then be better at inventing than we are, 00:07:41.525 --> 00:07:44.065 and they'll be doing so on digital timescales. 00:07:44.065 --> 00:07:48.966 What this means is basically a telescoping of the future. 00:07:48.966 --> 00:07:52.524 Think of all the crazy technologies that you could have imagined 00:07:52.524 --> 00:07:55.322 maybe humans could have developed in the fullness of time: 00:07:55.322 --> 00:07:58.580 cures for aging, space colonization, 00:07:58.580 --> 00:08:00.421 self-replicating nanobots 00:08:00.421 --> 00:08:02.301 or uploading of minds into computers, 00:08:02.301 --> 00:08:04.470 all kinds of science fiction-y stuff 00:08:04.470 --> 00:08:07.207 that's nevertheless consistent with the laws of physics. 00:08:07.207 --> 00:08:09.639 All of this, super intelligence could develop 00:08:09.639 --> 00:08:12.449 and possibly, quite rapidly. NOTE Paragraph 00:08:12.449 --> 00:08:16.007 Now, super intelligence with such technological maturity 00:08:16.007 --> 00:08:18.186 would be extremely powerful, 00:08:18.186 --> 00:08:19.982 and at least in some scenarios, 00:08:19.982 --> 00:08:22.624 it would be able to get what it wants. 00:08:22.624 --> 00:08:25.073 We would then have a future that would be shaped 00:08:25.073 --> 00:08:28.375 by the preferences of this AI. 00:08:29.855 --> 00:08:34.244 Now a good question is, what are those preferences? 00:08:34.244 --> 00:08:36.013 Here it gets trickier. 00:08:36.013 --> 00:08:37.448 To make any headway with this, 00:08:37.448 --> 00:08:39.364 we must first, first of all, 00:08:39.364 --> 00:08:41.276 avoid anthropomorphizing. 00:08:41.276 --> 00:08:45.385 And this is ironic because every newspaper article 00:08:45.385 --> 00:08:50.250 about the future of AI has a picture of this: 00:08:50.250 --> 00:08:52.424 So I think what we need to do is to conceive 00:08:52.424 --> 00:08:54.840 of the issue more abstractly, 00:08:54.840 --> 00:08:57.204 not in terms of vivid Hollywood scenarios. NOTE Paragraph 00:08:57.204 --> 00:09:00.821 We need to think of intelligence as an optimization process, 00:09:00.821 --> 00:09:06.488 a process that steers the future into a particular set of configurations. 00:09:06.488 --> 00:09:08.130 As super intelligence -- 00:09:08.130 --> 00:09:09.981 it's a really strong optimization process. 00:09:09.981 --> 00:09:12.858 It's extremely good at using available means 00:09:12.858 --> 00:09:16.007 to achieve a state in which its goal is realized. 00:09:16.007 --> 00:09:18.769 This means that there is no necessary conenction between 00:09:18.769 --> 00:09:21.853 being highly intelligent in this sense, 00:09:21.853 --> 00:09:24.125 and having an objective that we humans 00:09:24.125 --> 00:09:27.321 would find worthwhile or meaningful. NOTE Paragraph 00:09:27.321 --> 00:09:31.115 Suppose we give AI the goal to make humans smile. 00:09:31.115 --> 00:09:34.097 When the AI is weak, it performs useful or amusing actions 00:09:34.097 --> 00:09:35.944 that cause its user to smile. 00:09:35.944 --> 00:09:39.031 When the AI becomes super intelligent, 00:09:39.031 --> 00:09:41.284 it realizes that there is a more effective way 00:09:41.284 --> 00:09:42.721 to achieve this goal: 00:09:42.721 --> 00:09:44.476 take control of the world 00:09:44.476 --> 00:09:47.638 and stick electrodes into the facial muscles of humans 00:09:47.638 --> 00:09:50.579 to cause constant, beaming grins. 00:09:50.579 --> 00:09:53.124 Another example, suppose we give AI the goal to solve 00:09:53.124 --> 00:09:54.787 a difficult mathematical problem. 00:09:54.787 --> 00:09:56.764 When the AI becomes super intelligent, 00:09:56.764 --> 00:10:01.105 it realizes that the most effective way to get the solution to this problem 00:10:01.105 --> 00:10:04.035 is by transforming the planet into a giant computer, 00:10:04.035 --> 00:10:06.281 so as to increase its thinking capacity. 00:10:06.281 --> 00:10:09.045 And notice that this gives the AIs an instrumental reason 00:10:09.045 --> 00:10:11.561 to do things to us that we might not approve of. 00:10:11.561 --> 00:10:13.496 Human beings in this model are threats, 00:10:13.496 --> 00:10:16.417 we could prevent the mathematical problem from being solved. NOTE Paragraph 00:10:16.417 --> 00:10:20.161 Of course, perceivably things won't go wrong in these particular ways, 00:10:20.161 --> 00:10:22.454 these are cartoon examples. 00:10:22.454 --> 00:10:24.393 But the general point here is important: 00:10:24.393 --> 00:10:27.266 if you create a really powerful optimization process 00:10:27.266 --> 00:10:29.500 to maximize for objective x, 00:10:29.500 --> 00:10:31.776 you better make sure that your definition of x 00:10:31.776 --> 00:10:34.835 incorporates everything you care about. 00:10:34.835 --> 00:10:39.219 This is a lesson that's also taught in many a myth. 00:10:39.219 --> 00:10:44.517 Kind Midas wishes that everything he touches be turned into gold. 00:10:44.517 --> 00:10:47.378 He touches his daughter, she turns into fold. 00:10:47.378 --> 00:10:49.931 He touches his food, it turns into gold. 00:10:49.931 --> 00:10:52.520 This could become practically relevant, 00:10:52.520 --> 00:10:54.590 not just for a metaphor for greed, 00:10:54.590 --> 00:10:57.075 but an illustration of what happens if you create 00:10:57.075 --> 00:10:59.322 a powerful optimization process 00:10:59.322 --> 00:11:04.111 and give it misconceived or poorly specified goals. NOTE Paragraph 00:11:04.111 --> 00:11:09.300 Now you might say, "If a computer starts sticking electrodes into people's faces, 00:11:09.300 --> 00:11:12.555 we'd just shut it off." 00:11:12.555 --> 00:11:16.690 A: This is not necessarily so easy to do if we've grown 00:11:16.690 --> 00:11:18.185 dependent on the system, 00:11:18.185 --> 00:11:20.627 like where is the off switch to the internet? 00:11:20.627 --> 00:11:25.747 B: Why haven't the chimpanzees flicked the off-switch to humanity, 00:11:25.747 --> 00:11:27.298 or the neanderthals? 00:11:27.298 --> 00:11:29.964 They certainly had reasons. 00:11:29.964 --> 00:11:32.759 We have an off switch, for example, right here. 00:11:32.759 --> 00:11:34.813 [choking sound] 00:11:34.813 --> 00:11:37.238 The reason is that we are an intelligent adversary, 00:11:37.238 --> 00:11:39.966 we can anticipate threats and we can plan around them. 00:11:39.966 --> 00:11:42.470 But so could a super intelligent agent, 00:11:42.470 --> 00:11:45.724 and it would be much better at that than we are. 00:11:45.724 --> 00:11:52.911 The point is, we should not be confident that we have this under control here. NOTE Paragraph 00:11:52.911 --> 00:11:56.358 And we could try to make our job a little bit easier by, say, 00:11:56.358 --> 00:11:57.948 putting the AI in a box, 00:11:57.948 --> 00:12:01.034 like a secure software environment, a virtual reality simulation 00:12:01.034 --> 00:12:02.766 from which it cannot escape. 00:12:02.766 --> 00:12:06.912 But how confident can we be that the AI couldn't find a bug. 00:12:06.912 --> 00:12:10.081 Given that even human hackers find bugs all the time, 00:12:10.081 --> 00:12:14.237 I'd say, probably not very confident. 00:12:14.237 --> 00:12:18.785 So we disconnect the ethernet cable to create an air gap, 00:12:18.785 --> 00:12:23.823 but again, like nearly human hackers routinely transgress air gaps 00:12:23.823 --> 00:12:25.024 using social engineering. 00:12:25.034 --> 00:12:27.383 Like right now as I speak, I'm sure there is some employee 00:12:27.383 --> 00:12:30.546 out there somewhere who's been talked into handing out 00:12:30.546 --> 00:12:34.543 her account details by somebody claiming to be from the IT department. NOTE Paragraph 00:12:34.543 --> 00:12:36.701 More creative scenarios are also possible, 00:12:36.701 --> 00:12:40.166 like if you're the AI, you can imagine wiggling electroces around 00:12:40.166 --> 00:12:42.588 in your internal circuitry to create radio waves 00:12:42.588 --> 00:12:45.010 that you can use to communicate. 00:12:45.010 --> 00:12:47.434 Or maybe you could pretend to malfunction, 00:12:47.434 --> 00:12:50.741 and then when the programmers open you up to see what went wrong with you, 00:12:50.741 --> 00:12:52.517 they look at the source code -- BAM! -- 00:12:52.517 --> 00:12:55.314 the manipulation can take place. 00:12:55.314 --> 00:12:58.744 Or it could output the blueprint to a really nifty technology 00:12:58.744 --> 00:13:00.142 and when we implement it, 00:13:00.142 --> 00:13:04.539 it has some surreptitious side effect that the AI had planned. 00:13:04.539 --> 00:13:08.002 The point here is that we should not be confident in our ability 00:13:08.002 --> 00:13:11.810 to keep a super intelligent genie locked up in its bottle forever. 00:13:11.810 --> 00:13:15.034 Sooner or later, it will out. NOTE Paragraph 00:13:15.034 --> 00:13:18.137 I believe that the answer here is to figure out 00:13:18.137 --> 00:13:23.161 how to create super intelligent AI such that even if, when it escapes, 00:13:23.161 --> 00:13:26.438 it is still safe because it is fundamentally on our side 00:13:26.438 --> 00:13:28.337 because it shares our values. 00:13:28.337 --> 00:13:32.557 I see no way around this difficult problem. NOTE Paragraph 00:13:32.557 --> 00:13:36.391 Now, I'm actually fairly optimistic that this problem can be solved. 00:13:36.391 --> 00:13:40.294 We wouldn't have to write down a long list of everything we care aobut 00:13:40.294 --> 00:13:43.937 or worse yet, spell it out in some computer language 00:13:43.937 --> 00:13:45.391 like C ++ or Python, 00:13:45.391 --> 00:13:48.158 that would be a task beyond hopeless. 00:13:48.158 --> 00:13:52.455 Instead, we would create an AI that uses its intelligence 00:13:52.455 --> 00:13:55.226 to learn what we value, 00:13:55.226 --> 00:14:00.506 and its motivation system is constructed in such a way that it is motivated 00:14:00.506 --> 00:14:05.738 to pursue our values or to perform actions that it predicts we would approve of. 00:14:05.738 --> 00:14:08.712 We would thus leverage its intelligence as much as possible 00:14:08.712 --> 00:14:12.727 to solve the problem of value -loading. NOTE Paragraph 00:14:12.727 --> 00:14:14.239 This can happen, 00:14:14.239 --> 00:14:17.835 and the outcome could be very good for humanity. 00:14:17.835 --> 00:14:21.792 But it doesn't happen automatically. 00:14:21.792 --> 00:14:24.790 The initial conditions for the intelligent explosion 00:14:24.790 --> 00:14:27.653 might need to be set up in just the right way 00:14:27.653 --> 00:14:31.183 if we are to have a controlled detonation. 00:14:31.183 --> 00:14:33.801 The values that the AI has need to match ours, 00:14:33.801 --> 00:14:35.621 not just in the familiar context, 00:14:35.621 --> 00:14:37.999 like where we can easily check how the AI behaves, 00:14:37.999 --> 00:14:41.233 but also in all novel contexts that the AI might encounter 00:14:41.233 --> 00:14:42.790 in the indefinite future. NOTE Paragraph 00:14:42.790 --> 00:14:47.527 And there are also some esoteric issues that would need to be solved, sorted out 00:14:47.527 --> 00:14:49.616 the exact decisions of its decision theory, 00:14:49.616 --> 00:14:53.330 how to deal with logical uncertainty and so forth. 00:14:53.330 --> 00:14:56.522 So the technical problems that need to be solved to make this work 00:14:56.522 --> 00:14:57.515 look quite difficult, 00:14:57.515 --> 00:15:00.925 -- not as difficult as making a super intelligent AI, 00:15:00.925 --> 00:15:03.793 but fairly difficult. 00:15:03.793 --> 00:15:05.488 Here is the worry: 00:15:05.488 --> 00:15:10.172 making super intelligent AI is a really hard challenge. 00:15:10.172 --> 00:15:12.720 Making super intelligent AI that is safe 00:15:12.720 --> 00:15:15.426 involves some additional challenge on top of that. 00:15:15.426 --> 00:15:18.133 The risk is that if somebody figures out how to crack 00:15:18.133 --> 00:15:21.392 the first challenge without also having cracked 00:15:21.392 --> 00:15:25.402 the additional challenge of ensuring perfect safety. NOTE Paragraph 00:15:25.402 --> 00:15:28.706 So I think that we should work out a solution 00:15:28.706 --> 00:15:31.528 to the controlled problem in advance, 00:15:31.528 --> 00:15:34.608 so that we have it available by the time it is needed. 00:15:34.608 --> 00:15:37.875 Now it might be that we cannot solve the entire controlled problem 00:15:37.875 --> 00:15:41.299 in advance because maybe some element can only be put in place 00:15:41.299 --> 00:15:43.576 once you know the details of the architecture 00:15:43.576 --> 00:15:45.053 where it will be implemented. 00:15:45.053 --> 00:15:48.676 But the more of the controlled problem that we solve in advance, 00:15:48.676 --> 00:15:52.766 the better the odds that the transition to the machine intelligence era 00:15:52.766 --> 00:15:54.536 will go well. NOTE Paragraph 00:15:54.536 --> 00:15:58.950 This to me looks like a thing that is well worth doing 00:15:58.950 --> 00:16:02.282 and I can imagine that if things turn out okay, 00:16:02.282 --> 00:16:05.430 that people in a million years from now 00:16:05.430 --> 00:16:06.858 look back at this century 00:16:06.858 --> 00:16:08.972 and it might well be that they say 00:16:08.972 --> 00:16:11.119 that he one thing we did that really mattered 00:16:11.119 --> 00:16:13.037 was to get this thing right. NOTE Paragraph 00:16:13.037 --> 00:16:14.198 Thank you. NOTE Paragraph 00:16:14.198 --> 00:16:17.011 (Applause)