[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.57,0:00:04.78,Default,,0000,0000,0000,,I work with a bunch of mathematicians,\Nphilosophers and computer scientists
Dialogue: 0,0:00:04.78,0:00:09.99,Default,,0000,0000,0000,,and we sit around and think about\Nthe future of machine intelligence,
Dialogue: 0,0:00:09.99,0:00:12.20,Default,,0000,0000,0000,,among other things.
Dialogue: 0,0:00:12.20,0:00:16.76,Default,,0000,0000,0000,,Some people think that some of\Nthese people are science fiction-y
Dialogue: 0,0:00:16.76,0:00:19.86,Default,,0000,0000,0000,,far out there, crazy.
Dialogue: 0,0:00:19.86,0:00:21.33,Default,,0000,0000,0000,,But I like to say,
Dialogue: 0,0:00:21.33,0:00:24.93,Default,,0000,0000,0000,,"Okay, let's look at the modern\Nhuman condition."
Dialogue: 0,0:00:24.93,0:00:26.62,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:00:26.62,0:00:29.02,Default,,0000,0000,0000,,This is the normal way for things to be.
Dialogue: 0,0:00:29.02,0:00:31.31,Default,,0000,0000,0000,,But, if we think about it,
Dialogue: 0,0:00:31.31,0:00:34.60,Default,,0000,0000,0000,,we are actually recently arrived\Nguests on this planet.
Dialogue: 0,0:00:34.60,0:00:36.21,Default,,0000,0000,0000,,The human species --
Dialogue: 0,0:00:36.21,0:00:41.43,Default,,0000,0000,0000,,think of if earth was created\None year ago,
Dialogue: 0,0:00:41.43,0:00:44.98,Default,,0000,0000,0000,,the human species, then, \Nwould be 10-minutes-old.
Dialogue: 0,0:00:44.98,0:00:48.15,Default,,0000,0000,0000,,The industrial era started\Ntwo seconds ago.
Dialogue: 0,0:00:48.15,0:00:50.55,Default,,0000,0000,0000,,Another way to think of this,
Dialogue: 0,0:00:50.55,0:00:54.44,Default,,0000,0000,0000,,if you think of world GDP\Nover the last 10,000 years,
Dialogue: 0,0:00:54.44,0:00:57.53,Default,,0000,0000,0000,,I've actually taken the trouble\Nto plot this for you in a graph.\N
Dialogue: 0,0:00:57.53,0:00:59.30,Default,,0000,0000,0000,,It looks like this.
Dialogue: 0,0:00:59.30,0:01:00.67,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:01:00.67,0:01:02.82,Default,,0000,0000,0000,,It's a curious shape\Nfor a normal condition.
Dialogue: 0,0:01:02.82,0:01:04.52,Default,,0000,0000,0000,,I sure wouldn't want to sit on it.
Dialogue: 0,0:01:04.52,0:01:07.07,Default,,0000,0000,0000,,(Laughter)
Dialogue: 0,0:01:07.07,0:01:08.68,Default,,0000,0000,0000,,Let's ask ourselves,
Dialogue: 0,0:01:08.68,0:01:12.12,Default,,0000,0000,0000,,what is the cost of this current anomaly?
Dialogue: 0,0:01:12.12,0:01:14.39,Default,,0000,0000,0000,,Some people would say it's technology.
Dialogue: 0,0:01:14.39,0:01:15.85,Default,,0000,0000,0000,,Now it's true,
Dialogue: 0,0:01:15.85,0:01:19.48,Default,,0000,0000,0000,,technology has accumulated\Nthrough human history,
Dialogue: 0,0:01:19.48,0:01:23.71,Default,,0000,0000,0000,,and right now, technology\Nadvances extremely rapidly,
Dialogue: 0,0:01:23.71,0:01:25.28,Default,,0000,0000,0000,,that is the proximate cause,
Dialogue: 0,0:01:25.28,0:01:28.47,Default,,0000,0000,0000,,that's why we are currently \Nso very productive.
Dialogue: 0,0:01:28.47,0:01:32.87,Default,,0000,0000,0000,,But I like to think back further \Nto the ultimate cause.
Dialogue: 0,0:01:32.87,0:01:36.88,Default,,0000,0000,0000,,Look at these two \Nhighly distinguished gentlemen:
Dialogue: 0,0:01:36.88,0:01:38.62,Default,,0000,0000,0000,,We have Kanzi,
Dialogue: 0,0:01:38.62,0:01:41.26,Default,,0000,0000,0000,,he's mastered 200 lexical tokens,
Dialogue: 0,0:01:41.26,0:01:43.12,Default,,0000,0000,0000,,an incredible feat.
Dialogue: 0,0:01:43.12,0:01:46.82,Default,,0000,0000,0000,,And Ed Witten unleashed the second\Nsuper string revolution.
Dialogue: 0,0:01:46.82,0:01:49.14,Default,,0000,0000,0000,,If we look under the hood, \Nthis is what we find:
Dialogue: 0,0:01:49.14,0:01:50.71,Default,,0000,0000,0000,,basically the same thing.
Dialogue: 0,0:01:50.71,0:01:52.52,Default,,0000,0000,0000,,One is a little larger,
Dialogue: 0,0:01:52.52,0:01:55.28,Default,,0000,0000,0000,,it maybe also has a few tricks\Nin the exact way it's wired.
Dialogue: 0,0:01:55.28,0:01:59.09,Default,,0000,0000,0000,,These invisible differences cannot\Nbe too complicated, however,
Dialogue: 0,0:01:59.09,0:02:03.38,Default,,0000,0000,0000,,because they've only been\N250,000 generations since
Dialogue: 0,0:02:03.38,0:02:05.11,Default,,0000,0000,0000,,our last common ancestor.
Dialogue: 0,0:02:05.11,0:02:10.00,Default,,0000,0000,0000,,We know that complicated mechanisms\Nthat a long time to evolve.
Dialogue: 0,0:02:10.00,0:02:12.50,Default,,0000,0000,0000,,So a bunch of relatively minor changes
Dialogue: 0,0:02:12.50,0:02:15.57,Default,,0000,0000,0000,,take us from Kanzi to Witten.
Dialogue: 0,0:02:15.57,0:02:17.38,Default,,0000,0000,0000,,From broken-off tree branches,
Dialogue: 0,0:02:17.38,0:02:20.83,Default,,0000,0000,0000,,to intercontinental balistic missles.
Dialogue: 0,0:02:20.83,0:02:22.72,Default,,0000,0000,0000,,So this then seems pretty obvious that\N
Dialogue: 0,0:02:22.72,0:02:24.95,Default,,0000,0000,0000,,everything we've achieved, pretty much,
Dialogue: 0,0:02:24.95,0:02:27.21,Default,,0000,0000,0000,,and everything we care about\Ndepends crucially
Dialogue: 0,0:02:27.21,0:02:32.65,Default,,0000,0000,0000,,on relatively some minor changes \Nthat made the human mind.
Dialogue: 0,0:02:32.65,0:02:35.10,Default,,0000,0000,0000,,And the collaraly, of course, is that
Dialogue: 0,0:02:35.10,0:02:37.75,Default,,0000,0000,0000,,any further changes that could\Nsignificantly change
Dialogue: 0,0:02:37.75,0:02:39.78,Default,,0000,0000,0000,,the substrate of thinking
Dialogue: 0,0:02:39.78,0:02:43.65,Default,,0000,0000,0000,,could have potentially enormous consequences.
Dialogue: 0,0:02:43.65,0:02:46.58,Default,,0000,0000,0000,,Some of my colleagues think we're\Non the verge
Dialogue: 0,0:02:46.58,0:02:47.34,Default,,0000,0000,0000,,of something that could cause profound change
Dialogue: 0,0:02:47.34,0:02:51.38,Default,,0000,0000,0000,,in the substrate.
Dialogue: 0,0:02:51.38,0:02:54.35,Default,,0000,0000,0000,,That is, machine super intelligence.
Dialogue: 0,0:02:54.35,0:02:55.91,Default,,0000,0000,0000,,Artificual intelligence used to be
Dialogue: 0,0:02:55.91,0:02:59.06,Default,,0000,0000,0000,,putting commands in a box.
Dialogue: 0,0:02:59.06,0:03:01.50,Default,,0000,0000,0000,,You have human programmers that would
Dialogue: 0,0:03:01.50,0:03:04.92,Default,,0000,0000,0000,,painstakingly hand-craft items.
Dialogue: 0,0:03:04.92,0:03:06.40,Default,,0000,0000,0000,,You build up these expert systems,
Dialogue: 0,0:03:06.40,0:03:08.18,Default,,0000,0000,0000,,and they were kind of useful\Nfor some purposes,
Dialogue: 0,0:03:08.18,0:03:09.28,Default,,0000,0000,0000,,But they were very brittle,
Dialogue: 0,0:03:09.28,0:03:11.02,Default,,0000,0000,0000,,you couldn't scale them.
Dialogue: 0,0:03:11.02,0:03:14.41,Default,,0000,0000,0000,,Basically, you got out only\Nwhat you put in.
Dialogue: 0,0:03:14.41,0:03:17.41,Default,,0000,0000,0000,,But since then, a paradigm shift has\Ntaken place
Dialogue: 0,0:03:17.41,0:03:18.92,Default,,0000,0000,0000,,in the field of artificial intelligence.
Dialogue: 0,0:03:18.92,0:03:19.17,Default,,0000,0000,0000,,Today, the action is really around machine learning.
Dialogue: 0,0:03:19.17,0:03:28.06,Default,,0000,0000,0000,,So rather than handcrafting knowledge\Nrepresentations and features,
Dialogue: 0,0:03:28.06,0:03:31.71,Default,,0000,0000,0000,,we create algorithms that learn,
Dialogue: 0,0:03:31.71,0:03:34.28,Default,,0000,0000,0000,,often from raw perceptual data.
Dialogue: 0,0:03:34.28,0:03:39.06,Default,,0000,0000,0000,,Basically the same thing\Nthat the human infant does.
Dialogue: 0,0:03:39.06,0:03:43.27,Default,,0000,0000,0000,,The result is AI that is not limited \Nto one domain,\N
Dialogue: 0,0:03:43.27,0:03:47.90,Default,,0000,0000,0000,,the same system can learn to translate \Nbetween any pairs of languages,
Dialogue: 0,0:03:47.90,0:03:53.76,Default,,0000,0000,0000,,or learn to play any computer game\Nat the Atari console.
Dialogue: 0,0:03:53.76,0:03:59.07,Default,,0000,0000,0000,,Now of course, AI is still no where near having\Nthe same powerful, cross-domain
Dialogue: 0,0:03:59.07,0:04:02.06,Default,,0000,0000,0000,,ability to learn and plan as a human being has,
Dialogue: 0,0:04:02.06,0:04:03.97,Default,,0000,0000,0000,,the cortex still has some algorithmic tricks
Dialogue: 0,0:04:03.97,0:04:07.89,Default,,0000,0000,0000,,that we don't yet know\Nhow to match in machines.
Dialogue: 0,0:04:07.89,0:04:09.78,Default,,0000,0000,0000,,But so the question is,
Dialogue: 0,0:04:09.78,0:04:14.64,Default,,0000,0000,0000,,how far are we from being able\Nto match those tricks?
Dialogue: 0,0:04:14.64,0:04:15.80,Default,,0000,0000,0000,,A couple of years ago, we did a survey
Dialogue: 0,0:04:15.80,0:04:17.93,Default,,0000,0000,0000,,of some of the world's leading\NAI experts
Dialogue: 0,0:04:17.93,0:04:19.14,Default,,0000,0000,0000,,to see what they think
Dialogue: 0,0:04:19.14,0:04:21.44,Default,,0000,0000,0000,,and one of the questions we asked was,
Dialogue: 0,0:04:21.44,0:04:24.79,Default,,0000,0000,0000,,"By which year do you think\Nthere's a 50 percent probaility
Dialogue: 0,0:04:24.79,0:04:28.78,Default,,0000,0000,0000,,that we will have achieved \Nhuman-level machine intelligence?"
Dialogue: 0,0:04:28.78,0:04:31.64,Default,,0000,0000,0000,,We defined human-level here as the ability
Dialogue: 0,0:04:31.64,0:04:36.30,Default,,0000,0000,0000,,to perform any job at least as well\Nas an adult human level,
Dialogue: 0,0:04:36.30,0:04:39.84,Default,,0000,0000,0000,,not just within some limited domain.
Dialogue: 0,0:04:39.84,0:04:43.49,Default,,0000,0000,0000,,And the median answer was 2040 or 2050,
Dialogue: 0,0:04:43.49,0:04:46.30,Default,,0000,0000,0000,,depending on precisely which \Ngroup of experts we ask.
Dialogue: 0,0:04:46.30,0:04:49.23,Default,,0000,0000,0000,,Now, it could happen much, much later,
Dialogue: 0,0:04:49.23,0:04:52.28,Default,,0000,0000,0000,,or sooner, the truth is \Nnobody really knows.
Dialogue: 0,0:04:52.28,0:04:55.94,Default,,0000,0000,0000,,What we do know is that \Nthe ultimate limit
Dialogue: 0,0:04:55.94,0:04:58.80,Default,,0000,0000,0000,,to information processing in machine substrate,
Dialogue: 0,0:04:58.80,0:05:03.24,Default,,0000,0000,0000,,lie far outside the limits of biological tissue.
Dialogue: 0,0:05:03.24,0:05:05.62,Default,,0000,0000,0000,,This comes down to physics.
Dialogue: 0,0:05:05.62,0:05:10.34,Default,,0000,0000,0000,,A biological neuron fires, maybe, \Nat 200 Hertz, 200 times a second.
Dialogue: 0,0:05:10.34,0:05:13.93,Default,,0000,0000,0000,,But even a present-day transistor\Noperates at a giga-hert.
Dialogue: 0,0:05:13.93,0:05:16.64,Default,,0000,0000,0000,,Neurons propagate slowly in axons,
Dialogue: 0,0:05:16.64,0:05:19.57,Default,,0000,0000,0000,,100 meters per second, tops.
Dialogue: 0,0:05:19.57,0:05:23.08,Default,,0000,0000,0000,,But in computers, signals can travel\Nat the speed of light.
Dialogue: 0,0:05:23.08,0:05:24.95,Default,,0000,0000,0000,,There's also size limitations,
Dialogue: 0,0:05:24.95,0:05:27.98,Default,,0000,0000,0000,,a human brain has to fit inside a cranium,
Dialogue: 0,0:05:27.98,0:05:32.74,Default,,0000,0000,0000,,but a computer can be the size\Nof a warehouse or larger.
Dialogue: 0,0:05:32.74,0:05:38.34,Default,,0000,0000,0000,,So the potential of super intelligence \Nlies dormant in matter,
Dialogue: 0,0:05:38.34,0:05:44.05,Default,,0000,0000,0000,,much like the power of the atom lay dormant\Nthroughout human history,
Dialogue: 0,0:05:44.05,0:05:48.45,Default,,0000,0000,0000,,patiently waiting there until 1945.
Dialogue: 0,0:05:48.45,0:05:53.98,Default,,0000,0000,0000,,In this century, scientists may learn \Nto awake the power of artificial intelligence.
Dialogue: 0,0:05:53.98,0:05:58.41,Default,,0000,0000,0000,,And I think we might then see\Nan intelligence explosion.
Dialogue: 0,0:05:58.41,0:05:59.20,Default,,0000,0000,0000,,Now most people,
Dialogue: 0,0:05:59.20,0:06:02.35,Default,,0000,0000,0000,,when they think about what is smart\Nand what is dumb,
Dialogue: 0,0:06:02.35,0:06:05.39,Default,,0000,0000,0000,,I think I have in mind a picture\Nroughly like this.
Dialogue: 0,0:06:05.39,0:06:07.98,Default,,0000,0000,0000,,So at one hand, we have the village idiot,
Dialogue: 0,0:06:07.98,0:06:10.45,Default,,0000,0000,0000,,and then far over at the other side,
Dialogue: 0,0:06:10.45,0:06:11.86,Default,,0000,0000,0000,,we have Ed Witten,
Dialogue: 0,0:06:11.86,0:06:15.57,Default,,0000,0000,0000,,or Albert Einsten or whoever your favorite\Nguru is.
Dialogue: 0,0:06:15.57,0:06:19.06,Default,,0000,0000,0000,,But I think from the point of view\Nof Artificial Intelligence,
Dialogue: 0,0:06:19.06,0:06:23.26,Default,,0000,0000,0000,,the true picture is actually probably\Nmore like this:
Dialogue: 0,0:06:23.26,0:06:25.66,Default,,0000,0000,0000,,AI starts out at this point here,
Dialogue: 0,0:06:25.66,0:06:26.62,Default,,0000,0000,0000,,at zero intelligence,
Dialogue: 0,0:06:26.62,0:06:29.65,Default,,0000,0000,0000,,and then, after many, many \Nyears of really hard work,
Dialogue: 0,0:06:29.65,0:06:33.72,Default,,0000,0000,0000,,eventually we get to mouse-level\Nartificial intelligence,
Dialogue: 0,0:06:33.72,0:06:35.92,Default,,0000,0000,0000,,something that can navigate \Ncluttered environments
Dialogue: 0,0:06:35.92,0:06:37.91,Default,,0000,0000,0000,,as well as a mouse can.
Dialogue: 0,0:06:37.91,0:06:42.22,Default,,0000,0000,0000,,And then, after many, many more years\Nof really hard work, lots of investment,
Dialogue: 0,0:06:42.22,0:06:46.86,Default,,0000,0000,0000,,maybe eventually we get to \Nchimpanzee-level artificial intelligence.
Dialogue: 0,0:06:46.86,0:06:50.07,Default,,0000,0000,0000,,And then, after even more years \Nof really, really hard work,
Dialogue: 0,0:06:50.07,0:06:52.98,Default,,0000,0000,0000,,we get village idiot artificial intelligence.
Dialogue: 0,0:06:52.98,0:06:56.26,Default,,0000,0000,0000,,And a few moments later, \Nwe are beyond Ed Witten.
Dialogue: 0,0:06:56.26,0:06:59.22,Default,,0000,0000,0000,,The train doesn't stop at \Nhumanville station.
Dialogue: 0,0:06:59.22,0:07:02.25,Default,,0000,0000,0000,,It's likely, rather, to swoosh right by.
Dialogue: 0,0:07:02.25,0:07:04.23,Default,,0000,0000,0000,,Now this has profound implications,
Dialogue: 0,0:07:04.23,0:07:08.09,Default,,0000,0000,0000,,particularly when it comes \Nto questions of power.
Dialogue: 0,0:07:08.09,0:07:09.99,Default,,0000,0000,0000,,For example, chimpanzees are strong,
Dialogue: 0,0:07:09.99,0:07:15.21,Default,,0000,0000,0000,,pound for pound, a chimpanzee is about\Ntwice as strong as a fit human male.
Dialogue: 0,0:07:15.21,0:07:19.83,Default,,0000,0000,0000,,And yet, the fate of Kanzi and his pals\Ndepends a lot more \N
Dialogue: 0,0:07:19.83,0:07:24.26,Default,,0000,0000,0000,,on what we humans do than on \Nwhat the chimpanzees do themselves.
Dialogue: 0,0:07:24.26,0:07:27.54,Default,,0000,0000,0000,,Once there is super intelligence,
Dialogue: 0,0:07:27.54,0:07:32.04,Default,,0000,0000,0000,,the fate of humanity may depend\Non what the super intelligence does.
Dialogue: 0,0:07:32.04,0:07:36.69,Default,,0000,0000,0000,,Think about it: machine intelligence \Nis the last invention
Dialogue: 0,0:07:36.69,0:07:38.55,Default,,0000,0000,0000,,that humanity will ever need to make.
Dialogue: 0,0:07:38.55,0:07:41.52,Default,,0000,0000,0000,,Machines will then be better \Nat inventing than we are,
Dialogue: 0,0:07:41.52,0:07:44.06,Default,,0000,0000,0000,,and they'll be doing so on digital timescales.
Dialogue: 0,0:07:44.06,0:07:49.15,Default,,0000,0000,0000,,What this means is basically\Na telescoping of the future.
Dialogue: 0,0:07:49.15,0:07:52.86,Default,,0000,0000,0000,,Think of all the crazy technologies that you\Ncould have imaged
Dialogue: 0,0:07:52.86,0:07:55.32,Default,,0000,0000,0000,,that humans could have developed\Nin the fullness of time:
Dialogue: 0,0:07:55.32,0:07:58.58,Default,,0000,0000,0000,,cures for aging, space colonization,
Dialogue: 0,0:07:58.58,0:08:00.42,Default,,0000,0000,0000,,self-replicating nanobots
Dialogue: 0,0:08:00.42,0:08:02.30,Default,,0000,0000,0000,,or uploading of minds into computers,
Dialogue: 0,0:08:02.30,0:08:04.57,Default,,0000,0000,0000,,all kinds of science fiction-y stuff
Dialogue: 0,0:08:04.57,0:08:07.21,Default,,0000,0000,0000,,that's nevertheless consistent \Nwith the laws of physics.
Dialogue: 0,0:08:07.21,0:08:09.64,Default,,0000,0000,0000,,All of this, super intelligence \Ncould develop
Dialogue: 0,0:08:09.64,0:08:12.45,Default,,0000,0000,0000,,and possibly, quite rapidly.
Dialogue: 0,0:08:12.45,0:08:16.01,Default,,0000,0000,0000,,Now, super intelligence with such \Ntechnological maturity
Dialogue: 0,0:08:16.01,0:08:18.19,Default,,0000,0000,0000,,would be extremely powerful,
Dialogue: 0,0:08:18.19,0:08:19.98,Default,,0000,0000,0000,,and at least in some scenarios,
Dialogue: 0,0:08:19.98,0:08:23.22,Default,,0000,0000,0000,,it would be able to get \Nwhat it wants.
Dialogue: 0,0:08:23.22,0:08:24.87,Default,,0000,0000,0000,,We would then have a future\Nthat would be shaped
Dialogue: 0,0:08:24.87,0:08:29.86,Default,,0000,0000,0000,,by the preferences of this AI.
Dialogue: 0,0:08:29.86,0:08:34.24,Default,,0000,0000,0000,,Now a good question is, what are \Nthose preferences?
Dialogue: 0,0:08:34.24,0:08:36.01,Default,,0000,0000,0000,,Here it gets trickier.
Dialogue: 0,0:08:36.01,0:08:37.45,Default,,0000,0000,0000,,To make any headway with this,
Dialogue: 0,0:08:37.45,0:08:39.36,Default,,0000,0000,0000,,we must first, first of all,
Dialogue: 0,0:08:39.36,0:08:41.59,Default,,0000,0000,0000,,avoid anthropomorphizing.
Dialogue: 0,0:08:41.59,0:08:43.64,Default,,0000,0000,0000,,this is ironic because\Nevery newspaper article\N
Dialogue: 0,0:08:43.64,0:08:50.60,Default,,0000,0000,0000,,about the future of AI has a picture of this:
Dialogue: 0,0:08:50.60,0:08:54.17,Default,,0000,0000,0000,,So I think what we need to do is \Nto conceive of the issue
Dialogue: 0,0:08:54.17,0:08:54.84,Default,,0000,0000,0000,,more abstractly,
Dialogue: 0,0:08:54.84,0:08:56.64,Default,,0000,0000,0000,,not in terms of vivid Hollywood scenarios.
Dialogue: 0,0:08:56.64,0:09:00.82,Default,,0000,0000,0000,,We need to think of intelligence as an\Noptimization process,
Dialogue: 0,0:09:00.82,0:09:06.49,Default,,0000,0000,0000,,a process that steers the future\Ninto a particular set of configurations.
Dialogue: 0,0:09:06.49,0:09:08.13,Default,,0000,0000,0000,,As super intelligence --
Dialogue: 0,0:09:08.13,0:09:09.98,Default,,0000,0000,0000,,it's a really strong optimization process.
Dialogue: 0,0:09:09.98,0:09:12.86,Default,,0000,0000,0000,,It's extremely good at using \Navailable means
Dialogue: 0,0:09:12.86,0:09:16.01,Default,,0000,0000,0000,,to achieve a state in which its\Ngoal is realized.
Dialogue: 0,0:09:16.01,0:09:18.77,Default,,0000,0000,0000,,This means that there is no necessary\Nconenction between
Dialogue: 0,0:09:18.77,0:09:21.85,Default,,0000,0000,0000,,being highly intelligent in this sense,
Dialogue: 0,0:09:21.85,0:09:24.12,Default,,0000,0000,0000,,and having an objective that we humans
Dialogue: 0,0:09:24.12,0:09:27.32,Default,,0000,0000,0000,,would find worthwhile or meaningful.
Dialogue: 0,0:09:27.32,0:09:31.12,Default,,0000,0000,0000,,Suppose we give AI the goal \Nto make humans smile.
Dialogue: 0,0:09:31.12,0:09:34.10,Default,,0000,0000,0000,,When the AI is weak, it performs useful\Nor amusing actions
Dialogue: 0,0:09:34.10,0:09:35.94,Default,,0000,0000,0000,,that cause its user to smile.
Dialogue: 0,0:09:35.94,0:09:39.03,Default,,0000,0000,0000,,When the AI becomes super intelligent,
Dialogue: 0,0:09:39.03,0:09:41.28,Default,,0000,0000,0000,,it realizes that there is\Na more effective way
Dialogue: 0,0:09:41.28,0:09:42.72,Default,,0000,0000,0000,,to achieve this goal:
Dialogue: 0,0:09:42.72,0:09:44.48,Default,,0000,0000,0000,,take control of the world
Dialogue: 0,0:09:44.48,0:09:47.64,Default,,0000,0000,0000,,and stick electrodes into \Nthe facial muscles of humans
Dialogue: 0,0:09:47.64,0:09:50.58,Default,,0000,0000,0000,,to cause constant, beaming grins.
Dialogue: 0,0:09:50.58,0:09:53.12,Default,,0000,0000,0000,,Another example, suppose we give AI\Nthe goal to solve
Dialogue: 0,0:09:53.12,0:09:55.34,Default,,0000,0000,0000,,a difficult mathematical problem.
Dialogue: 0,0:09:55.34,0:09:56.76,Default,,0000,0000,0000,,When the AI becomes super intelligent,
Dialogue: 0,0:09:56.76,0:10:01.10,Default,,0000,0000,0000,,it realizes that the most effective way \Nto get the solution to this problem
Dialogue: 0,0:10:01.10,0:10:04.04,Default,,0000,0000,0000,,is to by transforming the planet\Ninto a giant computer,
Dialogue: 0,0:10:04.04,0:10:06.49,Default,,0000,0000,0000,,so as to increase its thinking capacity.
Dialogue: 0,0:10:06.49,0:10:09.04,Default,,0000,0000,0000,,And notice that this gives the AIs\Nan instrumental reason
Dialogue: 0,0:10:09.04,0:10:11.70,Default,,0000,0000,0000,,to do things to us that we\Nmight not approve us.
Dialogue: 0,0:10:11.70,0:10:13.50,Default,,0000,0000,0000,,Human beings in this model are threats,
Dialogue: 0,0:10:13.50,0:10:16.23,Default,,0000,0000,0000,,we could prevent the mathematical problem\Nfrom being solved.
Dialogue: 0,0:10:16.23,0:10:20.16,Default,,0000,0000,0000,,Of course, perceivably things won't \Ngo wrong in these particular ways,
Dialogue: 0,0:10:20.16,0:10:22.45,Default,,0000,0000,0000,,these are cartoon examples.
Dialogue: 0,0:10:22.45,0:10:24.39,Default,,0000,0000,0000,,But the general point here is important:
Dialogue: 0,0:10:24.39,0:10:27.27,Default,,0000,0000,0000,,if you create a really powerful\Noptimization process
Dialogue: 0,0:10:27.27,0:10:29.50,Default,,0000,0000,0000,,to maximize for objective x,
Dialogue: 0,0:10:29.50,0:10:31.78,Default,,0000,0000,0000,,you better make sure that \Nyour definition of x
Dialogue: 0,0:10:31.78,0:10:34.84,Default,,0000,0000,0000,,incorporates everything you care about.
Dialogue: 0,0:10:34.84,0:10:39.22,Default,,0000,0000,0000,,This is a lesson that's also taught\Nin many a myth.
Dialogue: 0,0:10:39.22,0:10:44.52,Default,,0000,0000,0000,,Kind Midas wishes that everything\Nhe touches be turned into gold.
Dialogue: 0,0:10:44.52,0:10:47.38,Default,,0000,0000,0000,,He touches his daughter, \Nshe turns into fold.
Dialogue: 0,0:10:47.38,0:10:49.93,Default,,0000,0000,0000,,He touches his food, it turns into gold.
Dialogue: 0,0:10:49.93,0:10:52.52,Default,,0000,0000,0000,,This could become practically relevant,
Dialogue: 0,0:10:52.52,0:10:54.59,Default,,0000,0000,0000,,not just for a metaphor for greed,
Dialogue: 0,0:10:54.59,0:10:57.08,Default,,0000,0000,0000,,but an illustration of what happens \Nif you create
Dialogue: 0,0:10:57.08,0:10:59.32,Default,,0000,0000,0000,,a powerful optimization process
Dialogue: 0,0:10:59.32,0:11:04.11,Default,,0000,0000,0000,,and give it misconceived \Nor poorly specified goals.
Dialogue: 0,0:11:04.11,0:11:09.30,Default,,0000,0000,0000,,Now you might say, "If a computer starts\Nsticking electrodes into people's faces,
Dialogue: 0,0:11:09.30,0:11:12.56,Default,,0000,0000,0000,,we'd just shut it off."
Dialogue: 0,0:11:12.56,0:11:16.69,Default,,0000,0000,0000,,A: This is not necessarily so easy to do\Nif we've grown dependent
Dialogue: 0,0:11:16.69,0:11:18.18,Default,,0000,0000,0000,,on the system,
Dialogue: 0,0:11:18.18,0:11:20.63,Default,,0000,0000,0000,,like where is the off switch to the internet?
Dialogue: 0,0:11:20.63,0:11:25.75,Default,,0000,0000,0000,,B: Why haven't the chimpanzees\Nflicked the off-switch to humanity,
Dialogue: 0,0:11:25.75,0:11:27.30,Default,,0000,0000,0000,,or the neandrathals?
Dialogue: 0,0:11:27.30,0:11:29.96,Default,,0000,0000,0000,,They certainly had reasons.
Dialogue: 0,0:11:29.96,0:11:32.76,Default,,0000,0000,0000,,We have an off switch, for example,\Nright here.
Dialogue: 0,0:11:32.76,0:11:34.81,Default,,0000,0000,0000,,[choking sound]
Dialogue: 0,0:11:34.81,0:11:37.24,Default,,0000,0000,0000,,The reason is that we are \Nan intelligent adversary,
Dialogue: 0,0:11:37.24,0:11:39.97,Default,,0000,0000,0000,,we can anticipate threats \Nand we can plan around them.
Dialogue: 0,0:11:39.97,0:11:42.47,Default,,0000,0000,0000,,But so could a super intelligent agent,
Dialogue: 0,0:11:42.47,0:11:45.72,Default,,0000,0000,0000,,and it would be much better at that\Nthan we are.
Dialogue: 0,0:11:45.72,0:11:52.91,Default,,0000,0000,0000,,The point is, we should not be confident\Nthat we have this under control here.
Dialogue: 0,0:11:52.91,0:11:56.36,Default,,0000,0000,0000,,And we could try to make our job\Na little bit easier by, say,
Dialogue: 0,0:11:56.36,0:11:58.15,Default,,0000,0000,0000,,putting the AI in a box,
Dialogue: 0,0:11:58.15,0:11:59.64,Default,,0000,0000,0000,,like a secure software environment,
Dialogue: 0,0:11:59.64,0:12:02.77,Default,,0000,0000,0000,,a virtual reality simulation \Nfrom which it cannot escape.
Dialogue: 0,0:12:02.77,0:12:06.91,Default,,0000,0000,0000,,But how confident can we be that\Nthe AI couldn't find a bug,
Dialogue: 0,0:12:06.91,0:12:10.08,Default,,0000,0000,0000,,given that even human hackers find bugs\Nall the time,
Dialogue: 0,0:12:10.08,0:12:14.24,Default,,0000,0000,0000,,I'd say, probably not very confident.
Dialogue: 0,0:12:14.24,0:12:18.78,Default,,0000,0000,0000,,So we disconnect the ethernet cable\Nto create an air gap,
Dialogue: 0,0:12:18.78,0:12:23.82,Default,,0000,0000,0000,,but again, like nearly human hackers\Nroutinely transgress air gaps
Dialogue: 0,0:12:23.82,0:12:25.17,Default,,0000,0000,0000,,using social engineering.
Dialogue: 0,0:12:25.17,0:12:26.42,Default,,0000,0000,0000,,Like right now as I speak,
Dialogue: 0,0:12:26.42,0:12:28.77,Default,,0000,0000,0000,,I"m sure there is some employee \Nout there somewhere
Dialogue: 0,0:12:28.77,0:12:31.30,Default,,0000,0000,0000,,who's been talked into handing out \Nher account details
Dialogue: 0,0:12:31.30,0:12:35.18,Default,,0000,0000,0000,,by some by somebody claiming\Nto be from the IT department.
Dialogue: 0,0:12:35.18,0:12:36.70,Default,,0000,0000,0000,,More creative scenarios are also possible,
Dialogue: 0,0:12:36.70,0:12:40.17,Default,,0000,0000,0000,,like if you're the AI, you can imagine\Nwiggling electroces around
Dialogue: 0,0:12:40.17,0:12:44.82,Default,,0000,0000,0000,,in your internal circuitry to create\Nradio waves that you can use to communicate.
Dialogue: 0,0:12:44.82,0:12:47.43,Default,,0000,0000,0000,,Or maybe you could pretend to malfunction,
Dialogue: 0,0:12:47.43,0:12:50.90,Default,,0000,0000,0000,,and then when the programmers open\Nyou up to see what went wrong with you,
Dialogue: 0,0:12:50.90,0:12:52.52,Default,,0000,0000,0000,,they look at the source code -- BAM! --
Dialogue: 0,0:12:52.52,0:12:55.31,Default,,0000,0000,0000,,the manipulation can take place.
Dialogue: 0,0:12:55.31,0:12:58.74,Default,,0000,0000,0000,,Or it could output the blueprint\Nto a really nifty technology
Dialogue: 0,0:12:58.74,0:13:00.14,Default,,0000,0000,0000,,and when we implement it,
Dialogue: 0,0:13:00.14,0:13:05.20,Default,,0000,0000,0000,,it has some surreptitious side effect\Nthat the AI had planned.
Dialogue: 0,0:13:05.20,0:13:08.00,Default,,0000,0000,0000,,The point here is that we should \Nnot be confident in our ability
Dialogue: 0,0:13:08.00,0:13:11.81,Default,,0000,0000,0000,,to keep a super intelligent genie\Nlocked up in its bottle forever.
Dialogue: 0,0:13:11.81,0:13:15.03,Default,,0000,0000,0000,,Sooner or later, it will out.
Dialogue: 0,0:13:15.03,0:13:18.14,Default,,0000,0000,0000,,I believe that the answer here\Nis to figure out
Dialogue: 0,0:13:18.14,0:13:21.23,Default,,0000,0000,0000,,how to create super intelligent AI\Nsuch that even if,
Dialogue: 0,0:13:21.23,0:13:23.15,Default,,0000,0000,0000,,when it escapes,
Dialogue: 0,0:13:23.15,0:13:26.44,Default,,0000,0000,0000,,it is still safe because it \Nis fundamentally on our side
Dialogue: 0,0:13:26.44,0:13:28.34,Default,,0000,0000,0000,,because it shares our values.
Dialogue: 0,0:13:28.34,0:13:32.56,Default,,0000,0000,0000,,I see no way around \Nthis difficult problem.
Dialogue: 0,0:13:32.56,0:13:36.39,Default,,0000,0000,0000,,Now, I'm actually fairly optimistic\Nthat this problem can be solved.
Dialogue: 0,0:13:36.39,0:13:40.29,Default,,0000,0000,0000,,We wouldn't have to write down \Na long list of everything we care aobut
Dialogue: 0,0:13:40.29,0:13:43.94,Default,,0000,0000,0000,,or worse yet, spell it out \Nin some computer language
Dialogue: 0,0:13:43.94,0:13:45.39,Default,,0000,0000,0000,,like C ++ or Python,
Dialogue: 0,0:13:45.39,0:13:48.16,Default,,0000,0000,0000,,that would be a task beyond hopeless.
Dialogue: 0,0:13:48.16,0:13:52.46,Default,,0000,0000,0000,,Instead, we would create an AI\Nthat uses its intelligence
Dialogue: 0,0:13:52.46,0:13:55.23,Default,,0000,0000,0000,,to learn what we value
Dialogue: 0,0:13:55.23,0:14:00.51,Default,,0000,0000,0000,,and its motivation system is constructed\Nin such a way that it is motivated
Dialogue: 0,0:14:00.51,0:14:06.30,Default,,0000,0000,0000,,to pursue our values or to perform actions\Nthat it predicts we would approve of.
Dialogue: 0,0:14:06.30,0:14:08.71,Default,,0000,0000,0000,,We would thus leverage its intelligence\Nas much as possible
Dialogue: 0,0:14:08.71,0:14:12.73,Default,,0000,0000,0000,,to solve the problem of value loading.
Dialogue: 0,0:14:12.73,0:14:14.24,Default,,0000,0000,0000,,This can happen,
Dialogue: 0,0:14:14.24,0:14:17.84,Default,,0000,0000,0000,,and the outcome could be \Nvery good for humanity.
Dialogue: 0,0:14:17.84,0:14:21.79,Default,,0000,0000,0000,,But it doesn't happen automatically.
Dialogue: 0,0:14:21.79,0:14:24.79,Default,,0000,0000,0000,,The initial conditions \Nfor the intelligent explosion
Dialogue: 0,0:14:24.79,0:14:27.65,Default,,0000,0000,0000,,might need to be set up \Nin just the right way
Dialogue: 0,0:14:27.65,0:14:31.18,Default,,0000,0000,0000,,if we are to have a controlled detonation.
Dialogue: 0,0:14:31.18,0:14:33.80,Default,,0000,0000,0000,,The values that the AI has need \Nto match ours,
Dialogue: 0,0:14:33.80,0:14:36.24,Default,,0000,0000,0000,,not just in the familiar context,
Dialogue: 0,0:14:36.24,0:14:37.100,Default,,0000,0000,0000,,like where we can easily check\Nhow the AI behaves,
Dialogue: 0,0:14:37.100,0:14:41.23,Default,,0000,0000,0000,,but also in all novel contexts\Nthat the AI might encounter
Dialogue: 0,0:14:41.23,0:14:42.79,Default,,0000,0000,0000,,in the indefinite future.
Dialogue: 0,0:14:42.79,0:14:45.08,Default,,0000,0000,0000,,And there are also some esoteric issues\Nthat would need to be solved,
Dialogue: 0,0:14:45.08,0:14:47.85,Default,,0000,0000,0000,,sorted out,
Dialogue: 0,0:14:47.85,0:14:49.62,Default,,0000,0000,0000,,the exact decisions of its decision theory,
Dialogue: 0,0:14:49.62,0:14:53.33,Default,,0000,0000,0000,,how to deal with logical uncertainty,\Nand so forth.
Dialogue: 0,0:14:53.33,0:14:56.52,Default,,0000,0000,0000,,So the technical problems that need\Nto be solved to make this work
Dialogue: 0,0:14:56.52,0:14:57.52,Default,,0000,0000,0000,,look quite difficult,
Dialogue: 0,0:14:57.52,0:15:00.92,Default,,0000,0000,0000,,-- not as difficult as making \Na super intelligent AI,
Dialogue: 0,0:15:00.92,0:15:03.79,Default,,0000,0000,0000,,but fairly difficult.
Dialogue: 0,0:15:03.79,0:15:05.49,Default,,0000,0000,0000,,Here is the worry:
Dialogue: 0,0:15:05.49,0:15:10.17,Default,,0000,0000,0000,,making super intelligent AI\Nis a super hard challenge.
Dialogue: 0,0:15:10.17,0:15:12.72,Default,,0000,0000,0000,,Making super intelligent AI\Nthat is safe
Dialogue: 0,0:15:12.72,0:15:16.76,Default,,0000,0000,0000,,involves some additional challenge \Non top of that.
Dialogue: 0,0:15:16.76,0:15:18.56,Default,,0000,0000,0000,,The risk is that if somebody\Nfigures out how to crack
Dialogue: 0,0:15:18.56,0:15:21.39,Default,,0000,0000,0000,,the first challenge without also\Nhaving cracked
Dialogue: 0,0:15:21.39,0:15:25.40,Default,,0000,0000,0000,,the additional challenge \Nof ensuring perfect safety.
Dialogue: 0,0:15:25.40,0:15:28.71,Default,,0000,0000,0000,,So I think that we should\Nwork out a solution
Dialogue: 0,0:15:28.71,0:15:31.53,Default,,0000,0000,0000,,to the controlled problem in advance,
Dialogue: 0,0:15:31.53,0:15:35.04,Default,,0000,0000,0000,,so that we have it available \Nby the time it is needed.
Dialogue: 0,0:15:35.04,0:15:38.24,Default,,0000,0000,0000,,Now it might be that we cannot\Nsolve the entire controlled problem
Dialogue: 0,0:15:38.24,0:15:41.30,Default,,0000,0000,0000,,in advance because maybe some element\Ncan only be put in place
Dialogue: 0,0:15:41.30,0:15:43.81,Default,,0000,0000,0000,,once you know the details of \Nthe architecture
Dialogue: 0,0:15:43.81,0:15:45.05,Default,,0000,0000,0000,,where it will be implemented.
Dialogue: 0,0:15:45.05,0:15:48.68,Default,,0000,0000,0000,,But the more of the controlled problem\Nthat we solve in advance,
Dialogue: 0,0:15:48.68,0:15:52.77,Default,,0000,0000,0000,,the better the odds that the transition\Nto the machine intelligence era
Dialogue: 0,0:15:52.77,0:15:54.54,Default,,0000,0000,0000,,will go well.
Dialogue: 0,0:15:54.54,0:15:58.95,Default,,0000,0000,0000,,This to me looks like a thing\Nthat is well worth doing
Dialogue: 0,0:15:58.95,0:16:02.28,Default,,0000,0000,0000,,and I can imagine that if \Nthings turn out okay,
Dialogue: 0,0:16:02.28,0:16:05.43,Default,,0000,0000,0000,,that people in a million years\Nfrom now
Dialogue: 0,0:16:05.43,0:16:06.86,Default,,0000,0000,0000,,look back at this century
Dialogue: 0,0:16:06.86,0:16:08.97,Default,,0000,0000,0000,,and it might well be that\Nthey say
Dialogue: 0,0:16:08.97,0:16:11.12,Default,,0000,0000,0000,,that he one thing we did\Nthat really mattered
Dialogue: 0,0:16:11.12,0:16:12.31,Default,,0000,0000,0000,,was to get this thing right.
Dialogue: 0,0:16:12.31,0:16:13.15,Default,,0000,0000,0000,,Thank you.
Dialogue: 0,0:16:13.15,0:16:18.38,Default,,0000,0000,0000,,(Applause)