1
00:00:00,570 --> 00:00:04,777
I work with a bunch of mathematicians,
philosophers and computer scientists

2
00:00:04,777 --> 00:00:09,986
and we sit around and think about
the future of machine intelligence,

3
00:00:09,986 --> 00:00:12,200
among other things.

4
00:00:12,200 --> 00:00:16,755
Some people think that some of
these people are science fiction-y

5
00:00:16,755 --> 00:00:19,856
far out there, crazy.

6
00:00:19,856 --> 00:00:21,326
But I like to say,

7
00:00:21,326 --> 00:00:24,930
"Okay, let's look at the modern
human condition."

8
00:00:24,930 --> 00:00:26,622
(Laughter)

9
00:00:26,622 --> 00:00:29,024
This is the normal way for things to be.

10
00:00:29,024 --> 00:00:31,309
But, if we think about it,

11
00:00:31,309 --> 00:00:34,602
we are actually recently arrived
guests on this planet.

12
00:00:34,602 --> 00:00:36,214
The human species --

13
00:00:36,214 --> 00:00:41,430
think of if earth was created
one year ago,

14
00:00:41,430 --> 00:00:44,978
the human species, then, 
would be 10-minutes-old.

15
00:00:44,978 --> 00:00:48,146
The industrial era started
two seconds ago.

16
00:00:48,146 --> 00:00:50,551
Another way to think of this,

17
00:00:50,551 --> 00:00:54,443
if you think of world GDP
over the last 10,000 years,

18
00:00:54,443 --> 00:00:57,530
I've actually taken the trouble
to plot this for you in a graph.


19
00:00:57,530 --> 00:00:59,304
It looks like this.

20
00:00:59,304 --> 00:01:00,667
(Laughter)

21
00:01:00,667 --> 00:01:02,818
It's a curious shape
for a normal condition.

22
00:01:02,818 --> 00:01:04,516
I sure wouldn't want to sit on it.

23
00:01:04,516 --> 00:01:07,067
(Laughter)

24
00:01:07,067 --> 00:01:08,681
Let's ask ourselves,

25
00:01:08,681 --> 00:01:12,122
what is the cost of this current anomaly?

26
00:01:12,122 --> 00:01:14,393
Some people would say it's technology.

27
00:01:14,393 --> 00:01:15,851
Now it's true,

28
00:01:15,851 --> 00:01:19,484
technology has accumulated
through human history,

29
00:01:19,484 --> 00:01:23,713
and right now, technology
advances extremely rapidly,

30
00:01:23,713 --> 00:01:25,278
that is the proximate cause,

31
00:01:25,278 --> 00:01:28,473
that's why we are currently 
so very productive.

32
00:01:28,473 --> 00:01:32,874
But I like to think back further 
to the ultimate cause.

33
00:01:32,874 --> 00:01:36,880
Look at these two 
highly distinguished gentlemen:

34
00:01:36,880 --> 00:01:38,623
We have Kanzi,

35
00:01:38,623 --> 00:01:41,260
he's mastered 200 lexical tokens,

36
00:01:41,260 --> 00:01:43,123
an incredible feat.

37
00:01:43,123 --> 00:01:46,817
And Ed Witten unleashed the second
super string revolution.

38
00:01:46,817 --> 00:01:49,141
If we look under the hood, 
this is what we find:

39
00:01:49,141 --> 00:01:50,711
basically the same thing.

40
00:01:50,711 --> 00:01:52,524
One is a little larger,

41
00:01:52,524 --> 00:01:55,282
it maybe also has a few tricks
in the exact way it's wired.

42
00:01:55,282 --> 00:01:59,094
These invisible differences cannot
be too complicated, however,

43
00:01:59,094 --> 00:02:03,379
because they've only been
250,000 generations since

44
00:02:03,379 --> 00:02:05,111
our last common ancestor.

45
00:02:05,111 --> 00:02:10,000
We know that complicated mechanisms
that a long time to evolve.

46
00:02:10,000 --> 00:02:12,499
So a bunch of relatively minor changes

47
00:02:12,499 --> 00:02:15,566
take us from Kanzi to Witten.

48
00:02:15,566 --> 00:02:17,377
From broken-off tree branches,

49
00:02:17,377 --> 00:02:20,829
to intercontinental balistic missles.

50
00:02:20,829 --> 00:02:22,724
So this then seems pretty obvious that


51
00:02:22,724 --> 00:02:24,954
everything we've achieved, pretty much,

52
00:02:24,954 --> 00:02:27,212
and everything we care about
depends crucially

53
00:02:27,212 --> 00:02:32,650
on relatively some minor changes 
that made the human mind.

54
00:02:32,650 --> 00:02:36,312
And the collaraly, of course, 
is that any further changes

55
00:02:36,312 --> 00:02:39,789
that could significantly change
the substrate of thinking

56
00:02:39,789 --> 00:02:43,651
could have potentially 
enormous consequences.

57
00:02:43,651 --> 00:02:46,576
Some of my colleagues 
think we're on the verge

58
00:02:46,576 --> 00:02:49,756
of something that could cause
a profound change

59
00:02:49,756 --> 00:02:51,384
in that substrate,

60
00:02:51,384 --> 00:02:54,347
and that is machine super intelligence.

61
00:02:54,347 --> 00:02:59,086
Artificial intelligence used to be
about putting commands in a box.

62
00:02:59,086 --> 00:03:04,295
You would have human programmers
that would painstakingly handcraft items,

63
00:03:04,295 --> 00:03:06,262
You build up these expert systems,

64
00:03:06,262 --> 00:03:07,916
and they were kind of useful
for some purposes,

65
00:03:07,916 --> 00:03:09,857
but they were very brittle,

66
00:03:09,867 --> 00:03:11,023
you couldn't scale them.

67
00:03:11,023 --> 00:03:14,410
Basically, you got out only
what you put in.

68
00:03:14,410 --> 00:03:16,967
But since then, a paradigm shift
has taken place

69
00:03:16,967 --> 00:03:18,874
in the field of artificial intelligence.

70
00:03:18,874 --> 00:03:21,894
Today, the action is really 
around machine learning.

71
00:03:21,894 --> 00:03:28,061
So rather than handcrafting knowledge
representations and features,

72
00:03:28,061 --> 00:03:31,709
we create algorithms that learn,

73
00:03:31,709 --> 00:03:34,275
often from raw perceptual data.

74
00:03:34,275 --> 00:03:39,063
Basically the same thing
that the human infant does.

75
00:03:39,063 --> 00:03:43,270
The result is AI that is 
not limited to one domain,

76
00:03:43,270 --> 00:03:47,901
the same system can learn to translate 
between any pairs of languages,

77
00:03:47,901 --> 00:03:53,068
or learn to play any computer game
at the Atari console.

78
00:03:53,068 --> 00:03:56,757
Now of course, AI is still 
no where near having

79
00:03:56,757 --> 00:04:00,701
the same powerful, cross-domain
ability to learn and plan

80
00:04:00,701 --> 00:04:02,335
as a human being has.

81
00:04:02,335 --> 00:04:04,461
The cortex still has some 
algorithmic tricks

82
00:04:04,461 --> 00:04:07,886
that we don't yet know
how to match in machines.

83
00:04:07,886 --> 00:04:09,785
But so the question is,

84
00:04:09,785 --> 00:04:13,885
how far are we from being able
to match those tricks?

85
00:04:13,885 --> 00:04:15,798
A couple of years ago, we did a survey

86
00:04:15,798 --> 00:04:17,927
of some of the world's leading AI experts

87
00:04:17,927 --> 00:04:19,136
to see what they think

88
00:04:19,136 --> 00:04:21,440
and one of the questions we asked was,

89
00:04:21,440 --> 00:04:24,793
"By which year do you think
there is a 50 percent probability

90
00:04:24,793 --> 00:04:28,785
that we will have achieved 
human-level machine intelligence?"

91
00:04:28,785 --> 00:04:31,898
We defined human-level here 
as the ability to perform

92
00:04:31,898 --> 00:04:35,839
almost any job at least as well
as an adult human,

93
00:04:35,839 --> 00:04:39,844
so real human-level, not just
within some limited domain.

94
00:04:39,844 --> 00:04:43,494
And the median answer was 2040 or 2050,

95
00:04:43,494 --> 00:04:46,300
depending on precisely which 
group of experts we asked.

96
00:04:46,300 --> 00:04:49,229
Now, it could happen much, much later,

97
00:04:49,229 --> 00:04:52,279
or sooner, the truth is 
nobody really knows.

98
00:04:52,279 --> 00:04:55,941
What we do know is that 
the ultimate limit

99
00:04:55,941 --> 00:04:58,802
to information processing 
in machine substrate,

100
00:04:58,802 --> 00:05:03,241
lie far outside the limits 
in biological tissue.

101
00:05:03,241 --> 00:05:05,619
This comes down to physics.

102
00:05:05,619 --> 00:05:10,337
A biological neuron fires, maybe, 
at 200 Hertz, 200 times a second.

103
00:05:10,337 --> 00:05:13,931
But even a present-day transistor
operates at a gigahert.

104
00:05:13,931 --> 00:05:16,640
Neurons propagate slowly in axons,

105
00:05:16,640 --> 00:05:19,568
100 meters per second, tops.

106
00:05:19,568 --> 00:05:23,079
But in computers, signals can travel
at the speed of light.

107
00:05:23,079 --> 00:05:24,948
There's also size limitations,

108
00:05:24,948 --> 00:05:27,975
a human brain has to fit inside a cranium,

109
00:05:27,975 --> 00:05:32,736
but a computer can be the size
of a warehouse or larger.

110
00:05:32,736 --> 00:05:38,335
So the potential of super intelligence 
lies dormant in matter,

111
00:05:38,335 --> 00:05:44,047
much like the power of the atom 
lay dormant throughout human history,

112
00:05:44,047 --> 00:05:48,452
patiently waiting there until 1945.

113
00:05:48,452 --> 00:05:50,920
In this century, scientists
may learn to awaken

114
00:05:50,920 --> 00:05:53,818
the power of artificial intelligence.

115
00:05:53,818 --> 00:05:58,406
And I think we might then see
an intelligence explosion.

116
00:05:58,406 --> 00:06:02,363
Now most people, when they think
about what is smart and what is dumb,

117
00:06:02,363 --> 00:06:05,386
I think I have in mind a picture
roughly like this.

118
00:06:05,386 --> 00:06:07,984
So at one hand, we have the village idiot,

119
00:06:07,984 --> 00:06:10,448
and then far over at the other side,

120
00:06:10,448 --> 00:06:11,857
we have Ed Witten,

121
00:06:11,857 --> 00:06:15,573
or Albert Einsten or whoever 
your favorite guru is.

122
00:06:15,573 --> 00:06:19,057
But I think that from the point of view
of artificial intelligence,

123
00:06:19,057 --> 00:06:23,258
the true picture is actually
probably more like this:

124
00:06:23,258 --> 00:06:26,636
AI starts out at this point here,
at zero intelligence,

125
00:06:26,636 --> 00:06:29,647
and then, after many, many 
years of really hard work,

126
00:06:29,647 --> 00:06:33,491
maybe eventually we get to
mouse-level artificial intelligence,

127
00:06:33,491 --> 00:06:35,921
something that can navigate 
cluttered environments

128
00:06:35,921 --> 00:06:37,908
as well as a mouse can.

129
00:06:37,908 --> 00:06:42,221
And then, after many, many more years
of really hard work, lots of investment,

130
00:06:42,221 --> 00:06:46,860
maybe eventually we get to 
chimpanzee-level artificial intelligence.

131
00:06:46,860 --> 00:06:50,070
And then, after even more years 
of really, really hard work,

132
00:06:50,070 --> 00:06:52,983
we get village idiot 
artificial intelligence.

133
00:06:52,983 --> 00:06:56,255
And a few moments later, 
we are beyond Ed Witten.

134
00:06:56,255 --> 00:06:59,225
The train doesn't stop at 
Human-ville Station.

135
00:06:59,225 --> 00:07:02,247
It's likely, rather, to swoosh right by.

136
00:07:02,247 --> 00:07:04,231
Now this has profound implications,

137
00:07:04,231 --> 00:07:08,093
particularly when it comes 
to questions of power.

138
00:07:08,093 --> 00:07:09,992
For example, chimpanzees are strong,

139
00:07:09,992 --> 00:07:15,214
pound for pound, a chimpanzee is about
twice as strong as a fit human male.

140
00:07:15,214 --> 00:07:19,828
And yet, the fate of Kanzi and his pals
depends a lot more 


141
00:07:19,828 --> 00:07:24,258
on what we humans do than on 
what the chimpanzees do themselves.

142
00:07:24,258 --> 00:07:27,542
Once there is super intelligence,

143
00:07:27,542 --> 00:07:32,041
the fate of humanity may depend
on what the super intelligence does.

144
00:07:32,041 --> 00:07:36,688
Think about it: machine intelligence 
is the last invention

145
00:07:36,688 --> 00:07:38,552
that humanity will ever need to make.

146
00:07:38,552 --> 00:07:41,525
Machines will then be better 
at inventing than we are,

147
00:07:41,525 --> 00:07:44,065
and they'll be doing so 
on digital timescales.

148
00:07:44,065 --> 00:07:48,966
What this means is basically
a telescoping of the future.

149
00:07:48,966 --> 00:07:52,524
Think of all the crazy technologies 
that you could have imagined

150
00:07:52,524 --> 00:07:55,322
maybe humans could have developed
in the fullness of time:

151
00:07:55,322 --> 00:07:58,580
cures for aging, space colonization,

152
00:07:58,580 --> 00:08:00,421
self-replicating nanobots

153
00:08:00,421 --> 00:08:02,301
or uploading of minds into computers,

154
00:08:02,301 --> 00:08:04,470
all kinds of science fiction-y stuff

155
00:08:04,470 --> 00:08:07,207
that's nevertheless consistent 
with the laws of physics.

156
00:08:07,207 --> 00:08:09,639
All of this, super intelligence 
could develop

157
00:08:09,639 --> 00:08:12,449
and possibly, quite rapidly.

158
00:08:12,449 --> 00:08:16,007
Now, super intelligence with such 
technological maturity

159
00:08:16,007 --> 00:08:18,186
would be extremely powerful,

160
00:08:18,186 --> 00:08:19,982
and at least in some scenarios,

161
00:08:19,982 --> 00:08:22,624
it would be able to get 
what it wants.

162
00:08:22,624 --> 00:08:25,073
We would then have a future
that would be shaped

163
00:08:25,073 --> 00:08:28,375
by the preferences of this AI.

164
00:08:29,855 --> 00:08:34,244
Now a good question is, what are 
those preferences?

165
00:08:34,244 --> 00:08:36,013
Here it gets trickier.

166
00:08:36,013 --> 00:08:37,448
To make any headway with this,

167
00:08:37,448 --> 00:08:39,364
we must first, first of all,

168
00:08:39,364 --> 00:08:41,276
avoid anthropomorphizing.

169
00:08:41,276 --> 00:08:45,385
And this is ironic because
every newspaper article

170
00:08:45,385 --> 00:08:50,250
about the future of AI 
has a picture of this:

171
00:08:50,250 --> 00:08:52,424
So I think what we need 
to do is to conceive

172
00:08:52,424 --> 00:08:54,840
of the issue more abstractly,

173
00:08:54,840 --> 00:08:57,204
not in terms of vivid Hollywood scenarios.

174
00:08:57,204 --> 00:09:00,821
We need to think of intelligence 
as an optimization process,

175
00:09:00,821 --> 00:09:06,488
a process that steers the future
into a particular set of configurations.

176
00:09:06,488 --> 00:09:08,130
As super intelligence --

177
00:09:08,130 --> 00:09:09,981
it's a really strong optimization process.

178
00:09:09,981 --> 00:09:12,858
It's extremely good at using 
available means

179
00:09:12,858 --> 00:09:16,007
to achieve a state in which its
goal is realized.

180
00:09:16,007 --> 00:09:18,769
This means that there is no necessary
conenction between

181
00:09:18,769 --> 00:09:21,853
being highly intelligent in this sense,

182
00:09:21,853 --> 00:09:24,125
and having an objective that we humans

183
00:09:24,125 --> 00:09:27,321
would find worthwhile or meaningful.

184
00:09:27,321 --> 00:09:31,115
Suppose we give AI the goal 
to make humans smile.

185
00:09:31,115 --> 00:09:34,097
When the AI is weak, it performs useful
or amusing actions

186
00:09:34,097 --> 00:09:35,944
that cause its user to smile.

187
00:09:35,944 --> 00:09:39,031
When the AI becomes super intelligent,

188
00:09:39,031 --> 00:09:41,284
it realizes that there is
a more effective way

189
00:09:41,284 --> 00:09:42,721
to achieve this goal:

190
00:09:42,721 --> 00:09:44,476
take control of the world

191
00:09:44,476 --> 00:09:47,638
and stick electrodes into 
the facial muscles of humans

192
00:09:47,638 --> 00:09:50,579
to cause constant, beaming grins.

193
00:09:50,579 --> 00:09:53,124
Another example, suppose
we give AI the goal to solve

194
00:09:53,124 --> 00:09:54,787
a difficult mathematical problem.

195
00:09:54,787 --> 00:09:56,764
When the AI becomes super intelligent,

196
00:09:56,764 --> 00:10:01,105
it realizes that the most effective way 
to get the solution to this problem

197
00:10:01,105 --> 00:10:04,035
is by transforming the planet
into a giant computer,

198
00:10:04,035 --> 00:10:06,281
so as to increase its thinking capacity.

199
00:10:06,281 --> 00:10:09,045
And notice that this gives the AIs
an instrumental reason

200
00:10:09,045 --> 00:10:11,561
to do things to us that we
might not approve of.

201
00:10:11,561 --> 00:10:13,496
Human beings in this model are threats,

202
00:10:13,496 --> 00:10:16,417
we could prevent the 
mathematical problem from being solved.

203
00:10:16,417 --> 00:10:20,161
Of course, perceivably things won't 
go wrong in these particular ways,

204
00:10:20,161 --> 00:10:22,454
these are cartoon examples.

205
00:10:22,454 --> 00:10:24,393
But the general point here is important:

206
00:10:24,393 --> 00:10:27,266
if you create a really powerful
optimization process

207
00:10:27,266 --> 00:10:29,500
to maximize for objective x,

208
00:10:29,500 --> 00:10:31,776
you better make sure that 
your definition of x

209
00:10:31,776 --> 00:10:34,835
incorporates everything you care about.

210
00:10:34,835 --> 00:10:39,219
This is a lesson that's also taught
in many a myth.

211
00:10:39,219 --> 00:10:44,517
Kind Midas wishes that everything
he touches be turned into gold.

212
00:10:44,517 --> 00:10:47,378
He touches his daughter, 
she turns into fold.

213
00:10:47,378 --> 00:10:49,931
He touches his food, it turns into gold.

214
00:10:49,931 --> 00:10:52,520
This could become practically relevant,

215
00:10:52,520 --> 00:10:54,590
not just for a metaphor for greed,

216
00:10:54,590 --> 00:10:57,075
but an illustration of what happens 
if you create

217
00:10:57,075 --> 00:10:59,322
a powerful optimization process

218
00:10:59,322 --> 00:11:04,111
and give it misconceived 
or poorly specified goals.

219
00:11:04,111 --> 00:11:09,300
Now you might say, "If a computer starts
sticking electrodes into people's faces,

220
00:11:09,300 --> 00:11:12,555
we'd just shut it off."

221
00:11:12,555 --> 00:11:16,690
A: This is not necessarily so easy 
to do if we've grown

222
00:11:16,690 --> 00:11:18,185
dependent on the system,

223
00:11:18,185 --> 00:11:20,627
like where is the off switch 
to the internet?

224
00:11:20,627 --> 00:11:25,747
B: Why haven't the chimpanzees
flicked the off-switch to humanity,

225
00:11:25,747 --> 00:11:27,298
or the neanderthals?

226
00:11:27,298 --> 00:11:29,964
They certainly had reasons.

227
00:11:29,964 --> 00:11:32,759
We have an off switch, 
for example, right here.

228
00:11:32,759 --> 00:11:34,813
[choking sound]

229
00:11:34,813 --> 00:11:37,238
The reason is that we are 
an intelligent adversary,

230
00:11:37,238 --> 00:11:39,966
we can anticipate threats 
and we can plan around them.

231
00:11:39,966 --> 00:11:42,470
But so could a super intelligent agent,

232
00:11:42,470 --> 00:11:45,724
and it would be much better 
at that than we are.

233
00:11:45,724 --> 00:11:52,911
The point is, we should not be confident
that we have this under control here.

234
00:11:52,911 --> 00:11:56,358
And we could try to make our job
a little bit easier by, say,

235
00:11:56,358 --> 00:11:57,948
putting the AI in a box,

236
00:11:57,948 --> 00:12:01,034
like a secure software environment,
a virtual reality simulation

237
00:12:01,034 --> 00:12:02,766
from which it cannot escape.

238
00:12:02,766 --> 00:12:06,912
But how confident can we be that
the AI couldn't find a bug.

239
00:12:06,912 --> 00:12:10,081
Given that even human hackers 
find bugs all the time,

240
00:12:10,081 --> 00:12:14,237
I'd say, probably not very confident.

241
00:12:14,237 --> 00:12:18,785
So we disconnect the ethernet cable
to create an air gap,

242
00:12:18,785 --> 00:12:23,823
but again, like nearly human hackers
routinely transgress air gaps

243
00:12:23,823 --> 00:12:25,024
using social engineering.

244
00:12:25,034 --> 00:12:27,383
Like right now as I speak, I'm sure
there is some employee

245
00:12:27,383 --> 00:12:30,546
out there somewhere who's been 
talked into handing out

246
00:12:30,546 --> 00:12:34,543
her account details by somebody
claiming to be from the IT department.

247
00:12:34,543 --> 00:12:36,701
More creative scenarios are also possible,

248
00:12:36,701 --> 00:12:40,166
like if you're the AI, you can imagine
wiggling electroces around

249
00:12:40,166 --> 00:12:42,588
in your internal circuitry 
to create radio waves

250
00:12:42,588 --> 00:12:45,010
that you can use to communicate.

251
00:12:45,010 --> 00:12:47,434
Or maybe you could pretend to malfunction,

252
00:12:47,434 --> 00:12:50,741
and then when the programmers open
you up to see what went wrong with you,

253
00:12:50,741 --> 00:12:52,517
they look at the source code -- BAM! --

254
00:12:52,517 --> 00:12:55,314
the manipulation can take place.

255
00:12:55,314 --> 00:12:58,744
Or it could output the blueprint
to a really nifty technology

256
00:12:58,744 --> 00:13:00,142
and when we implement it,

257
00:13:00,142 --> 00:13:04,539
it has some surreptitious side effect
that the AI had planned.

258
00:13:04,539 --> 00:13:08,002
The point here is that we should 
not be confident in our ability

259
00:13:08,002 --> 00:13:11,810
to keep a super intelligent genie
locked up in its bottle forever.

260
00:13:11,810 --> 00:13:15,034
Sooner or later, it will out.

261
00:13:15,034 --> 00:13:18,137
I believe that the answer here
is to figure out

262
00:13:18,137 --> 00:13:23,161
how to create super intelligent AI
such that even if, when it escapes,

263
00:13:23,161 --> 00:13:26,438
it is still safe because it 
is fundamentally on our side

264
00:13:26,438 --> 00:13:28,337
because it shares our values.

265
00:13:28,337 --> 00:13:32,557
I see no way around 
this difficult problem.

266
00:13:32,557 --> 00:13:36,391
Now, I'm actually fairly optimistic
that this problem can be solved.

267
00:13:36,391 --> 00:13:40,294
We wouldn't have to write down 
a long list of everything we care aobut

268
00:13:40,294 --> 00:13:43,937
or worse yet, spell it out 
in some computer language

269
00:13:43,937 --> 00:13:45,391
like C ++ or Python,

270
00:13:45,391 --> 00:13:48,158
that would be a task beyond hopeless.

271
00:13:48,158 --> 00:13:52,455
Instead, we would create an AI
that uses its intelligence

272
00:13:52,455 --> 00:13:55,226
to learn what we value,

273
00:13:55,226 --> 00:14:00,506
and its motivation system is constructed
in such a way that it is motivated

274
00:14:00,506 --> 00:14:05,738
to pursue our values or to perform actions
that it predicts we would approve of.

275
00:14:05,738 --> 00:14:08,712
We would thus leverage 
its intelligence as much as possible

276
00:14:08,712 --> 00:14:12,727
to solve the problem of value -loading.

277
00:14:12,727 --> 00:14:14,239
This can happen,

278
00:14:14,239 --> 00:14:17,835
and the outcome could be 
very good for humanity.

279
00:14:17,835 --> 00:14:21,792
But it doesn't happen automatically.

280
00:14:21,792 --> 00:14:24,790
The initial conditions 
for the intelligent explosion

281
00:14:24,790 --> 00:14:27,653
might need to be set up 
in just the right way

282
00:14:27,653 --> 00:14:31,183
if we are to have a controlled detonation.

283
00:14:31,183 --> 00:14:33,801
The values that the AI has
need to match ours,

284
00:14:33,801 --> 00:14:35,621
not just in the familiar context,

285
00:14:35,621 --> 00:14:37,999
like where we can easily check
how the AI behaves,

286
00:14:37,999 --> 00:14:41,233
but also in all novel contexts
that the AI might encounter

287
00:14:41,233 --> 00:14:42,790
in the indefinite future.

288
00:14:42,790 --> 00:14:47,527
And there are also some esoteric issues
that would need to be solved, sorted out

289
00:14:47,527 --> 00:14:49,616
the exact decisions 
of its decision theory,

290
00:14:49,616 --> 00:14:53,330
how to deal with 
logical uncertainty and so forth.

291
00:14:53,330 --> 00:14:56,522
So the technical problems that need
to be solved to make this work

292
00:14:56,522 --> 00:14:57,515
look quite difficult,

293
00:14:57,515 --> 00:15:00,925
-- not as difficult as making 
a super intelligent AI,

294
00:15:00,925 --> 00:15:03,793
but fairly difficult.

295
00:15:03,793 --> 00:15:05,488
Here is the worry:

296
00:15:05,488 --> 00:15:10,172
making super intelligent AI
is a really hard challenge.

297
00:15:10,172 --> 00:15:12,720
Making super intelligent AI that is safe

298
00:15:12,720 --> 00:15:15,426
involves some additional 
challenge on top of that.

299
00:15:15,426 --> 00:15:18,133
The risk is that if somebody
figures out how to crack

300
00:15:18,133 --> 00:15:21,392
the first challenge without also
having cracked

301
00:15:21,392 --> 00:15:25,402
the additional challenge 
of ensuring perfect safety.

302
00:15:25,402 --> 00:15:28,706
So I think that we should
work out a solution

303
00:15:28,706 --> 00:15:31,528
to the controlled problem in advance,

304
00:15:31,528 --> 00:15:34,608
so that we have it available 
by the time it is needed.

305
00:15:34,608 --> 00:15:37,875
Now it might be that we cannot
solve the entire controlled problem

306
00:15:37,875 --> 00:15:41,299
in advance because maybe some
element can only be put in place

307
00:15:41,299 --> 00:15:43,576
once you know the details of 
the architecture

308
00:15:43,576 --> 00:15:45,053
where it will be implemented.

309
00:15:45,053 --> 00:15:48,676
But the more of the controlled problem
that we solve in advance,

310
00:15:48,676 --> 00:15:52,766
the better the odds that the transition
to the machine intelligence era

311
00:15:52,766 --> 00:15:54,536
will go well.

312
00:15:54,536 --> 00:15:58,950
This to me looks like a thing
that is well worth doing

313
00:15:58,950 --> 00:16:02,282
and I can imagine that if 
things turn out okay,

314
00:16:02,282 --> 00:16:05,430
that people in a million years
from now

315
00:16:05,430 --> 00:16:06,858
look back at this century

316
00:16:06,858 --> 00:16:08,972
and it might well be 
that they say

317
00:16:08,972 --> 00:16:11,119
that he one thing we did
that really mattered

318
00:16:11,119 --> 00:16:13,037
was to get this thing right.

319
00:16:13,037 --> 00:16:14,198
Thank you.

320
00:16:14,198 --> 00:16:17,011
(Applause)