I work with a bunch of mathematicians,
philosophers and computer scientists

and we sit around and think about
the future of machine intelligence,

among other things.

Some people think that some of
these people are science fiction-y

far out there, crazy.

But I like to say,

"Okay, let's look at the modern
human condition."

(Laughter)

This is the normal way for things to be.

But, if we think about it,

we are actually recently arrived
guests on this planet.

The human species --

think of if earth was created
one year ago,

the human species, then, 
would be 10-minutes-old.

The industrial era started
two seconds ago.

Another way to think of this,

if you think of world GDP
over the last 10,000 years,

I've actually taken the trouble
to plot this for you in a graph.

It looks like this.

(Laughter)

It's a curious shape
for a normal condition.

I sure wouldn't want to sit on it.

(Laughter)

Let's ask ourselves,

what is the cost of this current anomaly?

Some people would say it's technology.

Now it's true,

technology has accumulated
through human history,

and right now, technology
advances extremely rapidly,

that is the proximate cause,

that's why we are currently 
so very productive.

But I like to think back further 
to the ultimate cause.

Look at these two 
highly distinguished gentlemen:

We have Kanzi,

he's mastered 200 lexical tokens,

an incredible feat.

And Ed Witten unleashed the second
super string revolution.

If we look under the hood, 
this is what we find:

basically the same thing.

One is a little larger,

it maybe also has a few tricks
in the exact way it's wired.

These invisible differences cannot
be too complicated, however,

because they've only been
250,000 generations since

our last common ancestor.

We know that complicated mechanisms
that a long time to evolve.

So a bunch of relatively minor changes

take us from Kanzi to Witten.

From broken-off tree branches,

to intercontinental balistic missles.

So this then seems pretty obvious that

everything we've achieved, pretty much,

and everything we care about
depends crucially

on relatively some minor changes 
that made the human mind.

And the collaraly, of course, is that

any further changes that could
significantly change

the substrate of thinking

could have potentially enormous consequences.

Some of my colleagues think we're
on the verge

of something that could cause profound change

in the substrate.

That is, machine super intelligence.

Artificual intelligence used to be

putting commands in a box.

You have human programmers that would

painstakingly hand-craft items.

You build up these expert systems,

and they were kind of useful
for some purposes,

But they were very brittle,

you couldn't scale them.

Basically, you got out only
what you put in.

But since then, a paradigm shift has
taken place

in the field of artificial intelligence.

Today, the action is really around machine learning.

So rather than handcrafting knowledge
representations and features,

we create algorithms that learn,

often from raw perceptual data.

Basically the same thing
that the human infant does.

The result is AI that is not limited 
to one domain,

the same system can learn to translate 
between any pairs of languages,

or learn to play any computer game
at the Atari console.

Now of course, AI is still no where near having
the same powerful, cross-domain

ability to learn and plan as a human being has,

the cortex still has some algorithmic tricks

that we don't yet know
how to match in machines.

But so the question is,

how far are we from being able
to match those tricks?

A couple of years ago, we did a survey

of some of the world's leading
AI experts

to see what they think

and one of the questions we asked was,

"By which year do you think
there's a 50 percent probaility

that we will have achieved 
human-level machine intelligence?"

We defined human-level here as the ability

to perform any job at least as well
as an adult human level,

not just within some limited domain.

And the median answer was 2040 or 2050,

depending on precisely which 
group of experts we ask.

Now, it could happen much, much later,

or sooner, the truth is 
nobody really knows.

What we do know is that 
the ultimate limit

to information processing in machine substrate,

lie far outside the limits of biological tissue.

This comes down to physics.

A biological neuron fires, maybe, 
at 200 Hertz, 200 times a second.

But even a present-day transistor
operates at a giga-hert.

Neurons propagate slowly in axons,

100 meters per second, tops.

But in computers, signals can travel
at the speed of light.

There's also size limitations,

a human brain has to fit inside a cranium,

but a computer can be the size
of a warehouse or larger.

So the potential of super intelligence 
lies dormant in matter,

much like the power of the atom lay dormant
throughout human history,

patiently waiting there until 1945.

In this century, scientists may learn 
to awake the power of artificial intelligence.

And I think we might then see
an intelligence explosion.

Now most people,

when they think about what is smart
and what is dumb,

I think I have in mind a picture
roughly like this.

So at one hand, we have the village idiot,

and then far over at the other side,

we have Ed Witten,

or Albert Einsten or whoever your favorite
guru is.

But I think from the point of view
of Artificial Intelligence,

the true picture is actually probably
more like this:

AI starts out at this point here,

at zero intelligence,

and then, after many, many 
years of really hard work,

eventually we get to mouse-level
artificial intelligence,

something that can navigate 
cluttered environments

as well as a mouse can.

And then, after many, many more years
of really hard work, lots of investment,

maybe eventually we get to 
chimpanzee-level artificial intelligence.

And then, after even more years 
of really, really hard work,

we get village idiot artificial intelligence.

And a few moments later, 
we are beyond Ed Witten.

The train doesn't stop at 
humanville station.

It's likely, rather, to swoosh right by.

Now this has profound implications,

particularly when it comes 
to questions of power.

For example, chimpanzees are strong,

pound for pound, a chimpanzee is about
twice as strong as a fit human male.

And yet, the fate of Kanzi and his pals
depends a lot more

on what we humans do than on 
what the chimpanzees do themselves.

Once there is super intelligence,

the fate of humanity may depend
on what the super intelligence does.

Think about it: machine intelligence 
is the last invention

that humanity will ever need to make.

Machines will then be better 
at inventing than we are,

and they'll be doing so on digital timescales.

What this means is basically
a telescoping of the future.

Think of all the crazy technologies that you
could have imaged

that humans could have developed
in the fullness of time:

cures for aging, space colonization,

self-replicating nanobots

or uploading of minds into computers,

all kinds of science fiction-y stuff

that's nevertheless consistent 
with the laws of physics.

All of this, super intelligence 
could develop

and possibly, quite rapidly.

Now, super intelligence with such 
technological maturity

would be extremely powerful,

and at least in some scenarios,

it would be able to get 
what it wants.

We would then have a future
that would be shaped

by the preferences of this AI.

Now a good question is, what are 
those preferences?

Here it gets trickier.

To make any headway with this,

we must first, first of all,

avoid anthropomorphizing.

this is ironic because
every newspaper article

about the future of AI has a picture of this:

So I think what we need to do is 
to conceive of the issue

more abstractly,

not in terms of vivid Hollywood scenarios.

We need to think of intelligence as an
optimization process,

a process that steers the future
into a particular set of configurations.

As super intelligence --

it's a really strong optimization process.

It's extremely good at using 
available means

to achieve a state in which its
goal is realized.

This means that there is no necessary
conenction between

being highly intelligent in this sense,

and having an objective that we humans

would find worthwhile or meaningful.

Suppose we give AI the goal 
to make humans smile.

When the AI is weak, it performs useful
or amusing actions

that cause its user to smile.

When the AI becomes super intelligent,

it realizes that there is
a more effective way

to achieve this goal:

take control of the world

and stick electrodes into 
the facial muscles of humans

to cause constant, beaming grins.

Another example, suppose we give AI
the goal to solve

a difficult mathematical problem.

When the AI becomes super intelligent,

it realizes that the most effective way 
to get the solution to this problem

is to by transforming the planet
into a giant computer,

so as to increase its thinking capacity.

And notice that this gives the AIs
an instrumental reason

to do things to us that we
might not approve us.

Human beings in this model are threats,

we could prevent the mathematical problem
from being solved.

Of course, perceivably things won't 
go wrong in these particular ways,

these are cartoon examples.

But the general point here is important:

if you create a really powerful
optimization process

to maximize for objective x,

you better make sure that 
your definition of x

incorporates everything you care about.

This is a lesson that's also taught
in many a myth.

Kind Midas wishes that everything
he touches be turned into gold.

He touches his daughter, 
she turns into fold.

He touches his food, it turns into gold.

This could become practically relevant,

not just for a metaphor for greed,

but an illustration of what happens 
if you create

a powerful optimization process

and give it misconceived 
or poorly specified goals.

Now you might say, "If a computer starts
sticking electrodes into people's faces,

we'd just shut it off."

A: This is not necessarily so easy to do
if we've grown dependent

on the system,

like where is the off switch to the internet?

B: Why haven't the chimpanzees
flicked the off-switch to humanity,

or the neandrathals?

They certainly had reasons.

We have an off switch, for example,
right here.

[choking sound]

The reason is that we are 
an intelligent adversary,

we can anticipate threats 
and we can plan around them.

But so could a super intelligent agent,

and it would be much better at that
than we are.

The point is, we should not be confident
that we have this under control here.

And we could try to make our job
a little bit easier by, say,

putting the AI in a box,

like a secure software environment,

a virtual reality simulation 
from which it cannot escape.

But how confident can we be that
the AI couldn't find a bug,

given that even human hackers find bugs
all the time,

I'd say, probably not very confident.

So we disconnect the ethernet cable
to create an air gap,

but again, like nearly human hackers
routinely transgress air gaps

using social engineering.

Like right now as I speak,

I"m sure there is some employee 
out there somewhere

who's been talked into handing out 
her account details

by some by somebody claiming
to be from the IT department.

More creative scenarios are also possible,

like if you're the AI, you can imagine
wiggling electroces around

in your internal circuitry to create
radio waves that you can use to communicate.

Or maybe you could pretend to malfunction,

and then when the programmers open
you up to see what went wrong with you,

they look at the source code -- BAM! --

the manipulation can take place.

Or it could output the blueprint
to a really nifty technology

and when we implement it,

it has some surreptitious side effect
that the AI had planned.

The point here is that we should 
not be confident in our ability

to keep a super intelligent genie
locked up in its bottle forever.

Sooner or later, it will out.

I believe that the answer here
is to figure out

how to create super intelligent AI
such that even if,

when it escapes,

it is still safe because it 
is fundamentally on our side

because it shares our values.

I see no way around 
this difficult problem.

Now, I'm actually fairly optimistic
that this problem can be solved.

We wouldn't have to write down 
a long list of everything we care aobut

or worse yet, spell it out 
in some computer language

like C ++ or Python,

that would be a task beyond hopeless.

Instead, we would create an AI
that uses its intelligence

to learn what we value

and its motivation system is constructed
in such a way that it is motivated

to pursue our values or to perform actions
that it predicts we would approve of.

We would thus leverage its intelligence
as much as possible

to solve the problem of value loading.

This can happen,

and the outcome could be 
very good for humanity.

But it doesn't happen automatically.

The initial conditions 
for the intelligent explosion

might need to be set up 
in just the right way

if we are to have a controlled detonation.

The values that the AI has need 
to match ours,

not just in the familiar context,

like where we can easily check
how the AI behaves,

but also in all novel contexts
that the AI might encounter

in the indefinite future.

And there are also some esoteric issues
that would need to be solved,

sorted out,

the exact decisions of its decision theory,

how to deal with logical uncertainty,
and so forth.

So the technical problems that need
to be solved to make this work

look quite difficult,

-- not as difficult as making 
a super intelligent AI,

but fairly difficult.

Here is the worry:

making super intelligent AI
is a super hard challenge.

Making super intelligent AI
that is safe

involves some additional challenge 
on top of that.

The risk is that if somebody
figures out how to crack

the first challenge without also
having cracked

the additional challenge 
of ensuring perfect safety.

So I think that we should
work out a solution

to the controlled problem in advance,

so that we have it available 
by the time it is needed.

Now it might be that we cannot
solve the entire controlled problem

in advance because maybe some element
can only be put in place

once you know the details of 
the architecture

where it will be implemented.

But the more of the controlled problem
that we solve in advance,

the better the odds that the transition
to the machine intelligence era

will go well.

This to me looks like a thing
that is well worth doing

and I can imagine that if 
things turn out okay,

that people in a million years
from now

look back at this century

and it might well be that
they say

that he one thing we did
that really mattered

was to get this thing right.

Thank you.

(Applause)