Mind reading with brain scanners | John-Dylan Hayes | TEDxBerlin

This talk was given at a local TEDx event, produced independently of the TED Conferences.

The research of John-Dylan Haynes focuses on the neural mechanisms underlying human cognitive processes. His special interests are technical and ethical foundations of mental state decoding, as well as the neuroscience of consciousness, intentions and free will. He talks about mind reading with brain scanners.

OK, so it's time
to let your trousers down.

Let's face it, we all have
a couple of secrets, yeah?

Everyone has a secret,

and I'd like to invite you to participate
in a simple experiment right now.

I'd like you to think one of your secrets.

Just think it by conjuring up
the image in your mind.

We tend to think these secrets are secret;
nobody knows about them.

You would think because it's in your mind

it doesn't mean
your neighbor sitting next you

can understand
or know about your secret.

That's why I want to challenge here.

I want to ask you about
whether your privacy is real,

the way you feel it.

Now your secret might
be something mundane.

It might be something simple such as,
for example, the PIN of your credit card.

It could be something embarrassing
you don't want people to know,

for example, it could be you enjoy
watching funny cat movies on the Internet,

something like that.

So, is it possible
to read out these thoughts?

In fact, we can tell about the person's
mental state to some degree

about their feelings
from their body language,

from the expressions on their faces.

If you look at these
famous Ekman faces here,

you can see that our facial expressions
give away what we feel,

but this is still severely limited.

If you look at the person on the left,
obviously she is surprised.

But what is she surprised about?
What is she looking at?

What is she hearing?
What is surprising her? We don't know.

We can't tell this
from the surface of the body,

from the body language or from face;
we somehow need to go deeper.

And deeper here means
that we need to go into that part

of the body that encodes
and stores your thoughts,

and that is the human brain.

Now, I am a brain scientist,
and for many years brain scientists

have established fact after fact
about the brain.

One thing that's always shown up

is that our thoughts are intimately
related to the brain processes.

So, it should in theory be possible -
if I put someone in a brain scanner -

to find out what they are thinking.

The question though is: How do I know
to interpret the language of the brain?

I somehow need to translate it
when I want to read it out.

I need some understanding
of the way information is stored.

So, a simple analogy can help us here.

We could think of the way
in which information is stored

on the surface of a CD.
How does this work?

It's the pattern of pits on the surface
of a CD that codes a piece of information,

say, for example, about a piece of music:

One piece of music,
one pattern of surface;

a different piece of music,
a different pattern of surface.

This is very similar to the way
your brain stores information.

It stores your thoughts
in spatial patterns of the brain activity.

So, how can we read these out?

How do we know what the spatial patterns
of the brain activity are?

For this we use so-called brain scanners.

It's a technology known as MRI,
Magnetic Resonance Imaging,

and basically involves
very strong magnetic fields.

It doesn't do you any harm,

but it gives you a quite well resolved map
of your brain activity.

So, that's what we do.

My job is people come to my lab,
at the Charité here in Berlin,

and we put people in brain scanners,

we read out the brain activity patterns,

and now we get something
that looks like this.

A brain image.

Don't confuse these brain images
with photographs.

They are not photographs.

They are not like accurate, perfect
depictions of some spatial events.

What they show is a spatial map
of probability

that tells you most likely
this part of the brain is involved

in some kinds of thought.

You can see some regions
are hot colored, red or orange;

they are the regions that become
more active than usual

when you engage in a specific thought.

The cold areas, blue areas are
where the activity is lower than normal.

But what does this specific pattern
of the brain activity mean?

Does anyone know what this person
is currently thinking about?

What do you think?

It's difficult.

It doesn't say on there
this person is thinking

about the Brandenburg Gate
or something like that.

It's not written in there.

Instead it's an abstract code
that the brain uses.

This is something
that's quite familiar to us.

We know this.

People faced this problem
over 200 years ago;

in fact, when they
encountered the hieroglyphs.

People 200 years ago
were puzzled by hieroglyphs;

they didn't know what hieroglyphs mean.

They saw this on the pyramids.

Some people must have said,
"That must be like a pretty wallpaper.

But instead, others might have said,
"I think it actually means something."

But what does it mean?

What do you think this means?
It's a very similar problem.

What does this mean?
Like the brain image, you can't know.

I can tell you this one here means
the brain as you might have guessed.

But how do you know this?

I don't have any ancient Egyptians
in my family. How do I know this?

It's because people
stumbled on the translation.

In 1799, someone found the translation,
the famous Rosetta Stone,

that you can see
in the British Museum in London.

This famous Rosetta Stone
contains a text in hieroglyphs

and a text in ancient Greek --
and Demotic as well,

that wasn't quite so much of interest.

So this allowed us to translate
hieroglyphs into ancient Greek,

a language that people
did know to understand.

This now makes the task very simple.

All we need to do is to find
the "Rosetta Stone" for the brain, right?

It sounds easy.

Where is it? Where do we
have to go and dig?

Do we have to go and dig
in North Africa or Southeast Asia?

Where is the "Rosetta Stone"
for the brain going to be?

Nobody so far unfortunately has found
the "Rosetta Stone" for the brain.

The reason is it is very very complicated,
much more complicated than hieroglyphs.

We need to use something different,
something called codebreaking.

Now, in the Second World War, the Germans
thought they were particularly clever

by encrypting their messages
on this famous Enigma machine.

They took a military text,
they put it through the machine,

and out came some scrambled version
that nobody could understand.

Then in the U-boats they
would find a scrambled text,

and they put it through the machine,
out they'd get the real message.

This is what you can do
if you know the code.

The British were trying to decipher it,
but they didn't know the code.

It was said they invented
brute force, statistical techniques

that allowed them to translate
this hidden code,

and that's what we do today
with brain images.

We try to understand
what the thoughts are a person has.

How does this work in detail?

l'll just give you a simple example.

What you can see here
are three images on the left

the person was looking at
inside the brain scanner:

the Brandenburg Gate, a bouquet
of flowers and a German shepherd dog.

On the right are the corresponding
patterns of the brain activity.

As you can see they are very different.

Every thought you have has its own
unique signature pattern of activity

in your brain.

You think one thing,
you get one pattern of activity;

you think something else,
you get a different pattern of activity.

Now what we do is we train computers

to recognize these patterns
of brain activity.

If I know to recognize
the pattern of activity in the top right,

I can tell that the person is thinking
about the Brandenburg Gate,

similar to recognizing an individual
based on their fingerprints.

If I find their fingerprints,
and I have them in a database,

I can use the software to detect
the fingerprints out of our database.

That's what we do.

We train the computers;
we train them to recognize

these brain activity patterns,

and then the computer has to prove
that it's worth the money,

or if the program we write
is worth its money,

that it can decipher
what you are thinking about,

and then you give the computer
some pattern of brain activity;

you might not know what it is,

and the proof is if the computer can
correctly tell what you're thinking about.

This works remarkably well.

If you do this simple game,
you get accuracy of up to 100%.

You get it right almost every time.

So this is for people in brain scanners
looking at visual images.

But you could say,

"If someone's in a brain scanner
looking at an image,

I just have to look at the image
they're looking at

and I'll know what
they're thinking about."

So, we need to go one step further.

The next step is we just
have to imagine an image.

Put them in a brain scanner,
we get them to imagine various things,

for example, watches, hands,
faces, all sorts of stuff,

we measure the brain activity patterns,
and we can decode

what they're thinking about
based on pure imagination.

It's just in their mind,
there's nothing in their environment

or on the computer screen to give it away,
they're just thinking about it.

This works incredibly well, as well.

But you could say,

"This is still rather academic. Let's look
for something even more interesting.

What about people's behavior,
their decisions,

things that are relevant?"

So we went ahead
and did a number of studies

where we tried to look
at people's decisions.

These are the decisions,
very simple decisions

among a few alternatives
for things either completely irrelevant,

such as, for example:

"Do you want to press
left or right button?",

or things that are a bit
more relevant, for example:

"Do you like this car?
Do you want to buy it?",

or "Do you like this car?
Do you want to buy it?"

So we read the brain activity
out of their brains.

In the first case, we looked
after they've made up their mind.

So people make up their mind,
we read their brain activity,

we try to decipher
until which choice they've made.

So everyone of you make a choice,

say, for example, between wanting
a BMW or a Mercedes.

You make your choice.

Can we read this
out of your brain activity?

The answer is yes,
with pretty decent accuracy.

The next question is, however,
what about the choice itself?

You're more or less free to choose
between one or the other.

What made you go one or the other way?

So we look backwards in time
before the decision.

And what we found was
that you could, to some degree,

predict a person's decision --
a free decision,

they can choose either alternative --

before they actually feel
that they're making up their mind,

up to 7 seconds before they think
they're making up their mind.

So you think you're deciding now,
but your brain has at least started

preparing this decision 7 seconds before.

How is that possible?

Well, it raises an important riddle.

That is, we normally think

of the sequence of events
when we make choices,

we decide in our mind,
we don't tell anyone, we just decide,

and then we use our brain
to put our body into motion,

for example, to tell people
what we're thinking about.

But this says it's quite different.

This says the brain
starts preparing the decision,

and then your mind comes
in a few seconds later,

and then you start moving
and doing things.

This is quite different, and it raises
lots of interesting questions.

The most important one is:
Who's responsible for the action?

If your conscious mind comes
after the decision,

that presumably has already been made,

how can you be held responsible
for the action?

If you commit a crime,
it was my unconscious brain activity

that happened before I made up my mind,

it's not my fault
that I committed this crime,

you could say.

So you can see there are lots
of interesting questions raised here

with respect to free will.

I'm going to leave that as an open teaser.

I'm going to focus
on a different question.

If you're interested in that,
go to our website.

I'm going to ask
a different question, and that is:

Can we read people's minds
in a technology

that is similar now to these stage acts
that people did in the 19th century?

What you can see here
is a depiction of a stage act,

a magician draws a person
from the audience,

they come on the stage and are blindfolded
and think about something,

say, for example, a shape or a person,

and the magician can
magically read their mind

and draw their thought on the board here.

Can we now replace this stage act
with something new?

Can we put a brain scanner on the stage?

And then the famous brain scanner,
you put people in it,

and it shows you on a board
what they're thinking about.

Is this possible?

The answer is: to some degree,
but not that well.

I'll explain to you why.

First of all, there's severe limitation
with the resolution of brain scanners.

They have a resolution
of a few millimeters,

so that means in every one of these
small measurement volumes we have,

we have up to a million nerve cells.

That's a lot of nerve cells
in one measurement pixel.

So basically we need to increase
the resolution of these brain scanners,

so we can get down
to the activity of single cells,

if you really want
to read out the full details,

but there are even worse issues
in this than just the resolution.

One big problem is that everyone of you
has that own signature brain language

that stores information.

You can see here 4 brain activity patterns

for the same thought
in 4 different individuals.

They're all different.

Everyone stores information differently,

which explains why we have
such a hard time understanding each other,

if our brain activity patterns
are so different.

So this is a severe challenge.

So we can't train our machine
on one person

and use it for another person.

It doesn't work that well
on the second person.

Another problem is:

The preferences and thoughts
change on the brain language

presumably changes
well across the lifespan.

When you're young
your favorite movie might be Lassie,

your favorite food might be a hamburger,

and your favorite music
might have been heavy metal.

Come on, we're talking about secrets here.
That's your secret, my secret.

When you're older, things might change,
now you like schmaltzy Austrian movies,

you like salad or at least pretend
to yourself that you like salad

and Bavarian brass music.

Come one, admit it.

So the problem is
that the brain activity patterns

causing this kind of information
change across a lifespan,

and we got no way of dealing with this.

But now the biggest problem is that we
can think so many different things.

One of my favorite sentences
is from Monty Python,

it's called, "My hovercraft
is full of eels."

Who would've ever thought that
someone was going to think this today?

If you're to program
the computer to predict

what you might be thinking about today,

I'm pretty sure
"My hovercraft is full of eels"

would've been very low on that list.

So we need to be able to account
for these situations as well.

We can think so many different things.

We could take a whole dictionary
full of different things.

So, we need a dictionary to translate
thoughts into brain activity patterns,

and we don't have this dictionary.

So, what could this look like?

Do we get a subject to come,
we pay them very well,

they come for 20 years
to the brain scanner.

The first day we start
reading the dictionary.

We begin with apple, and we continue,
and after 20 years we've finally done,

but with zed, we'd start
reading zebra and etc.

Is that how we measure
the brain activity patterns

for all these thoughts they might have?

Well, luckily we can take a shortcut.

The shortcut is that the brain’s
arranged information in a way

that is easier for us to understand.

And that is it uses
principles of similarity,

and I explain to you how this works.

You can see in the top right
a hypothetical brain activity pattern

while the person is thinking about a car,
and the bottom right,

a hypothetical brain activity pattern
while they're thinking about a bike.

But now we've measured,
with a brain scanner, a different pattern.

It looks like a mixture for the pattern
of a car and the pattern of a bike.

What could that be?
What do you think?

Well, it's a motorbike.

So you can exploit
these principles of similarity.

You don't need to know
what every thought looks like in the brain

and you can still read out
a large number of thoughts.

Now there are a number
of technical issues ensuing from this.

For example, potential lie detectors
or brain marketing,

lots of people are interested
in commercial applications

or forensic applications
of this technology,

and medical applications allowing people
who can't move their bodies

to suddenly move and control computers
and do things like that.

Now there's a big debate necessary

about what of these technologies we want
and what we think is ethically feasible.

But one important question
people are always interested in is:

Does that mean now
you open to manipulation?

Well, don't worry about that too much
because first of all, it would mean

that you could have to put someone
inside a brain scanner.

So to some degree,
your thoughts are private.

If you're just walking through the world,

nobody has a mobile brain scanner
that works remotely.

You can think about your PIN
of your credit card,

nobody is going to be able to read it out.
Don't worry about that.

Now, one thing that people
are still quite interested in

is whether you can be manipulated,
"manipulated" here means:

Is it possible to write thoughts
into a person's brain

to make them think certain things

as you can see
in some science fiction movies

like in "Strange Days,"
or "Inception," and etc.?

The answer is today this is not possible.

And the reason is that the way
our thoughts are coded in our brain

is in very intricate fine grain patterns
of brain activity,

there's really lots of detail
in these brain activity patterns.

Today, there are techniques
for stimulating the brain,

but they're very diffuse;

there's no technique available
that would allow us to write

these very detailed patterns
of activity into the brain.

So, brain scanners can be used to read out
a person's thoughts to some degree.

Don't worry about the technology,

and especially you don't have to worry
about people programming thoughts

into your brains.

Thanks very much.

(Applause)