Machine intelligence makes human morals more important

So I started my first job
as a computer programmer

in my very first year of college,

basically as a teenager.

Soon after I started working,

writing software in a company,

a manager who worked at the company
came down to where I was,

and he whispered to me,

"Can he tell if I'm lying?"

There was nobody else in the room.

"Can who tell if you're lying,
and why are we whispering?"

The manager pointed
at the computer in the room.

"Can he tell if I'm lying?"

Well, that manager was having
an affair with the receptionist,

and I was still a teenager,

so I whisper-shouted back to him,

"Yes, the computer can tell
if you're lying."

(Laughter)

Well, I laughed, but actually
the laugh's on me.

Nowadays, there are computational systems
that can suss out emotional states,

and even lying,

from processing human faces.

Advertisers and even governments
are very interested.

I had become a computer programmer
because I was one of those kids

crazed about math and science,

but somewhere along the line
I'd learned about nuclear weapons,

and I'd gotten really concerned
with the ethics of science.

I was troubled.

However, because of family circumstances,

I also needed to start working
as soon as possible.

So I thought to myself, hey,
let me pick a technical field

where I can get a job easily

and where I don't have to deal
with any troublesome questions of ethics.

So I picked computer.

Well ha ha ha, all the laughs are on me.

Nowadays, computer scientists
are building platforms that control

what a billion people see every day.

They're developing cars that
could decide who to run over.

They're even building machines, weapons,
that might kill human beings in war.

It's ethics all the way down.

Machine intelligence is here.

We're now using computation
to make all sort of decisions,

but also new kinds of decisions.

We're asking questions to computation
that have no single right answers,

that are subjective and open-ended
and value-laden.

We're asking questions like,

"Who should the company hire?"

"Which update from which friend
should you be shown?"

"Which convict is more
likely to re-offend?"

"Which news item or movie
should be recommended to people?"

Look, yes, we've been using
computers for a while,

but this is different.

This is a historical twist,

because we cannot anchor
computation for such subjective decisions

the way we can anchor computation

for flying airplanes, building bridges,
going to the moon.

Are airplanes safer?
Did the bridge sway and fall?

There we have agreed-upon,
fairly clear benchmarks,

and we have laws of nature to guide us.

We have no such anchors and benchmarks

for decisions in messy human affairs.

To make things more complicated,
our software is getting more powerful,

but it's also getting less transparent
and more complex.

Recently, in the past decade,

complex algorithms have made
great strides.

They can recognize human faces.

They can decipher handwriting.

They can detect credit card fraud
and block spam

and they can translate between languages.

They can detect tumors in medical imaging.

They can beat humans in chess and go.

Much of this progress comes from
a method called machine learning.

Machine learning is different
than traditional programming,

where you give the computer
detailed, exact, painstaking instructions.

It's more like you take the system
and you feed it lot of data,

including unstructured data,

like the kind we generate
in our digital lives.

And the system learns
by churning through this data.

And also crucially, these systems
don't operate under a single answer logic.

They don't produce a simple answer.

It's more probabilistic.

This one is probably more
like what you're looking for.

Now the upside is, this method
is really powerful.

The head of Google's AI systems called it
the unreasonable effectiveness of data.

The downside is we don't really
understand what the system learned.

In fact, that's its power.

This is less like giving instructions
to a computer.

It's more like training a puppy
machine creature

we don't really understand or control.

So this is our problem.

It's a problem when this artificial
intelligence system gets things wrong.

It's also a problem

when it gets things right,

because we don't even know which is which
when it's a subjective problem.

We don't know what this thing is thinking.

So consider a hiring algorithm,

a system used to hire people
using machine learning systems.

Such a system would have been trained
on previous employees' data

an instructed to find and hire
people like the existing

high performers in the company.

Sounds good.

I once attended a conference
that brought together

human resources, managers,
and executives, high-level people,

using such systems in hiring.

They were super-excited.

They thought that this would make hiring
more objective, less biased,

and give women
and minorities a better shot

against biased human managers.

And look, human hiring is biased.

I know.

In one of my early jobs as a programmer,

my immediate manager would sometimes
come down to where I was

really early in the morning

or really late in the afternoon,

and she'd say, "Zeynep,
let's go to lunch!"

I'd be puzzled by the weird timing.

It's 4 pm? Lunch?

I was broke, so free lunch. I always went.

I later realized what was happening.

My immediate managers
had not confessed to their higher-ups

that the programmer they hired
for a serious job was a teen girl

who wore jeans and sneakers to work.

I was doing a good job.

I just looked wrong and was
the wrong age and gender.

So hiring in a gender- and race-blind way

certainly sounds good to me.

But with these systems,
it is more complicated, and here's why.

Currently, computational systems
can infer all sorts of things about you

from your digital crumbs,

even if you have not
disclosed those things.

They can infer your sexual orientation,

your personality traits,

your political leanings.

They have predictive power
with high levels of accuracy,

remember for things
you haven't even disclosed.

This is inference.

I have a friend who developed
such computational systems

to predict the likelihood of clinical
or post-partum depression

from social media data.

The results were impressive.

Her system can predict
the likelihood of depression

months before the onset of any symptoms,

months before.

No symptoms, there's prediction.

She hopes it will be used
for early intervention.

Great.

But now put this in the context of hiring.

So at this human resources
manager's conference,

I approached a high-level manager

in a very large company,

and I said to her, "Look,
what if, unbeknownst to you,

your system is weeding out people with
high future likelihood of depression?

They're not depressed now,
just maybe in the future, more likely.

What if it's weeding out women
more likely to be pregnant

in the next year or two
but aren't pregnant now?

What if it's hiring aggressive people
because that's your workplace culture?"

You can't tell this by looking
at gender breakdowns.

Those may be balanced.

And since this is machine learning,
not traditional coding,

there is no variable there

labeled "higher risk of depression,"

"higher risk of pregnancy,"

"aggressive guy scale."

Not only do you not know
what your system is selecting on,

you don't even know
where to begin to look.

It's a black box.

It has predictive power,
but you don't understand it.

"What safeguards," I asked, "do you have
to make sure that your black box

isn't doing something shady?"

So she looked at me

as if I had just stepped
on 10 puppy tails.

She stared at me and she said,

"I don't want to hear
another word about this."

And she turned around and walked away.

Mind you, she wasn't rude.

It was clearly what I don't know
isn't my problem, go away, death stare.

Look, such a system may even be less
biased than human managers in some ways,

and it could make monetary sense,

but it could also lead to a steady but
stealthy shutting out of the job market

of people with higher risk of depression.

Is this the kind of society
we want to build

without even knowing we've done this

because we turned decision-making
to machines we don't totally understand?

Another problem is this:

these systems are often trained
on data generated by our actions,

human imprints.

Well, they could just be
reflecting our biases,

and these systems could be
picking up on our biases

and amplifying them

and showing them back to us,

while we're telling ourselves, "We're just
doing objective neutral computation."

Researchers found that on Google,

women are less likely than men

to be shown job ads for high-paying jobs,

and searching for African-American names

is more likely to bring up ads
suggesting criminal history,

even when there is none.

Such hidden biases
and black box algorithms

that researchers uncover sometimes

but sometimes we don't know

can have life-altering consequences.