Jennifer Helsby: Prediction and Control

Musik

Herald: Who of you is using Facebook? Twitter?
Diaspora?

concerned noise And all of that data
you enter there

gets to server, gets into the hand of somebody
who's using it

and the next talk
is especially about that,

because there's also intelligent machines
and intelligent algorithms

that try to make something
out of that data.

So the post-doc researcher Jennifer Helsby

of the University of Chicago,
which works in this

intersection between policy and
technology,

will now ask you the question:
To who would we give that power?

Dr. Helsby: Thanks.
applause

Okay, so, today I'm gonna do a brief tour
of intelligent systems

and how they're currently used

and then we're gonna look at some examples
with respect

to the properties that we might care about

these systems having,
and I'll talk a little bit about

some of the work that's been done in academia

on these topics.

And then we'll talk about some
promising paths forward.

So, I wanna start with this:
Kranzberg's First Law of Technology

So, it's not good or bad,
but it also isn't neutral.

Technology shapes our world,
and it can act as

a liberating force-- or an oppressive and
controlling force.

So, in this talk, I'm gonna go
towards some of the aspects

of intelligent systems that might be more
controlling in nature.

So, as we all know,

because of the rapidly decreasing cost
of storage and computation,

along with the rise of new sensor technologies,

data collection devices
are being pushed into every

aspect of our lives: in our homes, our cars,

in our pockets, on our wrists.

And data collection systems act as intermediaries

for a huge amount of human communication.

And much of this data sits in government

and corporate databases.

So, in order to make use of this data,

we need to be able to make some inferences.

So, one way of approaching this is I can hire

a lot of humans, and I can have these humans

manually examine the data, and they can acquire

expert knowledge of the domain, and then

perhaps they can make some decisions

or at least some recommendations
based on it.

However, there's some problems with this.

One is that it's slow, and thus expensive.

It's also biased. We know that humans have

all sorts of biases, both conscious and unconscious,

and it would be nice to have a system
that did not have

these inaccuracies.

It's also not very transparent: I might

not really know the factors that led to

some decisions being made.

Even humans themselves
often don't really understand

why they came to a given decision, because

of their being emotional in nature.

And, thus, these human decision making systems

are often difficult to audit.

So, another way to proceed is maybe instead

I study the system and the data carefully

and I write down the best rules
for making a decision

or, I can have a machine
dynamically figure out

the best rules, as in machine learning.

So, maybe this is a better approach.

It's certainly fast, and thus cheap.

And maybe I can construct
the system in such a way

that it doesn't have the biases that are inherent

in human decision making.

And, since I've written these rules down,

or a computer has learned these rules,

then I can just show them to somebody, right?

And then they can audit it.

So, more and more decision making is being

done in this way.

And so, in this model, we take data

we make an inference based on that data

using these algorithms, and then

we can take actions.

And, when we take this more scientific approach

to making decisions and optimizing for

a desired outcome,
we can take an experimental approach

so we can determine
which actions are most effective

in achieving a desired outcome.

Maybe there are some types of communication

styles that are most effective
with certain people.

I can perhaps deploy some individualized incentives

to get the outcome that I desire.

And, maybe even if I carefully design an experiment

with the environment in which people make

these decisions, perhaps even very small changes

can introduce significant changes
in peoples' behavior.

So, through these mechanisms,
and this experimental approach,

I can maximize the probability
that humans do

what I want.

So, algorithmic decision making is being used

in industry, and is used
in lots of other areas,

from astrophysics to medicine, and is now

moving into new domains, including

government applications.

So, we have recommendation engines like
Netflix, Yelp, SoundCloud,

that direct our attention to what we should

watch and listen to.

Since 2009, Google uses
personalized searched results,

including if you're not logged in
into your Google account.

And we also have algorithm curation and filtering,

as in the case of Facebook News Feed,

Google News, Yahoo News,

which shows you what news articles, for example,

you should be looking at.

And this is important, because a lot of people

get news from these media.

We even have algorithmic journalists!

So, automatic systems generate articles

about weather, traffic, or sports

instead of a human.

And, another application that's more recent

is the use of predictive systems

in political campaigns.

So, political campaigns also now take this

approach to predict on an individual basis

which candidate voters
are likely to vote for.

And then they can target,
on an individual basis,

those that can be persuaded otherwise.

And, finally, in the public sector,

we're starting to use predictive systems

in areas from policing, to health,
to education and energy.

So, there are some advantages to this.

So, one thing is that we can automate

aspects of our lives
that we consider to be mundane

using systems that are intelligent

and adaptive enough.

We can make use of all the data

and really get the pieces of information we

really care about.

We can spend money in the most effective way,

and we can do this with this experimental

approach to optimize actions to produce

desired outcomes.

So, we can embed intelligence

into all of these mundane objects

and enable them to make decisions for us,

and so that's what we're doing more and more,

and we can have an object
that decides for us

what temperature we should set our house,

what we should be doing, etc.

So, there might be some implications here.

We want these systems
that do work on this data

to increase the opportunities
available to us.

But it might be that there are some implications

that we have not carefully thought through.

This is a new area, and people are only

starting to scratch the surface of what the

problems might be.

In some cases, they might narrow the options

available to people,

and this approach subjects people to

suggestive messaging intended to nudge them

to a desired outcome.

Some people may have a problem with that.

Values we care about are not gonna be

baked into these systems by default.

It's also the case that some algorithmic systems

facilitate work that we do not like.

For example, in the case of mass surveillance.

And even the same systems,

used by different people or organizations,

have very different consequences.

For example, if I can predict

with high accuracy, based on say search queries,

who's gonna be admitted to a hospital,

some people would be interested
in knowing that.

You might be interested
in having your doctor know that.

But that same predictive model
in the hands of

an insurance company
has a very different implication.

So, the point here is that these systems

structure and influence how humans interact

with each other, how they interact with society,

and how they interact with government.

And if they constrain what people can do,

we should really care about this.

So now I'm gonna go to
sort of an extreme case,

just as an example, and that's this
Chinese Social Credit System.

And so this is probably one of the more

ambitious uses of data,

that is used to rank each citizen

based on their behavior, in China.

So right now, there are various pilot systems

deployed by various companies doing this in
China.

They're currently voluntary, and by 2020

this system is gonna be decided on,

or a combination of the systems,

that is gonna be mandatory for everyone.

And so, in this system, there are some citizens,

and a huge range of data sources are used.

So, some of the data sources are

your financial data,

your criminal history,

how many points you have
on your driver's license,

medical information-- for example,
if you take birth control pills,

that's incorporated.

Your purchase history-- for example,
if you purchase games,

you are down-ranked in the system.

Some of the systems, not all of them,

incorporate social media monitoring,

which makes sense if you're a state like China,

you probably want to know about

political statements that people
are saying on social media.

And, one of the more interesting parts is

social network analysis:
looking at the relationships between people.

So, if you have a close relationship with
somebody

and they have a low credit score,

that can have implications on your credit
score.

So, the way that these scores
are generated is secret.

And, according to the call for these systems

put out by the government,

the goal is to
"carry forward the sincerity and

traditional virtues" and
establish the idea of a

"sincerity culture."

But wait, it gets better:

so, there's a portal that enables citizens

to look up the citizen score of anyone.

And many people like this system,

they think it's a fun game.

They boast about it on social media,

they put their score in their dating profile,

because if you're ranked highly you're

part of an exclusive club.

You can get VIP treatment
at hotels and other companies.

But the downside is that, if you're excluded

from that club, your weak score
may have other implications,

like being unable to get access
to credit, housing, jobs.

There is some reporting that even travel visas

might be restricted
if your score is particularly low.

So, a system like this, for a state, is really

the optimal solution
to the problem of the public.

It constitutes a very subtle and insiduous

mechanism of social control.

You don't need to spend a lot of money on

police or prisons if you can set up a system

where people discourage one another from

anti-social acts like political action
in exchange for

a coupon for a free Uber ride.

So, there are a lot of
legitimate questions here:

What protections does
user data have in this scheme?

Do any safeguards exist to prevent tampering?

What mechanism, if any, is there to prevent

false input data from creating erroneous inferences?

Is there any way that people can fix

their score once they're ranked poorly?

Or does it end up becoming a

self-fulfilling prophecy?

Your weak score means you have less access

to jobs and credit, and now you will have

limited access to opportunity.

So, let's take a step back.

So, what do we want?

So, we probably don't want that,

but as advocates we really wanna

understand what questions we should be asking

of these systems. Right now there's

very little oversight,

and we wanna make sure that we don't

sort of sleepwalk our way to a situation

where we've lost even more power

to these centralized systems of control.

And if you're an implementer, we wanna understand

what can we be doing better.

Are there better ways that we can be implementing

these systems?

Are there values that, as humans,

we care about that we should make sure

these systems have?

So, the first thing
that most people in the room

might think about is privacy.

Which is, of course, of the utmost importance.

We need privacy, and there is a good discussion

on the importance of protecting
user data where possible.

So, in this talk, I'm gonna focus
on the other aspects of

algorithmic decision making,

that I think have got less attention.

Because it's not just privacy
that we need to worry about here.

We also want systems that are fair and equitable.

We want transparent systems,

we don't want opaque decisions
to be made about us,

decisions that might have serious impacts

on our lives.

And we need some accountability mechanisms.

So, for the rest of this talk

we're gonna go through each one of these things

and look at some examples.

So, the first thing is fairness.

And so, as I said in the beginning,
this is one area

where there might be an advantage

to making decisions by machine,

especially in areas where there have

historically been fairness issues with

decision making, such as law enforcement.

So, this is one way that police departments

use predictive models.

The idea here is police would like to

allocate resources in a more effective way,

and they would also like to enable

proactive policing.

So, if you can predict where crimes
are going to occur,

or who is going to commit crimes,

then you can put cops in those places,

or perhaps following these people,

and then the crimes will not occur.

So, it's sort of the pre-crime approach.

So, there are a few ways of going about this.

One way is doing this individual-level prediction.

So you take each citizen
and estimate the risk

that each citizen will participate,
say, in violence

based on some data.

And then you can flag those people that are

considered particularly violent.

So, this is currently done.

This is done in the U.S.

It's done in Chicago,
by the Chicago Police Department.

And they maintain a heat list of individuals

that are considered most likely to commit,

or be the victim of, violence.

And this is done using data
that the police maintain.

So, the features that are used
in this predictive model

include things that are derived from

individuals' criminal history.

So, for example, have they been involved in

gun violence in the past?

Do they have narcotics arrests? And so on.

But another thing that's incorporated

in the Chicago Police Department model is

information derived from
social media network analysis.

So, who you interact with,

as noted in police data.

So, for example, your co-arrestees.

When officers conduct field interviews,

who are people interacting with?

And then this is all incorporated
into this risk score.

So another way to proceed,

which is the method that most companies

that sell products like this
to the police have taken,

is instead predicting which areas

are likely to have crimes committed in them.

So, take my city, I put a grid down,

and then I use crime statistics

and maybe some ancillary data sources,

to determine which areas have

the highest risk of crimes occurring in them,

and I can flag those areas and send

police officers to them.

So now, let's look at some of the tools

that are used for this geographic-level prediction.

So, here are 3 companies that sell these

geographic-level predictive policing systems.

So, PredPol has a system that uses

primarily crime statistics:

only the time, place, and type of crime

to predict where crimes will occur.

HunchLab uses a wider range of data sources

including, for example, weather

and then Hitachi is a newer system

that has a predictive crime analytics tool

that also incorporates social media.

The first one, to my knowledge, to do so.

And these systems are in use

in 50+ cities in the U.S.

So, why do police departments buy this?

Some police departments are interesting in

buying systems like this, because they're marketed

as impartial systems,

so it's a way to police in an unbiased way.

And so, these companies make

statements like this--

by the way, the references
will all be at the end,

and they'll be on the slides--

So, for example

the predictive crime analytics from Hitachi

claims that the system is anonymous,

because it shows you an area,

it doesn't show you
to look for a particular person.

and PredPol reassures people that

it eliminates any liberties or profiling concerns.

And HunchLab notes that the system

fairly represents priorities for public safety

and is unbiased by race
or ethnicity, for example.

So, let's take a minute
to describe in more detail

what we mean when we talk about fairness.

So, when we talk about fairness,

we mean a few things.

So, one is fairness with respect to individuals:

so if I'm very similar to somebody

and we go through some process

and there is two very different
outcomes to that process

we would consider that to be unfair.

So, we want similar people to be treated

in a similar way.

But, there are certain protected attributes

that we wouldn't want someone

to discriminate based on.

And so, there's this other property,
Group Fairness.

So, we can look at the statistical parity

between groups, based on gender, race, etc.

and see if they're treated in a similar way.

And we might not expect that in some cases,

for example if the base rates in each group

are very different.

And then there's also Fairness in Errors.

All predictive systems are gonna make errors,

and if the errors are concentrated,

then that may also represent unfairness.

And so this concern arose recently with Facebook

because people with Native American names

had their profiles flagged as fraudulent

far more often than those
with White American names.

So these are the sorts of things
that we worry about

and each of these are metrics,

and if you're interested more you should

check those 2 papers out.

So, how can potential issues
with predictive policing

have implications for these principles?

So, one problem is
the training data that's used.

Some of these systems only use crime statistics,

other systems-- all of them use crime statistics

in some way.

So, one problem is that crime databases

contain only crimes that've been detected.

Right? So, the police are only gonna detect

crimes that they know are happening,

either through patrol and their own investigation

or because they've been alerted to crime,

for example by a citizen calling the police.

So, a citizen has to feel like
they can call the police,

like that's a good idea.

So, some crimes suffer
from this problem less than others:

for example, gun violence
is much easier to detect

relative to fraud, for example,

which is very difficult to detect.

Now the racial profiling aspect
of this might come in

because of biased policing in the past.

So, for example, for marijuana arrests,

black people are arrested in the U.S. at rates

4 times that of white people,

even though there is statistical parity

with these 2 groups, to within a few percent.

So, this is where problems can arise.

So, let's go back to this

geographic-level predictive policing.

So the danger here is that, unless this system

is very carefully constructed,

this sort of crime area ranking might

again become a self-fulling prophecy.

If you send police officers to these areas,

you further scrutinize them,

and then again you're only detecting a subset

of crimes, and the cycle continues.

So, one obvious issue is that

this statement about geographic-based
crime prediction

being anonymous is not true,

because race and location are very strongly

correlated in the U.S.

And this is something that machine-learning
systems

can potentially learn.

Another issue is that, for example,

for individual fairness, one of my homes

sits within one of these boxes.

Some of these boxes
in these systems are very small,

for example PredPol is 500ft x 500ft,

so it's maybe only a few houses.

So, the implications of this system are that

you have police officers maybe sitting

in a police cruiser outside your home

and a few doors down someone

may not be within that box,

and doesn't have this.

So, that may represent unfairness.

So, there are real questions here,

especially because there's no opt-out.

There's no way to opt-out of this system:

if you live in a city that has this,

then you have to deal with it.

So, it's quite difficult to find out

what's really going on

because the algorithm is secret.

And, in most cases, we don't know

the full details of the inputs.

We have some idea
about what features are used,

but that's about it.

We also don't know the output.

That would be knowing police allocation,

police strategies,

and in order to nail down
what's really going on here

in order to verify the validity of

these companies' claims,

it may be necessary
to have a 3rd party come in,

examine the inputs and outputs of the system,

and say concretely what's going on.

And if everything is fine and dandy

then this shouldn't be a problem.

So, that's potentially one role that

advocates can play.

Maybe we should start pushing for audits

of systems that are used in this way.

These could have serious implications

for peoples' lives.

So, we'll return
to this idea a little bit later,

but for now this leads us
nicely to Transparency.

So, we wanna know

what these systems are doing.

But it's very hard,
for the reasons described earlier,

but even in the case of something like

trying to understand Google's search algorithm,

it's difficult because it's personalized.

So, by construction, each user is

only seeing one endpoint.

So, it's a very isolating system.

What do other people see?

And one reason it's difficult to make

some of these systems transparent

is because of, simply, the complexity

of the algorithms.

So, an algorithm can become so complex that

it's difficult to comprehend,

even for the designer of the system,

or the implementer of the system.

The designed might know that this algorithm

maximizes some metric-- say, accuracy,

but they may not always have a solid

understanding of what the algorithm is doing

for all inputs.

Certainly with respect to fairness.

So, in some cases,
it might not be appropriate to use

an extremely complex model.

It might be better to use a simpler system

with human-interpretable features.

Another issue that arises

from the opacity of these systems

and the centralized control

is that it makes them very influential.

And thus, an excellent target

for manipulation or tampering.

So, this might be tampering that is done

from an organization that controls the system,

or an insider at one of the organizations,

or anyone who's able to compromise their security.

So, this is an interesting academic work

that looked at the possibility of

slightly modifying search rankings

to shift people's political views.

So, since people are most likely to

click on the top search results,

so 90% of clicks go to the
first page of search results,

then perhaps by reshuffling
things a little bit,

or maybe dropping some search results,

you can influence people's views

in a coherent way,

and maybe you can make it so subtle

that no one is able to notice.

So in this academic study,

they did an experiment

in the 2014 Indian election.

So they used real voters,

and they kept the size
of the experiment small enough

that it was not going to influence the outcome

of the election.

So the researchers took people,

they determined their political leaning,

and they segmented them into
control and treatment groups,

where the treatment was manipulation

of the search ranking results,

And then they had these people
browse the web.

And what they found, is that

this mechanism is very effective at shifting

people's voter preferences.

So, in this study, they were able to introduce

a 20% shift in voter preferences.

Even alerting users to the fact that this

was going to be done, telling them

"we are going to manipulate your search results,"

"really pay attention,"

they were totally unable to decrease

the magnitude of the effect.

So, the margins of error in many elections

is incredibly small,

and the authors estimate that this shift

could change the outcome of about

25% of elections worldwide, if this were done.

And the bias is so small that no one can tell.

So, all humans, no matter how smart

and resistant to manipulation
we think we are,

all of us are subject to this sort of manipulation,

and we really can't tell.

So, I'm not saying that this is occurring,

but right now there is no
regulation to stop this,

there is no way we could reliably detect this,

so there's a huge amount of power here.

So, something to think about.

But it's not only corporations that are interested

in this sort of behavioral manipulation.

In 2010, UK Prime Minister David Cameron

created this UK Behavioural Insights Team,

which is informally called the Nudge Unit.

And so what they do is
they use behavioral science

and this predictive analytics approach,

with experimentation,

to have people make better decisions

for themselves and society--

as determined by the UK government.

And as of a few months ago,

after an executive order signed by Obama

in September, the United States now has

its own Nudge Unit.

So, to be clear, I don't think that this is

some sort of malicious plot.

I think that there can be huge value

in these sorts of initiatives,

positively impacting people's lives,

but when this sort of behavioral manipulation

is being done, in part openly,

oversight is pretty important,

and we really need to consider

what these systems are optimizing for.

And that's something that we might

not always know, or at least understand,

so for example, for industry,

we do have a pretty good understanding there:

industry cares about optimizing for

the time spent on the website,

Facebook wants you to spend more time on Facebook,

they want you to click on ads,

click on newsfeed items,

they want you to like things.

And, fundamentally: profit.

So, already this has some serious implications,

and this had pretty serious implications

in the last 10 years, in media for example.

The optimizing for click-through rate in journalism

has produced a race to the bottom

in terms of quality.

And another issue is that optimizing

for what people like might not always be

the best approach.

So, Facebook officials have said publicly

about how Facebook's goal is to make you happy,

they want you to open that newsfeed

and just feel great.

But, there's an issue there, right?

Because people get their news,

like 40% of people according to Pew Research,

get their news from Facebook.

So, if people don't want to see

war and corpses,
because it makes them feel sad,

so this is not a system that is gonna optimize

for an informed population.

It's not gonna produce a population that is

ready to engage in civic life.

It's gonna produce an amused populations

whose time is occupied by cat pictures.

So, in politics, we have a similar

optimization problem that's occurring.

So, these political campaigns that use

these predictive systems,

are optimizing for votes for the desired candidate,

of course.

So, instead of a political campaign being

--well, maybe this is a naive view, but--

being an open discussion of the issues

facing the country,

it becomes this micro-targeted
persuasion game,

and the people that get targeted

are a very small subset of all people,

and it's only gonna be people that are

you know, on the edge, maybe disinterested,

those are the people that are gonna get attention

from political candidates.

In policy, as with these Nudge Units,

they're being used to enable

better use of government services.

There are some good projects that have

come out of this:

increasing voter registration,

improving health outcomes,

improving education outcomes.

But some of these predictive systems

that we're starting to see in government

are optimizing for compliance,

as is the case with predictive policing.

So this is something that we need to

watch carefully.

I think this is a nice quote that

sort of describes the problem.

In some ways me might be narrowing

our horizon, and the danger is that

these tools are separating people.

And this is particularly bad

for political action, because political action

requires people to have shared experience,

and thus are able to collectively act

to exert pressure to fix problems.

So, finally: accountability.

So, we need some oversight mechanisms.

For example, in the case of errors--

so this is particularly important for

civil or bureaucratic systems.

So, when an algorithm produces some decision,

we don't always want humans to just

defer to the machine,

and that might represent one of the problems.

So, there are starting to be some cases

of computer algorithms yielding a decision,

and then humans being unable to correct

an obvious error.

So there's this case in Georgia,
in the United States,

where 2 young people went to

the Department of Motor Vehicles,

they're twins, and they went

to get their driver's license.

However, they were both flagged by

a fraud algorithm that uses facial recognition

to look for similar faces,

and I guess the people that designed the system

didn't think of the possibility of twins.

Yeah.
So, they just left

without their driver's licenses.

The people in the Department of Motor Vehicles

were unable to correct this.

So, this is one implication--

it's like something out of Kafka.

But there are also cases of errors being made,

and people not noticing until

after actions have been taken,

some of them very serious--

because people simply deferred

to the machine.

So, this is an example from San Francisco.

So, an ALPR-- an Automated License Plate Reader--

is a device that uses image recognition

to detect and read license plates,

and usually to compare license plates

with a known list of plates of interest.

And, so, San Francisco uses these

and they're mounted on police cars.

So, in this case, San Francisco ALPR

got a hit on a car,

and it was the car of a 47-year-old woman,

with no criminal history.

And so it was a false hit

because it was a blurry image,

and it matched erroneously with

one of the plates of interest

that happened to be a stolen vehicle.

So, they conducted a traffic stop on her,

and they take her out of the vehicle,

they search her and the vehicle,

she gets a pat-down,

and they have her kneel

at gunpoint, in the street.

So, how much oversight should be present

depends on the implications of the system.

It's certainly the case that

for some of these decision-making systems,

an error might not be that important,

it could be relatively harmless,

but in this case,
an error in this algorithmic decision

led to this totally innocent person

literally having a gun pointed at her.

So, that brings us to: we need some way of

getting some information about

what is going on here.

We don't wanna have to wait for these events

before we are able to determine

some information about the system.

So, auditing is one option:

to independently verify the statements

of companies, in situations where we have

inputs and outputs.

So, for example, this could be done with

Google, Facebook.

If you have the inputs of a system,

say you have test accounts,

or real accounts,

maybe you can collect
people's information together.

So that was something that was done

during the 2012 Obama campaign

by ProPublica.

People noticed that they were getting

different emails from the Obama campaign,

and were interested to see

based on what factors

the emails were changing.

So, I think about 200 people submitted emails

and they were able to determine some information

about what the emails
were being varied based on.

So there have been some successful

attempts at this.

So, compare inputs and then look at

why one item was shown to one user

and not another, and see if there's

any statistical differences.

So, there's some potential legal issues

with the test accounts, so that's something

to think about-- I'm not a lawyer.

So, for example, if you wanna examine

ad-targeting algorithms,

one way to proceed is to construct

a browsing profile, and then examine

what ads are served back to you.

And so this is something that

academic researchers have looked at,

because, at the time at least,

you didn't need to make an account to do this.

So, this was a study that was presented at

Privacy Enhancing Technologies last year,

and in this study, the researchers

generate some browsing profiles

that differ only by one characteristic,

so they're basically identical in every way

except for one thing.

And that is denoted by Treatment 1 and 2.

So this is a randomized, controlled trial,

but I left out the randomization part

for simplicity.

So, in one study,
they applied a treatment of gender.

So, they had the browsing profiles

in Treatment 1 be male browsing profiles,

and the browsing profiles in Treatment 2
be female.

And they wanted to see: is there any difference

in the way that ads are targeted

if browsing profiles are effectively identical

except for gender?

So, it turns out that there was.

So, a 3rd-party site was showing Google ads

for senior executive positions

at a rate 6 times higher to the fake men

than for the fake women in this study.

So, this sort of auditing is not going to

be able to determine everything

that algorithms are doing, but they can

sometimes uncover interesting,

at least statistical differences.

So, this leads us to the fundamental issue:

Right now, we're really not in control

of some of these systems,

and we really need these predictive systems

to be controlled by us,

in order for them not to be used

as a system of control.

So there are some technologies that I'd like

to point you all to.

We need tools in the digital commons

that can help address some of these concerns.

So, the first thing is that of course

we known that minimizing the amount of

data available can help in some contexts,

which we can do by making systems

that are private by design, and by default.

Another thing is that these audit tools

might be useful.

And, so, these 2 nice examples in academia...

the ad experiment that I just showed was done

using AdFisher.

So, these are 2 toolkits that you can use

to start doing this sort of auditing.

Another technology that is generally useful,

but particularly in the case of prediction

it's useful to maintain access to

as many sites as possible,

through anonymity systems like Tor,

because it's impossible to personalize

when everyone looks the same.

So this is a very important technology.

Something that doesn't really exist,

but that I think is pretty important,

is having some tool to view the landscape.

So, as we know from these few studies

that have been done,

different people are not seeing the internet

in the same way.

This is one reason why we don't like censorship.

But, rich and poor people,

from academic research we know that

there is widespread price discrimination
on the internet,

so rich and poor people see a different view

of the Internet,

men and women see a different view

of the Internet.

We wanna know how different people

see the same site,

and this could be the beginning of

a defense system for this sort of

manipulation/tampering that I showed earlier.

Another interesting approach is obfuscation:

injecting noise into the system.

So there's an interesting browser extension

called Adnauseum, that's for Firefox,

which clicks on every single ad you're served,

to inject noise.

So that's, I think, an interesting approach

that people haven't looked at too much.

So in terms of policy,

Facebook and Google, these internet giants,

have billions of users,

and sometimes they like to call themselves

new public utilities,

and if that's the case then

it might be necessary to subject them

to additional regulation.

Another problem that's come up,

for example with some of the studies

that Facebook has done,

is sometimes a lack of ethics review.

So, for example, in academia,

if you're gonna do research involving humans,

there's an Institutional Review Board

that you go to that verifies that

you're doing things in an ethical manner.

And some companies do have internal

review processes like this, but it might

be important to have an independent

ethics board that does this sort of thing.

And we really need 3rd-party auditing.

So, for example, some companies

don't want auditing to be done

because of IP concerns,

and if that's the concern

maybe having a set of people

that are not paid by the company

to check how some of these systems

are being implemented,

could help give us confidence that

things are being done in a reasonable way.

So, in closing,

algorithmic decision making is here,

and it's barreling forward
at a very fast rate,

and we need to figure out what

the guide rails should be,

and how to install them

to handle some of the potential threats.

There's a huge amount of power here.

We need more openness in these systems.

And, right now,

with the intelligent systems that do exist,

we don't know what's occurring really,

and we need to watch carefully

where and how these systems are being used.

And I think this community has

an important role to play in this fight,

to study what's being done,

to show people what's being done,

to raise the debate and advocate,

and, where necessary, to resist.

Thanks.

applause

Herald: So, let's have a question and answer.

Microphone 2, please.

Mic 2: Hi there.

Thanks for the talk.

Since these pre-crime softwares also

arrived here in Germany

with the start of the so-called CopWatch system

in southern Germany,
and Bavaria and Nuremberg especially,

where they try to predict burglary crime

using that criminal record

geographical analysis, like you explained,

leads me to a 2-fold question:

first, have you heard of any research

that measures the effectiveness

of such measures, at all?

And, second:

What do you think of the game theory

if the thieves or the bad guys

know the system, and when they
game the system,

they will probably win,

since one police officer in an interview said

this system is used to reduce

the personal costs of policing,

so they just send the guys
where the red flags are,

and the others take the day off.

Dr. Helsby: Yup.

Um, so, with respect to

testing the effectiveness of predictive policing,

the companies,

some of them do randomized, controlled trials

and claim a reduction in policing.

The best independent study that I've seen

is by this RAND Corporation

that did a study in, I think,

Shreveport, Louisiana,

and in their report they claim

that there was no statistically significant

difference, they didn't find any reduction.

And it was specifically looking at

property crime, which I think you mentioned.

So, I think right now there's sort of

conflicting reports between

the independent auditors
and these company claims.

So there definitely needs to be more study.

And then, the 2nd thing...sorry,
remind me what it was?

Mic 2: What about the guys gaming the system?

Dr. Helsby: Oh, yeah.

I think it's a legitimate concern.

Like, if all the outputs
were just immediately public,

then, yes, everyone knows the location

of all police officers,

and I imagine that people would have

a problem with that.

Yup.

Heraldl: Microphone #4, please.

Mic 4: Yeah, this is not actually a question,

but just a comment.

I've enjoyed your talk very much,

in particular after watching

the talk in Hall 1 earlier in the afternoon.

The "Say Hi to Your New Boss", about

algorithms that are trained with big data,

and finally make decisions.

And I think these 2 talks are kind of complementary,

and if people are interested in the topic

they might want to check out the other talk

and watch it later, because these

fit very well together.

Dr. Helsby: Yeah, it was a great talk.

Herald: Microphone #2, please.

Mic 2: Um, yeah, you mentioned

the need to have some kind of 3rd-party auditing

or some kind of way to

peek into these algorithms

and to see what they're doing,

and to see if they're being fair.

Can you talk a little bit more about that?

Like, going forward,

some kind of regulatory structures

would probably have to emerge

to analyze and to look at

these black boxes that are just sort of

popping up everywhere and, you know,

controlling more and more of the things

in our lives, and important decisions.

So, just, what kind of discussions

are there for that?

And what kind of possibility
is there for that?

And, I'm sure that companies would be

very, very resistant to

any kind of attempt to look into

algorithms, and to...

Dr. Helsby: Yeah, I mean, definitely

companies would be very resistant to

having people look into their algorithms.

So, if you wanna do a very rigorous

audit of what's going on

then it's probably necessary to have

a few people come in

and sign NDAs, and then

look through the systems.

So, that's one way to proceed.

But, another way to proceed that--

so, these academic researchers have done

a few experiments

and found some interesting things,

and that's sort all the attempts at auditing

that we've seen:

there was 1 attempt in 2012
for the Obama campaign,

but there's really not been any

sort of systematic attempt--

you know, like, in censorship

we see a systematic attempt to

do measurement as often as possible,

check what's going on,

and that itself, you know,

can act as an oversight mechanism.

But, right now,

I think many of these companies

realize no one is watching,

so there's no real push to have

people verify: are you being fair when you

implement this system?

Because no one's really checking.

Mic 2: Do you think that,

at some point, it would be like

an FDA or SEC, to give some American examples...

an actual government regulatory agency

that has the power and ability to

not just sort of look and try to

reverse engineer some of these algorithms,

but actually peek in there and make sure

that things are fair, because it seems like

there's just-- it's so important now

that, again, it could be the difference between

life and death, between

getting a job, not getting a job,

being pulled over,
not being pulled over,

being racially profiled,
not racially profiled,

things like that.
Dr. Helsby: Right.

Mic 2: Is it moving in that direction?

Or is it way too early for it?

Dr. Helsby: I mean, so some people have...

someone has called for, like,

a Federal Search Commission,

or like a Federal Algorithms Commission,

that would do this sort of oversight work,

but it's in such early stages right now

that there's no real push for that.

But I think it's a good idea.

Herald: And again, #2 please.

Mic 2: Thank you again for your talk.

I was just curious if you can point

to any examples of

either current producers or consumers

of these algorithmic systems

who are actively and publicly trying

to do so in a responsible manner

by describing what they're trying to do

and how they're going about it?

Dr. Helsby: So, yeah, there are some companies,

for example, like DataKind,

that try to deploy algorithmic systems

in as responsible a way as possible,

for like public policy.

Like, I actually also implement systems

for public policy in a transparent way.

Like, all the code is in GitHub, etc.

And it is also the case to give credit to

Google, and these giants,

they're trying to implement transparency systems

that help you understand.

This has been done with respect to

how your data is being collected,

but for example if you go on Amazon.com

you can see a recommendation has been made,

and that is pretty transparent.

You can see "this item
was recommended to me,"

so you know that prediction
is being used in this case,

and it will say why prediction is being used:

because you purchased some item.

And Google has a similar thing,

if you go to like Google Ad Settings,

you can even turn off personalization of ads

if you want,

and you can also see some of the inferences

that have been learned about you.

A subset of the inferences that have been

learned about you.

So, like, what interests...

Herald: A question from the internet, please?

Signal Angel: Yes, billetQ is asking

how do you avoid biases in machine learning?

I asume analysis system, for example,

could be biased against women and minorities,

if used for hiring decisions
based on known data.

Dr. Helsby: Yeah, so one thing is to

just explicitly check.

So, you can check to see how

positive outcomes are being distributed

among those protected classes.

You could also incorporate these sort of

fairness constraints in the function

that you optimize when you train the system,

and so, if you're interested in reading more

about this, the 2 papers--

let me go to References--

there's a good paper called

Fairness Through Awareness that describes

how to go about doing this,

so I recommend this person read that.

It's good.

Herald: Microphone 2, please.

Mic2: Thanks again for your talk.

Umm, hello?

Okay.

Umm, I see of course a problem with

all the black boxes that you describe

with regards for the crime systems,

but when we look at the advertising systems

in many cases they are very networked.

There are many different systems collaborating

and exchanging data via open APIs:

RESTful APIs, and various

demand-side platforms
and audience-exchange platforms,

and everything.

So, can that help to at least

increase awareness on where targeting, personalization

might be happening?

I mean, I'm looking at systems like

BuiltWith, that surface what kind of

JavaScript libraries are used elsewhere.

So, is that something that could help

at least to give a better awareness

and listing all the points where

you might be targeted...

Dr. Helsby: So, like, with respect to

advertising, the fact that
there is behind the scenes

this like complicated auction process

that's occurring, just makes things

a lot more complicated.

So, for example, I said briefly

that they found that there's this
statistical difference

between how men and women are treated,

but it doesn't necessarily mean that

"Oh, the algorithm is definitely biased."

It could be because of this auction process,

it could be that women are considered

more valuable when it comes to advertising,

and so these executive ads are getting

outbid by some other ads,

and so there's a lot of potential

causes for that.

So, I think it just makes things
a lot more complicated.

I don't know if it helps
with the bias at all.

Mic 2: Well, the question was more

a direction... can it help to surface

and make people aware of that fact?

I mean, I can talk to my kids probably,

and they will probably understand,

but I can't explain that to my grandma,

who's also, umm, looking at an iPad.

Dr. Helsby: So, the fact that

the systems are...

I don't know if I understand.

Mic 2: OK. I think that the main problem

is that we are behind the industry efforts

to being targeted at, and many people

do know, but a lot more people don't know,

and making them aware of the fact

that they are a target, in a way,

is something that can only be shown

by a 3rd party that disposed that data,

and make audits in a way--

maybe in an automated way.

Dr. Helsby: Right.

Yeah, I think it certainly
could help with advocacy

if that's the point, yeah.

Herald: Another question
from the internet, please.

Signal Angel: Yes, on IRC they are asking

if we know that prediction in some cases

provides an influence that cannot be controlled.

So, r4v5 would like to know from you

if there are some cases or areas where

machine learning simply shouldn't go?

Dr. Helsby: Umm, so I think...

I mean, yes, I think that it is the case

that in some cases machine learning

might not be appropriate.

For example, if you use machine learning

to decide who should be searched.

I don't think it should be the case that

machine learning algorithms should

ever be used to determine

probable cause, or something like that.

So, if it's just one piece of evidence

that you consider,

and there's human oversight always,

maybe it's fine, but

we should be very suspicious and hesitant

in certain contexts where

the ramifications are very serious.

Like the No Fly List, and so on.

Herald: And #2 again.

Mic 2: A second question

that just occurred to me, if you don't mind.

Umm, until the advent of

algorithmic systems,

when there've been cases of serious harm

that's been resulted in individuals or groups,

and it's been demonstrated that

it's occurred because of

an individual or a system of people

being systematically biased, then often

one of the actions that's taken is

pressure's applied, and then

people are required to change,

and hopely be held responsible,

and then change the way that they do things

to try to remove bias from that system.

What's the current thinking about

how we can go about doing that

when the systems that are doing that

are algorithmic?

Is it just going to be human oversight,

and humans are gonna have to be

held responsible for the oversight?

Dr. Helsby: So, in terms of bias,

if we're concerned about bias towards

particular types of people,

that's something that we can optimize for.

So, we can train systems that are unbiased

in this way.

So that's one way to deal with it.

But there's always gonna be errors,

so that's sort of a separate issue

from the bias, and in the case

where there are errors,

there must be oversight.

So, one way that one could improve

the way that this is done

is by making sure that you're

keeping track of confidence of decisions.

So, if you have a low confidence prediction,

then maybe a human
should come in and check things.

So, that might be one way to proceed.

Herald: So, there's no more question.

I close this talk now,

and thank you very much

and a big applause to

Jennifer Helsby!

roaring applause

subtitles created by c3subtitles.de
Join, and help us!