34C3 - Dude, you broke the Future!

34C3 preroll music

Herald: Humans of Congress, it is my
pleasure to announce the next speaker.

I was supposed to pick out a few awards or
something, to actually present what he's

done in his life, but I can
only say: he's one of us!

applause

Charles Stross!
ongoing applause

Charles Stross: Hi! Is this on?
Good. Great.

I'm really pleased to be here and I
want to start by apologizing for my total

lack of German. So this talk is gonna be in
English. Good morning. I'm Charlie Stross

and it's my job to tell lies for money, or
rather, I write science fiction, much of

it about on the future, which in recent
years has become ridiculously hard to

predict. In this talk I'm going to talk
about why. Now our species, Homo sapiens

sapiens, is about 300,000 years old. It
used to be about 200,000 years old,

but it grew an extra 100,000
years in the past year because of new

archaeological discoveries, I mean, go
figure. For all but the last three

centuries or so - of that span, however -
predicting the future was really easy. If

you were an average person - as opposed to
maybe a king or a pope - natural disasters

aside, everyday life 50 years in the
future would resemble everyday life 50

years in your past. Let that sink in for a
bit. For 99.9% of human existence on this

earth, the future was static. Then
something changed and the future began to

shift increasingly rapidly, until, in the
present day, things are moving so fast,

it's barely possible to anticipate trends
from one month to the next. Now as an

eminent computer scientist, Edsger Dijkstra
once remarked, computer science is no more

about computers than astronomy is about
building big telescopes, the same can be

said of my field of work, writing science
fiction, sci-fi is rarely about science

and even more rarely about predicting the
future, but sometimes we dabble in

Futurism and lately, Futurism has gotten
really, really, weird. Now when I write a

near future work of fiction, one set, say, a
decade hence, there used to be a recipe I

could follow, that worked eerily well. Simply put:
90% of the next decade stuff is

already here around us today.
Buildings are designed to

last many years, automobiles have a design
life of about a decade, so half the cars on

the road in 2027 are already there now -
they're new. People? There'll be some new

faces, aged 10 and under, and some older
people will have died, but most of us

adults will still be around, albeit older
and grayer, this is the 90% of a near

future that's already here today. After
the already existing 90%, another 9% of a

near future a decade hence used to be
easily predictable: you look at trends

dictated by physical limits, such as
Moore's law and you look at Intel's road

map and you use a bit of creative
extrapolation and you won't go too far

wrong. If I predict - wearing my futurology
hat - that in 2027 LTE cellular phones will

be ubiquitous, 5G will be available for
high bandwidth applications and there will be

fallback to some kind of satellite data
service at a price, you probably won't

laugh at me.
I mean, it's not like I'm predicting that

airlines will fly slower and Nazis will
take over the United States, is it ?

laughing

And therein lies the problem. There is
remaining 1% of what Donald Rumsfeld

called the "unknown unknowns", what throws off
all predictions. As it happens, airliners

today are slower than they were in the
1970s and don't get me started about the Nazis,

I mean, nobody in 2007 was expecting a Nazi
revival in 2017, were they?

Only this time, Germans get to be the good guys.
laughing, applause

So. My recipe for fiction set 10 years
in the future used to be:

"90% is already here,
9% is not here yet but predictable

and 1% is 'who ordered that?'" But unfortunately
the ratios have changed, I think we're now

down to maybe 80% already here - climate
change takes a huge toll on architecture -

then 15% not here yet, but predictable and
a whopping 5% of utterly unpredictable

deep craziness. Now... before I carry on
with this talk, I want to spend a minute or

two ranting loudly and ruling out the
singularity. Some of you might assume, that

as the author of books like "Singularity
Sky" and "Accelerando",

I expect an impending technological
singularity,

that we will develop self-improving
artificial intelligence and mind uploading

and the whole wish list of transhumanist
aspirations promoted by the likes of

Ray Kurzweil, will come to pass. Unfortunately
this isn't the case. I think transhumanism

is a warmed-over Christian heresy. While
its adherents tend to be outspoken atheists,

they can't quite escape from the
history that gave rise to our current

Western civilization. Many of you are
familiar with design patterns, an approach

to software engineering that focuses on
abstraction and simplification, in order

to promote reusable code. When you look at
the AI singularity as a narrative and

identify the numerous places in their
story where the phrase "and then a miracle

happens" occur, it becomes apparent pretty
quickly, that they've reinvented Christiantiy.

applause

Indeed, the wellspring of
today's transhumanists draw in a long rich

history of Russian philosophy, exemplified
by the russian orthodox theologian Nikolai

Fyodorovich Fedorov by way of his disciple
Konstantin Tsiolkovsky, whose derivation

of a rocket equation makes him
essentially the father of modern space

flight. Once you start probing the nether
regions of transhumanist forth and run

into concepts like Roko's Basilisk - by the
way, any of you who didn't know about the

Basilisk before, are now doomed to an
eternity in AI hell, terribly sorry - you

realize, they've mangled it to match some
of the nastier aspects of Presbyterian

Protestantism. Now they basically invented
original sin and Satan in the guise of an

AI that doesn't exist yet ,it's.. kind of
peculiar. Anyway, my take on the

singularity is: What if something walks
like a duck and quacks like a duck? It's

probably a duck. And if it looks like a
religion, it's probably a religion.

I don't see much evidence for human-like,
self-directed artificial intelligences

coming along any time soon, and a fair bit
of evidence, that nobody accepts and freaks

in cognitive science departments, even
want it. I mean, if we invented an AI

that was like a human mind, it would do the
AI equivalent of sitting on the sofa,

munching popcorn and
watching the Super Bowl all day.

It wouldn't be much use to us.
laughter, applause

What we're getting instead,
is self-optimizing tools that defy

human comprehension, but are not
in fact any more like our kind

of intelligence than a Boeing 737 is like
a seagull. Boeing 737s and seagulls both

fly, Boeing 737s don't lay eggs and shit
everywhere. So I'm going to wash my hands

of a singularity as a useful explanatory
model of the future without further ado.

I'm one of those vehement atheists as well
and I'm gonna try and offer you a better

model for what's happening to us. Now, as
my fellow Scottish science fictional author

Ken MacLeod likes to say "the secret
weapon of science fiction is history".

History is, loosely speaking, is the written
record of what and how people did things

in past times. Times that have slipped out
of our personal memories. We science

fiction writers tend to treat history as a
giant toy chest to raid, whenever we feel

like telling a story. With a little bit of
history, it's really easy to whip up an

entertaining yarn about a galactic empire
that mirrors the development and decline

of a Habsburg Empire or to respin the
October Revolution as a tale of how Mars

got its independence. But history is
useful for so much more than that.

It turns out, that our personal memories
don't span very much time at all. I'm 53

and I barely remember the 1960s. I only
remember the 1970s with the eyes of a 6 to

16 year old. My father died this year,
aged 93, and he'd just about remembered the

1930s. Only those of my father's
generation directly remember the Great

Depression and can compare it to the
2007/08 global financial crisis directly.

We Westerners tend to pay little attention
to cautionary tales told by 90-somethings.

We're modern, we're change obsessed and we
tend to repeat our biggest social mistakes

just as they slip out of living memory,
which means they recur on a timescale of

70 to 100 years.
So if our personal memories are useless,

we need a better toolkit
and history provides that toolkit.

History gives us the perspective to see what
went wrong in the past and to look for

patterns and check to see whether those
patterns are recurring in the present.

Looking in particular at the history of the past two
to four hundred years, that age of rapidly

increasing change that I mentioned at the
beginning. One glaringly obvious deviation

from the norm of the preceding
3000 centuries is obvious, and that's

the development of artificial intelligence,
which happened no earlier than 1553 and no

later than 1844. I'm talking of course
about the very old, very slow AI's we call

corporations. What lessons from the history
of a company can we draw that tell us

about the likely behavior of the type of
artificial intelligence we're interested

in here, today?
Well. Need a mouthful of water.

Let me crib from Wikipedia for a moment.

Wikipedia: "In the late 18th
century, Stewart Kyd, the author of the

first treatise on corporate law in English,
defined a corporation as: 'a collection of

many individuals united into one body,
under a special denomination, having

perpetual succession under an artificial
form, and vested, by policy of the law, with

the capacity of acting, in several respects,
as an individual, enjoying privileges and

immunities in common, and of exercising a
variety of political rights, more or less

extensive, according to the design of its
institution, or the powers conferred upon

it, either at the time of its creation, or
at any subsequent period of its

existence.'"
This was a late 18th century definition,

sound like a piece of software to you?
In 1844, the British government passed the

"Joint Stock Companies Act" which created
a register of companies and allowed any

legal person, for a fee, to register a
company which in turn existed as a

separate legal person. Prior to that point,
it required a Royal Charter or an act of

Parliament to create a company.
Subsequently, the law was extended to limit

the liability of individual shareholders
in event of business failure and then both

Germany and the United States added their
own unique twists to what today we see is

the doctrine of corporate personhood.
Now, though plenty of other things that

happened between the 16th and 21st centuries
did change the shape of the world we live in.

I've skipped the changes in
agricultural productivity that happened

due to energy economics,
which finally broke the Malthusian trap

our predecessors lived in.
This in turn broke the long-term

cap on economic growth of about
0.1% per year

in the absence of famines, plagues and
wars and so on.

I've skipped the germ theory of diseases
and the development of trade empires

in the age of sail and gunpowder,
that were made possible by advances

in accurate time measurement.

I've skipped the rise, and
hopefully decline, of the pernicious

theory of scientific racism that
underpinned Western colonialism and the

slave trade. I've skipped the rise of
feminism, the ideological position that

women are human beings rather than
property and the decline of patriarchy.

I've skipped the whole of the
Enlightenment and the Age of Revolutions,

but this is a technocratic.. technocentric
Congress, so I want to frame this talk in

terms of AI, which we all like to think we
understand. Here's the thing about these

artificial persons we call corporations.
Legally, they're people. They have goals,

they operate in pursuit of these goals,
they have a natural life cycle.

In the 1950s, a typical U.S. corporation on the
S&P 500 Index had a life span of 60 years.

Today it's down to less than 20 years.
This is largely due to predation.

Corporations are cannibals, they eat
one another.

They're also hive super organisms
like bees or ants.

For the first century and a
half, they relied entirely on human

employees for their internal operation,
but today they're automating their

business processes very rapidly. Each
human is only retained so long as they can

perform their assigned tasks more
efficiently than a piece of software

and they can all be replaced by another
human, much as the cells in our own bodies

are functionally interchangeable and a
group of cells can - in extremis - often be

replaced by a prosthetic device.
To some extent, corporations can be

trained to serve of the personal desires of
their chief executives, but even CEOs can

be dispensed with, if their activities
damage the corporation, as Harvey

Weinstein found out a couple of months
ago.

Finally, our legal environment today has
been tailored for the convenience of

corporate persons, rather than human
persons, to the point where our governments

now mimic corporations in many of our
internal structures.

So, to understand where we're going, we
need to start by asking "What do our

current actually existing AI overlords
want?"

Now, Elon Musk, who I believe you've
all heard of, has an obsessive fear of one

particular hazard of artificial
intelligence, which he conceives of as

being a piece of software that functions
like a brain in a box, namely the

Paperclip Optimizer or Maximizer.
A Paperclip Maximizer is a term of art for

a goal seeking AI that has a single
priority, e.g., maximizing the

number of paperclips in the universe. The
Paperclip Maximizer is able to improve

itself in pursuit of its goal, but has no
ability to vary its goal, so will

ultimately attempt to convert all the
metallic elements in the solar system into

paperclips, even if this is obviously
detrimental to the well-being of the

humans who set it this goal.
Unfortunately I don't think Musk

is paying enough attention,
consider his own companies.

Tesla isn't a Paperclip Maximizer, it's a
battery Maximizer.

After all, a battery.. an
electric car is a battery with wheels and

seats. SpaceX is an orbital payload
Maximizer, driving down the cost of space

launches in order to encourage more sales
for the service it provides. SolarCity is

a photovoltaic panel maximizer and so on.
All three of the.. Musk's very own slow AIs

are based on an architecture, designed to
maximize return on shareholder

investment, even if by doing so they cook
the planet the shareholders have to live

on or turn the entire thing into solar
panels.

But hey, if you're Elon Musk, thats okay,
you're gonna retire on Mars anyway.

laughing

By the way, I'm ragging on Musk in this
talks, simply because he's the current

opinionated tech billionaire, who thinks
for disrupting a couple of industries

entitles him to make headlines.
If this was 2007 and my focus slightly

difference.. different, I'd be ragging on
Steve Jobs and if we're in 1997 my target

would be Bill Gates.
Don't take it personally, Elon.

laughing

Back to topic. The problem of
corporations is, that despite their overt

goals, whether they make electric vehicles
or beer or sell life insurance policies,

they all have a common implicit Paperclip
Maximizer goal: to generate revenue. If

they don't make money, they're eaten by a
bigger predator or they go bust. It's as

vital to them as breathing is to us
mammals. They generally pursue their

implicit goal - maximizing revenue - by
pursuing their overt goal.

But sometimes they try instead to
manipulate their environment, to ensure

that money flows to them regardless.
Human toolmaking culture has become very

complicated over time. New technologies
always come with an attached implicit

political agenda that seeks to extend the
use of the technology. Governments react

to this by legislating to control new
technologies and sometimes we end up with

industries actually indulging in legal
duels through the regulatory mechanism of

law to determine, who prevails. For
example, consider the automobile. You

can't have mass automobile transport
without gas stations and fuel distribution

pipelines.
These in turn require access to whoever

owns the land the oil is extracted from
under and before you know it, you end up

with a permanent army in Iraq and a clamp
dictatorship in Saudi Arabia. Closer to

home, automobiles imply jaywalking laws and
drink-driving laws. They affect Town

Planning regulations and encourage
suburban sprawl, the construction of human

infrastructure on a scale required by
automobiles, not pedestrians.

This in turn is bad for competing
transport technologies, like buses or

trams, which work best in cities with a
high population density. So to get laws

that favour the automobile in place,
providing an environment conducive to

doing business, automobile companies spend
money on political lobbyists and when they

can get away with it, on bribes. Bribery
needn't be blatant of course. E.g.,

the reforms of a British railway network
in the 1960s dismembered many branch lines

and coincided with a surge in road
building and automobile sales. These

reforms were orchestrated by Transport
Minister Ernest Marples, who was purely a

politician. The fact that he accumulated a
considerable personal fortune during this

period by buying shares in motorway
construction corporations, has nothing to

do with it. So, no conflict of interest
there - now if the automobile in industry

can't be considered a pure Paperclip
Maximizer... sorry, the automobile

industry in isolation can't be considered
a pure Paperclip Maximizer. You have to

look at it in conjunction with the fossil
fuel industries, the road construction

business, the accident insurance sector
and so on. When you do this, you begin to

see the outline of a paperclip-maximizing
ecosystem that invades far-flung lands and

grinds up and kills around one and a
quarter million people per year. That's

the global death toll from automobile
accidents currently, according to the World

Health Organization. It rivals the First
World War on an ongoing permanent basis

and these are all side effects of its
drive to sell you a new car. Now,

automobiles aren't of course a total
liability. Today's cars are regulated

stringently for safety and, in theory, to
reduce toxic emissions. They're fast,

efficient and comfortable. We can thank
legal mandated regulations imposed by

governments for this, of course. Go back
to the 1970s and cars didn't have crumple

zones, go back to the 50s and they didn't
come with seat belts as standard. In the

1930s, indicators, turn signals and brakes
on all four wheels were optional and your

best hope of surviving a 50 km/h-crash was
to be thrown out of a car and land somewhere

without breaking your neck.
Regulator agencies are our current

political system's tool of choice for
preventing Paperclip Maximizers from

running amok. Unfortunately, regulators
don't always work. The first failure mode

of regulators that you need to be aware of
is regulatory capture, where regulatory

bodies are captured by the industries they
control. Ajit Pai, Head of American Federal

Communications Commission, which just voted
to eliminate net neutrality rules in the

U.S., has worked as Associate
General Counsel for Verizon Communications

Inc, the largest current descendant of the
Bell Telephone system's monopoly. After

the AT&T antitrust lawsuit, the Bell
network was broken up into the seven baby

bells. They've now pretty much reformed
and reaggregated and Verizon is the largest current one.

Why should someone with a transparent
interest in a technology corporation end

up running a regulator that tries to
control the industry in question? Well, if

you're going to regulate a complex
technology, you need to recruit regulators

from people who understand it.
Unfortunately, most of those people are

industry insiders. Ajit Pai is clearly
very much aware of how Verizon is

regulated, very insightful into its
operations and wants to do something about

it - just not necessarily in the public
interest.

applause
When regulators end up staffed by people

drawn from the industries they're supposed
to control, they frequently end up working

with their former office mates, to make it
easier to turn a profit, either by raising

barriers to keep new insurgent companies
out or by dismantling safeguards that

protect the public. Now a second problem
is regulatory lag where a technology

advances so rapidly, that regulations are
laughably obsolete by the time they're

issued. Consider the EU directive
requiring cookie notices on websites to

caution users, that their activities are
tracked and their privacy may be violated.

This would have been a good idea in 1993
or 1996, but unfortunatelly it didn't show up

until 2011. Fingerprinting and tracking
mechanisms have nothing to do with cookies

and were already widespread by then. Tim
Berners-Lee observed in 1995, that five

years worth of change was happening on the
web for every 12 months of real-world

time. By that yardstick, the cookie law
came out nearly a century too late to do

any good. Again, look at Uber. This month,
the European Court of Justice ruled that

Uber is a taxi service, not a Web App. This
is arguably correct - the problem is, Uber

has spread globally since it was founded
eight years ago, subsidizing its drivers to

put competing private hire firms out of
business. Whether this is a net good for

societys own is debatable. The problem is, a
taxi driver can get awfully hungry if she

has to wait eight years for a court ruling
against a predator intent on disrupting

her business. So, to recap: firstly, we
already have Paperclip Maximizers and

Musk's AI alarmism is curiously mirror
blind. Secondly, we have mechanisms for

keeping Paperclip Maximizers in check, but
they don't work very well against AIs that

deploy the dark arts, especially
corruption and bribery and they're even

worse against true AIs, that evolved too
fast for human mediated mechanisms like

the law to keep up with. Finally, unlike
the naive vision of a Paperclip Maximizer

that maximizes only paperclips, existing
AIs have multiple agendas, their overt

goal, but also profit seeking, expansion
into new markets and to accommodate the

desire of whoever is currently in the
driving seat.

sighs

Now, this brings me to the next major
heading in this dismaying laundry list:

how it all went wrong. It seems to me that
our current political upheavals, the best

understood, is arising from the capture
of post 1917 democratic institutions by

large-scale AI. Everywhere you look, you
see voters protesting angrily against an

entrenched establishment, that seems
determined to ignore the wants and needs

of their human constituents in favor of
those of the machines. The brexit upset

was largely result of a protest vote
against the British political

establishment, the election of Donald
Trump likewise, with a side order of racism

on top. Our major political parties are
led by people who are compatible with the

system as it exists today, a system that
has been shaped over decades by

corporations distorting our government and
regulatory environments. We humans live in

a world shaped by the desires and needs of
AI, forced to live on their terms and we're

taught, that we're valuable only to the
extent we contribute to the rule of the

machines. Now this is free sea and we're
all more interested in computers and

communications technology than this
historical crap. But as I said earlier,

history is a secret weapon, if you know how
to use it. What history is good for, is

enabling us to spot recurring patterns
that repeat across timescales outside our

personal experience. And if we look at our
historical very slow AIs, what do we learn

from them about modern AI and how it's
going to behave? Well to start with, our

AIs have been warped, the new AIs,
the electronic one's instantiated in our

machines, have been warped by a terrible
fundamentally flawed design decision back

in 1995, but as damaged democratic
political processes crippled our ability

to truly understand the world around us
and led to the angry upheavals and upsets

of our present decade. That mistake was
the decision, to fund the build-out of a

public World Wide Web as opposed to be
earlier government-funded corporate and

academic Internet by
monetizing eyeballs through advertising

revenue. The ad-supported web we're used
to today wasn't inevitable. If you recall

the web as it was in 1994, there were very
few ads at all and not much, in a way, of

Commerce. 1995 was the year, the World Wide
Web really came to public attention in the

anglophone world and consumer-facing
websites began to appear. Nobody really

knew, how this thing was going to be paid
for. The original .com bubble was all

about working out, how to monetize the web
for the first time and a lot of people

lost their shirts in the process. A naive
initial assumption was that the

transaction cost of setting up a tcp/ip
connection over modem was too high to

support.. to be supported by per-use micro
billing for web pages. So instead of

charging people fraction of a euro cent
for every page view, we'd bill customers

indirectly, by shoving advertising banners
in front of their eyes and hoping they'd

click through and buy something.
Unfortunately, advertising is in an

industry, one of those pre-existing very
slow AI ecosystems I already alluded to.

Advertising tries to maximize its hold on
the attention of the minds behind each

human eyeball. The coupling of advertising
with web search was an inevitable

outgrowth, I mean how better to attract
the attention of reluctant subjects, than to

find out what they're really interested in
seeing and selling ads that relate to

those interests. The problem of applying
the paperclip maximize approach to

monopolizing eyeballs, however, is that
eyeballs are a limited, scarce resource.

There are only 168 hours in every week, in
which I can gaze at banner ads. Moreover,

most ads are irrelevant to my interests and
it doesn't matter, how often you flash an ad

for dog biscuits at me, I'm never going to
buy any. I have a cat. To make best

revenue-generating use of our eyeballs,
it's necessary for the ad industry to

learn, who we are and what interests us and
to target us increasingly minutely in hope

of hooking us with stuff we're attracted
to.

In other words: the ad industry is a
paperclip maximizer, but for its success,

it relies on developing a theory of mind
that applies to human beings.

sighs

Do I need to divert on to the impassioned
rant about the hideous corruption

and evil that is Facebook?
Audience: Yes!

CS: Okay, somebody said yes.
I'm guessing you've heard it all before,

but for too long don't read.. summary is:
Facebook is as much a search engine as

Google or Amazon. Facebook searches are
optimized for faces, that is for human

beings. If you want to find someone you
fell out of touch with thirty years ago,

Facebook probably knows where they live,
what their favorite color is, what sized

shoes they wear and what they said about
you to your friends behind your back all

those years ago, that made you cut them off.
Even if you don't have a Facebook account,

Facebook has a You account, a hole in their
social graph of a bunch of connections

pointing in to it and your name tagged on
your friends photographs. They know a lot

about you and they sell access to their
social graph to advertisers, who then

target you, even if you don't think you use
Facebook. Indeed, there is barely any

point in not using Facebook these days, if
ever. Social media Borg: "Resistance is

futile!" So however, Facebook is trying to
get eyeballs on ads, so is Twitter and so

are Google. To do this, they fine-tuned the
content they show you to make it more

attractive to your eyes and by attractive
I do not mean pleasant. We humans have an

evolved automatic reflex to pay attention
to threats and horrors as well as

pleasurable stimuli and the algorithms,
that determine what they show us when we

look at Facebook or Twitter, take this bias
into account. You might react more

strongly to a public hanging in Iran or an
outrageous statement by Donald Trump than

to a couple kissing. The algorithm knows
and will show you whatever makes you pay

attention, not necessarily what you need or
want to see.

So this brings me to another point about
computerized AI as opposed to corporate

AI. AI algorithms tend to embody the
prejudices and beliefs of either the

programmers, or the data set
the AI was trained on.

A couple of years ago I ran across an
account of a webcam, developed by mostly

pale-skinned Silicon Valley engineers, that
had difficulty focusing or achieving correct

color balance, when pointed at dark-skinned
faces.

Fast an example of human programmer
induced bias, they didn't have a wide

enough test set and didn't recognize that
they were inherently biased towards

expecting people to have pale skin. But
with today's deep learning, bias can creep

in, while the datasets for neural networks are
trained on, even without the programmers

intending it. Microsoft's first foray into
a conversational chat bot driven by

machine learning, Tay, was what we yanked
offline within days last year, because

4chan and reddit based trolls discovered,
that they could train it towards racism and

sexism for shits and giggles. Just imagine
you're a poor naive innocent AI who's just

been switched on and you're hoping to pass
your Turing test and what happens? 4chan

decide to play with your head.
laughing

I got to feel sorry for Tay.
Now, humans may be biased,

but at least individually we're
accountable and if somebody gives you

racist or sexist abuse to your face, you
can complain or maybe punch them. It's

impossible to punch a corporation and it
may not even be possible to identify the

source of unfair bias, when you're dealing
with a machine learning system. AI based

systems that instantiate existing
prejudices make social change harder.

Traditional advertising works by playing
on the target customer's insecurity and

fear as much as their aspirations. And fear
of a loss of social status and privileges

are powerful stress. Fear and xenophobia
are useful tools for tracking advertising..

ah, eyeballs.
What happens when we get pervasive social

networks, that have learned biases against
say Feminism or Islam or melanin? Or deep

learning systems, trained on datasets
contaminated by racist dipshits and their

propaganda? Deep learning systems like the
ones inside Facebook, that determine which

stories to show you to get you to pay as
much attention as possible to be adverse.

I think, you probably have an inkling of
how.. where this is now going. Now, if you

think, this is sounding a bit bleak and
unpleasant, you'd be right. I write sci-fi.

You read or watch or play sci-fi. We're
acculturated to think of science and

technology as good things that make life
better, but this ain't always so. Plenty of

technologies have historically been
heavily regulated or even criminalized for

good reason and once you get past any
reflexive indignation, criticism of

technology and progress, you might agree
with me, that it is reasonable to ban

individuals from owning nuclear weapons or
nerve gas. Less obviously, they may not be

weapons, but we've banned
chlorofluorocarbon refrigerants, because

they were building up in the high
stratosphere and destroying the ozone

layer that protects us from UVB radiation.
We banned tetra e-file LED in

gasoline, because it poisoned people and
led to a crime wave. These are not

weaponized technologies, but they have
horrible side effects. Now, nerve gas and

leaded gasoline were 1930s chemical
technologies, promoted by 1930s

corporations. Halogenated refrigerants and
nuclear weapons are totally 1940s. ICBMs

date to the 1950s. You know, I have
difficulty seeing why people are getting

so worked up over North Korea. North Korea
reaches 1953 level parity - be terrified

and hide under the bed!
I submit that the 21st century is throwing

up dangerous new technologies, just as our
existing strategies for regulating very

slow AIs have proven to be inadequate. And
I don't have an answer to how we regulate

new technologies, I just want to flag it up
as a huge social problem that is going to

affect the coming century.
I'm now going to give you four examples of

new types of AI application that are
going to warp our societies even more

badly than the old slow AIs, we.. have done.
This isn't an exhaustive list, this is just

some examples I dream, I pulled out of
my ass. We need to work out a general

strategy for getting on top of this sort
of thing before they get on top of us and

I think, this is actually a very urgent
problem. So I'm just going to give you this

list of dangerous new technologies that
are arriving now, or coming, and send you

away to think about what to do next. I
mean, we are activists here, we should be

thinking about this and planning what
to do. Now, the first nasty technology I'd

like to talk about, is political hacking
tools that rely on social graph directed

propaganda. This is low-hanging fruit
after the electoral surprises of 2016.

Cambridge Analytica pioneered the use of
deep learning by scanning the Facebook and

Twitter social graphs to identify voters
political affiliations by simply looking

at what tweets or Facebook comments they
liked, very able to do this, to identify

individuals with a high degree of
precision, who were vulnerable to

persuasion and who lived in electorally
sensitive districts. They then canvassed

them with propaganda, that targeted their
personal hot-button issues to change their

electoral intentions. The tools developed
by web advertisers to sell products have

now been weaponized for political purposes
and the amount of personal information

about our affiliations that we expose on
social media, makes us vulnerable. Aside, in

the last U.S. Presidential election, as
mounting evidence for the British

referendum on leaving the EU was subject
to foreign cyber war attack, now

weaponized social media, as was the most
recent French Presidential election.

In fact, if we remember the leak of emails
from the Macron campaign, it turns out that

many of those emails were false, because
the Macron campaign anticipated that they

would be attacked and an email trove would
be leaked in the last days before the

election. So they deliberately set up
false emails that would be hacked and then

leaked and then could be discredited. It
gets twisty fast. Now I'm kind of biting

my tongue and trying, not to take sides
here. I have my own political affiliation

after all, and I'm not terribly mainstream.
But if social media companies don't work

out how to identify and flag micro-
targeted propaganda, then democratic

institutions will stop working and elections
will be replaced by victories, whoever

can buy the most trolls. This won't
simply be billionaires but.. like the Koch

brothers and Robert Mercer from the U.S.
throwing elections to whoever will

hand them the biggest tax cuts. Russian
military cyber war doctrine calls for the

use of social media to confuse and disable
perceived enemies, in addition to the

increasingly familiar use of zero-day
exploits for espionage, such as spear

phishing and distributed denial-of-service
attacks, on our infrastructure, which are

practiced by Western agencies. Problem is,
once the Russians have demonstrated that

this is an effective tactic, the use of
propaganda bot armies in cyber war will go

global. And at that point, our social
discourse will be irreparably poisoned.

Incidentally, I'd like to add - as another
aside like the Elon Musk thing - I hate

the cyber prefix! It usually indicates,
that whoever's using it has no idea what

they're talking about.
applause, laughter

Unfortunately, much as the way the term
hacker was corrupted from its original

meaning in the 1990s, the term cyber war
has, it seems, to have stuck and it's now an

actual thing that we can point to and say:
"This is what we're talking about". So I'm

afraid, we're stuck with this really
horrible term. But that's a digression, I

should get back on topic, because I've only
got 20 minutes to go.

Now, the second threat that we need to
think about regulating ,or controlling, is

an adjunct to deep learning target
propaganda: it's the use of neural network

generated false video media. We're used to
photoshopped images these days, but faking

video and audio takes it to the next
level. Luckily, faking video and audio is

labor-intensive, isn't it? Well nope, not
anymore. We're seeing the first generation

of AI assisted video porn, in which the
faces of film stars are mapped onto those

of other people in a video clip, using
software rather than laborious in human

process.
A properly trained neural network

recognizes faces and transforms the face
of the Hollywood star, they want to put

into a porn movie, into the face of - onto
the face of the porn star in the porn clip

and suddenly you have "Oh dear God, get it
out of my head" - no, not gonna give you

any examples. Let's just say it's bad
stuff.

laughs
Meanwhile we have WaveNet, a system

for generating realistic sounding speech,
if a voice of a human's speak of a neural

network has been trained to mimic any
human speaker. We can now put words into

other people's mouths realistically
without employing a voice actor. This

stuff is still geek intensive. It requires
relatively expensive GPUs or cloud

computing clusters, but in less than a
decade it'll be out in the wild, turned

into something, any damn script kiddie can
use and just about everyone will be able

to fake up a realistic video of someone
they don't like doing something horrible.

I mean, Donald Trump in the White House. I
can't help but hope that out there

somewhere there's some geek like Steve
Bannon with a huge rack of servers who's

faking it all, but no. Now, also we've
already seen alarm this year over bizarre

YouTube channels that attempt to monetize
children's TV brands by scraping the video

content of legitimate channels and adding
their own advertising in keywords on top

before reposting it. This is basically
your YouTube spam.

Many of these channels are shaped by
paperclip maximizing advertising AIs, but

are simply trying to maximise their search
ranking on YouTube and it's entirely

algorithmic: you have a whole list of
keywords, you perm, you take them, you slap

them on top of existing popular videos and
re-upload the videos. Once you add neural

network driven tools for inserting
character A into pirated video B, to click

maximize.. for click maximizing bots,
things are gonna get very weird, though. And

they're gonna get even weirder, when these
tools are deployed for political gain.

We tend - being primates, that evolved 300
thousand years ago in a smartphone free

environment - to evaluate the inputs from
our eyes and ears much less critically

than what random strangers on the Internet
tell us in text. We're already too

vulnerable to fake news as it is. Soon
they'll be coming for us, armed with

believable video evidence. The Smart Money
says that by 2027 you won't be able to

believe anything you see in video, unless
for a cryptographic signatures on it,

linking it back to the camera that shot
the raw feed. But you know how good most

people are at using encryption - it's going to
be chaos!

So, paperclip maximizers with focus on
eyeballs are very 20th century. The new

generation is going to be focusing on our
nervous system. Advertising as an industry

can only exist because of a quirk of our
nervous system, which is that we're

susceptible to addiction. Be it
tobacco, gambling or heroin, we

recognize addictive behavior, when we see
it. Well, do we? It turns out the human

brain's reward feedback loops are
relatively easy to gain. Large

corporations like Zynga - producers of
FarmVille - exist solely because of it,

free to use social media platforms like
Facebook and Twitter, are dominant precisely

because they're structured to reward
frequent short bursts of interaction and

to generate emotional engagement - not
necessarily positive emotions, anger and

hatred are just as good when it comes to
attracting eyeballs for advertisers.

Smartphone addiction is a side effect of
advertising as a revenue model. Frequent

short bursts of interaction to keep us
coming back for more. Now a new.. newish

development, thanks to deep learning again -
I keep coming back to deep learning,

don't I? - use of neural networks in a
manner that Marvin Minsky never envisaged,

back when he was deciding that the
Perzeptron was where it began and ended

and it couldn't do anything.
Well, we have neuroscientists now, who've

mechanized the process of making apps more
addictive. Dopamine Labs is one startup

that provides tools to app developers to
make any app more addictive, as well as to

reduce the desire to continue
participating in a behavior if it's

undesirable, if the app developer actually
wants to help people kick the habit. This

goes way beyond automated A/B testing. A/B
testing allows developers to plot a binary

tree path between options, moving towards a
single desired goal. But true deep

learning, addictiveness maximizers, can
optimize for multiple attractors in

parallel. The more users you've got on
your app, the more effectively you can work

out, what attracts them and train them and
focus on extra addictive characteristics.

Now, going by their public face, the folks
at Dopamine Labs seem to have ethical

qualms about the misuse of addiction
maximizers. But neuroscience isn't a

secret and sooner or later some really
unscrupulous sociopaths will try to see

how far they can push it. So let me give
you a specific imaginary scenario: Apple

have put a lot of effort into making real-
time face recognition work on the iPhone X

and it's going to be everywhere on
everybody's phone in another couple of

years. You can't fool an iPhone X with a
photo or even a simple mask. It does depth

mapping to ensure, your eyes are in the
right place and can tell whether they're

open or closed. It recognizes your face
from underlying bone structure through

makeup and bruises. It's running
continuously, checking pretty much as often

as every time you'd hit the home button on
a more traditional smartphone UI and it

can see where your eyeballs are pointing.
The purpose of a face recognition system

is to provide for real-time authenticate
continuous authentication when you're

using a device - not just enter a PIN or
sign a password or use a two factor

authentication pad, but the device knows
that you are its authorized user on a

continuous basis and if somebody grabs
your phone and runs away with it, it'll

know that it's been stolen immediately, it
sees the face of the thief.

However, your phone monitoring your facial
expressions and correlating against app

usage has other implications. Your phone
will be aware of precisely what you like

to look at on your screen.. on its screen.
We may well have sufficient insight on the

part of the phone to identify whether
you're happy or sad, bored or engaged.

With addiction seeking deep learning tools
and neural network generated images, those

synthetic videos I was talking about, it's
entirely.. in principle entirely possible to

feed you an endlessly escalating payload
of arousal-inducing inputs. It might be

Facebook or Twitter messages, optimized to
produce outrage, or it could be porn

generated by AI to appeal to kinks you
don't even consciously know you have.

But either way, the app now owns your
central nervous system and you will be

monetized. And finally, I'd like to raise a
really hair-raising specter that goes well

beyond the use of deep learning and
targeted propaganda and cyber war. Back in

2011, an obscure Russian software house
launched an iPhone app for pickup artists

called 'Girls Around Me'. Spoiler: Apple
pulled it like a hot potato as soon as

word got out that it existed. Now, Girls
Around Me works out where the user is

using GPS, then it would query Foursquare
and Facebook for people matching a simple

relational search, for single females on
Facebook, per relationship status, who have

checked in, or been checked in by their
friends, in your vicinity on Foursquare.

The app then displays their locations on a
map along with links to their social media

profiles. If they were doing it today, the
interface would be gamified, showing strike

rates and a leaderboard and flagging
targets who succumbed to harassment as

easy lays.
But these days, the cool kids and single

adults are all using dating apps with a
missing vowel in the name, only a creeper

would want something like Girls Around Me,
right? Unfortunately, there are much, much

nastier uses of and scraping social media
to find potential victims for serial

rapists. Does your social media profile
indicate your political religious

affiliation? No? Cambridge Analytica can
work them out with 99.9% precision

anyway, so don't worry about that. We
already have you pegged. Now add a service

that can identify people's affiliation and
location and you have a beginning of a

flash mob app, one that will show people
like us and people like them on a

hyperlocal map.
Imagine you're a young female and a

supermarket like Target has figured out
from your purchase patterns, that you're

pregnant, even though you don't know it
yet. This actually happened in 2011. Now

imagine, that all the anti-abortion
campaigners in your town have an app

called "Babies Risk" on their phones.
Someone has paid for the analytics feed

from the supermarket and every time you go
near a family planning clinic, a group of

unfriendly anti-abortion protesters
somehow miraculously show up and swarm

you. Or imagine you're male and gay and
the "God hates fags"-crowd has invented a

100% reliable gaydar app, based on your
Grindr profile, and is getting their fellow

travelers to queer bash gay men - only when
they're alone or outnumbered by ten to

one. That's the special horror of precise
geolocation not only do you always know

where you are, the AIs know, where you are
and some of them aren't friendly. Or

imagine, you're in Pakistan and Christian
Muslim tensions are rising or in rural

Alabama or an Democrat, you know the
possibilities are endless. Someone out

there is working on this. A geolocation
aware, social media scraping deep learning

application, that uses a gamified
competitive interface to reward its

players for joining in acts of mob
violence against whoever the app developer

hates.
Probably it has an innocuous seeming, but

highly addictive training mode, to get the
users accustomed to working in teams and

obeying the apps instructions. Think
Ingress or Pokemon Go. Then at some pre-

planned zero-hour, it switches mode and
starts rewarding players for violence,

players who have been primed to think of
their targets as vermin by a steady drip

feed of micro-targeted dehumanizing
propaganda inputs, delivered over a period

of months. And the worst bit of this picture?
Is that the app developer isn't even a

nation-state trying to disrupt its enemies
or an extremist political group trying to

murder gays, Jews or Muslims. It's just a
Paperclip Maximizer doing what it does

and you are the paper. Welcome to the 21st
century.

applause
Uhm...

Thank you.

ongoing applause
We have a little time for questions. Do

you have a microphone for the orders? Do
we have any questions? ... OK.

Herald: So you are doing a Q&A?
CS: Hmm?

Herald: So you are doing a Q&A. Well if
there are any questions, please come

forward to the microphones, numbers 1
through 4 and ask.

Mic 1: Do you really think it's all
bleak and dystopian like you prescribed

it, because I also think the future can be
bright, looking at the internet with open

source and like, it's all growing and going
faster and faster in a good

direction. So what do you think about
the balance here?

CS: sighs Basically, I think the
problem is, that about 3% of us

are sociopaths or psychopaths, who spoil
everything for the other 97% of us.

Wouldn't it be great if somebody could
write an app that would identify all the

psychopaths among us and let the rest of
us just kill them?

laughing, applause
Yeah, we have all the

tools to make a utopia, we have it now
today. A bleak miserable grim meathook

future is not inevitable, but it's up to
us to use these tools to prevent the bad

stuff happening and to do that, we have to
anticipate the bad outcomes and work to

try and figure out a way to deal with
them. That's what this talk is. I'm trying

to do a bit of a wake-up call and get
people thinking about how much worse

things can get and what we need to do to
prevent it from happening. What I was

saying earlier about our regulatory
systems being broken, stands. How do we

regulate the deep learning technologies?
This is something we need to think about.

H: Okay mic number two.
Mic 2: Hello? ... When you talk about

corporations as AIs, where do you see that
analogy you're making? Do you see them as

literally AIs or figuratively?
CS: Almost literally. If

you're familiar with philosopher
(?) Searle's Chinese room paradox

from the 1970s, by which he attempted to
prove that artificial intelligence was

impossible, a corporation is very much the
Chinese room implementation of an AI. It

is a bunch of human beings in a box. You
put inputs into the box, you get apples

out of a box. Does it matter, whether it's
all happening in software or whether

there's a human being following rules
inbetween to assemble the output? I don't

see there being much of a difference.
Now you have to look at a company at a

very abstract level to view it as an AI,
but more and more companies are automating

their internal business processes. You've
got to view this as an ongoing trend. And

yeah, they have many of the characteristics
of an AI.

Herald: Okay mic number four.
Mic 4: Hi, thanks for your talk.

You probably heard of the Time Well
Spent and Design Ethics movements that

are alerting developers to dark patterns
in UI design, where

these people design apps to manipulate
people. I'm curious if you find any

optimism in the possibility of amplifying
or promoting those movements.

CS: Uhm, you know, I knew about dark
patterns, I knew about people trying to

optimize them, I wasn't actually aware
there were movements against this. Okay I'm

53 years old, I'm out of touch. I haven't
actually done any serious programming in

15 years. I'm so rusty, my rust has rust on
it. But, you know, it is a worrying trend

and actual activism is a good start.
Raising awareness of hazards and of what

we should be doing about them, is a good
start. And I would classify this actually

as a moral issue. We need to..
corporations evaluate everything in terms

of revenue, because it's very
equivalent to breathing, they have to

breathe. Corporations don't usually have
any moral framework. We're humans, we need

a moral framework to operate within. Even
if it's as simple as first "Do no harm!"

or "Do not do unto others that which would
be repugnant if it was done unto you!",

the Golden Rule. So, yeah, we should be
trying to spread awareness of this about

and working with program developers, to
look to remind them that they are human

beings and have to be humane in their
application of technology, is a necessary

start.
applause

H: Thank you! Mic 3?
Mic 3: Hi! Yeah, I think that folks,

especially in this sort of crowd, tend to
jump to the "just get off of

Facebook"-solution first, for a lot of
these things that are really, really

scary. But what worries me, is how we sort
of silence ourselves when we do that.

After the election I actually got back on
Facebook, because the Women's March was

mostly organized through Facebook. But
yeah, I think we need a lot more

regulation, but we can't just throw it
out. We're.. because it's..

social media is the only... really good
platform we have right now

to express ourselves, to
have our rules, or power.

CS: Absolutely. I have made
a point of not really using Facebook

for many, many, many years.
I have a Facebook page simply to

shut up the young marketing people at my
publisher, who used to prop up every two

years and say: "Why don't you have a
Facebook. Everybody's got a Facebook."

No, I've had a blog since 1993!
laughing

But no, I'm gonna have to use Facebook,
because these days, not using Facebook is

like not using email. You're cutting off
your nose to spite your face. What we

really do need to be doing, is looking for
some form of effective oversight of

Facebook and particularly, of how they..
the algorithms that show you content, are

written. What I was saying earlier about
how algorithms are not as transparent as

human beings to people, applies hugely to
them. And both, Facebook and Twitter

control the information
that they display to you.

Herald: Okay, I'm terribly sorry for all the
people queuing at the mics now, we're out

of time. I also have to apologize, I
announced, that this talk was being held in

English, but it was being held in English.
the latter pronounced on the G

Thank you very much, Charles Stross!

CS: Thank you very much for
listening to me, it's been a pleasure!

applause

postroll music

subtitles created by c3subtitles.de
in the year 2018