34C3 - Gamified Control?

34c3 intro
Herald: Okay, so ladies and gentlemen

Katika Kühnreich. So did you all
thank the angels today? No, yes, maybe?

[Applause]
Herald: Did you all took your prosĂm two

today? No? Okay, see, Katika will tell you
about the social credit system from China,

and will tell you a little bit more about
how a human being can be valued. She is

the magista in political science and
modern and ancient Chinese science. She

not only has her theoretical knowledge
from books, but she always also combines

it with the street study. So please
welcome, with a very warm hand of

applause, Katika!
[Applause]

Katika: Hey guys, I'm happy to be here,
and first I want to thank all the people

who made that possible. This conference
and this talk; so thanks to all you, and a

special thanks to all the people
translating.

[Applause]
Katika: Before we start, I like to say a

few points. The first was the thank you,
we already have. The second is I have to

apologize: English is not my mother tongue
so you will have some funny mistakes that

I make, so the Internet has a bit more
fun. Because the world can be (nice). The

second thing, is I will take you on a
journey through China's social credit

system - its current state. And because we
only have half an hour, more or less,

because I'd really like to spend a lot of
time with you guys on talks, on discussion

and QA, we have to leave a lot of
interesting sites out. So there were some

excellent talks on the CCC about
technological aspects of Internet and

Internet usage in China. I will
concentrate really hard on this social

credit system. But if you have questions,
please ask in the Q&A. The Q&A will not be

recorded, so you can feel free. And if you
have more questions, just contact me after

the talk. And, it's a China topic and
often when you talk about China, people

say "oh yeah, oh yeah: that's China." This
is no China bashing session - I'm sorry

for anyone who expected that. China has a
government, and a government wants to

control something. So this talk is about
social credit systems in China but it is

about a tendency I see internationally.
It's the tendency to answer social

problems with technological solutions, or
at least it's the try. So please keep in

mind: it's happening in China, but it's
not that we are safe in the West or cozy

or anything else. So it's meant as an
example of an international tendency. And

I prepared some cards, so I will stick to
the topic. I have to look at them from

time to time. This talk presents my
current state of research; it's not the

truth. And this talked sadly represents my
current state of my cold as well, so I'm

not totally fed please apologized that as
well. For a little journey I prepared a

couple of stops. The first one will be a
video from 2015 made by Extra Credit on

social credit systems in China. So if you
don't like the talk, you have seven and a

half minutes to just vanish - nobody will
notice. The next point will be "Updates

and Corrections" - it's a video from
2015, and there's some faulty information

in it. And to do that, I will give you a
quick look at the Chinese Internet

landscape. The next point will be the
current state of social credit systems. If

I say SCS: it's the same - I'm just lazy.
And there's one question often not

answered when talking about the Chinese
credit system - it's why is it happening.

And again it's not because it's an evil
government. It's a government. So we'll

dive in some ideological aspects of
Chinese politics today. And in the end,

I'm really looking forward to that point
because I can shut a bit up and listen to

your thoughts and we're really interested
in that. So, everyone ready? Cozy? That's

wonderful, thanks! Then we can start with
the video; that isn't it shown right now.

Now we need some technical support. It's
much more fun when you see the pictures.

But we have time, so.
Herald: Do we have the (tv guys) here?

Katika: We got free water!
Herald: Gentlemen, I think we have a

technical problem here, so while the VOC
and my stage manager is trying to get some

people from there. Yes he will do this -
he's a good guy. He's the best guy!

Gentlemen, ladies, applause for the VOC!
[Applause]

Herald: So while we're waiting to have the
video; and trust me it looked really good

from here. Let's play a little game -
later we will have a Q&A session but

unfortunately we are not allowed or we we
can't record the Q&A session, and we will

also not video stream the Q&A session. So
I would like you to have a quick hand of

applause for all of you, because you will
have a great discussion with this nice

lady and therefore we try to make applause
- we test the applause.

Katika: Okay
[Applause]

Herald: No no no, that should be louder -
come on!

[Applause]
Herald: Okay we still have no video, but I

think over there I did not hear any
applause. Could you, could you please try

it again?
[Applause]

Herald: Yeah, better, better. I think
these guys are way better. Are they? Come

on!
[Applause]

Katika: Otherwise I can use the time to
give you some information I'd present you

after the video. Thank you. The
information you get in the video is based

on a press release of the American Civil
Liberties Union, and that contains some

mistakes. One thing - when you want to
know something about China, the government

is really open in publishing what they're
doing, and what they're planning to do.

And it's seems to be much more comfortable
to just copy, but maybe have a look into

what the government says because it's a
good way to find out what they are doing

and they're scaring the open. It works
with a lot of governments. And I realized

that because this video had a lot of
impact, and there was a lot of news report

on that scary system in China but when you
looked at the news report and you had the

video before, you found out that it was
more or less the same, the very same, as

in the video even years later. So please
if you want to find something out - check

for facts. They might be online too. So
that's the picture, and that's my break.

Video: Okay, so one you recently brought
this to our attention, and when we first

started looking into it, we couldn't
believe it. Even now looking at this I

keep thinking that there's got to be
something about this that I just don't

understand, like, I must be reading this
wrong. It's just so clearly something out

of dystopian science fiction. And yet,
here it is in our world, and people are

actually embracing it. Apparently China
has gamified being an obedient citizen.

[Music]
Video: Going under the innocuous name of

Sesame Credit, China has created a score
for how good a citizen you are, and that's

one of the scariest things I've heard in
quite a while. It's jointly run by

Tencent, yes that Tencent - the one that
owns Riot Games and has a significant

share in Epic and Activision-Blizzard, and
also the ascendant Chinese competitor to

Amazon, Alibaba, hence the name Sesame. So
the owners of China's largest social

networks have partnered with the
governments to create something akin to

the US credit score. But instead of
measuring how regularly you pay your

bills, it measures how obediently you
follow the party line. They dredge data

from your social network, so if you post
pictures of Tiananmen Square, or share a

link about the recent stock market
collapse, your Sesame Credit score goes

down. Share a link from the state-
sponsored news agency about how good the

economy is doing and your score goes up.
The Alibaba and Tencent are also the

largest online retailers in China so
Sesame Credit is also able to pull data

from your purchases. If you're making
purchases the state deems valuable like

work shoes or local agricultural products,
your score goes up. If you import anime

from Japan, though, down the score goes.
And this score has real-world

consequences. Like many games, Sesame
Credit has tiers and levels, and having a

high score gives you special benefits -
like making it easier to get the paperwork

you need to travel. Or making it easier to
get a loan. Now currently, there are no

consequences for having a low score but
there's been talk about implementing

penalties once the system becomes
mandatory in 2020. Penalties like slower

Internet speeds for low scoring citizens.
Or even restricting the jobs that a low

scoring person's allowed to hold. But
there's one more layer to Sesame Credit,

and here's where this goes from being
repulsive to downright insidious. Because

this is all part of a social network it
also scans your friends -

so you will lose points for having friends
with low obedience scores. And it tells

you this. At any point, anybody can check
anyone elses score. And when you check

your own score, Sesame Credit provides a
handy map of your friends to show you who

is dragging your score down. Have you ever
had that thing where you play a game with

somebody who wasn't doing very well and
you've tried to change their behavior to

make them do better? Or maybe after a
while you just sort of stopped playing

with the people who are holding you back.
That's at the heart of how this system

works, and it's also what makes this one
of the most terrifying tools of

authoritarian oppression I've ever read
about. Because like mass censorship, jail

time, assassinations - those are all big
messy implements for keeping a population

in line. That messiness and severity
foster resentment - eventually rebellion.

They're expensive, they're unwieldy - in
the end, those tools are impossible to

maintain. But social pressure?
Ostracization? Those things are free -

they happen on their own. And as a
government tool they don't have nearly the

same potential for going embarrassingly
disastrously wrong. With a system like

this in place, a government doesn't even
have to tell neighbor to spy on neighbor;

to rat each other out. Because that's all
built into a seemingly innocuous game

system. The government need not step in.
Re-education will be handled for them by

friends, classmates, and relatives who
want to maintain a high score. And if that

doesn't work then potentially dangerous
ideas still end up quarantined by the

social isolation this game system causes.
Express or help to spread to many radical

ideas, and people will stop associating
with you. And not because some jackbooted

thugs showed up at the door with threats,
but simply because associating with

somebody with those ideas will lose them
all the privileges they've worked so hard

to obtain. It recontextualizes obedience
to an authoritarian regime.

In the past, you obeyed such powers
because you were afraid. Fear kept you

motivated. But fear is negative. It
fosters resentment. The world we're

stepping into instead uses positive
reinforcement to promote being subservient

to the will of the regime. It's Big
Brother's kinder, gentler hand. And the

things that make this scary is that we've
seen the efficacy of this only too well in

games. You may not actually know this, but
when World of Warcraft was in its early

stages of development, it had an unrested
penalty mechanic that started limiting

experienced gains for players who had
played too much. And players hated it -

they resented it they complained about it
every day. So after brainstorming on this

for a while, Blizzard had the idea to
simply change how the mechanic was

presented. Without changing any of the
existing numbers or systems, they started

referring to the unrested experience state
as normal, and made it the default. And

they started calling the original normal
experience gained state as rested. That's

all they changed, and everybody loved it.
People would log on every day just to get

that bonus. Positive reinforcement works
wonders. But unlike World of Warcraft,

which built a system just to get people to
embrace meaningless grinding, Sesame

Credit has built a system to get people to
enjoy falling into line. Now the system's

not mandatory yet - it's opt-in right now.
But it's going to be mandatory in 2020,

and there's a terrible brilliance to
phasing that in. The early adopters are

going to be people excited about this
system. People who are already patriotic

and are eager for anything that'll help
display that patriotism to the world. And,

as early adopters they're gonna talk it
up. They're gonna give it an air of being

positive and fun.
Then it'll be foisted on the society as a

whole. More than that though, the early
adopters are gonna compete. Already you

can see hundreds of thousands of tweets
displaying people's high scores or showing

off the new milestones they've hit. Giving
a hard numbers to their patriotism and

giving them bragging rights for being the
most patriotic, most right-thinking person

they know. And that's gonna set the tone
for how Sesame Credit is intended to be

used - as a competition to see who can
agree with the government the most. We've

talked about propaganda games on this
channel before but for all the time we've

spent examining and deconstructing
terrible games that espouse hate and for

all the studies on propaganda games James
has done, this is the use of game systems

that frightens me the most. Because to
most people Sesame Credit will seem

benign. Perhaps even fun - it's a
conversation starter, something to share

with your friends. But it's making heavy
use of all the psychological motivators

that we Gamemakers deploy in scoring
systems, and ladders and levels. Systems

that we built to shape play habits and to
keep people coming back. Like I said, I'm

still kind of in a state of disbelief
looking at this. If any of you are

watching this from China, please tell me
if I'm misunderstanding this thing,

because I would love to be wrong about it.
If not, well, I hope this episode can do

some small amount to help the fight to
keep such a system from becoming

mandatory. For everyone in the rest of the
world, I hope this helps remind us all how

important it is to be aware and to be
vigilant. All of these gamification

techniques we've learned through making
games offer incredible opportunities for

making this world a better more engaging
place. But every great tool carries with

it the potential for misuse. And it's on
us as a community who understands this

amazing new medium, to do what we can to
stop that. We'll see you next week.

[Applause]
Katika: So, we had the shedule, and thank

you to Extra Credit for this video, and
thank you for concentrating on

gamification. Because a lot of time, when
I talk to social scientists, without a lot

of technological knowledge, it was like
okay - it's a new system. And the whole

gamification thing triggered me, and I
wondered about a lot of people who didn't

seem too interested in it. Because I said
that the video contains some faulty

information. First, I'd like to
concentrate on key aspects they told us in

with this video. There is a social rating
system in China now. The system will

become mandatory in 2020. That's all
right. It works on Big Data algorithms,

artificial intelligence, and your online
and offline data. And it uses technologies

like gamification and nudging. And if you
hear it's an Orwellian system - no it's

not. 1984 was about force. Gamification is
about feeling good and cozy. So, in the

updates and corrections, I promise you a
quick look at the end Chinese internet

landscape. What most people already know.
From the beginning the Chinese government

started to control the internet as much as
it could and its users. So some

vocabularies you might have heard are the
Great Firewall, or its new extra the

Golden Shield, the Great Cannon, which is
a DDoS weapon. It would be really

fascinating to dive in there, but we have
to watch from far. Another point that's

really interesting for us looking Social
Credit System is true name or real name

registration. Which means you can't use a
lot of services if you don't give them

your ID number. And the mandatory system
in 2020 will work on ID numbers as far as

we know. So, we watched the other side:
the users, their millions of users - it's

one more than 1.4 billion people in China.
And ICTs are really popular. Much more

than in Germany, for example. And another
thing that please keep in mind - cash is

vanishing in China.
A couple of years ago, for example, when I

lived in China 2004: I paid my rent in
cash half a year in advance. When I paid

for anything I paid cash. You could
couldn't find a lot of places accepting

cards. But now China didn't have this
development of cards - they just jumped

right through paying with mobile apps. And
you can buy noodles or snack or magazine

with your mobile phone. And the Chinese
users found multiple ways to deal with

censorship, and to trespass it. It's a
wonderful thing to look up.

And now, the big players of the Chinese
Internet, because we learned some names in

the film. I made something very Chinese -
I made something colorful. Okay, you have

three big players: its Alibaba and Tencent
- we heard about them, and Baidu is the

third biggest player. Each millions of
users, or billions. So to explain this, I

will use some American companies names to
help understanding. And Baidu is known for

that; was the wrong; yeah, Baidu is known
for Baidu Barker - this is like the

Chinese Wikipedia, it's just a bit much
controlled. And a have Baidu Pay, and

Baidu is huge in artificial intelligence.
So the next one is Tencent - we learned

about the gaming thing. And developed QQ -
this is a messenger than some of you might

remember: ICQ. Yeah, it's just a bit
bigger. Another app a lot of people don't

think it's Chinese, it is Chinese, it's
called WeChat. It's I can say they even

mmm WhatsApp, right? And guess what WeChat
developed - WeChat Pay. And there's

another little company Tencent holds some
parts in its SnapChat - it's twelve

percent owned by Tencent. Alibaba has
Alipay which is one of the biggest payment

apps. And you can even use it in Germany
now I think, in some shops. It's running

Taobao Wang, which is like eBay just a bit
bigger. And Sina Weibo. Weibo mains micro

blogging or twittering. And Sina is the
biggest company in China, and they hold

more or less a third. Now we come to
something really interesting - it's Ant

Financials. Because Ant Financials created
something called Sesame Credit. Sesame

Credit is Ali Baba's child, not Tencent,
and Ali Baba's. Tencent developed their

own system. So, updates and corrections
without colors. 2014 was the year the

Chinese state council released a planning
outline on the construction of social

credit systems in China. And it gave us a
lot of information but only very few

specific information. So, they started,
and in 2020 it will be managed mandatory.

But they worked on for as long as at least
2005.

It allows governmental and private SCS. So
you have both. And eight companies are

allowed to form their own private social
credit systems. We learned Tencent and

Alibaba are two of them. In different
regions, Hainan for example, it's an

island - you have social credit systems
run by the government. So you see you have

a lot of different systems running at the
same time in different regions and

sometimes you can choose between different
social credit systems at the same place.

And Sesame Credit we have learned is the
biggest and run by Alibaba. And Sesame

Credit is not the one who will be
mandatory. I have one other point that I

forgot it might come back later. So, to
give you a bit more information, let's

have a quick look at recent developments
that I thought of would be really

interesting for you when you have that
scenario of a society that is formed by a

writing system. In 2016 there were some
new guidelines on the SCS. New

cybersecurity laws and you might expect it
they are getting a bit harder and harder

every time. And there was a police robot -
I don't know if you heard about it I was

going to write an article but then we had
a really nice conference that made us a

lot of work and my article died in the
way. So you might have watched Wall-E -

what was the movie couple of years ago.
You might remember Eve - there was a

little cute white robot that had a
specialty - it could taser. Someone in

China must have watched that movie too,
because guess what - it looks like Eve and

it can tazer. You find pictures and
clips online. And the People's Bank of

China announced its own digital currency
to come soon. They are working on it for

years. And the real name registrations for
online comments. So no real name

registrations, no online comments. And in
2020 - the mandatory Social Credit System.

The next question is usually: how does the
social criticism work? And we take a quick

look at Sesame Credit because they added
more and more services that Extra Credit

guys could know about. So we learned a lot
of data mining they do.

For example, when you use AliPay it goes
to your scores. So your score gets higher

and higher the more you shop with AliPay.
Then other Alibaba services - we know

there are different and millions of
services. And now it's getting special

because it draws information from
officials as well. So we have information

from courts, debtors registries, and
because love always makes everything

better, you get your data from the biggest
dating app in China - it's Baihe. So you

can see if the person you want to fall in
love with have a good score. And of

course, they're including more and more
services. And there are penalties now. If

you forgot to pay your child support for a
couple of month or years or; I have heard

people can forget that for quite a long
time. There are penalties: for example,

you can't take high-speed trains or first-
class flight tickets. Which is a very

intelligent way. Again, it's a very
intelligent way to add penalties because

if you start with people nobody likes,
nobody likes openly, it's clever. And of

course, the companies do not tell you a
lot about the technologies like they use

in the systems. So and then people want to
know about the mandatory system, what it

can do, how will it look and so on, and
there's a very simple answer - we don't

know. The Chinese government doesn't give
you a lot of information on how it will

look. But we can guess. In having a look
on how the government worked in the last

20-30 years. So, we can have a look how
they prepare the mandatory system. Like I

said, different systems at the same time,
and they will learn from this. And my

guess is they will have like a little
medley of the best features of the

existing systems and get rid of the faulty
ones. So, by best, I mean best from the

point of view of the government of course.
So now we are bit later because of our

technological difficulties, thanks again
for solving. Why do you have a Social

Credit System?
Let's dive a bit back in history. Some

people might remember the year 1989.
Before you had a lot of countries that

called themselves socialists. But after,
you had only a few and China is one of

them. And how did they make it? In the 80s
they had something called the era of

reform and opening-up, where they opened
up to the West mostly, and to capitalism.

And you might know what that means.
China's called the transformation society.

And with some people getting rich quickly,
and a lot of people getting poorer and

poorer and watching the rich - you have a
lot of social tension. And remember, China

is a huge country. A vast amount on
people, and know people are really good in

getting on each other's nerves. So you
have problems. I give you one number that

I really like - in 2014, the official
count of mass incidents everyday was

nearly 250. And the number is rising, so
it will be much more today. And a mass

incident can be anything from pensioners
holding up banners demanding more money

because they worked their whole lives, I'm
sorry, I really have to concentrate on not

swearing, to like, riots, full-grown
riots, petrol bombs, burning buildings.

For example, G20 in Hamburg, you might
have heard. That's a mass incident. That's

nothing to get crazy about, that's just a
little thing that happened. Hamburg is

still standing. And, thank you, hello
Hamburg.

[Applause]
And keep in mind organized societies are

controlled on different levels. That's not
only China. And social control is usually

done by groups, so it's much easier if we
control each other. And it has the long

history in China too, and a lot of
socialist countries develop new forms of

social control. So in China, you have a
Hukou - that means you can only get social

services where you're permitted to live.
I just give you a really really short

version of everything it's not it's a bit
more complex as you know. The Tang on

which you can say it's an analog version a
lot of your stuff what you've done goes in

there. You needed a marriage permit if you
wanted to get married and it wasn't easy

to live without a married permit and as
China became more and more digitized you

might know what comes next. Oh too anyone
control became digitized as well and as

people are using ICTs more and more they
create a huge amount of data. And the

bloody thing with data is you don't feel,
smell or see anything vanishing, do you?

And people it's helpful as humans to

develop something towards something we
can't see smell or fear. And more and more

technologies are used in civilians and
social control. But that still leaves the

question: why is China doing it? Let's
go to another point I'd really interested

in: ideological influences the way we see
the world and the way we think about it

is formed by theories, by ideologies and
what's the Chinese flag ? You see it

again, maybe you remember, some of you,
it's red with yellow stars - that usually

is a code for socialists and this is Mao
Zedong. He was kind of big in China. And

so you have - here you have three
socialist symbols. It really helps to

know a bit about Marxism in researching
China and especially in Mao Zedong

thought it's not called maoism it's
called more Zedong thought it's about what

he thought and in this reform of opening
up - I'm trying to feed you a bit of

Chinese history - the Chinese government
opened up to more. For example they opened

up to nationalism. And if you know a bit
about socialism, usually you wouldn't

put socialism and nationalism in one room,
because they didn't like each other. The

Chinese government found a way they like
each other. They got really interested

in PR and they really research other
countries how governments there stood in

power. And some are not so well known to
most of us this is Confucius.

Confucianism shaped and molded the
Chinese societies for more than 2,000

years. And if you have that big
influence, you can't just get rid of it.

It's in people's memories. It's in the
whole society, and Confusianism came back

as an ideology, where some parts of the
Socialists ideology vanished. Confucianism

came back and Confucianism is about a
society where everyone is on her or his

place and her place is always below. And
with all these new ideological

influences, the way of communication
changed as well.

Before it was the masses and now it's the
public. All campaigns and programs had

revolutionary names or poetic forms like
hundred flowers yet the Great Leap

Forward, and all these campaigns molded
our picture of China, or the picture

most people have in their hearts people in
bloom, red flags. But now it's changed you

have new campaigns and programs and they
have other names as well. You have

socialist harmonious society as a goal for
the society and a couple of years ago

before, you couldn't put socialism and
harmony in one room either. And you have

the Chinese dream. Yes, it sounds a bit
like American dream, yeah. And we learned

that all governments are interested in
control and feedback because you can't

control something, if you don't pay
attention and can't blow up anyway. So

like we have an autonomous nervous system
that tells every part what's going on, you

have one electronic autonomous nervous
system now in many societies not only

China.
People using ICTs and a social credit

system just creates a wonderful
opportunity of feedback and on the same

time on influence. So, we come to the end
of this talk and to the summary. So, again

this I think is really important: it's an
international tendency to solve social

problems with technological solutions. And
as far as I am concerned this is not the

best way maybe. And the Chinese government
- they do not count and think in years,

they think in decades at least. And they
looked at the internet and found out

that this is a good tool and they're using
it. Successfully. But we know at least

since Edward Snowden, that other
governments are capable of using the

Internet as a tool as well. A social
credit system exists of the use of big

data, algorithm, ICTs and gamification.
And it will become mandatory. And this

game, when it becomes mandatory, will
define your life. And not only your life,

it will define the life of the people
surrounding you as well. The schools your

children for example will go to will be
chosen on your rate. And there's the

opportunity to put this together with a
national cryptocurrency. So, thank you

all for listening that what's my current
state of research and my current state of

my code, we both presented to you this
talk and I'm really looking forward to QA

and discussing. Thanks.
34c3 outro

subtitles created by c3subtitles.de
in the year 2018. Join, and help us!