34c3 intro
Herald: Okay, so ladies and gentlemen
Katika Kühnreich. So did you all
thank the angels today? No, yes, maybe?
[Applause]
Herald: Did you all took your prosĂm two
today? No? Okay, see, Katika will tell you
about the social credit system from China,
and will tell you a little bit more about
how a human being can be valued. She is
the magista in political science and
modern and ancient Chinese science. She
not only has her theoretical knowledge
from books, but she always also combines
it with the street study. So please
welcome, with a very warm hand of
applause, Katika!
[Applause]
Katika: Hey guys, I'm happy to be here,
and first I want to thank all the people
who made that possible. This conference
and this talk; so thanks to all you, and a
special thanks to all the people
translating.
[Applause]
Katika: Before we start, I like to say a
few points. The first was the thank you,
we already have. The second is I have to
apologize: English is not my mother tongue
so you will have some funny mistakes that
I make, so the Internet has a bit more
fun. Because the world can be (nice). The
second thing, is I will take you on a
journey through China's social credit
system - its current state. And because we
only have half an hour, more or less,
because I'd really like to spend a lot of
time with you guys on talks, on discussion
and QA, we have to leave a lot of
interesting sites out. So there were some
excellent talks on the CCC about
technological aspects of Internet and
Internet usage in China. I will
concentrate really hard on this social
credit system. But if you have questions,
please ask in the Q&A. The Q&A will not be
recorded, so you can feel free. And if you
have more questions, just contact me after
the talk. And, it's a China topic and
often when you talk about China, people
say "oh yeah, oh yeah: that's China." This
is no China bashing session - I'm sorry
for anyone who expected that. China has a
government, and a government wants to
control something. So this talk is about
social credit systems in China but it is
about a tendency I see internationally.
It's the tendency to answer social
problems with technological solutions, or
at least it's the try. So please keep in
mind: it's happening in China, but it's
not that we are safe in the West or cozy
or anything else. So it's meant as an
example of an international tendency. And
I prepared some cards, so I will stick to
the topic. I have to look at them from
time to time. This talk presents my
current state of research; it's not the
truth. And this talked sadly represents my
current state of my cold as well, so I'm
not totally fed please apologized that as
well. For a little journey I prepared a
couple of stops. The first one will be a
video from 2015 made by Extra Credit on
social credit systems in China. So if you
don't like the talk, you have seven and a
half minutes to just vanish - nobody will
notice. The next point will be "Updates
and Corrections" - it's a video from
2015, and there's some faulty information
in it. And to do that, I will give you a
quick look at the Chinese Internet
landscape. The next point will be the
current state of social credit systems. If
I say SCS: it's the same - I'm just lazy.
And there's one question often not
answered when talking about the Chinese
credit system - it's why is it happening.
And again it's not because it's an evil
government. It's a government. So we'll
dive in some ideological aspects of
Chinese politics today. And in the end,
I'm really looking forward to that point
because I can shut a bit up and listen to
your thoughts and we're really interested
in that. So, everyone ready? Cozy? That's
wonderful, thanks! Then we can start with
the video; that isn't it shown right now.
Now we need some technical support. It's
much more fun when you see the pictures.
But we have time, so.
Herald: Do we have the (tv guys) here?
Katika: We got free water!
Herald: Gentlemen, I think we have a
technical problem here, so while the VOC
and my stage manager is trying to get some
people from there. Yes he will do this -
he's a good guy. He's the best guy!
Gentlemen, ladies, applause for the VOC!
[Applause]
Herald: So while we're waiting to have the
video; and trust me it looked really good
from here. Let's play a little game -
later we will have a Q&A session but
unfortunately we are not allowed or we we
can't record the Q&A session, and we will
also not video stream the Q&A session. So
I would like you to have a quick hand of
applause for all of you, because you will
have a great discussion with this nice
lady and therefore we try to make applause
- we test the applause.
Katika: Okay
[Applause]
Herald: No no no, that should be louder -
come on!
[Applause]
Herald: Okay we still have no video, but I
think over there I did not hear any
applause. Could you, could you please try
it again?
[Applause]
Herald: Yeah, better, better. I think
these guys are way better. Are they? Come
on!
[Applause]
Katika: Otherwise I can use the time to
give you some information I'd present you
after the video. Thank you. The
information you get in the video is based
on a press release of the American Civil
Liberties Union, and that contains some
mistakes. One thing - when you want to
know something about China, the government
is really open in publishing what they're
doing, and what they're planning to do.
And it's seems to be much more comfortable
to just copy, but maybe have a look into
what the government says because it's a
good way to find out what they are doing
and they're scaring the open. It works
with a lot of governments. And I realized
that because this video had a lot of
impact, and there was a lot of news report
on that scary system in China but when you
looked at the news report and you had the
video before, you found out that it was
more or less the same, the very same, as
in the video even years later. So please
if you want to find something out - check
for facts. They might be online too. So
that's the picture, and that's my break.
Video: Okay, so one you recently brought
this to our attention, and when we first
started looking into it, we couldn't
believe it. Even now looking at this I
keep thinking that there's got to be
something about this that I just don't
understand, like, I must be reading this
wrong. It's just so clearly something out
of dystopian science fiction. And yet,
here it is in our world, and people are
actually embracing it. Apparently China
has gamified being an obedient citizen.
[Music]
Video: Going under the innocuous name of
Sesame Credit, China has created a score
for how good a citizen you are, and that's
one of the scariest things I've heard in
quite a while. It's jointly run by
Tencent, yes that Tencent - the one that
owns Riot Games and has a significant
share in Epic and Activision-Blizzard, and
also the ascendant Chinese competitor to
Amazon, Alibaba, hence the name Sesame. So
the owners of China's largest social
networks have partnered with the
governments to create something akin to
the US credit score. But instead of
measuring how regularly you pay your
bills, it measures how obediently you
follow the party line. They dredge data
from your social network, so if you post
pictures of Tiananmen Square, or share a
link about the recent stock market
collapse, your Sesame Credit score goes
down. Share a link from the state-
sponsored news agency about how good the
economy is doing and your score goes up.
The Alibaba and Tencent are also the
largest online retailers in China so
Sesame Credit is also able to pull data
from your purchases. If you're making
purchases the state deems valuable like
work shoes or local agricultural products,
your score goes up. If you import anime
from Japan, though, down the score goes.
And this score has real-world
consequences. Like many games, Sesame
Credit has tiers and levels, and having a
high score gives you special benefits -
like making it easier to get the paperwork
you need to travel. Or making it easier to
get a loan. Now currently, there are no
consequences for having a low score but
there's been talk about implementing
penalties once the system becomes
mandatory in 2020. Penalties like slower
Internet speeds for low scoring citizens.
Or even restricting the jobs that a low
scoring person's allowed to hold. But
there's one more layer to Sesame Credit,
and here's where this goes from being
repulsive to downright insidious. Because
this is all part of a social network it
also scans your friends -
so you will lose points for having friends
with low obedience scores. And it tells
you this. At any point, anybody can check
anyone elses score. And when you check
your own score, Sesame Credit provides a
handy map of your friends to show you who
is dragging your score down. Have you ever
had that thing where you play a game with
somebody who wasn't doing very well and
you've tried to change their behavior to
make them do better? Or maybe after a
while you just sort of stopped playing
with the people who are holding you back.
That's at the heart of how this system
works, and it's also what makes this one
of the most terrifying tools of
authoritarian oppression I've ever read
about. Because like mass censorship, jail
time, assassinations - those are all big
messy implements for keeping a population
in line. That messiness and severity
foster resentment - eventually rebellion.
They're expensive, they're unwieldy - in
the end, those tools are impossible to
maintain. But social pressure?
Ostracization? Those things are free -
they happen on their own. And as a
government tool they don't have nearly the
same potential for going embarrassingly
disastrously wrong. With a system like
this in place, a government doesn't even
have to tell neighbor to spy on neighbor;
to rat each other out. Because that's all
built into a seemingly innocuous game
system. The government need not step in.
Re-education will be handled for them by
friends, classmates, and relatives who
want to maintain a high score. And if that
doesn't work then potentially dangerous
ideas still end up quarantined by the
social isolation this game system causes.
Express or help to spread to many radical
ideas, and people will stop associating
with you. And not because some jackbooted
thugs showed up at the door with threats,
but simply because associating with
somebody with those ideas will lose them
all the privileges they've worked so hard
to obtain. It recontextualizes obedience
to an authoritarian regime.
In the past, you obeyed such powers
because you were afraid. Fear kept you
motivated. But fear is negative. It
fosters resentment. The world we're
stepping into instead uses positive
reinforcement to promote being subservient
to the will of the regime. It's Big
Brother's kinder, gentler hand. And the
things that make this scary is that we've
seen the efficacy of this only too well in
games. You may not actually know this, but
when World of Warcraft was in its early
stages of development, it had an unrested
penalty mechanic that started limiting
experienced gains for players who had
played too much. And players hated it -
they resented it they complained about it
every day. So after brainstorming on this
for a while, Blizzard had the idea to
simply change how the mechanic was
presented. Without changing any of the
existing numbers or systems, they started
referring to the unrested experience state
as normal, and made it the default. And
they started calling the original normal
experience gained state as rested. That's
all they changed, and everybody loved it.
People would log on every day just to get
that bonus. Positive reinforcement works
wonders. But unlike World of Warcraft,
which built a system just to get people to
embrace meaningless grinding, Sesame
Credit has built a system to get people to
enjoy falling into line. Now the system's
not mandatory yet - it's opt-in right now.
But it's going to be mandatory in 2020,
and there's a terrible brilliance to
phasing that in. The early adopters are
going to be people excited about this
system. People who are already patriotic
and are eager for anything that'll help
display that patriotism to the world. And,
as early adopters they're gonna talk it
up. They're gonna give it an air of being
positive and fun.
Then it'll be foisted on the society as a
whole. More than that though, the early
adopters are gonna compete. Already you
can see hundreds of thousands of tweets
displaying people's high scores or showing
off the new milestones they've hit. Giving
a hard numbers to their patriotism and
giving them bragging rights for being the
most patriotic, most right-thinking person
they know. And that's gonna set the tone
for how Sesame Credit is intended to be
used - as a competition to see who can
agree with the government the most. We've
talked about propaganda games on this
channel before but for all the time we've
spent examining and deconstructing
terrible games that espouse hate and for
all the studies on propaganda games James
has done, this is the use of game systems
that frightens me the most. Because to
most people Sesame Credit will seem
benign. Perhaps even fun - it's a
conversation starter, something to share
with your friends. But it's making heavy
use of all the psychological motivators
that we Gamemakers deploy in scoring
systems, and ladders and levels. Systems
that we built to shape play habits and to
keep people coming back. Like I said, I'm
still kind of in a state of disbelief
looking at this. If any of you are
watching this from China, please tell me
if I'm misunderstanding this thing,
because I would love to be wrong about it.
If not, well, I hope this episode can do
some small amount to help the fight to
keep such a system from becoming
mandatory. For everyone in the rest of the
world, I hope this helps remind us all how
important it is to be aware and to be
vigilant. All of these gamification
techniques we've learned through making
games offer incredible opportunities for
making this world a better more engaging
place. But every great tool carries with
it the potential for misuse. And it's on
us as a community who understands this
amazing new medium, to do what we can to
stop that. We'll see you next week.
[Applause]
Katika: So, we had the shedule, and thank
you to Extra Credit for this video, and
thank you for concentrating on
gamification. Because a lot of time, when
I talk to social scientists, without a lot
of technological knowledge, it was like
okay - it's a new system. And the whole
gamification thing triggered me, and I
wondered about a lot of people who didn't
seem too interested in it. Because I said
that the video contains some faulty
information. First, I'd like to
concentrate on key aspects they told us in
with this video. There is a social rating
system in China now. The system will
become mandatory in 2020. That's all
right. It works on Big Data algorithms,
artificial intelligence, and your online
and offline data. And it uses technologies
like gamification and nudging. And if you
hear it's an Orwellian system - no it's
not. 1984 was about force. Gamification is
about feeling good and cozy. So, in the
updates and corrections, I promise you a
quick look at the end Chinese internet
landscape. What most people already know.
From the beginning the Chinese government
started to control the internet as much as
it could and its users. So some
vocabularies you might have heard are the
Great Firewall, or its new extra the
Golden Shield, the Great Cannon, which is
a DDoS weapon. It would be really
fascinating to dive in there, but we have
to watch from far. Another point that's
really interesting for us looking Social
Credit System is true name or real name
registration. Which means you can't use a
lot of services if you don't give them
your ID number. And the mandatory system
in 2020 will work on ID numbers as far as
we know. So, we watched the other side:
the users, their millions of users - it's
one more than 1.4 billion people in China.
And ICTs are really popular. Much more
than in Germany, for example. And another
thing that please keep in mind - cash is
vanishing in China.
A couple of years ago, for example, when I
lived in China 2004: I paid my rent in
cash half a year in advance. When I paid
for anything I paid cash. You could
couldn't find a lot of places accepting
cards. But now China didn't have this
development of cards - they just jumped
right through paying with mobile apps. And
you can buy noodles or snack or magazine
with your mobile phone. And the Chinese
users found multiple ways to deal with
censorship, and to trespass it. It's a
wonderful thing to look up.
And now, the big players of the Chinese
Internet, because we learned some names in
the film. I made something very Chinese -
I made something colorful. Okay, you have
three big players: its Alibaba and Tencent
- we heard about them, and Baidu is the
third biggest player. Each millions of
users, or billions. So to explain this, I
will use some American companies names to
help understanding. And Baidu is known for
that; was the wrong; yeah, Baidu is known
for Baidu Barker - this is like the
Chinese Wikipedia, it's just a bit much
controlled. And a have Baidu Pay, and
Baidu is huge in artificial intelligence.
So the next one is Tencent - we learned
about the gaming thing. And developed QQ -
this is a messenger than some of you might
remember: ICQ. Yeah, it's just a bit
bigger. Another app a lot of people don't
think it's Chinese, it is Chinese, it's
called WeChat. It's I can say they even
mmm WhatsApp, right? And guess what WeChat
developed - WeChat Pay. And there's
another little company Tencent holds some
parts in its SnapChat - it's twelve
percent owned by Tencent. Alibaba has
Alipay which is one of the biggest payment
apps. And you can even use it in Germany
now I think, in some shops. It's running
Taobao Wang, which is like eBay just a bit
bigger. And Sina Weibo. Weibo mains micro
blogging or twittering. And Sina is the
biggest company in China, and they hold
more or less a third. Now we come to
something really interesting - it's Ant
Financials. Because Ant Financials created
something called Sesame Credit. Sesame
Credit is Ali Baba's child, not Tencent,
and Ali Baba's. Tencent developed their
own system. So, updates and corrections
without colors. 2014 was the year the
Chinese state council released a planning
outline on the construction of social
credit systems in China. And it gave us a
lot of information but only very few
specific information. So, they started,
and in 2020 it will be managed mandatory.
But they worked on for as long as at least
2005.
It allows governmental and private SCS. So
you have both. And eight companies are
allowed to form their own private social
credit systems. We learned Tencent and
Alibaba are two of them. In different
regions, Hainan for example, it's an
island - you have social credit systems
run by the government. So you see you have
a lot of different systems running at the
same time in different regions and
sometimes you can choose between different
social credit systems at the same place.
And Sesame Credit we have learned is the
biggest and run by Alibaba. And Sesame
Credit is not the one who will be
mandatory. I have one other point that I
forgot it might come back later. So, to
give you a bit more information, let's
have a quick look at recent developments
that I thought of would be really
interesting for you when you have that
scenario of a society that is formed by a
writing system. In 2016 there were some
new guidelines on the SCS. New
cybersecurity laws and you might expect it
they are getting a bit harder and harder
every time. And there was a police robot -
I don't know if you heard about it I was
going to write an article but then we had
a really nice conference that made us a
lot of work and my article died in the
way. So you might have watched Wall-E -
what was the movie couple of years ago.
You might remember Eve - there was a
little cute white robot that had a
specialty - it could taser. Someone in
China must have watched that movie too,
because guess what - it looks like Eve and
it can tazer. You find pictures and
clips online. And the People's Bank of
China announced its own digital currency
to come soon. They are working on it for
years. And the real name registrations for
online comments. So no real name
registrations, no online comments. And in
2020 - the mandatory Social Credit System.
The next question is usually: how does the
social criticism work? And we take a quick
look at Sesame Credit because they added
more and more services that Extra Credit
guys could know about. So we learned a lot
of data mining they do.
For example, when you use AliPay it goes
to your scores. So your score gets higher
and higher the more you shop with AliPay.
Then other Alibaba services - we know
there are different and millions of
services. And now it's getting special
because it draws information from
officials as well. So we have information
from courts, debtors registries, and
because love always makes everything
better, you get your data from the biggest
dating app in China - it's Baihe. So you
can see if the person you want to fall in
love with have a good score. And of
course, they're including more and more
services. And there are penalties now. If
you forgot to pay your child support for a
couple of month or years or; I have heard
people can forget that for quite a long
time. There are penalties: for example,
you can't take high-speed trains or first-
class flight tickets. Which is a very
intelligent way. Again, it's a very
intelligent way to add penalties because
if you start with people nobody likes,
nobody likes openly, it's clever. And of
course, the companies do not tell you a
lot about the technologies like they use
in the systems. So and then people want to
know about the mandatory system, what it
can do, how will it look and so on, and
there's a very simple answer - we don't
know. The Chinese government doesn't give
you a lot of information on how it will
look. But we can guess. In having a look
on how the government worked in the last
20-30 years. So, we can have a look how
they prepare the mandatory system. Like I
said, different systems at the same time,
and they will learn from this. And my
guess is they will have like a little
medley of the best features of the
existing systems and get rid of the faulty
ones. So, by best, I mean best from the
point of view of the government of course.
So now we are bit later because of our
technological difficulties, thanks again
for solving. Why do you have a Social
Credit System?
Let's dive a bit back in history. Some
people might remember the year 1989.
Before you had a lot of countries that
called themselves socialists. But after,
you had only a few and China is one of
them. And how did they make it? In the 80s
they had something called the era of
reform and opening-up, where they opened
up to the West mostly, and to capitalism.
And you might know what that means.
China's called the transformation society.
And with some people getting rich quickly,
and a lot of people getting poorer and
poorer and watching the rich - you have a
lot of social tension. And remember, China
is a huge country. A vast amount on
people, and know people are really good in
getting on each other's nerves. So you
have problems. I give you one number that
I really like - in 2014, the official
count of mass incidents everyday was
nearly 250. And the number is rising, so
it will be much more today. And a mass
incident can be anything from pensioners
holding up banners demanding more money
because they worked their whole lives, I'm
sorry, I really have to concentrate on not
swearing, to like, riots, full-grown
riots, petrol bombs, burning buildings.
For example, G20 in Hamburg, you might
have heard. That's a mass incident. That's
nothing to get crazy about, that's just a
little thing that happened. Hamburg is
still standing. And, thank you, hello
Hamburg.
[Applause]
And keep in mind organized societies are
controlled on different levels. That's not
only China. And social control is usually
done by groups, so it's much easier if we
control each other. And it has the long
history in China too, and a lot of
socialist countries develop new forms of
social control. So in China, you have a
Hukou - that means you can only get social
services where you're permitted to live.
I just give you a really really short
version of everything it's not it's a bit
more complex as you know. The Tang on
which you can say it's an analog version a
lot of your stuff what you've done goes in
there. You needed a marriage permit if you
wanted to get married and it wasn't easy
to live without a married permit and as
China became more and more digitized you
might know what comes next. Oh too anyone
control became digitized as well and as
people are using ICTs more and more they
create a huge amount of data. And the
bloody thing with data is you don't feel,
smell or see anything vanishing, do you?
And people it's helpful as humans to
develop something towards something we
can't see smell or fear. And more and more
technologies are used in civilians and
social control. But that still leaves the
question: why is China doing it? Let's
go to another point I'd really interested
in: ideological influences the way we see
the world and the way we think about it
is formed by theories, by ideologies and
what's the Chinese flag ? You see it
again, maybe you remember, some of you,
it's red with yellow stars - that usually
is a code for socialists and this is Mao
Zedong. He was kind of big in China. And
so you have - here you have three
socialist symbols. It really helps to
know a bit about Marxism in researching
China and especially in Mao Zedong
thought it's not called maoism it's
called more Zedong thought it's about what
he thought and in this reform of opening
up - I'm trying to feed you a bit of
Chinese history - the Chinese government
opened up to more. For example they opened
up to nationalism. And if you know a bit
about socialism, usually you wouldn't
put socialism and nationalism in one room,
because they didn't like each other. The
Chinese government found a way they like
each other. They got really interested
in PR and they really research other
countries how governments there stood in
power. And some are not so well known to
most of us this is Confucius.
Confucianism shaped and molded the
Chinese societies for more than 2,000
years. And if you have that big
influence, you can't just get rid of it.
It's in people's memories. It's in the
whole society, and Confusianism came back
as an ideology, where some parts of the
Socialists ideology vanished. Confucianism
came back and Confucianism is about a
society where everyone is on her or his
place and her place is always below. And
with all these new ideological
influences, the way of communication
changed as well.
Before it was the masses and now it's the
public. All campaigns and programs had
revolutionary names or poetic forms like
hundred flowers yet the Great Leap
Forward, and all these campaigns molded
our picture of China, or the picture
most people have in their hearts people in
bloom, red flags. But now it's changed you
have new campaigns and programs and they
have other names as well. You have
socialist harmonious society as a goal for
the society and a couple of years ago
before, you couldn't put socialism and
harmony in one room either. And you have
the Chinese dream. Yes, it sounds a bit
like American dream, yeah. And we learned
that all governments are interested in
control and feedback because you can't
control something, if you don't pay
attention and can't blow up anyway. So
like we have an autonomous nervous system
that tells every part what's going on, you
have one electronic autonomous nervous
system now in many societies not only
China.
People using ICTs and a social credit
system just creates a wonderful
opportunity of feedback and on the same
time on influence. So, we come to the end
of this talk and to the summary. So, again
this I think is really important: it's an
international tendency to solve social
problems with technological solutions. And
as far as I am concerned this is not the
best way maybe. And the Chinese government
- they do not count and think in years,
they think in decades at least. And they
looked at the internet and found out
that this is a good tool and they're using
it. Successfully. But we know at least
since Edward Snowden, that other
governments are capable of using the
Internet as a tool as well. A social
credit system exists of the use of big
data, algorithm, ICTs and gamification.
And it will become mandatory. And this
game, when it becomes mandatory, will
define your life. And not only your life,
it will define the life of the people
surrounding you as well. The schools your
children for example will go to will be
chosen on your rate. And there's the
opportunity to put this together with a
national cryptocurrency. So, thank you
all for listening that what's my current
state of research and my current state of
my code, we both presented to you this
talk and I'm really looking forward to QA
and discussing. Thanks.
34c3 outro
subtitles created by c3subtitles.de
in the year 2018. Join, and help us!