-
34c3 intro
Herald: Okay, so ladies and gentlemen
-
Katika Kühnreich. So did you all
thank the angels today? No, yes, maybe?
-
[Applause]
Herald: Did you all took your prosĂm two
-
today? No? Okay, see, Katika will tell you
about the social credit system from China,
-
and will tell you a little bit more about
how a human being can be valued. She is
-
the magista in political science and
modern and ancient Chinese science. She
-
not only has her theoretical knowledge
from books, but she always also combines
-
it with the street study. So please
welcome, with a very warm hand of
-
applause, Katika!
[Applause]
-
Katika: Hey guys, I'm happy to be here,
and first I want to thank all the people
-
who made that possible. This conference
and this talk; so thanks to all you, and a
-
special thanks to all the people
translating.
-
[Applause]
Katika: Before we start, I like to say a
-
few points. The first was the thank you,
we already have. The second is I have to
-
apologize: English is not my mother tongue
so you will have some funny mistakes that
-
I make, so the Internet has a bit more
fun. Because the world can be (nice). The
-
second thing, is I will take you on a
journey through China's social credit
-
system - its current state. And because we
only have half an hour, more or less,
-
because I'd really like to spend a lot of
time with you guys on talks, on discussion
-
and QA, we have to leave a lot of
interesting sites out. So there were some
-
excellent talks on the CCC about
technological aspects of Internet and
-
Internet usage in China. I will
concentrate really hard on this social
-
credit system. But if you have questions,
please ask in the Q&A. The Q&A will not be
-
recorded, so you can feel free. And if you
have more questions, just contact me after
-
the talk. And, it's a China topic and
often when you talk about China, people
-
say "oh yeah, oh yeah: that's China." This
is no China bashing session - I'm sorry
-
for anyone who expected that. China has a
government, and a government wants to
-
control something. So this talk is about
social credit systems in China but it is
-
about a tendency I see internationally.
It's the tendency to answer social
-
problems with technological solutions, or
at least it's the try. So please keep in
-
mind: it's happening in China, but it's
not that we are safe in the West or cozy
-
or anything else. So it's meant as an
example of an international tendency. And
-
I prepared some cards, so I will stick to
the topic. I have to look at them from
-
time to time. This talk presents my
current state of research; it's not the
-
truth. And this talked sadly represents my
current state of my cold as well, so I'm
-
not totally fed please apologized that as
well. For a little journey I prepared a
-
couple of stops. The first one will be a
video from 2015 made by Extra Credit on
-
social credit systems in China. So if you
don't like the talk, you have seven and a
-
half minutes to just vanish - nobody will
notice. The next point will be "Updates
-
and Corrections" - it's a video from
2015, and there's some faulty information
-
in it. And to do that, I will give you a
quick look at the Chinese Internet
-
landscape. The next point will be the
current state of social credit systems. If
-
I say SCS: it's the same - I'm just lazy.
And there's one question often not
-
answered when talking about the Chinese
credit system - it's why is it happening.
-
And again it's not because it's an evil
government. It's a government. So we'll
-
dive in some ideological aspects of
Chinese politics today. And in the end,
-
I'm really looking forward to that point
because I can shut a bit up and listen to
-
your thoughts and we're really interested
in that. So, everyone ready? Cozy? That's
-
wonderful, thanks! Then we can start with
the video; that isn't it shown right now.
-
Now we need some technical support. It's
much more fun when you see the pictures.
-
But we have time, so.
Herald: Do we have the (tv guys) here?
-
Katika: We got free water!
Herald: Gentlemen, I think we have a
-
technical problem here, so while the VOC
and my stage manager is trying to get some
-
people from there. Yes he will do this -
he's a good guy. He's the best guy!
-
Gentlemen, ladies, applause for the VOC!
[Applause]
-
Herald: So while we're waiting to have the
video; and trust me it looked really good
-
from here. Let's play a little game -
later we will have a Q&A session but
-
unfortunately we are not allowed or we we
can't record the Q&A session, and we will
-
also not video stream the Q&A session. So
I would like you to have a quick hand of
-
applause for all of you, because you will
have a great discussion with this nice
-
lady and therefore we try to make applause
- we test the applause.
-
Katika: Okay
[Applause]
-
Herald: No no no, that should be louder -
come on!
-
[Applause]
Herald: Okay we still have no video, but I
-
think over there I did not hear any
applause. Could you, could you please try
-
it again?
[Applause]
-
Herald: Yeah, better, better. I think
these guys are way better. Are they? Come
-
on!
[Applause]
-
Katika: Otherwise I can use the time to
give you some information I'd present you
-
after the video. Thank you. The
information you get in the video is based
-
on a press release of the American Civil
Liberties Union, and that contains some
-
mistakes. One thing - when you want to
know something about China, the government
-
is really open in publishing what they're
doing, and what they're planning to do.
-
And it's seems to be much more comfortable
to just copy, but maybe have a look into
-
what the government says because it's a
good way to find out what they are doing
-
and they're scaring the open. It works
with a lot of governments. And I realized
-
that because this video had a lot of
impact, and there was a lot of news report
-
on that scary system in China but when you
looked at the news report and you had the
-
video before, you found out that it was
more or less the same, the very same, as
-
in the video even years later. So please
if you want to find something out - check
-
for facts. They might be online too. So
that's the picture, and that's my break.
-
Video: Okay, so one you recently brought
this to our attention, and when we first
-
started looking into it, we couldn't
believe it. Even now looking at this I
-
keep thinking that there's got to be
something about this that I just don't
-
understand, like, I must be reading this
wrong. It's just so clearly something out
-
of dystopian science fiction. And yet,
here it is in our world, and people are
-
actually embracing it. Apparently China
has gamified being an obedient citizen.
-
[Music]
Video: Going under the innocuous name of
-
Sesame Credit, China has created a score
for how good a citizen you are, and that's
-
one of the scariest things I've heard in
quite a while. It's jointly run by
-
Tencent, yes that Tencent - the one that
owns Riot Games and has a significant
-
share in Epic and Activision-Blizzard, and
also the ascendant Chinese competitor to
-
Amazon, Alibaba, hence the name Sesame. So
the owners of China's largest social
-
networks have partnered with the
governments to create something akin to
-
the US credit score. But instead of
measuring how regularly you pay your
-
bills, it measures how obediently you
follow the party line. They dredge data
-
from your social network, so if you post
pictures of Tiananmen Square, or share a
-
link about the recent stock market
collapse, your Sesame Credit score goes
-
down. Share a link from the state-
sponsored news agency about how good the
-
economy is doing and your score goes up.
The Alibaba and Tencent are also the
-
largest online retailers in China so
Sesame Credit is also able to pull data
-
from your purchases. If you're making
purchases the state deems valuable like
-
work shoes or local agricultural products,
your score goes up. If you import anime
-
from Japan, though, down the score goes.
And this score has real-world
-
consequences. Like many games, Sesame
Credit has tiers and levels, and having a
-
high score gives you special benefits -
like making it easier to get the paperwork
-
you need to travel. Or making it easier to
get a loan. Now currently, there are no
-
consequences for having a low score but
there's been talk about implementing
-
penalties once the system becomes
mandatory in 2020. Penalties like slower
-
Internet speeds for low scoring citizens.
Or even restricting the jobs that a low
-
scoring person's allowed to hold. But
there's one more layer to Sesame Credit,
-
and here's where this goes from being
repulsive to downright insidious. Because
-
this is all part of a social network it
also scans your friends -
-
so you will lose points for having friends
with low obedience scores. And it tells
-
you this. At any point, anybody can check
anyone elses score. And when you check
-
your own score, Sesame Credit provides a
handy map of your friends to show you who
-
is dragging your score down. Have you ever
had that thing where you play a game with
-
somebody who wasn't doing very well and
you've tried to change their behavior to
-
make them do better? Or maybe after a
while you just sort of stopped playing
-
with the people who are holding you back.
That's at the heart of how this system
-
works, and it's also what makes this one
of the most terrifying tools of
-
authoritarian oppression I've ever read
about. Because like mass censorship, jail
-
time, assassinations - those are all big
messy implements for keeping a population
-
in line. That messiness and severity
foster resentment - eventually rebellion.
-
They're expensive, they're unwieldy - in
the end, those tools are impossible to
-
maintain. But social pressure?
Ostracization? Those things are free -
-
they happen on their own. And as a
government tool they don't have nearly the
-
same potential for going embarrassingly
disastrously wrong. With a system like
-
this in place, a government doesn't even
have to tell neighbor to spy on neighbor;
-
to rat each other out. Because that's all
built into a seemingly innocuous game
-
system. The government need not step in.
Re-education will be handled for them by
-
friends, classmates, and relatives who
want to maintain a high score. And if that
-
doesn't work then potentially dangerous
ideas still end up quarantined by the
-
social isolation this game system causes.
Express or help to spread to many radical
-
ideas, and people will stop associating
with you. And not because some jackbooted
-
thugs showed up at the door with threats,
but simply because associating with
-
somebody with those ideas will lose them
all the privileges they've worked so hard
-
to obtain. It recontextualizes obedience
to an authoritarian regime.
-
In the past, you obeyed such powers
because you were afraid. Fear kept you
-
motivated. But fear is negative. It
fosters resentment. The world we're
-
stepping into instead uses positive
reinforcement to promote being subservient
-
to the will of the regime. It's Big
Brother's kinder, gentler hand. And the
-
things that make this scary is that we've
seen the efficacy of this only too well in
-
games. You may not actually know this, but
when World of Warcraft was in its early
-
stages of development, it had an unrested
penalty mechanic that started limiting
-
experienced gains for players who had
played too much. And players hated it -
-
they resented it they complained about it
every day. So after brainstorming on this
-
for a while, Blizzard had the idea to
simply change how the mechanic was
-
presented. Without changing any of the
existing numbers or systems, they started
-
referring to the unrested experience state
as normal, and made it the default. And
-
they started calling the original normal
experience gained state as rested. That's
-
all they changed, and everybody loved it.
People would log on every day just to get
-
that bonus. Positive reinforcement works
wonders. But unlike World of Warcraft,
-
which built a system just to get people to
embrace meaningless grinding, Sesame
-
Credit has built a system to get people to
enjoy falling into line. Now the system's
-
not mandatory yet - it's opt-in right now.
But it's going to be mandatory in 2020,
-
and there's a terrible brilliance to
phasing that in. The early adopters are
-
going to be people excited about this
system. People who are already patriotic
-
and are eager for anything that'll help
display that patriotism to the world. And,
-
as early adopters they're gonna talk it
up. They're gonna give it an air of being
-
positive and fun.
Then it'll be foisted on the society as a
-
whole. More than that though, the early
adopters are gonna compete. Already you
-
can see hundreds of thousands of tweets
displaying people's high scores or showing
-
off the new milestones they've hit. Giving
a hard numbers to their patriotism and
-
giving them bragging rights for being the
most patriotic, most right-thinking person
-
they know. And that's gonna set the tone
for how Sesame Credit is intended to be
-
used - as a competition to see who can
agree with the government the most. We've
-
talked about propaganda games on this
channel before but for all the time we've
-
spent examining and deconstructing
terrible games that espouse hate and for
-
all the studies on propaganda games James
has done, this is the use of game systems
-
that frightens me the most. Because to
most people Sesame Credit will seem
-
benign. Perhaps even fun - it's a
conversation starter, something to share
-
with your friends. But it's making heavy
use of all the psychological motivators
-
that we Gamemakers deploy in scoring
systems, and ladders and levels. Systems
-
that we built to shape play habits and to
keep people coming back. Like I said, I'm
-
still kind of in a state of disbelief
looking at this. If any of you are
-
watching this from China, please tell me
if I'm misunderstanding this thing,
-
because I would love to be wrong about it.
If not, well, I hope this episode can do
-
some small amount to help the fight to
keep such a system from becoming
-
mandatory. For everyone in the rest of the
world, I hope this helps remind us all how
-
important it is to be aware and to be
vigilant. All of these gamification
-
techniques we've learned through making
games offer incredible opportunities for
-
making this world a better more engaging
place. But every great tool carries with
-
it the potential for misuse. And it's on
us as a community who understands this
-
amazing new medium, to do what we can to
stop that. We'll see you next week.
-
[Applause]
Katika: So, we had the shedule, and thank
-
you to Extra Credit for this video, and
thank you for concentrating on
-
gamification. Because a lot of time, when
I talk to social scientists, without a lot
-
of technological knowledge, it was like
okay - it's a new system. And the whole
-
gamification thing triggered me, and I
wondered about a lot of people who didn't
-
seem too interested in it. Because I said
that the video contains some faulty
-
information. First, I'd like to
concentrate on key aspects they told us in
-
with this video. There is a social rating
system in China now. The system will
-
become mandatory in 2020. That's all
right. It works on Big Data algorithms,
-
artificial intelligence, and your online
and offline data. And it uses technologies
-
like gamification and nudging. And if you
hear it's an Orwellian system - no it's
-
not. 1984 was about force. Gamification is
about feeling good and cozy. So, in the
-
updates and corrections, I promise you a
quick look at the end Chinese internet
-
landscape. What most people already know.
From the beginning the Chinese government
-
started to control the internet as much as
it could and its users. So some
-
vocabularies you might have heard are the
Great Firewall, or its new extra the
-
Golden Shield, the Great Cannon, which is
a DDoS weapon. It would be really
-
fascinating to dive in there, but we have
to watch from far. Another point that's
-
really interesting for us looking Social
Credit System is true name or real name
-
registration. Which means you can't use a
lot of services if you don't give them
-
your ID number. And the mandatory system
in 2020 will work on ID numbers as far as
-
we know. So, we watched the other side:
the users, their millions of users - it's
-
one more than 1.4 billion people in China.
And ICTs are really popular. Much more
-
than in Germany, for example. And another
thing that please keep in mind - cash is
-
vanishing in China.
A couple of years ago, for example, when I
-
lived in China 2004: I paid my rent in
cash half a year in advance. When I paid
-
for anything I paid cash. You could
couldn't find a lot of places accepting
-
cards. But now China didn't have this
development of cards - they just jumped
-
right through paying with mobile apps. And
you can buy noodles or snack or magazine
-
with your mobile phone. And the Chinese
users found multiple ways to deal with
-
censorship, and to trespass it. It's a
wonderful thing to look up.
-
And now, the big players of the Chinese
Internet, because we learned some names in
-
the film. I made something very Chinese -
I made something colorful. Okay, you have
-
three big players: its Alibaba and Tencent
- we heard about them, and Baidu is the
-
third biggest player. Each millions of
users, or billions. So to explain this, I
-
will use some American companies names to
help understanding. And Baidu is known for
-
that; was the wrong; yeah, Baidu is known
for Baidu Barker - this is like the
-
Chinese Wikipedia, it's just a bit much
controlled. And a have Baidu Pay, and
-
Baidu is huge in artificial intelligence.
So the next one is Tencent - we learned
-
about the gaming thing. And developed QQ -
this is a messenger than some of you might
-
remember: ICQ. Yeah, it's just a bit
bigger. Another app a lot of people don't
-
think it's Chinese, it is Chinese, it's
called WeChat. It's I can say they even
-
mmm WhatsApp, right? And guess what WeChat
developed - WeChat Pay. And there's
-
another little company Tencent holds some
parts in its SnapChat - it's twelve
-
percent owned by Tencent. Alibaba has
Alipay which is one of the biggest payment
-
apps. And you can even use it in Germany
now I think, in some shops. It's running
-
Taobao Wang, which is like eBay just a bit
bigger. And Sina Weibo. Weibo mains micro
-
blogging or twittering. And Sina is the
biggest company in China, and they hold
-
more or less a third. Now we come to
something really interesting - it's Ant
-
Financials. Because Ant Financials created
something called Sesame Credit. Sesame
-
Credit is Ali Baba's child, not Tencent,
and Ali Baba's. Tencent developed their
-
own system. So, updates and corrections
without colors. 2014 was the year the
-
Chinese state council released a planning
outline on the construction of social
-
credit systems in China. And it gave us a
lot of information but only very few
-
specific information. So, they started,
and in 2020 it will be managed mandatory.
-
But they worked on for as long as at least
2005.
-
It allows governmental and private SCS. So
you have both. And eight companies are
-
allowed to form their own private social
credit systems. We learned Tencent and
-
Alibaba are two of them. In different
regions, Hainan for example, it's an
-
island - you have social credit systems
run by the government. So you see you have
-
a lot of different systems running at the
same time in different regions and
-
sometimes you can choose between different
social credit systems at the same place.
-
And Sesame Credit we have learned is the
biggest and run by Alibaba. And Sesame
-
Credit is not the one who will be
mandatory. I have one other point that I
-
forgot it might come back later. So, to
give you a bit more information, let's
-
have a quick look at recent developments
that I thought of would be really
-
interesting for you when you have that
scenario of a society that is formed by a
-
writing system. In 2016 there were some
new guidelines on the SCS. New
-
cybersecurity laws and you might expect it
they are getting a bit harder and harder
-
every time. And there was a police robot -
I don't know if you heard about it I was
-
going to write an article but then we had
a really nice conference that made us a
-
lot of work and my article died in the
way. So you might have watched Wall-E -
-
what was the movie couple of years ago.
You might remember Eve - there was a
-
little cute white robot that had a
specialty - it could taser. Someone in
-
China must have watched that movie too,
because guess what - it looks like Eve and
-
it can tazer. You find pictures and
clips online. And the People's Bank of
-
China announced its own digital currency
to come soon. They are working on it for
-
years. And the real name registrations for
online comments. So no real name
-
registrations, no online comments. And in
2020 - the mandatory Social Credit System.
-
The next question is usually: how does the
social criticism work? And we take a quick
-
look at Sesame Credit because they added
more and more services that Extra Credit
-
guys could know about. So we learned a lot
of data mining they do.
-
For example, when you use AliPay it goes
to your scores. So your score gets higher
-
and higher the more you shop with AliPay.
Then other Alibaba services - we know
-
there are different and millions of
services. And now it's getting special
-
because it draws information from
officials as well. So we have information
-
from courts, debtors registries, and
because love always makes everything
-
better, you get your data from the biggest
dating app in China - it's Baihe. So you
-
can see if the person you want to fall in
love with have a good score. And of
-
course, they're including more and more
services. And there are penalties now. If
-
you forgot to pay your child support for a
couple of month or years or; I have heard
-
people can forget that for quite a long
time. There are penalties: for example,
-
you can't take high-speed trains or first-
class flight tickets. Which is a very
-
intelligent way. Again, it's a very
intelligent way to add penalties because
-
if you start with people nobody likes,
nobody likes openly, it's clever. And of
-
course, the companies do not tell you a
lot about the technologies like they use
-
in the systems. So and then people want to
know about the mandatory system, what it
-
can do, how will it look and so on, and
there's a very simple answer - we don't
-
know. The Chinese government doesn't give
you a lot of information on how it will
-
look. But we can guess. In having a look
on how the government worked in the last
-
20-30 years. So, we can have a look how
they prepare the mandatory system. Like I
-
said, different systems at the same time,
and they will learn from this. And my
-
guess is they will have like a little
medley of the best features of the
-
existing systems and get rid of the faulty
ones. So, by best, I mean best from the
-
point of view of the government of course.
So now we are bit later because of our
-
technological difficulties, thanks again
for solving. Why do you have a Social
-
Credit System?
Let's dive a bit back in history. Some
-
people might remember the year 1989.
Before you had a lot of countries that
-
called themselves socialists. But after,
you had only a few and China is one of
-
them. And how did they make it? In the 80s
they had something called the era of
-
reform and opening-up, where they opened
up to the West mostly, and to capitalism.
-
And you might know what that means.
China's called the transformation society.
-
And with some people getting rich quickly,
and a lot of people getting poorer and
-
poorer and watching the rich - you have a
lot of social tension. And remember, China
-
is a huge country. A vast amount on
people, and know people are really good in
-
getting on each other's nerves. So you
have problems. I give you one number that
-
I really like - in 2014, the official
count of mass incidents everyday was
-
nearly 250. And the number is rising, so
it will be much more today. And a mass
-
incident can be anything from pensioners
holding up banners demanding more money
-
because they worked their whole lives, I'm
sorry, I really have to concentrate on not
-
swearing, to like, riots, full-grown
riots, petrol bombs, burning buildings.
-
For example, G20 in Hamburg, you might
have heard. That's a mass incident. That's
-
nothing to get crazy about, that's just a
little thing that happened. Hamburg is
-
still standing. And, thank you, hello
Hamburg.
-
[Applause]
And keep in mind organized societies are
-
controlled on different levels. That's not
only China. And social control is usually
-
done by groups, so it's much easier if we
control each other. And it has the long
-
history in China too, and a lot of
socialist countries develop new forms of
-
social control. So in China, you have a
Hukou - that means you can only get social
-
services where you're permitted to live.
I just give you a really really short
-
version of everything it's not it's a bit
more complex as you know. The Tang on
-
which you can say it's an analog version a
lot of your stuff what you've done goes in
-
there. You needed a marriage permit if you
wanted to get married and it wasn't easy
-
to live without a married permit and as
China became more and more digitized you
-
might know what comes next. Oh too anyone
control became digitized as well and as
-
people are using ICTs more and more they
create a huge amount of data. And the
-
bloody thing with data is you don't feel,
smell or see anything vanishing, do you?
-
And people it's helpful as humans to
-
develop something towards something we
can't see smell or fear. And more and more
-
technologies are used in civilians and
social control. But that still leaves the
-
question: why is China doing it? Let's
go to another point I'd really interested
-
in: ideological influences the way we see
the world and the way we think about it
-
is formed by theories, by ideologies and
what's the Chinese flag ? You see it
-
again, maybe you remember, some of you,
it's red with yellow stars - that usually
-
is a code for socialists and this is Mao
Zedong. He was kind of big in China. And
-
so you have - here you have three
socialist symbols. It really helps to
-
know a bit about Marxism in researching
China and especially in Mao Zedong
-
thought it's not called maoism it's
called more Zedong thought it's about what
-
he thought and in this reform of opening
up - I'm trying to feed you a bit of
-
Chinese history - the Chinese government
opened up to more. For example they opened
-
up to nationalism. And if you know a bit
about socialism, usually you wouldn't
-
put socialism and nationalism in one room,
because they didn't like each other. The
-
Chinese government found a way they like
each other. They got really interested
-
in PR and they really research other
countries how governments there stood in
-
power. And some are not so well known to
most of us this is Confucius.
-
Confucianism shaped and molded the
Chinese societies for more than 2,000
-
years. And if you have that big
influence, you can't just get rid of it.
-
It's in people's memories. It's in the
whole society, and Confusianism came back
-
as an ideology, where some parts of the
Socialists ideology vanished. Confucianism
-
came back and Confucianism is about a
society where everyone is on her or his
-
place and her place is always below. And
with all these new ideological
-
influences, the way of communication
changed as well.
-
Before it was the masses and now it's the
public. All campaigns and programs had
-
revolutionary names or poetic forms like
hundred flowers yet the Great Leap
-
Forward, and all these campaigns molded
our picture of China, or the picture
-
most people have in their hearts people in
bloom, red flags. But now it's changed you
-
have new campaigns and programs and they
have other names as well. You have
-
socialist harmonious society as a goal for
the society and a couple of years ago
-
before, you couldn't put socialism and
harmony in one room either. And you have
-
the Chinese dream. Yes, it sounds a bit
like American dream, yeah. And we learned
-
that all governments are interested in
control and feedback because you can't
-
control something, if you don't pay
attention and can't blow up anyway. So
-
like we have an autonomous nervous system
that tells every part what's going on, you
-
have one electronic autonomous nervous
system now in many societies not only
-
China.
People using ICTs and a social credit
-
system just creates a wonderful
opportunity of feedback and on the same
-
time on influence. So, we come to the end
of this talk and to the summary. So, again
-
this I think is really important: it's an
international tendency to solve social
-
problems with technological solutions. And
as far as I am concerned this is not the
-
best way maybe. And the Chinese government
- they do not count and think in years,
-
they think in decades at least. And they
looked at the internet and found out
-
that this is a good tool and they're using
it. Successfully. But we know at least
-
since Edward Snowden, that other
governments are capable of using the
-
Internet as a tool as well. A social
credit system exists of the use of big
-
data, algorithm, ICTs and gamification.
And it will become mandatory. And this
-
game, when it becomes mandatory, will
define your life. And not only your life,
-
it will define the life of the people
surrounding you as well. The schools your
-
children for example will go to will be
chosen on your rate. And there's the
-
opportunity to put this together with a
national cryptocurrency. So, thank you
-
all for listening that what's my current
state of research and my current state of
-
my code, we both presented to you this
talk and I'm really looking forward to QA
-
and discussing. Thanks.
34c3 outro
-
subtitles created by c3subtitles.de
in the year 2018. Join, and help us!