34c3 intro Herald: Okay, so ladies and gentlemen Katika Kühnreich. So did you all thank the angels today? No, yes, maybe? [Applause] Herald: Did you all took your prosĂm two today? No? Okay, see, Katika will tell you about the social credit system from China, and will tell you a little bit more about how a human being can be valued. She is the magista in political science and modern and ancient Chinese science. She not only has her theoretical knowledge from books, but she always also combines it with the street study. So please welcome, with a very warm hand of applause, Katika! [Applause] Katika: Hey guys, I'm happy to be here, and first I want to thank all the people who made that possible. This conference and this talk; so thanks to all you, and a special thanks to all the people translating. [Applause] Katika: Before we start, I like to say a few points. The first was the thank you, we already have. The second is I have to apologize: English is not my mother tongue so you will have some funny mistakes that I make, so the Internet has a bit more fun. Because the world can be (nice). The second thing, is I will take you on a journey through China's social credit system - its current state. And because we only have half an hour, more or less, because I'd really like to spend a lot of time with you guys on talks, on discussion and QA, we have to leave a lot of interesting sites out. So there were some excellent talks on the CCC about technological aspects of Internet and Internet usage in China. I will concentrate really hard on this social credit system. But if you have questions, please ask in the Q&A. The Q&A will not be recorded, so you can feel free. And if you have more questions, just contact me after the talk. And, it's a China topic and often when you talk about China, people say "oh yeah, oh yeah: that's China." This is no China bashing session - I'm sorry for anyone who expected that. China has a government, and a government wants to control something. So this talk is about social credit systems in China but it is about a tendency I see internationally. It's the tendency to answer social problems with technological solutions, or at least it's the try. So please keep in mind: it's happening in China, but it's not that we are safe in the West or cozy or anything else. So it's meant as an example of an international tendency. And I prepared some cards, so I will stick to the topic. I have to look at them from time to time. This talk presents my current state of research; it's not the truth. And this talked sadly represents my current state of my cold as well, so I'm not totally fed please apologized that as well. For a little journey I prepared a couple of stops. The first one will be a video from 2015 made by Extra Credit on social credit systems in China. So if you don't like the talk, you have seven and a half minutes to just vanish - nobody will notice. The next point will be "Updates and Corrections" - it's a video from 2015, and there's some faulty information in it. And to do that, I will give you a quick look at the Chinese Internet landscape. The next point will be the current state of social credit systems. If I say SCS: it's the same - I'm just lazy. And there's one question often not answered when talking about the Chinese credit system - it's why is it happening. And again it's not because it's an evil government. It's a government. So we'll dive in some ideological aspects of Chinese politics today. And in the end, I'm really looking forward to that point because I can shut a bit up and listen to your thoughts and we're really interested in that. So, everyone ready? Cozy? That's wonderful, thanks! Then we can start with the video; that isn't it shown right now. Now we need some technical support. It's much more fun when you see the pictures. But we have time, so. Herald: Do we have the (tv guys) here? Katika: We got free water! Herald: Gentlemen, I think we have a technical problem here, so while the VOC and my stage manager is trying to get some people from there. Yes he will do this - he's a good guy. He's the best guy! Gentlemen, ladies, applause for the VOC! [Applause] Herald: So while we're waiting to have the video; and trust me it looked really good from here. Let's play a little game - later we will have a Q&A session but unfortunately we are not allowed or we we can't record the Q&A session, and we will also not video stream the Q&A session. So I would like you to have a quick hand of applause for all of you, because you will have a great discussion with this nice lady and therefore we try to make applause - we test the applause. Katika: Okay [Applause] Herald: No no no, that should be louder - come on! [Applause] Herald: Okay we still have no video, but I think over there I did not hear any applause. Could you, could you please try it again? [Applause] Herald: Yeah, better, better. I think these guys are way better. Are they? Come on! [Applause] Katika: Otherwise I can use the time to give you some information I'd present you after the video. Thank you. The information you get in the video is based on a press release of the American Civil Liberties Union, and that contains some mistakes. One thing - when you want to know something about China, the government is really open in publishing what they're doing, and what they're planning to do. And it's seems to be much more comfortable to just copy, but maybe have a look into what the government says because it's a good way to find out what they are doing and they're scaring the open. It works with a lot of governments. And I realized that because this video had a lot of impact, and there was a lot of news report on that scary system in China but when you looked at the news report and you had the video before, you found out that it was more or less the same, the very same, as in the video even years later. So please if you want to find something out - check for facts. They might be online too. So that's the picture, and that's my break. Video: Okay, so one you recently brought this to our attention, and when we first started looking into it, we couldn't believe it. Even now looking at this I keep thinking that there's got to be something about this that I just don't understand, like, I must be reading this wrong. It's just so clearly something out of dystopian science fiction. And yet, here it is in our world, and people are actually embracing it. Apparently China has gamified being an obedient citizen. [Music] Video: Going under the innocuous name of Sesame Credit, China has created a score for how good a citizen you are, and that's one of the scariest things I've heard in quite a while. It's jointly run by Tencent, yes that Tencent - the one that owns Riot Games and has a significant share in Epic and Activision-Blizzard, and also the ascendant Chinese competitor to Amazon, Alibaba, hence the name Sesame. So the owners of China's largest social networks have partnered with the governments to create something akin to the US credit score. But instead of measuring how regularly you pay your bills, it measures how obediently you follow the party line. They dredge data from your social network, so if you post pictures of Tiananmen Square, or share a link about the recent stock market collapse, your Sesame Credit score goes down. Share a link from the state- sponsored news agency about how good the economy is doing and your score goes up. The Alibaba and Tencent are also the largest online retailers in China so Sesame Credit is also able to pull data from your purchases. If you're making purchases the state deems valuable like work shoes or local agricultural products, your score goes up. If you import anime from Japan, though, down the score goes. And this score has real-world consequences. Like many games, Sesame Credit has tiers and levels, and having a high score gives you special benefits - like making it easier to get the paperwork you need to travel. Or making it easier to get a loan. Now currently, there are no consequences for having a low score but there's been talk about implementing penalties once the system becomes mandatory in 2020. Penalties like slower Internet speeds for low scoring citizens. Or even restricting the jobs that a low scoring person's allowed to hold. But there's one more layer to Sesame Credit, and here's where this goes from being repulsive to downright insidious. Because this is all part of a social network it also scans your friends - so you will lose points for having friends with low obedience scores. And it tells you this. At any point, anybody can check anyone elses score. And when you check your own score, Sesame Credit provides a handy map of your friends to show you who is dragging your score down. Have you ever had that thing where you play a game with somebody who wasn't doing very well and you've tried to change their behavior to make them do better? Or maybe after a while you just sort of stopped playing with the people who are holding you back. That's at the heart of how this system works, and it's also what makes this one of the most terrifying tools of authoritarian oppression I've ever read about. Because like mass censorship, jail time, assassinations - those are all big messy implements for keeping a population in line. That messiness and severity foster resentment - eventually rebellion. They're expensive, they're unwieldy - in the end, those tools are impossible to maintain. But social pressure? Ostracization? Those things are free - they happen on their own. And as a government tool they don't have nearly the same potential for going embarrassingly disastrously wrong. With a system like this in place, a government doesn't even have to tell neighbor to spy on neighbor; to rat each other out. Because that's all built into a seemingly innocuous game system. The government need not step in. Re-education will be handled for them by friends, classmates, and relatives who want to maintain a high score. And if that doesn't work then potentially dangerous ideas still end up quarantined by the social isolation this game system causes. Express or help to spread to many radical ideas, and people will stop associating with you. And not because some jackbooted thugs showed up at the door with threats, but simply because associating with somebody with those ideas will lose them all the privileges they've worked so hard to obtain. It recontextualizes obedience to an authoritarian regime. In the past, you obeyed such powers because you were afraid. Fear kept you motivated. But fear is negative. It fosters resentment. The world we're stepping into instead uses positive reinforcement to promote being subservient to the will of the regime. It's Big Brother's kinder, gentler hand. And the things that make this scary is that we've seen the efficacy of this only too well in games. You may not actually know this, but when World of Warcraft was in its early stages of development, it had an unrested penalty mechanic that started limiting experienced gains for players who had played too much. And players hated it - they resented it they complained about it every day. So after brainstorming on this for a while, Blizzard had the idea to simply change how the mechanic was presented. Without changing any of the existing numbers or systems, they started referring to the unrested experience state as normal, and made it the default. And they started calling the original normal experience gained state as rested. That's all they changed, and everybody loved it. People would log on every day just to get that bonus. Positive reinforcement works wonders. But unlike World of Warcraft, which built a system just to get people to embrace meaningless grinding, Sesame Credit has built a system to get people to enjoy falling into line. Now the system's not mandatory yet - it's opt-in right now. But it's going to be mandatory in 2020, and there's a terrible brilliance to phasing that in. The early adopters are going to be people excited about this system. People who are already patriotic and are eager for anything that'll help display that patriotism to the world. And, as early adopters they're gonna talk it up. They're gonna give it an air of being positive and fun. Then it'll be foisted on the society as a whole. More than that though, the early adopters are gonna compete. Already you can see hundreds of thousands of tweets displaying people's high scores or showing off the new milestones they've hit. Giving a hard numbers to their patriotism and giving them bragging rights for being the most patriotic, most right-thinking person they know. And that's gonna set the tone for how Sesame Credit is intended to be used - as a competition to see who can agree with the government the most. We've talked about propaganda games on this channel before but for all the time we've spent examining and deconstructing terrible games that espouse hate and for all the studies on propaganda games James has done, this is the use of game systems that frightens me the most. Because to most people Sesame Credit will seem benign. Perhaps even fun - it's a conversation starter, something to share with your friends. But it's making heavy use of all the psychological motivators that we Gamemakers deploy in scoring systems, and ladders and levels. Systems that we built to shape play habits and to keep people coming back. Like I said, I'm still kind of in a state of disbelief looking at this. If any of you are watching this from China, please tell me if I'm misunderstanding this thing, because I would love to be wrong about it. If not, well, I hope this episode can do some small amount to help the fight to keep such a system from becoming mandatory. For everyone in the rest of the world, I hope this helps remind us all how important it is to be aware and to be vigilant. All of these gamification techniques we've learned through making games offer incredible opportunities for making this world a better more engaging place. But every great tool carries with it the potential for misuse. And it's on us as a community who understands this amazing new medium, to do what we can to stop that. We'll see you next week. [Applause] Katika: So, we had the shedule, and thank you to Extra Credit for this video, and thank you for concentrating on gamification. Because a lot of time, when I talk to social scientists, without a lot of technological knowledge, it was like okay - it's a new system. And the whole gamification thing triggered me, and I wondered about a lot of people who didn't seem too interested in it. Because I said that the video contains some faulty information. First, I'd like to concentrate on key aspects they told us in with this video. There is a social rating system in China now. The system will become mandatory in 2020. That's all right. It works on Big Data algorithms, artificial intelligence, and your online and offline data. And it uses technologies like gamification and nudging. And if you hear it's an Orwellian system - no it's not. 1984 was about force. Gamification is about feeling good and cozy. So, in the updates and corrections, I promise you a quick look at the end Chinese internet landscape. What most people already know. From the beginning the Chinese government started to control the internet as much as it could and its users. So some vocabularies you might have heard are the Great Firewall, or its new extra the Golden Shield, the Great Cannon, which is a DDoS weapon. It would be really fascinating to dive in there, but we have to watch from far. Another point that's really interesting for us looking Social Credit System is true name or real name registration. Which means you can't use a lot of services if you don't give them your ID number. And the mandatory system in 2020 will work on ID numbers as far as we know. So, we watched the other side: the users, their millions of users - it's one more than 1.4 billion people in China. And ICTs are really popular. Much more than in Germany, for example. And another thing that please keep in mind - cash is vanishing in China. A couple of years ago, for example, when I lived in China 2004: I paid my rent in cash half a year in advance. When I paid for anything I paid cash. You could couldn't find a lot of places accepting cards. But now China didn't have this development of cards - they just jumped right through paying with mobile apps. And you can buy noodles or snack or magazine with your mobile phone. And the Chinese users found multiple ways to deal with censorship, and to trespass it. It's a wonderful thing to look up. And now, the big players of the Chinese Internet, because we learned some names in the film. I made something very Chinese - I made something colorful. Okay, you have three big players: its Alibaba and Tencent - we heard about them, and Baidu is the third biggest player. Each millions of users, or billions. So to explain this, I will use some American companies names to help understanding. And Baidu is known for that; was the wrong; yeah, Baidu is known for Baidu Barker - this is like the Chinese Wikipedia, it's just a bit much controlled. And a have Baidu Pay, and Baidu is huge in artificial intelligence. So the next one is Tencent - we learned about the gaming thing. And developed QQ - this is a messenger than some of you might remember: ICQ. Yeah, it's just a bit bigger. Another app a lot of people don't think it's Chinese, it is Chinese, it's called WeChat. It's I can say they even mmm WhatsApp, right? And guess what WeChat developed - WeChat Pay. And there's another little company Tencent holds some parts in its SnapChat - it's twelve percent owned by Tencent. Alibaba has Alipay which is one of the biggest payment apps. And you can even use it in Germany now I think, in some shops. It's running Taobao Wang, which is like eBay just a bit bigger. And Sina Weibo. Weibo mains micro blogging or twittering. And Sina is the biggest company in China, and they hold more or less a third. Now we come to something really interesting - it's Ant Financials. Because Ant Financials created something called Sesame Credit. Sesame Credit is Ali Baba's child, not Tencent, and Ali Baba's. Tencent developed their own system. So, updates and corrections without colors. 2014 was the year the Chinese state council released a planning outline on the construction of social credit systems in China. And it gave us a lot of information but only very few specific information. So, they started, and in 2020 it will be managed mandatory. But they worked on for as long as at least 2005. It allows governmental and private SCS. So you have both. And eight companies are allowed to form their own private social credit systems. We learned Tencent and Alibaba are two of them. In different regions, Hainan for example, it's an island - you have social credit systems run by the government. So you see you have a lot of different systems running at the same time in different regions and sometimes you can choose between different social credit systems at the same place. And Sesame Credit we have learned is the biggest and run by Alibaba. And Sesame Credit is not the one who will be mandatory. I have one other point that I forgot it might come back later. So, to give you a bit more information, let's have a quick look at recent developments that I thought of would be really interesting for you when you have that scenario of a society that is formed by a writing system. In 2016 there were some new guidelines on the SCS. New cybersecurity laws and you might expect it they are getting a bit harder and harder every time. And there was a police robot - I don't know if you heard about it I was going to write an article but then we had a really nice conference that made us a lot of work and my article died in the way. So you might have watched Wall-E - what was the movie couple of years ago. You might remember Eve - there was a little cute white robot that had a specialty - it could taser. Someone in China must have watched that movie too, because guess what - it looks like Eve and it can tazer. You find pictures and clips online. And the People's Bank of China announced its own digital currency to come soon. They are working on it for years. And the real name registrations for online comments. So no real name registrations, no online comments. And in 2020 - the mandatory Social Credit System. The next question is usually: how does the social criticism work? And we take a quick look at Sesame Credit because they added more and more services that Extra Credit guys could know about. So we learned a lot of data mining they do. For example, when you use AliPay it goes to your scores. So your score gets higher and higher the more you shop with AliPay. Then other Alibaba services - we know there are different and millions of services. And now it's getting special because it draws information from officials as well. So we have information from courts, debtors registries, and because love always makes everything better, you get your data from the biggest dating app in China - it's Baihe. So you can see if the person you want to fall in love with have a good score. And of course, they're including more and more services. And there are penalties now. If you forgot to pay your child support for a couple of month or years or; I have heard people can forget that for quite a long time. There are penalties: for example, you can't take high-speed trains or first- class flight tickets. Which is a very intelligent way. Again, it's a very intelligent way to add penalties because if you start with people nobody likes, nobody likes openly, it's clever. And of course, the companies do not tell you a lot about the technologies like they use in the systems. So and then people want to know about the mandatory system, what it can do, how will it look and so on, and there's a very simple answer - we don't know. The Chinese government doesn't give you a lot of information on how it will look. But we can guess. In having a look on how the government worked in the last 20-30 years. So, we can have a look how they prepare the mandatory system. Like I said, different systems at the same time, and they will learn from this. And my guess is they will have like a little medley of the best features of the existing systems and get rid of the faulty ones. So, by best, I mean best from the point of view of the government of course. So now we are bit later because of our technological difficulties, thanks again for solving. Why do you have a Social Credit System? Let's dive a bit back in history. Some people might remember the year 1989. Before you had a lot of countries that called themselves socialists. But after, you had only a few and China is one of them. And how did they make it? In the 80s they had something called the era of reform and opening-up, where they opened up to the West mostly, and to capitalism. And you might know what that means. China's called the transformation society. And with some people getting rich quickly, and a lot of people getting poorer and poorer and watching the rich - you have a lot of social tension. And remember, China is a huge country. A vast amount on people, and know people are really good in getting on each other's nerves. So you have problems. I give you one number that I really like - in 2014, the official count of mass incidents everyday was nearly 250. And the number is rising, so it will be much more today. And a mass incident can be anything from pensioners holding up banners demanding more money because they worked their whole lives, I'm sorry, I really have to concentrate on not swearing, to like, riots, full-grown riots, petrol bombs, burning buildings. For example, G20 in Hamburg, you might have heard. That's a mass incident. That's nothing to get crazy about, that's just a little thing that happened. Hamburg is still standing. And, thank you, hello Hamburg. [Applause] And keep in mind organized societies are controlled on different levels. That's not only China. And social control is usually done by groups, so it's much easier if we control each other. And it has the long history in China too, and a lot of socialist countries develop new forms of social control. So in China, you have a Hukou - that means you can only get social services where you're permitted to live. I just give you a really really short version of everything it's not it's a bit more complex as you know. The Tang on which you can say it's an analog version a lot of your stuff what you've done goes in there. You needed a marriage permit if you wanted to get married and it wasn't easy to live without a married permit and as China became more and more digitized you might know what comes next. Oh too anyone control became digitized as well and as people are using ICTs more and more they create a huge amount of data. And the bloody thing with data is you don't feel, smell or see anything vanishing, do you? And people it's helpful as humans to develop something towards something we can't see smell or fear. And more and more technologies are used in civilians and social control. But that still leaves the question: why is China doing it? Let's go to another point I'd really interested in: ideological influences the way we see the world and the way we think about it is formed by theories, by ideologies and what's the Chinese flag ? You see it again, maybe you remember, some of you, it's red with yellow stars - that usually is a code for socialists and this is Mao Zedong. He was kind of big in China. And so you have - here you have three socialist symbols. It really helps to know a bit about Marxism in researching China and especially in Mao Zedong thought it's not called maoism it's called more Zedong thought it's about what he thought and in this reform of opening up - I'm trying to feed you a bit of Chinese history - the Chinese government opened up to more. For example they opened up to nationalism. And if you know a bit about socialism, usually you wouldn't put socialism and nationalism in one room, because they didn't like each other. The Chinese government found a way they like each other. They got really interested in PR and they really research other countries how governments there stood in power. And some are not so well known to most of us this is Confucius. Confucianism shaped and molded the Chinese societies for more than 2,000 years. And if you have that big influence, you can't just get rid of it. It's in people's memories. It's in the whole society, and Confusianism came back as an ideology, where some parts of the Socialists ideology vanished. Confucianism came back and Confucianism is about a society where everyone is on her or his place and her place is always below. And with all these new ideological influences, the way of communication changed as well. Before it was the masses and now it's the public. All campaigns and programs had revolutionary names or poetic forms like hundred flowers yet the Great Leap Forward, and all these campaigns molded our picture of China, or the picture most people have in their hearts people in bloom, red flags. But now it's changed you have new campaigns and programs and they have other names as well. You have socialist harmonious society as a goal for the society and a couple of years ago before, you couldn't put socialism and harmony in one room either. And you have the Chinese dream. Yes, it sounds a bit like American dream, yeah. And we learned that all governments are interested in control and feedback because you can't control something, if you don't pay attention and can't blow up anyway. So like we have an autonomous nervous system that tells every part what's going on, you have one electronic autonomous nervous system now in many societies not only China. People using ICTs and a social credit system just creates a wonderful opportunity of feedback and on the same time on influence. So, we come to the end of this talk and to the summary. So, again this I think is really important: it's an international tendency to solve social problems with technological solutions. And as far as I am concerned this is not the best way maybe. And the Chinese government - they do not count and think in years, they think in decades at least. And they looked at the internet and found out that this is a good tool and they're using it. Successfully. But we know at least since Edward Snowden, that other governments are capable of using the Internet as a tool as well. A social credit system exists of the use of big data, algorithm, ICTs and gamification. And it will become mandatory. And this game, when it becomes mandatory, will define your life. And not only your life, it will define the life of the people surrounding you as well. The schools your children for example will go to will be chosen on your rate. And there's the opportunity to put this together with a national cryptocurrency. So, thank you all for listening that what's my current state of research and my current state of my code, we both presented to you this talk and I'm really looking forward to QA and discussing. Thanks. 34c3 outro subtitles created by c3subtitles.de in the year 2018. Join, and help us!