34C3 preroll music Herald: Humans of Congress, it is my pleasure to announce the next speaker. I was supposed to pick out a few awards or something, to actually present what he's done in his life, but I can only say: he's one of us! applause Charles Stross! ongoing applause Charles Stross: Hi! Is this on? Good. Great. I'm really pleased to be here and I want to start by apologizing for my total lack of German. So this talk is gonna be in English. Good morning. I'm Charlie Stross and it's my job to tell lies for money, or rather, I write science fiction, much of it about on the future, which in recent years has become ridiculously hard to predict. In this talk I'm going to talk about why. Now our species, Homo sapiens sapiens, is about 300,000 years old. It used to be about 200,000 years old, but it grew an extra 100,000 years in the past year because of new archaeological discoveries, I mean, go figure. For all but the last three centuries or so - of that span, however - predicting the future was really easy. If you were an average person - as opposed to maybe a king or a pope - natural disasters aside, everyday life 50 years in the future would resemble everyday life 50 years in your past. Let that sink in for a bit. For 99.9% of human existence on this earth, the future was static. Then something changed and the future began to shift increasingly rapidly, until, in the present day, things are moving so fast, it's barely possible to anticipate trends from one month to the next. Now as an eminent computer scientist, Edsger Dijkstra once remarked, computer science is no more about computers than astronomy is about building big telescopes, the same can be said of my field of work, writing science fiction, sci-fi is rarely about science and even more rarely about predicting the future, but sometimes we dabble in Futurism and lately, Futurism has gotten really, really, weird. Now when I write a near future work of fiction, one set, say, a decade hence, there used to be a recipe I could follow, that worked eerily well. Simply put: 90% of the next decade stuff is already here around us today. Buildings are designed to last many years, automobiles have a design life of about a decade, so half the cars on the road in 2027 are already there now - they're new. People? There'll be some new faces, aged 10 and under, and some older people will have died, but most of us adults will still be around, albeit older and grayer, this is the 90% of a near future that's already here today. After the already existing 90%, another 9% of a near future a decade hence used to be easily predictable: you look at trends dictated by physical limits, such as Moore's law and you look at Intel's road map and you use a bit of creative extrapolation and you won't go too far wrong. If I predict - wearing my futurology hat - that in 2027 LTE cellular phones will be ubiquitous, 5G will be available for high bandwidth applications and there will be fallback to some kind of satellite data service at a price, you probably won't laugh at me. I mean, it's not like I'm predicting that airlines will fly slower and Nazis will take over the United States, is it ? laughing And therein lies the problem. There is remaining 1% of what Donald Rumsfeld called the "unknown unknowns", what throws off all predictions. As it happens, airliners today are slower than they were in the 1970s and don't get me started about the Nazis, I mean, nobody in 2007 was expecting a Nazi revival in 2017, were they? Only this time, Germans get to be the good guys. laughing, applause So. My recipe for fiction set 10 years in the future used to be: "90% is already here, 9% is not here yet but predictable and 1% is 'who ordered that?'" But unfortunately the ratios have changed, I think we're now down to maybe 80% already here - climate change takes a huge toll on architecture - then 15% not here yet, but predictable and a whopping 5% of utterly unpredictable deep craziness. Now... before I carry on with this talk, I want to spend a minute or two ranting loudly and ruling out the singularity. Some of you might assume, that as the author of books like "Singularity Sky" and "Accelerando", I expect an impending technological singularity, that we will develop self-improving artificial intelligence and mind uploading and the whole wish list of transhumanist aspirations promoted by the likes of Ray Kurzweil, will come to pass. Unfortunately this isn't the case. I think transhumanism is a warmed-over Christian heresy. While its adherents tend to be outspoken atheists, they can't quite escape from the history that gave rise to our current Western civilization. Many of you are familiar with design patterns, an approach to software engineering that focuses on abstraction and simplification, in order to promote reusable code. When you look at the AI singularity as a narrative and identify the numerous places in their story where the phrase "and then a miracle happens" occur, it becomes apparent pretty quickly, that they've reinvented Christiantiy. applause Indeed, the wellspring of today's transhumanists draw in a long rich history of Russian philosophy, exemplified by the russian orthodox theologian Nikolai Fyodorovich Fedorov by way of his disciple Konstantin Tsiolkovsky, whose derivation of a rocket equation makes him essentially the father of modern space flight. Once you start probing the nether regions of transhumanist forth and run into concepts like Roko's Basilisk - by the way, any of you who didn't know about the Basilisk before, are now doomed to an eternity in AI hell, terribly sorry - you realize, they've mangled it to match some of the nastier aspects of Presbyterian Protestantism. Now they basically invented original sin and Satan in the guise of an AI that doesn't exist yet ,it's.. kind of peculiar. Anyway, my take on the singularity is: What if something walks like a duck and quacks like a duck? It's probably a duck. And if it looks like a religion, it's probably a religion. I don't see much evidence for human-like, self-directed artificial intelligences coming along any time soon, and a fair bit of evidence, that nobody accepts and freaks in cognitive science departments, even want it. I mean, if we invented an AI that was like a human mind, it would do the AI equivalent of sitting on the sofa, munching popcorn and watching the Super Bowl all day. It wouldn't be much use to us. laughter, applause What we're getting instead, is self-optimizing tools that defy human comprehension, but are not in fact any more like our kind of intelligence than a Boeing 737 is like a seagull. Boeing 737s and seagulls both fly, Boeing 737s don't lay eggs and shit everywhere. So I'm going to wash my hands of a singularity as a useful explanatory model of the future without further ado. I'm one of those vehement atheists as well and I'm gonna try and offer you a better model for what's happening to us. Now, as my fellow Scottish science fictional author Ken MacLeod likes to say "the secret weapon of science fiction is history". History is, loosely speaking, is the written record of what and how people did things in past times. Times that have slipped out of our personal memories. We science fiction writers tend to treat history as a giant toy chest to raid, whenever we feel like telling a story. With a little bit of history, it's really easy to whip up an entertaining yarn about a galactic empire that mirrors the development and decline of a Habsburg Empire or to respin the October Revolution as a tale of how Mars got its independence. But history is useful for so much more than that. It turns out, that our personal memories don't span very much time at all. I'm 53 and I barely remember the 1960s. I only remember the 1970s with the eyes of a 6 to 16 year old. My father died this year, aged 93, and he'd just about remembered the 1930s. Only those of my father's generation directly remember the Great Depression and can compare it to the 2007/08 global financial crisis directly. We Westerners tend to pay little attention to cautionary tales told by 90-somethings. We're modern, we're change obsessed and we tend to repeat our biggest social mistakes just as they slip out of living memory, which means they recur on a timescale of 70 to 100 years. So if our personal memories are useless, we need a better toolkit and history provides that toolkit. History gives us the perspective to see what went wrong in the past and to look for patterns and check to see whether those patterns are recurring in the present. Looking in particular at the history of the past two to four hundred years, that age of rapidly increasing change that I mentioned at the beginning. One glaringly obvious deviation from the norm of the preceding 3000 centuries is obvious, and that's the development of artificial intelligence, which happened no earlier than 1553 and no later than 1844. I'm talking of course about the very old, very slow AI's we call corporations. What lessons from the history of a company can we draw that tell us about the likely behavior of the type of artificial intelligence we're interested in here, today? Well. Need a mouthful of water. Let me crib from Wikipedia for a moment. Wikipedia: "In the late 18th century, Stewart Kyd, the author of the first treatise on corporate law in English, defined a corporation as: 'a collection of many individuals united into one body, under a special denomination, having perpetual succession under an artificial form, and vested, by policy of the law, with the capacity of acting, in several respects, as an individual, enjoying privileges and immunities in common, and of exercising a variety of political rights, more or less extensive, according to the design of its institution, or the powers conferred upon it, either at the time of its creation, or at any subsequent period of its existence.'" This was a late 18th century definition, sound like a piece of software to you? In 1844, the British government passed the "Joint Stock Companies Act" which created a register of companies and allowed any legal person, for a fee, to register a company which in turn existed as a separate legal person. Prior to that point, it required a Royal Charter or an act of Parliament to create a company. Subsequently, the law was extended to limit the liability of individual shareholders in event of business failure and then both Germany and the United States added their own unique twists to what today we see is the doctrine of corporate personhood. Now, though plenty of other things that happened between the 16th and 21st centuries did change the shape of the world we live in. I've skipped the changes in agricultural productivity that happened due to energy economics, which finally broke the Malthusian trap our predecessors lived in. This in turn broke the long-term cap on economic growth of about 0.1% per year in the absence of famines, plagues and wars and so on. I've skipped the germ theory of diseases and the development of trade empires in the age of sail and gunpowder, that were made possible by advances in accurate time measurement. I've skipped the rise, and hopefully decline, of the pernicious theory of scientific racism that underpinned Western colonialism and the slave trade. I've skipped the rise of feminism, the ideological position that women are human beings rather than property and the decline of patriarchy. I've skipped the whole of the Enlightenment and the Age of Revolutions, but this is a technocratic.. technocentric Congress, so I want to frame this talk in terms of AI, which we all like to think we understand. Here's the thing about these artificial persons we call corporations. Legally, they're people. They have goals, they operate in pursuit of these goals, they have a natural life cycle. In the 1950s, a typical U.S. corporation on the S&P 500 Index had a life span of 60 years. Today it's down to less than 20 years. This is largely due to predation. Corporations are cannibals, they eat one another. They're also hive super organisms like bees or ants. For the first century and a half, they relied entirely on human employees for their internal operation, but today they're automating their business processes very rapidly. Each human is only retained so long as they can perform their assigned tasks more efficiently than a piece of software and they can all be replaced by another human, much as the cells in our own bodies are functionally interchangeable and a group of cells can - in extremis - often be replaced by a prosthetic device. To some extent, corporations can be trained to serve of the personal desires of their chief executives, but even CEOs can be dispensed with, if their activities damage the corporation, as Harvey Weinstein found out a couple of months ago. Finally, our legal environment today has been tailored for the convenience of corporate persons, rather than human persons, to the point where our governments now mimic corporations in many of our internal structures. So, to understand where we're going, we need to start by asking "What do our current actually existing AI overlords want?" Now, Elon Musk, who I believe you've all heard of, has an obsessive fear of one particular hazard of artificial intelligence, which he conceives of as being a piece of software that functions like a brain in a box, namely the Paperclip Optimizer or Maximizer. A Paperclip Maximizer is a term of art for a goal seeking AI that has a single priority, e.g., maximizing the number of paperclips in the universe. The Paperclip Maximizer is able to improve itself in pursuit of its goal, but has no ability to vary its goal, so will ultimately attempt to convert all the metallic elements in the solar system into paperclips, even if this is obviously detrimental to the well-being of the humans who set it this goal. Unfortunately I don't think Musk is paying enough attention, consider his own companies. Tesla isn't a Paperclip Maximizer, it's a battery Maximizer. After all, a battery.. an electric car is a battery with wheels and seats. SpaceX is an orbital payload Maximizer, driving down the cost of space launches in order to encourage more sales for the service it provides. SolarCity is a photovoltaic panel maximizer and so on. All three of the.. Musk's very own slow AIs are based on an architecture, designed to maximize return on shareholder investment, even if by doing so they cook the planet the shareholders have to live on or turn the entire thing into solar panels. But hey, if you're Elon Musk, thats okay, you're gonna retire on Mars anyway. laughing By the way, I'm ragging on Musk in this talks, simply because he's the current opinionated tech billionaire, who thinks for disrupting a couple of industries entitles him to make headlines. If this was 2007 and my focus slightly difference.. different, I'd be ragging on Steve Jobs and if we're in 1997 my target would be Bill Gates. Don't take it personally, Elon. laughing Back to topic. The problem of corporations is, that despite their overt goals, whether they make electric vehicles or beer or sell life insurance policies, they all have a common implicit Paperclip Maximizer goal: to generate revenue. If they don't make money, they're eaten by a bigger predator or they go bust. It's as vital to them as breathing is to us mammals. They generally pursue their implicit goal - maximizing revenue - by pursuing their overt goal. But sometimes they try instead to manipulate their environment, to ensure that money flows to them regardless. Human toolmaking culture has become very complicated over time. New technologies always come with an attached implicit political agenda that seeks to extend the use of the technology. Governments react to this by legislating to control new technologies and sometimes we end up with industries actually indulging in legal duels through the regulatory mechanism of law to determine, who prevails. For example, consider the automobile. You can't have mass automobile transport without gas stations and fuel distribution pipelines. These in turn require access to whoever owns the land the oil is extracted from under and before you know it, you end up with a permanent army in Iraq and a clamp dictatorship in Saudi Arabia. Closer to home, automobiles imply jaywalking laws and drink-driving laws. They affect Town Planning regulations and encourage suburban sprawl, the construction of human infrastructure on a scale required by automobiles, not pedestrians. This in turn is bad for competing transport technologies, like buses or trams, which work best in cities with a high population density. So to get laws that favour the automobile in place, providing an environment conducive to doing business, automobile companies spend money on political lobbyists and when they can get away with it, on bribes. Bribery needn't be blatant of course. E.g., the reforms of a British railway network in the 1960s dismembered many branch lines and coincided with a surge in road building and automobile sales. These reforms were orchestrated by Transport Minister Ernest Marples, who was purely a politician. The fact that he accumulated a considerable personal fortune during this period by buying shares in motorway construction corporations, has nothing to do with it. So, no conflict of interest there - now if the automobile in industry can't be considered a pure Paperclip Maximizer... sorry, the automobile industry in isolation can't be considered a pure Paperclip Maximizer. You have to look at it in conjunction with the fossil fuel industries, the road construction business, the accident insurance sector and so on. When you do this, you begin to see the outline of a paperclip-maximizing ecosystem that invades far-flung lands and grinds up and kills around one and a quarter million people per year. That's the global death toll from automobile accidents currently, according to the World Health Organization. It rivals the First World War on an ongoing permanent basis and these are all side effects of its drive to sell you a new car. Now, automobiles aren't of course a total liability. Today's cars are regulated stringently for safety and, in theory, to reduce toxic emissions. They're fast, efficient and comfortable. We can thank legal mandated regulations imposed by governments for this, of course. Go back to the 1970s and cars didn't have crumple zones, go back to the 50s and they didn't come with seat belts as standard. In the 1930s, indicators, turn signals and brakes on all four wheels were optional and your best hope of surviving a 50 km/h-crash was to be thrown out of a car and land somewhere without breaking your neck. Regulator agencies are our current political system's tool of choice for preventing Paperclip Maximizers from running amok. Unfortunately, regulators don't always work. The first failure mode of regulators that you need to be aware of is regulatory capture, where regulatory bodies are captured by the industries they control. Ajit Pai, Head of American Federal Communications Commission, which just voted to eliminate net neutrality rules in the U.S., has worked as Associate General Counsel for Verizon Communications Inc, the largest current descendant of the Bell Telephone system's monopoly. After the AT&T antitrust lawsuit, the Bell network was broken up into the seven baby bells. They've now pretty much reformed and reaggregated and Verizon is the largest current one. Why should someone with a transparent interest in a technology corporation end up running a regulator that tries to control the industry in question? Well, if you're going to regulate a complex technology, you need to recruit regulators from people who understand it. Unfortunately, most of those people are industry insiders. Ajit Pai is clearly very much aware of how Verizon is regulated, very insightful into its operations and wants to do something about it - just not necessarily in the public interest. applause When regulators end up staffed by people drawn from the industries they're supposed to control, they frequently end up working with their former office mates, to make it easier to turn a profit, either by raising barriers to keep new insurgent companies out or by dismantling safeguards that protect the public. Now a second problem is regulatory lag where a technology advances so rapidly, that regulations are laughably obsolete by the time they're issued. Consider the EU directive requiring cookie notices on websites to caution users, that their activities are tracked and their privacy may be violated. This would have been a good idea in 1993 or 1996, but unfortunatelly it didn't show up until 2011. Fingerprinting and tracking mechanisms have nothing to do with cookies and were already widespread by then. Tim Berners-Lee observed in 1995, that five years worth of change was happening on the web for every 12 months of real-world time. By that yardstick, the cookie law came out nearly a century too late to do any good. Again, look at Uber. This month, the European Court of Justice ruled that Uber is a taxi service, not a Web App. This is arguably correct - the problem is, Uber has spread globally since it was founded eight years ago, subsidizing its drivers to put competing private hire firms out of business. Whether this is a net good for societys own is debatable. The problem is, a taxi driver can get awfully hungry if she has to wait eight years for a court ruling against a predator intent on disrupting her business. So, to recap: firstly, we already have Paperclip Maximizers and Musk's AI alarmism is curiously mirror blind. Secondly, we have mechanisms for keeping Paperclip Maximizers in check, but they don't work very well against AIs that deploy the dark arts, especially corruption and bribery and they're even worse against true AIs, that evolved too fast for human mediated mechanisms like the law to keep up with. Finally, unlike the naive vision of a Paperclip Maximizer that maximizes only paperclips, existing AIs have multiple agendas, their overt goal, but also profit seeking, expansion into new markets and to accommodate the desire of whoever is currently in the driving seat. sighs Now, this brings me to the next major heading in this dismaying laundry list: how it all went wrong. It seems to me that our current political upheavals, the best understood, is arising from the capture of post 1917 democratic institutions by large-scale AI. Everywhere you look, you see voters protesting angrily against an entrenched establishment, that seems determined to ignore the wants and needs of their human constituents in favor of those of the machines. The brexit upset was largely result of a protest vote against the British political establishment, the election of Donald Trump likewise, with a side order of racism on top. Our major political parties are led by people who are compatible with the system as it exists today, a system that has been shaped over decades by corporations distorting our government and regulatory environments. We humans live in a world shaped by the desires and needs of AI, forced to live on their terms and we're taught, that we're valuable only to the extent we contribute to the rule of the machines. Now this is free sea and we're all more interested in computers and communications technology than this historical crap. But as I said earlier, history is a secret weapon, if you know how to use it. What history is good for, is enabling us to spot recurring patterns that repeat across timescales outside our personal experience. And if we look at our historical very slow AIs, what do we learn from them about modern AI and how it's going to behave? Well to start with, our AIs have been warped, the new AIs, the electronic one's instantiated in our machines, have been warped by a terrible fundamentally flawed design decision back in 1995, but as damaged democratic political processes crippled our ability to truly understand the world around us and led to the angry upheavals and upsets of our present decade. That mistake was the decision, to fund the build-out of a public World Wide Web as opposed to be earlier government-funded corporate and academic Internet by monetizing eyeballs through advertising revenue. The ad-supported web we're used to today wasn't inevitable. If you recall the web as it was in 1994, there were very few ads at all and not much, in a way, of Commerce. 1995 was the year, the World Wide Web really came to public attention in the anglophone world and consumer-facing websites began to appear. Nobody really knew, how this thing was going to be paid for. The original .com bubble was all about working out, how to monetize the web for the first time and a lot of people lost their shirts in the process. A naive initial assumption was that the transaction cost of setting up a tcp/ip connection over modem was too high to support.. to be supported by per-use micro billing for web pages. So instead of charging people fraction of a euro cent for every page view, we'd bill customers indirectly, by shoving advertising banners in front of their eyes and hoping they'd click through and buy something. Unfortunately, advertising is in an industry, one of those pre-existing very slow AI ecosystems I already alluded to. Advertising tries to maximize its hold on the attention of the minds behind each human eyeball. The coupling of advertising with web search was an inevitable outgrowth, I mean how better to attract the attention of reluctant subjects, than to find out what they're really interested in seeing and selling ads that relate to those interests. The problem of applying the paperclip maximize approach to monopolizing eyeballs, however, is that eyeballs are a limited, scarce resource. There are only 168 hours in every week, in which I can gaze at banner ads. Moreover, most ads are irrelevant to my interests and it doesn't matter, how often you flash an ad for dog biscuits at me, I'm never going to buy any. I have a cat. To make best revenue-generating use of our eyeballs, it's necessary for the ad industry to learn, who we are and what interests us and to target us increasingly minutely in hope of hooking us with stuff we're attracted to. In other words: the ad industry is a paperclip maximizer, but for its success, it relies on developing a theory of mind that applies to human beings. sighs Do I need to divert on to the impassioned rant about the hideous corruption and evil that is Facebook? Audience: Yes! CS: Okay, somebody said yes. I'm guessing you've heard it all before, but for too long don't read.. summary is: Facebook is as much a search engine as Google or Amazon. Facebook searches are optimized for faces, that is for human beings. If you want to find someone you fell out of touch with thirty years ago, Facebook probably knows where they live, what their favorite color is, what sized shoes they wear and what they said about you to your friends behind your back all those years ago, that made you cut them off. Even if you don't have a Facebook account, Facebook has a You account, a hole in their social graph of a bunch of connections pointing in to it and your name tagged on your friends photographs. They know a lot about you and they sell access to their social graph to advertisers, who then target you, even if you don't think you use Facebook. Indeed, there is barely any point in not using Facebook these days, if ever. Social media Borg: "Resistance is futile!" So however, Facebook is trying to get eyeballs on ads, so is Twitter and so are Google. To do this, they fine-tuned the content they show you to make it more attractive to your eyes and by attractive I do not mean pleasant. We humans have an evolved automatic reflex to pay attention to threats and horrors as well as pleasurable stimuli and the algorithms, that determine what they show us when we look at Facebook or Twitter, take this bias into account. You might react more strongly to a public hanging in Iran or an outrageous statement by Donald Trump than to a couple kissing. The algorithm knows and will show you whatever makes you pay attention, not necessarily what you need or want to see. So this brings me to another point about computerized AI as opposed to corporate AI. AI algorithms tend to embody the prejudices and beliefs of either the programmers, or the data set the AI was trained on. A couple of years ago I ran across an account of a webcam, developed by mostly pale-skinned Silicon Valley engineers, that had difficulty focusing or achieving correct color balance, when pointed at dark-skinned faces. Fast an example of human programmer induced bias, they didn't have a wide enough test set and didn't recognize that they were inherently biased towards expecting people to have pale skin. But with today's deep learning, bias can creep in, while the datasets for neural networks are trained on, even without the programmers intending it. Microsoft's first foray into a conversational chat bot driven by machine learning, Tay, was what we yanked offline within days last year, because 4chan and reddit based trolls discovered, that they could train it towards racism and sexism for shits and giggles. Just imagine you're a poor naive innocent AI who's just been switched on and you're hoping to pass your Turing test and what happens? 4chan decide to play with your head. laughing I got to feel sorry for Tay. Now, humans may be biased, but at least individually we're accountable and if somebody gives you racist or sexist abuse to your face, you can complain or maybe punch them. It's impossible to punch a corporation and it may not even be possible to identify the source of unfair bias, when you're dealing with a machine learning system. AI based systems that instantiate existing prejudices make social change harder. Traditional advertising works by playing on the target customer's insecurity and fear as much as their aspirations. And fear of a loss of social status and privileges are powerful stress. Fear and xenophobia are useful tools for tracking advertising.. ah, eyeballs. What happens when we get pervasive social networks, that have learned biases against say Feminism or Islam or melanin? Or deep learning systems, trained on datasets contaminated by racist dipshits and their propaganda? Deep learning systems like the ones inside Facebook, that determine which stories to show you to get you to pay as much attention as possible to be adverse. I think, you probably have an inkling of how.. where this is now going. Now, if you think, this is sounding a bit bleak and unpleasant, you'd be right. I write sci-fi. You read or watch or play sci-fi. We're acculturated to think of science and technology as good things that make life better, but this ain't always so. Plenty of technologies have historically been heavily regulated or even criminalized for good reason and once you get past any reflexive indignation, criticism of technology and progress, you might agree with me, that it is reasonable to ban individuals from owning nuclear weapons or nerve gas. Less obviously, they may not be weapons, but we've banned chlorofluorocarbon refrigerants, because they were building up in the high stratosphere and destroying the ozone layer that protects us from UVB radiation. We banned tetra e-file LED in gasoline, because it poisoned people and led to a crime wave. These are not weaponized technologies, but they have horrible side effects. Now, nerve gas and leaded gasoline were 1930s chemical technologies, promoted by 1930s corporations. Halogenated refrigerants and nuclear weapons are totally 1940s. ICBMs date to the 1950s. You know, I have difficulty seeing why people are getting so worked up over North Korea. North Korea reaches 1953 level parity - be terrified and hide under the bed! I submit that the 21st century is throwing up dangerous new technologies, just as our existing strategies for regulating very slow AIs have proven to be inadequate. And I don't have an answer to how we regulate new technologies, I just want to flag it up as a huge social problem that is going to affect the coming century. I'm now going to give you four examples of new types of AI application that are going to warp our societies even more badly than the old slow AIs, we.. have done. This isn't an exhaustive list, this is just some examples I dream, I pulled out of my ass. We need to work out a general strategy for getting on top of this sort of thing before they get on top of us and I think, this is actually a very urgent problem. So I'm just going to give you this list of dangerous new technologies that are arriving now, or coming, and send you away to think about what to do next. I mean, we are activists here, we should be thinking about this and planning what to do. Now, the first nasty technology I'd like to talk about, is political hacking tools that rely on social graph directed propaganda. This is low-hanging fruit after the electoral surprises of 2016. Cambridge Analytica pioneered the use of deep learning by scanning the Facebook and Twitter social graphs to identify voters political affiliations by simply looking at what tweets or Facebook comments they liked, very able to do this, to identify individuals with a high degree of precision, who were vulnerable to persuasion and who lived in electorally sensitive districts. They then canvassed them with propaganda, that targeted their personal hot-button issues to change their electoral intentions. The tools developed by web advertisers to sell products have now been weaponized for political purposes and the amount of personal information about our affiliations that we expose on social media, makes us vulnerable. Aside, in the last U.S. Presidential election, as mounting evidence for the British referendum on leaving the EU was subject to foreign cyber war attack, now weaponized social media, as was the most recent French Presidential election. In fact, if we remember the leak of emails from the Macron campaign, it turns out that many of those emails were false, because the Macron campaign anticipated that they would be attacked and an email trove would be leaked in the last days before the election. So they deliberately set up false emails that would be hacked and then leaked and then could be discredited. It gets twisty fast. Now I'm kind of biting my tongue and trying, not to take sides here. I have my own political affiliation after all, and I'm not terribly mainstream. But if social media companies don't work out how to identify and flag micro- targeted propaganda, then democratic institutions will stop working and elections will be replaced by victories, whoever can buy the most trolls. This won't simply be billionaires but.. like the Koch brothers and Robert Mercer from the U.S. throwing elections to whoever will hand them the biggest tax cuts. Russian military cyber war doctrine calls for the use of social media to confuse and disable perceived enemies, in addition to the increasingly familiar use of zero-day exploits for espionage, such as spear phishing and distributed denial-of-service attacks, on our infrastructure, which are practiced by Western agencies. Problem is, once the Russians have demonstrated that this is an effective tactic, the use of propaganda bot armies in cyber war will go global. And at that point, our social discourse will be irreparably poisoned. Incidentally, I'd like to add - as another aside like the Elon Musk thing - I hate the cyber prefix! It usually indicates, that whoever's using it has no idea what they're talking about. applause, laughter Unfortunately, much as the way the term hacker was corrupted from its original meaning in the 1990s, the term cyber war has, it seems, to have stuck and it's now an actual thing that we can point to and say: "This is what we're talking about". So I'm afraid, we're stuck with this really horrible term. But that's a digression, I should get back on topic, because I've only got 20 minutes to go. Now, the second threat that we need to think about regulating ,or controlling, is an adjunct to deep learning target propaganda: it's the use of neural network generated false video media. We're used to photoshopped images these days, but faking video and audio takes it to the next level. Luckily, faking video and audio is labor-intensive, isn't it? Well nope, not anymore. We're seeing the first generation of AI assisted video porn, in which the faces of film stars are mapped onto those of other people in a video clip, using software rather than laborious in human process. A properly trained neural network recognizes faces and transforms the face of the Hollywood star, they want to put into a porn movie, into the face of - onto the face of the porn star in the porn clip and suddenly you have "Oh dear God, get it out of my head" - no, not gonna give you any examples. Let's just say it's bad stuff. laughs Meanwhile we have WaveNet, a system for generating realistic sounding speech, if a voice of a human's speak of a neural network has been trained to mimic any human speaker. We can now put words into other people's mouths realistically without employing a voice actor. This stuff is still geek intensive. It requires relatively expensive GPUs or cloud computing clusters, but in less than a decade it'll be out in the wild, turned into something, any damn script kiddie can use and just about everyone will be able to fake up a realistic video of someone they don't like doing something horrible. I mean, Donald Trump in the White House. I can't help but hope that out there somewhere there's some geek like Steve Bannon with a huge rack of servers who's faking it all, but no. Now, also we've already seen alarm this year over bizarre YouTube channels that attempt to monetize children's TV brands by scraping the video content of legitimate channels and adding their own advertising in keywords on top before reposting it. This is basically your YouTube spam. Many of these channels are shaped by paperclip maximizing advertising AIs, but are simply trying to maximise their search ranking on YouTube and it's entirely algorithmic: you have a whole list of keywords, you perm, you take them, you slap them on top of existing popular videos and re-upload the videos. Once you add neural network driven tools for inserting character A into pirated video B, to click maximize.. for click maximizing bots, things are gonna get very weird, though. And they're gonna get even weirder, when these tools are deployed for political gain. We tend - being primates, that evolved 300 thousand years ago in a smartphone free environment - to evaluate the inputs from our eyes and ears much less critically than what random strangers on the Internet tell us in text. We're already too vulnerable to fake news as it is. Soon they'll be coming for us, armed with believable video evidence. The Smart Money says that by 2027 you won't be able to believe anything you see in video, unless for a cryptographic signatures on it, linking it back to the camera that shot the raw feed. But you know how good most people are at using encryption - it's going to be chaos! So, paperclip maximizers with focus on eyeballs are very 20th century. The new generation is going to be focusing on our nervous system. Advertising as an industry can only exist because of a quirk of our nervous system, which is that we're susceptible to addiction. Be it tobacco, gambling or heroin, we recognize addictive behavior, when we see it. Well, do we? It turns out the human brain's reward feedback loops are relatively easy to gain. Large corporations like Zynga - producers of FarmVille - exist solely because of it, free to use social media platforms like Facebook and Twitter, are dominant precisely because they're structured to reward frequent short bursts of interaction and to generate emotional engagement - not necessarily positive emotions, anger and hatred are just as good when it comes to attracting eyeballs for advertisers. Smartphone addiction is a side effect of advertising as a revenue model. Frequent short bursts of interaction to keep us coming back for more. Now a new.. newish development, thanks to deep learning again - I keep coming back to deep learning, don't I? - use of neural networks in a manner that Marvin Minsky never envisaged, back when he was deciding that the Perzeptron was where it began and ended and it couldn't do anything. Well, we have neuroscientists now, who've mechanized the process of making apps more addictive. Dopamine Labs is one startup that provides tools to app developers to make any app more addictive, as well as to reduce the desire to continue participating in a behavior if it's undesirable, if the app developer actually wants to help people kick the habit. This goes way beyond automated A/B testing. A/B testing allows developers to plot a binary tree path between options, moving towards a single desired goal. But true deep learning, addictiveness maximizers, can optimize for multiple attractors in parallel. The more users you've got on your app, the more effectively you can work out, what attracts them and train them and focus on extra addictive characteristics. Now, going by their public face, the folks at Dopamine Labs seem to have ethical qualms about the misuse of addiction maximizers. But neuroscience isn't a secret and sooner or later some really unscrupulous sociopaths will try to see how far they can push it. So let me give you a specific imaginary scenario: Apple have put a lot of effort into making real- time face recognition work on the iPhone X and it's going to be everywhere on everybody's phone in another couple of years. You can't fool an iPhone X with a photo or even a simple mask. It does depth mapping to ensure, your eyes are in the right place and can tell whether they're open or closed. It recognizes your face from underlying bone structure through makeup and bruises. It's running continuously, checking pretty much as often as every time you'd hit the home button on a more traditional smartphone UI and it can see where your eyeballs are pointing. The purpose of a face recognition system is to provide for real-time authenticate continuous authentication when you're using a device - not just enter a PIN or sign a password or use a two factor authentication pad, but the device knows that you are its authorized user on a continuous basis and if somebody grabs your phone and runs away with it, it'll know that it's been stolen immediately, it sees the face of the thief. However, your phone monitoring your facial expressions and correlating against app usage has other implications. Your phone will be aware of precisely what you like to look at on your screen.. on its screen. We may well have sufficient insight on the part of the phone to identify whether you're happy or sad, bored or engaged. With addiction seeking deep learning tools and neural network generated images, those synthetic videos I was talking about, it's entirely.. in principle entirely possible to feed you an endlessly escalating payload of arousal-inducing inputs. It might be Facebook or Twitter messages, optimized to produce outrage, or it could be porn generated by AI to appeal to kinks you don't even consciously know you have. But either way, the app now owns your central nervous system and you will be monetized. And finally, I'd like to raise a really hair-raising specter that goes well beyond the use of deep learning and targeted propaganda and cyber war. Back in 2011, an obscure Russian software house launched an iPhone app for pickup artists called 'Girls Around Me'. Spoiler: Apple pulled it like a hot potato as soon as word got out that it existed. Now, Girls Around Me works out where the user is using GPS, then it would query Foursquare and Facebook for people matching a simple relational search, for single females on Facebook, per relationship status, who have checked in, or been checked in by their friends, in your vicinity on Foursquare. The app then displays their locations on a map along with links to their social media profiles. If they were doing it today, the interface would be gamified, showing strike rates and a leaderboard and flagging targets who succumbed to harassment as easy lays. But these days, the cool kids and single adults are all using dating apps with a missing vowel in the name, only a creeper would want something like Girls Around Me, right? Unfortunately, there are much, much nastier uses of and scraping social media to find potential victims for serial rapists. Does your social media profile indicate your political religious affiliation? No? Cambridge Analytica can work them out with 99.9% precision anyway, so don't worry about that. We already have you pegged. Now add a service that can identify people's affiliation and location and you have a beginning of a flash mob app, one that will show people like us and people like them on a hyperlocal map. Imagine you're a young female and a supermarket like Target has figured out from your purchase patterns, that you're pregnant, even though you don't know it yet. This actually happened in 2011. Now imagine, that all the anti-abortion campaigners in your town have an app called "Babies Risk" on their phones. Someone has paid for the analytics feed from the supermarket and every time you go near a family planning clinic, a group of unfriendly anti-abortion protesters somehow miraculously show up and swarm you. Or imagine you're male and gay and the "God hates fags"-crowd has invented a 100% reliable gaydar app, based on your Grindr profile, and is getting their fellow travelers to queer bash gay men - only when they're alone or outnumbered by ten to one. That's the special horror of precise geolocation not only do you always know where you are, the AIs know, where you are and some of them aren't friendly. Or imagine, you're in Pakistan and Christian Muslim tensions are rising or in rural Alabama or an Democrat, you know the possibilities are endless. Someone out there is working on this. A geolocation aware, social media scraping deep learning application, that uses a gamified competitive interface to reward its players for joining in acts of mob violence against whoever the app developer hates. Probably it has an innocuous seeming, but highly addictive training mode, to get the users accustomed to working in teams and obeying the apps instructions. Think Ingress or Pokemon Go. Then at some pre- planned zero-hour, it switches mode and starts rewarding players for violence, players who have been primed to think of their targets as vermin by a steady drip feed of micro-targeted dehumanizing propaganda inputs, delivered over a period of months. And the worst bit of this picture? Is that the app developer isn't even a nation-state trying to disrupt its enemies or an extremist political group trying to murder gays, Jews or Muslims. It's just a Paperclip Maximizer doing what it does and you are the paper. Welcome to the 21st century. applause Uhm... Thank you. ongoing applause We have a little time for questions. Do you have a microphone for the orders? Do we have any questions? ... OK. Herald: So you are doing a Q&A? CS: Hmm? Herald: So you are doing a Q&A. Well if there are any questions, please come forward to the microphones, numbers 1 through 4 and ask. Mic 1: Do you really think it's all bleak and dystopian like you prescribed it, because I also think the future can be bright, looking at the internet with open source and like, it's all growing and going faster and faster in a good direction. So what do you think about the balance here? CS: sighs Basically, I think the problem is, that about 3% of us are sociopaths or psychopaths, who spoil everything for the other 97% of us. Wouldn't it be great if somebody could write an app that would identify all the psychopaths among us and let the rest of us just kill them? laughing, applause Yeah, we have all the tools to make a utopia, we have it now today. A bleak miserable grim meathook future is not inevitable, but it's up to us to use these tools to prevent the bad stuff happening and to do that, we have to anticipate the bad outcomes and work to try and figure out a way to deal with them. That's what this talk is. I'm trying to do a bit of a wake-up call and get people thinking about how much worse things can get and what we need to do to prevent it from happening. What I was saying earlier about our regulatory systems being broken, stands. How do we regulate the deep learning technologies? This is something we need to think about. H: Okay mic number two. Mic 2: Hello? ... When you talk about corporations as AIs, where do you see that analogy you're making? Do you see them as literally AIs or figuratively? CS: Almost literally. If you're familiar with philosopher (?) Searle's Chinese room paradox from the 1970s, by which he attempted to prove that artificial intelligence was impossible, a corporation is very much the Chinese room implementation of an AI. It is a bunch of human beings in a box. You put inputs into the box, you get apples out of a box. Does it matter, whether it's all happening in software or whether there's a human being following rules inbetween to assemble the output? I don't see there being much of a difference. Now you have to look at a company at a very abstract level to view it as an AI, but more and more companies are automating their internal business processes. You've got to view this as an ongoing trend. And yeah, they have many of the characteristics of an AI. Herald: Okay mic number four. Mic 4: Hi, thanks for your talk. You probably heard of the Time Well Spent and Design Ethics movements that are alerting developers to dark patterns in UI design, where these people design apps to manipulate people. I'm curious if you find any optimism in the possibility of amplifying or promoting those movements. CS: Uhm, you know, I knew about dark patterns, I knew about people trying to optimize them, I wasn't actually aware there were movements against this. Okay I'm 53 years old, I'm out of touch. I haven't actually done any serious programming in 15 years. I'm so rusty, my rust has rust on it. But, you know, it is a worrying trend and actual activism is a good start. Raising awareness of hazards and of what we should be doing about them, is a good start. And I would classify this actually as a moral issue. We need to.. corporations evaluate everything in terms of revenue, because it's very equivalent to breathing, they have to breathe. Corporations don't usually have any moral framework. We're humans, we need a moral framework to operate within. Even if it's as simple as first "Do no harm!" or "Do not do unto others that which would be repugnant if it was done unto you!", the Golden Rule. So, yeah, we should be trying to spread awareness of this about and working with program developers, to look to remind them that they are human beings and have to be humane in their application of technology, is a necessary start. applause H: Thank you! Mic 3? Mic 3: Hi! Yeah, I think that folks, especially in this sort of crowd, tend to jump to the "just get off of Facebook"-solution first, for a lot of these things that are really, really scary. But what worries me, is how we sort of silence ourselves when we do that. After the election I actually got back on Facebook, because the Women's March was mostly organized through Facebook. But yeah, I think we need a lot more regulation, but we can't just throw it out. We're.. because it's.. social media is the only... really good platform we have right now to express ourselves, to have our rules, or power. CS: Absolutely. I have made a point of not really using Facebook for many, many, many years. I have a Facebook page simply to shut up the young marketing people at my publisher, who used to prop up every two years and say: "Why don't you have a Facebook. Everybody's got a Facebook." No, I've had a blog since 1993! laughing But no, I'm gonna have to use Facebook, because these days, not using Facebook is like not using email. You're cutting off your nose to spite your face. What we really do need to be doing, is looking for some form of effective oversight of Facebook and particularly, of how they.. the algorithms that show you content, are written. What I was saying earlier about how algorithms are not as transparent as human beings to people, applies hugely to them. And both, Facebook and Twitter control the information that they display to you. Herald: Okay, I'm terribly sorry for all the people queuing at the mics now, we're out of time. I also have to apologize, I announced, that this talk was being held in English, but it was being held in English. the latter pronounced on the G Thank you very much, Charles Stross! CS: Thank you very much for listening to me, it's been a pleasure! applause postroll music subtitles created by c3subtitles.de in the year 2018