[Script Info] Title: [Events] Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text Dialogue: 0,0:00:00.10,0:00:16.60,Default,,0000,0000,0000,,{\i1}Music{\i0}\NHerald: The next talk is about how risky Dialogue: 0,0:00:16.60,0:00:23.21,Default,,0000,0000,0000,,is software you use. So you may be heard\Nabout Trump versus a Russian security Dialogue: 0,0:00:23.21,0:00:30.95,Default,,0000,0000,0000,,company. We won't judge this, we won't\Ncomment this, but we dislike the Dialogue: 0,0:00:30.95,0:00:36.59,Default,,0000,0000,0000,,prejudgments of this case. Tim Carstens\Nand Parker Thompson will tell you a little Dialogue: 0,0:00:36.59,0:00:43.30,Default,,0000,0000,0000,,bit more about how risky the software is\Nyou use. Tim Carstens is CITL's Acting Dialogue: 0,0:00:43.30,0:00:48.35,Default,,0000,0000,0000,,Director and Parker Thompson is CITL's\Nlead engineer. Please welcome with a very, Dialogue: 0,0:00:48.35,0:00:53.88,Default,,0000,0000,0000,,very warm applause: Tim and Parker!\NThanks. Dialogue: 0,0:00:53.88,0:01:05.41,Default,,0000,0000,0000,,{\i1}Applause{\i0}\NTim Carstens: Howdy, howdy. So my name is Dialogue: 0,0:01:05.41,0:01:13.01,Default,,0000,0000,0000,,Tim Carstens. I'm the acting director of\Nthe cyber independent testing lab. It's Dialogue: 0,0:01:13.01,0:01:19.04,Default,,0000,0000,0000,,four words there, we'll talk about all for\Ntoday, especially cyber. With me today as Dialogue: 0,0:01:19.04,0:01:25.76,Default,,0000,0000,0000,,our lead engineer Parker Thompson. Not on\Nstage or our other collaborators: Patrick Dialogue: 0,0:01:25.76,0:01:32.93,Default,,0000,0000,0000,,Stach, Sarah Zatko, and present in the\Nroom but not on stage - Mudge. So today Dialogue: 0,0:01:32.93,0:01:37.01,Default,,0000,0000,0000,,we're going to be talking about our work,\Nthe lead in. The introduction that was Dialogue: 0,0:01:37.01,0:01:40.29,Default,,0000,0000,0000,,given is phrased in terms of Kaspersky and\Nall of that, I'm not gonna be speaking Dialogue: 0,0:01:40.29,0:01:45.37,Default,,0000,0000,0000,,about Kaspersky and I guarantee you I'm\Nnot gonna be speaking about my president. Dialogue: 0,0:01:45.37,0:01:50.01,Default,,0000,0000,0000,,Right, yeah? Okay. Thank you.\N{\i1}Applause{\i0} Dialogue: 0,0:01:50.01,0:01:55.29,Default,,0000,0000,0000,,All right, so why don't we go ahead and\Nkick off: I'll mention now parts of this Dialogue: 0,0:01:55.29,0:02:00.54,Default,,0000,0000,0000,,presentation are going to be quite\Ntechnical. Not most of it, and I will Dialogue: 0,0:02:00.54,0:02:04.03,Default,,0000,0000,0000,,always include analogies and all these\Nother things if you are here in security Dialogue: 0,0:02:04.03,0:02:10.53,Default,,0000,0000,0000,,but not a bit-twiddler. But if you do want\Nto be able to review some of the technical Dialogue: 0,0:02:10.53,0:02:14.81,Default,,0000,0000,0000,,material, if I go through it too fast you\Nlike to read if you're a mathematician or Dialogue: 0,0:02:14.81,0:02:20.51,Default,,0000,0000,0000,,if you are a computer scientist, our sides\Nare already available for download at this Dialogue: 0,0:02:20.51,0:02:25.40,Default,,0000,0000,0000,,site here. We think our pal our partners\Nat power door for getting that set up for Dialogue: 0,0:02:25.40,0:02:31.63,Default,,0000,0000,0000,,us. Let's let's get started on the real\Nmaterial here. Alright, so we are CITL: a Dialogue: 0,0:02:31.63,0:02:35.77,Default,,0000,0000,0000,,nonprofit organization based in the United\NStates founded by our chief scientist Dialogue: 0,0:02:35.77,0:02:43.02,Default,,0000,0000,0000,,Sarah Zatko and our board chair Mudge. And\Nour mission is a public good mission - we Dialogue: 0,0:02:43.02,0:02:47.40,Default,,0000,0000,0000,,are hackers but our mission here is\Nactually to look out for people who do not Dialogue: 0,0:02:47.40,0:02:50.46,Default,,0000,0000,0000,,know very much about machines\Nor as much as the other hackers do. Dialogue: 0,0:02:50.46,0:02:56.03,Default,,0000,0000,0000,,Specifically, we seek to improve the state\Nof software security by providing the Dialogue: 0,0:02:56.03,0:03:01.34,Default,,0000,0000,0000,,public with accurate reporting on the\Nsecurity of popular software, right? And Dialogue: 0,0:03:01.34,0:03:05.52,Default,,0000,0000,0000,,so there was a mouthful for you. But no\Ndoubt, no doubt, every single one of you Dialogue: 0,0:03:05.52,0:03:10.70,Default,,0000,0000,0000,,has received questions of the form: what\Ndo I run on my phone, what do I do with Dialogue: 0,0:03:10.70,0:03:13.95,Default,,0000,0000,0000,,this, what do I do with that, how do I\Nprotect myself - all these other things Dialogue: 0,0:03:13.95,0:03:19.77,Default,,0000,0000,0000,,lots of people in the general public\Nlooking for agency in computing. No one's Dialogue: 0,0:03:19.77,0:03:25.00,Default,,0000,0000,0000,,offering it to them, and so we're trying\Nto go ahead and provide a forcing function Dialogue: 0,0:03:25.00,0:03:29.98,Default,,0000,0000,0000,,on the software field in order to, you\Nknow, again be able to enable consumers Dialogue: 0,0:03:29.98,0:03:36.48,Default,,0000,0000,0000,,and users and all these things. Our social\Ngood work is funded largely by charitable Dialogue: 0,0:03:36.48,0:03:40.82,Default,,0000,0000,0000,,monies from the Ford Foundation whom we\Nthank a great deal, but we also have major Dialogue: 0,0:03:40.82,0:03:44.92,Default,,0000,0000,0000,,partnerships with Consumer Reports, which\Nis a major organization in the United Dialogue: 0,0:03:44.92,0:03:51.62,Default,,0000,0000,0000,,States that generally, broadly, looks at\Nconsumer goods for safety and performance. Dialogue: 0,0:03:51.62,0:03:55.69,Default,,0000,0000,0000,,But also partners with The Digital\NStandard, which probably would be of great Dialogue: 0,0:03:55.69,0:03:59.46,Default,,0000,0000,0000,,interest to many people here at Congress\Nas it is a holistic standard for Dialogue: 0,0:03:59.46,0:04:04.34,Default,,0000,0000,0000,,protecting user rights. We'll talk about\Nsome of the work that goes into those Dialogue: 0,0:04:04.34,0:04:10.25,Default,,0000,0000,0000,,things here in a bit, but first I want to\Ngive the big picture of what it is we're Dialogue: 0,0:04:10.25,0:04:17.94,Default,,0000,0000,0000,,really trying to do in one one short\Nlittle sentence. Something like this but Dialogue: 0,0:04:17.94,0:04:23.71,Default,,0000,0000,0000,,for security, right? What are the\Nimportant facts, how does it rate, you Dialogue: 0,0:04:23.71,0:04:26.81,Default,,0000,0000,0000,,know, is it easy to consume, is it easy to\Ngo ahead and look and say this thing is Dialogue: 0,0:04:26.81,0:04:31.30,Default,,0000,0000,0000,,good this thing is not good. Something\Nlike this, but for software security. Dialogue: 0,0:04:33.12,0:04:39.19,Default,,0000,0000,0000,,Sounds hard doesn't it? So I want to talk\Na little bit about what I mean by Dialogue: 0,0:04:39.19,0:04:44.87,Default,,0000,0000,0000,,something like this.\NThere are lots of consumer outlook and Dialogue: 0,0:04:44.87,0:04:50.27,Default,,0000,0000,0000,,watchdog and protection groups - some\Nprivate, some government, which are Dialogue: 0,0:04:50.27,0:04:54.82,Default,,0000,0000,0000,,looking to do this for various things that\Nare not a software security. And you can Dialogue: 0,0:04:54.82,0:04:58.21,Default,,0000,0000,0000,,see some examples here that are big in the\NUnited States - I happen to not like these Dialogue: 0,0:04:58.21,0:05:02.12,Default,,0000,0000,0000,,as much as some of the newer consumer\Nlabels coming out from the EU. But Dialogue: 0,0:05:02.12,0:05:05.08,Default,,0000,0000,0000,,nonetheless they are examples of the kinds\Nof things people have done in other Dialogue: 0,0:05:05.08,0:05:10.87,Default,,0000,0000,0000,,fields, fields that are not security to\Ntry to achieve that same end. And when Dialogue: 0,0:05:10.87,0:05:17.41,Default,,0000,0000,0000,,these things work well, it is for three\Nreasons: One, it has to contain the Dialogue: 0,0:05:17.41,0:05:22.96,Default,,0000,0000,0000,,relevant information. Two: it has to be\Nbased in fact, we're not talking opinions, Dialogue: 0,0:05:22.96,0:05:28.80,Default,,0000,0000,0000,,this is not a book club or something like\Nthat. And three: it has to be actionable, Dialogue: 0,0:05:28.80,0:05:32.52,Default,,0000,0000,0000,,it has to be actionable - you have to be\Nable to know how to make a decision based Dialogue: 0,0:05:32.52,0:05:36.37,Default,,0000,0000,0000,,on it. How do you do that for software\Nsecurity? How {\i1}do{\i0} you do that for Dialogue: 0,0:05:36.37,0:05:43.76,Default,,0000,0000,0000,,software security? So the rest of the talk\Nis going to go in three parts. Dialogue: 0,0:05:43.76,0:05:49.45,Default,,0000,0000,0000,,First, we're going to give a bit of an\Noverview for more of the consumer facing Dialogue: 0,0:05:49.45,0:05:52.82,Default,,0000,0000,0000,,side of things for that we do: look at\Nsome data that we have reported on early Dialogue: 0,0:05:52.82,0:05:57.26,Default,,0000,0000,0000,,and all these other kinds of good things.\NWe're then going to go ahead and get Dialogue: 0,0:05:57.94,0:06:06.01,Default,,0000,0000,0000,,terrifyingly, terrifyingly technical. And\Nthen after that we'll talk about tools to Dialogue: 0,0:06:06.01,0:06:09.56,Default,,0000,0000,0000,,actually implement all this stuff. The\Ntechnical part comes before the tools. So Dialogue: 0,0:06:09.56,0:06:12.17,Default,,0000,0000,0000,,it just tells you how terrifyingly\Ntechnical we're gonna get. It's gonna be Dialogue: 0,0:06:12.17,0:06:19.68,Default,,0000,0000,0000,,fun right. So how do you do this for\Nsoftware security: a consumer version. So, Dialogue: 0,0:06:19.68,0:06:25.01,Default,,0000,0000,0000,,if you set forth to the task of trying to\Nmeasure software security, right, many Dialogue: 0,0:06:25.01,0:06:27.68,Default,,0000,0000,0000,,people here probably do work in the\Nsecurity field perhaps as consultants Dialogue: 0,0:06:27.68,0:06:32.28,Default,,0000,0000,0000,,doing reviews; certainly I used to. Then\Nprobably what you're thinking to yourself Dialogue: 0,0:06:32.28,0:06:38.65,Default,,0000,0000,0000,,right now is that there are lots and lots\Nand lots and lots of things that affect Dialogue: 0,0:06:38.65,0:06:44.15,Default,,0000,0000,0000,,the security of a piece of software. Some\Nof which are, mmm, you're only gonna see Dialogue: 0,0:06:44.15,0:06:47.64,Default,,0000,0000,0000,,them if you go reversing. And some of\Nwhich are just you know kicking around on Dialogue: 0,0:06:47.64,0:06:51.58,Default,,0000,0000,0000,,the ground waiting for you to notice,\Nright. So we're going to talk about both Dialogue: 0,0:06:51.58,0:06:55.92,Default,,0000,0000,0000,,of those kinds of things that you might\Nmeasure. But here you see these giant Dialogue: 0,0:06:55.92,0:07:03.38,Default,,0000,0000,0000,,charts that basically go through on the\Nleft - on the left we have Microsoft Excel Dialogue: 0,0:07:03.38,0:07:08.31,Default,,0000,0000,0000,,on OS X on the right Google Chrome for OS\NX this is a couple years old at this point Dialogue: 0,0:07:08.31,0:07:12.85,Default,,0000,0000,0000,,maybe one and a half years old but over\Nhere I'm not expecting you to be able to Dialogue: 0,0:07:12.85,0:07:16.25,Default,,0000,0000,0000,,read these - the real point is to say look\Nat all of the different things you can Dialogue: 0,0:07:16.25,0:07:20.49,Default,,0000,0000,0000,,measure very easily.\NHow do you distill, it how do you boil it Dialogue: 0,0:07:20.49,0:07:26.77,Default,,0000,0000,0000,,down, right. So this is a the opposite of\Na good consumer safety label. This is just Dialogue: 0,0:07:26.77,0:07:29.78,Default,,0000,0000,0000,,um if you ever done any consulting this is\Nthe kind of report you hand a client to Dialogue: 0,0:07:29.78,0:07:32.87,Default,,0000,0000,0000,,tell them how good their software is,\Nright? It's the opposite of consumer Dialogue: 0,0:07:32.87,0:07:39.80,Default,,0000,0000,0000,,grade. But the reason I'm showing it here\Nis because, you know, I'm gonna call out Dialogue: 0,0:07:39.80,0:07:42.65,Default,,0000,0000,0000,,some things and maybe you can't process\Nall of this because it's too much Dialogue: 0,0:07:42.65,0:07:46.91,Default,,0000,0000,0000,,material, you know. But I'm gonna call it\Nsome things and once I call them out just Dialogue: 0,0:07:46.91,0:07:52.95,Default,,0000,0000,0000,,like NP you're gonna recognize them\Ninstantly. So for example, Excel, at the Dialogue: 0,0:07:52.95,0:07:56.82,Default,,0000,0000,0000,,time of this review - look at this column\Nof dots. What's this dots telling you? Dialogue: 0,0:07:56.82,0:07:59.99,Default,,0000,0000,0000,,It's telling you look at all these\Nlibraries -all of them are 32-bit only. Dialogue: 0,0:07:59.99,0:08:07.18,Default,,0000,0000,0000,,Not 64 bits, not 64 bits. Take a look at\NChrome - exact opposite, exact opposite Dialogue: 0,0:08:07.18,0:08:14.02,Default,,0000,0000,0000,,64-bit binary, right? What are some other\Nthings? Excel, again, on OSX maybe you can Dialogue: 0,0:08:14.02,0:08:19.55,Default,,0000,0000,0000,,see these danger warning signs that go\Nstraight straight up the whole thing. Dialogue: 0,0:08:19.55,0:08:27.52,Default,,0000,0000,0000,,That's the the absence of major heat\Nprotection flags in the binary headers. Dialogue: 0,0:08:27.52,0:08:31.92,Default,,0000,0000,0000,,We'll talk about some what that means\Nexactly in a bit. But also if you hop over Dialogue: 0,0:08:31.92,0:08:35.64,Default,,0000,0000,0000,,here you'll see like yeah yeah yeah like\NChrome has all the different heat Dialogue: 0,0:08:35.64,0:08:41.58,Default,,0000,0000,0000,,protections that a binary might enable, on\NOSX that is, but it also has more dots in Dialogue: 0,0:08:41.58,0:08:44.65,Default,,0000,0000,0000,,this column here off to the right. And\Nwhat do those dots represent? Dialogue: 0,0:08:44.65,0:08:52.05,Default,,0000,0000,0000,,Those dots represent functions, functions\Nthat historically have been the source of Dialogue: 0,0:08:52.05,0:08:54.46,Default,,0000,0000,0000,,you know if you call these functions are\Nvery hard to call correctly - if you're a Dialogue: 0,0:08:54.46,0:08:59.03,Default,,0000,0000,0000,,C programmer the "gets" function is a good\Nexample. But there are lots of them. And Dialogue: 0,0:08:59.03,0:09:03.28,Default,,0000,0000,0000,,you can see here the Chrome doesn't mind,\Nit uses them all a bunch. And Excel not so Dialogue: 0,0:09:03.28,0:09:08.36,Default,,0000,0000,0000,,much. And if you know the history of\NMicrosoft and the trusted computing Dialogue: 0,0:09:08.36,0:09:12.38,Default,,0000,0000,0000,,initiative and the SDO and all of that you\Nwill know that a very long time ago Dialogue: 0,0:09:12.38,0:09:17.18,Default,,0000,0000,0000,,Microsoft made the decision and they said\Nwe're gonna start purging some of these Dialogue: 0,0:09:17.18,0:09:22.01,Default,,0000,0000,0000,,risky functions from our code bases\Nbecause we think it's easier to ban them Dialogue: 0,0:09:22.01,0:09:24.99,Default,,0000,0000,0000,,than teach our devs to use them correctly.\NAnd you see that reverberating out in Dialogue: 0,0:09:24.99,0:09:28.98,Default,,0000,0000,0000,,their software. Google on the other hand\Nsays yeah yeah yeah those functions can be Dialogue: 0,0:09:28.98,0:09:31.92,Default,,0000,0000,0000,,dangerous to use but if you know how to\Nuse them they can be very good and so Dialogue: 0,0:09:31.92,0:09:38.96,Default,,0000,0000,0000,,they're permitted. The point all of this\Nis building to is that if you start by Dialogue: 0,0:09:38.96,0:09:42.54,Default,,0000,0000,0000,,just measuring every little thing that\Nlike your static analyzers can detect in a Dialogue: 0,0:09:42.54,0:09:47.76,Default,,0000,0000,0000,,piece of software. Two things: one, you\Nwind up with way more data than you can Dialogue: 0,0:09:47.76,0:09:55.27,Default,,0000,0000,0000,,show in a slide. And two: the engineering\Nprocess, the software development life Dialogue: 0,0:09:55.27,0:09:59.78,Default,,0000,0000,0000,,cycle that went into the software will\Nleave behind artifacts that tell you Dialogue: 0,0:09:59.78,0:10:05.17,Default,,0000,0000,0000,,something about the decisions that went\Ninto designing that engineering process. Dialogue: 0,0:10:05.17,0:10:10.18,Default,,0000,0000,0000,,And so you know, Google for example:\Nquite rigorous as far as hitting you know Dialogue: 0,0:10:10.18,0:10:14.38,Default,,0000,0000,0000,,GCC dash, and then enable all of the\Ncompiler protections. Microsoft may be Dialogue: 0,0:10:14.38,0:10:19.95,Default,,0000,0000,0000,,less good at that, but much more rigid in\Nthings that's were very popular ideas when Dialogue: 0,0:10:19.95,0:10:24.20,Default,,0000,0000,0000,,they introduced trusted computing,\Nalright. So the big takeaway from this Dialogue: 0,0:10:24.20,0:10:29.04,Default,,0000,0000,0000,,material is that again the software\Nengineering process results in artifacts Dialogue: 0,0:10:29.04,0:10:35.61,Default,,0000,0000,0000,,in the software that people can find.\NAlright. Ok, so that's that's a whole Dialogue: 0,0:10:35.61,0:10:40.58,Default,,0000,0000,0000,,bunch of data, certainly it's not a\Nconsumer-friendly label. So how do you Dialogue: 0,0:10:40.58,0:10:45.90,Default,,0000,0000,0000,,start to get in towards the consumer zone?\NWell, the main defect of the big reports Dialogue: 0,0:10:45.90,0:10:51.24,Default,,0000,0000,0000,,that we just saw is that it's too much\Ninformation. It's a very dense on data but Dialogue: 0,0:10:51.24,0:10:55.65,Default,,0000,0000,0000,,it's very hard to distill it to the "so\Nwhat" of it, right? Dialogue: 0,0:10:55.65,0:11:00.47,Default,,0000,0000,0000,,And so this here is one of our earlier\Nattempts to go ahead and do that Dialogue: 0,0:11:00.47,0:11:04.99,Default,,0000,0000,0000,,distillation. What are these charts how\Ndid we come up with these? Well on the Dialogue: 0,0:11:04.99,0:11:08.49,Default,,0000,0000,0000,,previous slide when we saw all these\Ndifferent factors that you can analyze in Dialogue: 0,0:11:08.49,0:11:14.19,Default,,0000,0000,0000,,software, basically here's whose views\Nthat we arrive at this. For each of those Dialogue: 0,0:11:14.19,0:11:18.64,Default,,0000,0000,0000,,things: pick a weight. Go ahead and\Ncompute a score, average against the Dialogue: 0,0:11:18.64,0:11:22.11,Default,,0000,0000,0000,,weight: tada, now you have some number.\NYou can do that for each of the libraries Dialogue: 0,0:11:22.11,0:11:25.82,Default,,0000,0000,0000,,and the piece of software. And if you do\Nthat for each of the libraries in the Dialogue: 0,0:11:25.82,0:11:29.40,Default,,0000,0000,0000,,software you can then go ahead and produce\Nthese histograms to show, you know, like Dialogue: 0,0:11:29.40,0:11:35.62,Default,,0000,0000,0000,,this percentage of the DLLs had a score in\Nthis range. Boom, there's a bar, right. Dialogue: 0,0:11:35.62,0:11:39.27,Default,,0000,0000,0000,,How do you pick those weights? We'll talk\Nabout that in a sec - it's very technical. Dialogue: 0,0:11:39.27,0:11:45.34,Default,,0000,0000,0000,,But the the takeaway though, is you know\Nthat you wind up with these charts. Now Dialogue: 0,0:11:45.34,0:11:48.33,Default,,0000,0000,0000,,I've obscured the labels, I've obscured\Nthe labels and the reason I've done that Dialogue: 0,0:11:48.33,0:11:52.33,Default,,0000,0000,0000,,is because I don't really care that much\Nabout the actual counts. I want to talk Dialogue: 0,0:11:52.33,0:11:57.42,Default,,0000,0000,0000,,about the shapes, the shapes of these\Ncharts: it's a qualitative thing. Dialogue: 0,0:11:57.42,0:12:02.54,Default,,0000,0000,0000,,So here: good scores appear on the right,\Nbad scores appear on the left. The Dialogue: 0,0:12:02.54,0:12:06.27,Default,,0000,0000,0000,,histogram measuring all the libraries and\Ncomponents and so a very secure piece of Dialogue: 0,0:12:06.27,0:12:12.88,Default,,0000,0000,0000,,software in this model manifests as a tall\Nbar far to the right. And you can see a Dialogue: 0,0:12:12.88,0:12:17.91,Default,,0000,0000,0000,,clear example at in our custom Gentoo\Nbuild. Anyone here is a Gentoo fan knows - Dialogue: 0,0:12:17.91,0:12:21.19,Default,,0000,0000,0000,,hey I'm going to install this thing, I\Nthink I'm going to go ahead and turn on Dialogue: 0,0:12:21.19,0:12:25.12,Default,,0000,0000,0000,,every single one of those flags, and lo\Nand behold if you do that yeah you wind up Dialogue: 0,0:12:25.12,0:12:30.52,Default,,0000,0000,0000,,with tall bar far to the right. Here's in\NUbuntu 16, I bet it's 16.04 but I don't Dialogue: 0,0:12:30.52,0:12:35.96,Default,,0000,0000,0000,,recall exactly, 16 LTS. Here you see a lot\Nof tall bars to the right - not quite as Dialogue: 0,0:12:35.96,0:12:39.62,Default,,0000,0000,0000,,consolidated as a custom Gentoo build, but\Nthat makes sense doesn't it right? Because Dialogue: 0,0:12:39.62,0:12:44.77,Default,,0000,0000,0000,,then you know you don't do your whole\NUbuntu build. Now I want to contrast. I Dialogue: 0,0:12:44.77,0:12:50.36,Default,,0000,0000,0000,,want to contrast. So over here on the\Nright we see in the same model, an Dialogue: 0,0:12:50.36,0:12:55.93,Default,,0000,0000,0000,,analysis of the firmware obtained from two\Nsmart televisions. Last year's models from Dialogue: 0,0:12:55.93,0:12:59.92,Default,,0000,0000,0000,,Samsung and LG. And here the model\Nnumbers. We did this work in concert with Dialogue: 0,0:12:59.92,0:13:05.04,Default,,0000,0000,0000,,Consumer Reports. And what do you notice\Nabout these histograms, right. Are the Dialogue: 0,0:13:05.04,0:13:11.79,Default,,0000,0000,0000,,bars tall and to the right? No, they look\Nalmost normal, not quite, but that doesn't Dialogue: 0,0:13:11.79,0:13:16.62,Default,,0000,0000,0000,,really matter. The main thing that matters\Nis that this is the shape you would expect Dialogue: 0,0:13:16.62,0:13:23.65,Default,,0000,0000,0000,,to get if you were playing a random game\Nbasically to decide what security features Dialogue: 0,0:13:23.65,0:13:27.88,Default,,0000,0000,0000,,to enable in your software. This is the\Nshape of not having a security program, is Dialogue: 0,0:13:27.88,0:13:33.54,Default,,0000,0000,0000,,my bet. That's my bet. And so what do you\Nsee? You see heavy concentration here in Dialogue: 0,0:13:33.54,0:13:38.80,Default,,0000,0000,0000,,the middle, right, that seems fair, and\Nlike it tails off. On the Samsung nothing Dialogue: 0,0:13:38.80,0:13:43.55,Default,,0000,0000,0000,,scored all that great, same on the LG.\NBoth of them are you know running their Dialogue: 0,0:13:43.55,0:13:46.64,Default,,0000,0000,0000,,respective operating systems and they're\Nbasically just inheriting whatever Dialogue: 0,0:13:46.64,0:13:51.25,Default,,0000,0000,0000,,security came from whatever open source\Nthing they forked, right. Dialogue: 0,0:13:51.25,0:13:55.00,Default,,0000,0000,0000,,So this is this is the kind of message,\Nthis right here is the kind of thing that Dialogue: 0,0:13:55.00,0:14:01.93,Default,,0000,0000,0000,,we serve to exist for. This is us\Nproducing charts showing that the current Dialogue: 0,0:14:01.93,0:14:08.02,Default,,0000,0000,0000,,practices in the not-so consumer-friendly\Nspace of running your own Linux distros Dialogue: 0,0:14:08.02,0:14:13.29,Default,,0000,0000,0000,,far exceed the products being delivered,\Ncertainly in this case in the smart TV Dialogue: 0,0:14:13.29,0:14:24.94,Default,,0000,0000,0000,,market. But I think you might agree with\Nme, it's much worse than this. So let's Dialogue: 0,0:14:24.94,0:14:28.32,Default,,0000,0000,0000,,dig into that a little bit more, I have a\Ndifferent point that I want to make about Dialogue: 0,0:14:28.32,0:14:33.96,Default,,0000,0000,0000,,that same data set - so this table here\Nthis table is again looking at the LG Dialogue: 0,0:14:33.96,0:14:39.77,Default,,0000,0000,0000,,Samsung and Gentoo Linux installations.\NAnd on this table we're just pulling out Dialogue: 0,0:14:39.77,0:14:43.84,Default,,0000,0000,0000,,some of the easy to identify security\Nfeatures you might enable in a binary Dialogue: 0,0:14:43.84,0:14:49.99,Default,,0000,0000,0000,,right. So percentage of binaries with\Naddress space layout randomization, right? Dialogue: 0,0:14:49.99,0:14:56.43,Default,,0000,0000,0000,,Let's talk about that on our Gentoo build\Nit's over 99%. That also holds for the Dialogue: 0,0:14:56.43,0:15:02.70,Default,,0000,0000,0000,,Amazon Linux AMI - it holds in Ubuntu.\NASLR is incredibly common in modern Linux. Dialogue: 0,0:15:02.70,0:15:09.29,Default,,0000,0000,0000,,And despite that, fewer than 70 percent of\Nthe binaries on the LG television had it Dialogue: 0,0:15:09.29,0:15:13.74,Default,,0000,0000,0000,,enabled. Fewer than 70 percent. And the\NSamsung was doing, you know, better than Dialogue: 0,0:15:13.74,0:15:19.78,Default,,0000,0000,0000,,that I guess, but 80 percent is a pretty\Ndisappointing when a default Linux Dialogue: 0,0:15:19.78,0:15:25.19,Default,,0000,0000,0000,,install, you know, mainstream Linux distro\Nis going to get you 99, right? And it only Dialogue: 0,0:15:25.19,0:15:28.08,Default,,0000,0000,0000,,gets worse, it only gets worse right you\Nknow? Dialogue: 0,0:15:28.08,0:15:32.38,Default,,0000,0000,0000,,RELRO support, if you don't know what that\Nis that's ok but if you do, look abysmal Dialogue: 0,0:15:32.38,0:15:37.81,Default,,0000,0000,0000,,coverage look at this abysmal coverage\Ncoming out of these IOT devices very sad. Dialogue: 0,0:15:37.81,0:15:40.75,Default,,0000,0000,0000,,And you see it over and over and over\Nagain. I'm showing this because some Dialogue: 0,0:15:40.75,0:15:46.34,Default,,0000,0000,0000,,people in this room or watching this video\Nship software - and I have a message, I Dialogue: 0,0:15:46.34,0:15:50.31,Default,,0000,0000,0000,,have a message to those people who ship\Nsoftware who aren't working on say Chrome Dialogue: 0,0:15:50.31,0:15:58.61,Default,,0000,0000,0000,,or any of the other big-name Pwn2Own kinds\Nof targets. Look at this: you can be Dialogue: 0,0:15:58.61,0:16:02.48,Default,,0000,0000,0000,,leading the pack by mastering the\Nfundamentals. You can be leading the pack Dialogue: 0,0:16:02.48,0:16:07.08,Default,,0000,0000,0000,,by mastering the fundamentals. This is a\Npoint that really as a security field we Dialogue: 0,0:16:07.08,0:16:11.18,Default,,0000,0000,0000,,really need to be driving home. You know,\None of the things that we're seeing here Dialogue: 0,0:16:11.18,0:16:15.71,Default,,0000,0000,0000,,in our data is that if you're the vendor\Nwho is shipping the product everyone has Dialogue: 0,0:16:15.71,0:16:19.39,Default,,0000,0000,0000,,heard of in the security field and maybe\Nyour game is pretty decent right? If Dialogue: 0,0:16:19.39,0:16:23.60,Default,,0000,0000,0000,,you're shipping say Windows or if you're\Nshipping Firefox or whatever. But if Dialogue: 0,0:16:23.60,0:16:26.15,Default,,0000,0000,0000,,you're if you're doing one of these things\Nwhere people are just kind of beating you Dialogue: 0,0:16:26.15,0:16:30.62,Default,,0000,0000,0000,,up for default passwords, then your\Nproblems are way further than just default Dialogue: 0,0:16:30.62,0:16:35.40,Default,,0000,0000,0000,,passwords, right? Like the house, the\Nhouse is messy it needs to be cleaned, Dialogue: 0,0:16:35.40,0:16:43.19,Default,,0000,0000,0000,,needs to be cleaned. So the rest of the\Ntalk like I said we're going to be Dialogue: 0,0:16:43.19,0:16:47.02,Default,,0000,0000,0000,,discussing a lot of other things that\Namount to getting you know a peek behind Dialogue: 0,0:16:47.02,0:16:50.69,Default,,0000,0000,0000,,the curtain and where some of these things\Ncome from and getting very specific about Dialogue: 0,0:16:50.69,0:16:54.42,Default,,0000,0000,0000,,how this business works, but if you're\Ninterested in more of the high level Dialogue: 0,0:16:54.42,0:16:58.98,Default,,0000,0000,0000,,material - especially if you're interested\Nin interesting results and insights, some Dialogue: 0,0:16:58.98,0:17:01.95,Default,,0000,0000,0000,,of which I'm going to have here later. But\NI really encourage you though to take a Dialogue: 0,0:17:01.95,0:17:06.75,Default,,0000,0000,0000,,look at the talk from this past summer by\Nour chief scientist Sarah Zatko, which is Dialogue: 0,0:17:06.75,0:17:11.22,Default,,0000,0000,0000,,predominantly on the topic of surprising\Nresults in the data. Dialogue: 0,0:17:14.89,0:17:18.54,Default,,0000,0000,0000,,Today, though, this being our first time\Npresenting here in Europe, we figured we Dialogue: 0,0:17:18.54,0:17:22.87,Default,,0000,0000,0000,,would take more of an overarching kind of\Nview. What we're doing and why we're Dialogue: 0,0:17:22.87,0:17:26.62,Default,,0000,0000,0000,,excited about it and where it's headed. So\Nwe're about to move into a little bit of Dialogue: 0,0:17:26.62,0:17:31.60,Default,,0000,0000,0000,,the underlying theory, you know. Why do I\Nthink it's reasonable to even try to Dialogue: 0,0:17:31.60,0:17:35.43,Default,,0000,0000,0000,,measure the security of software from a\Ntechnical perspective. But before we can Dialogue: 0,0:17:35.43,0:17:39.31,Default,,0000,0000,0000,,get into that I need to talk a little bit\Nabout our goals, so that the decisions and Dialogue: 0,0:17:39.31,0:17:45.38,Default,,0000,0000,0000,,the theory; the motivation is clear,\Nright. Our goals are really simple: it's a Dialogue: 0,0:17:45.38,0:17:51.40,Default,,0000,0000,0000,,very easy organization to run because of\Nthat. Goal number one: remain independent Dialogue: 0,0:17:51.40,0:17:56.26,Default,,0000,0000,0000,,of vendor influence. We are not the first\Norganization to purport to be looking out Dialogue: 0,0:17:56.26,0:18:02.47,Default,,0000,0000,0000,,for the consumer. But unlike many of our\Npredecessors, we are not taking money from Dialogue: 0,0:18:02.47,0:18:09.92,Default,,0000,0000,0000,,the people we review, right? Seems like\Nsome basic stuff. Seems like some basic Dialogue: 0,0:18:09.92,0:18:17.54,Default,,0000,0000,0000,,stuff right? Thank you, okay.\NTwo: automated, comparable, quantitative Dialogue: 0,0:18:17.54,0:18:23.79,Default,,0000,0000,0000,,analysis. Why automated? Well, we need our\Ntest results to be reproducible. And Tim Dialogue: 0,0:18:23.79,0:18:27.72,Default,,0000,0000,0000,,goes in opens up your software in IDA and\Nfinds a bunch of stuff that makes them all Dialogue: 0,0:18:27.72,0:18:32.62,Default,,0000,0000,0000,,stoped - that's not a very repeatable kind\Nof a standard for things. And so we're Dialogue: 0,0:18:32.62,0:18:36.44,Default,,0000,0000,0000,,interested in things which are automated.\NWe'll talk about, maybe a few hackers in Dialogue: 0,0:18:36.44,0:18:39.94,Default,,0000,0000,0000,,here know how hard that is. We'll talk\Nabout that, but then last we also we're Dialogue: 0,0:18:39.94,0:18:43.54,Default,,0000,0000,0000,,well acting as a watchdog - we're\Nprotecting the interests of the user, the Dialogue: 0,0:18:43.54,0:18:47.63,Default,,0000,0000,0000,,consumer, however you would like to look\Nat it. But we also have three non goals, Dialogue: 0,0:18:47.63,0:18:52.51,Default,,0000,0000,0000,,three non goals that are equally\Nimportant. One: we have a non goal of Dialogue: 0,0:18:52.51,0:18:56.86,Default,,0000,0000,0000,,finding and disclosing vulnerabilities. I\Nreserve the right to find and disclose Dialogue: 0,0:18:56.86,0:19:01.37,Default,,0000,0000,0000,,vulnerabilities. But that's not my goal,\Nit's not my goal. Another non goal is to Dialogue: 0,0:19:01.37,0:19:04.84,Default,,0000,0000,0000,,tell software vendors what to do. If a\Nvendor asks me how to remediate their Dialogue: 0,0:19:04.84,0:19:08.50,Default,,0000,0000,0000,,terrible score, I will tell them what we\Nare measuring but I'm not there to help Dialogue: 0,0:19:08.50,0:19:11.95,Default,,0000,0000,0000,,them remediate. It's on them to be able to\Nship a secure product without me holding Dialogue: 0,0:19:11.95,0:19:19.05,Default,,0000,0000,0000,,their hand. We'll see. And then three:\Nnon-goal, perform free security testing Dialogue: 0,0:19:19.05,0:19:24.09,Default,,0000,0000,0000,,for vendors. Our testing happens after you\Nrelease. Because when you release your Dialogue: 0,0:19:24.09,0:19:28.98,Default,,0000,0000,0000,,software you are telling people it is\Nready to be used. Is it really though, is Dialogue: 0,0:19:28.98,0:19:31.80,Default,,0000,0000,0000,,it really though, right?\N{\i1}Applause{\i0} Dialogue: 0,0:19:31.80,0:19:37.31,Default,,0000,0000,0000,,Yeah, thank you. Yeah, so we are not there\Nto give you a preview of what your score Dialogue: 0,0:19:37.31,0:19:42.27,Default,,0000,0000,0000,,will be. There is no sum of money you can\Nhand me that will get you an early preview Dialogue: 0,0:19:42.27,0:19:46.17,Default,,0000,0000,0000,,of what your score is - you can try me,\Nyou can try me: there's a fee for trying Dialogue: 0,0:19:46.17,0:19:50.26,Default,,0000,0000,0000,,me. There's a fee for trying me. But I'm\Nnot gonna look at your stuff until I'm Dialogue: 0,0:19:50.26,0:19:58.55,Default,,0000,0000,0000,,ready to drop it, right. Yeah bitte, yeah.\NAll right. So moving into this theory Dialogue: 0,0:19:58.55,0:20:02.77,Default,,0000,0000,0000,,territory. Three big questions, three big\Nquestions that need to be addressed if you Dialogue: 0,0:20:02.77,0:20:06.99,Default,,0000,0000,0000,,want to do our work efficiently: what\Nworks, what works for improving security, Dialogue: 0,0:20:06.99,0:20:13.03,Default,,0000,0000,0000,,what are the things that need or that you\Nreally want to see in software. Two: how Dialogue: 0,0:20:13.03,0:20:17.12,Default,,0000,0000,0000,,do you recognize when it's being done?\NIt's no good if someone hands you a piece Dialogue: 0,0:20:17.12,0:20:20.17,Default,,0000,0000,0000,,of software and says, "I've done all the\Nlatest things" and it's a complete black Dialogue: 0,0:20:20.17,0:20:24.53,Default,,0000,0000,0000,,box. If you can't check the claim, the\Nclaim is as good as false, in practical Dialogue: 0,0:20:24.53,0:20:30.21,Default,,0000,0000,0000,,terms, period, right. Software has to be\Nreviewable or a priori, I'll think you're Dialogue: 0,0:20:30.21,0:20:35.73,Default,,0000,0000,0000,,full of it. And then three: who's doing it\N- of all the things that work, that you Dialogue: 0,0:20:35.73,0:20:39.82,Default,,0000,0000,0000,,can recognize, who's actually doing it.\NYou know, let's go ahead - our field is Dialogue: 0,0:20:39.82,0:20:47.43,Default,,0000,0000,0000,,famous for ruining people's holidays and\Nweekends over Friday bug disclosures, you Dialogue: 0,0:20:47.43,0:20:51.80,Default,,0000,0000,0000,,know New Year's Eve bug disclosures. I\Nwould like us to also be famous for Dialogue: 0,0:20:51.80,0:20:59.25,Default,,0000,0000,0000,,calling out those teams and those software\Norganizations which are being as good as Dialogue: 0,0:20:59.25,0:21:04.24,Default,,0000,0000,0000,,the bad guys are being bad, yeah? So\Nprovide someone an incentive to be maybe Dialogue: 0,0:21:04.24,0:21:19.46,Default,,0000,0000,0000,,happy to see us for a change, right. Okay,\Nso thank you. Yeah, all right. So how do Dialogue: 0,0:21:19.46,0:21:26.12,Default,,0000,0000,0000,,we actually pull these things off; the\Nbasic idea. So, I'm going to get into some Dialogue: 0,0:21:26.12,0:21:29.47,Default,,0000,0000,0000,,deeper theory: if you're not a theorist I\Nwant you to focus on this slide. Dialogue: 0,0:21:29.47,0:21:33.43,Default,,0000,0000,0000,,And I'm gonna bring it back, it's not all\Ntheory from here on out after this but if Dialogue: 0,0:21:33.43,0:21:39.29,Default,,0000,0000,0000,,you're not a theorist I really want you to\Nfocus on this slide. The basic motivation, Dialogue: 0,0:21:39.29,0:21:42.56,Default,,0000,0000,0000,,the basic motivation behind what we're\Ndoing; the technical motivation - why we Dialogue: 0,0:21:42.56,0:21:47.02,Default,,0000,0000,0000,,think that it's possible to measure and\Nreport on security. It all boils down to Dialogue: 0,0:21:47.02,0:21:53.02,Default,,0000,0000,0000,,this right. So we start with a thought\Nexperiment, a gedankent, right? Given a Dialogue: 0,0:21:53.02,0:21:58.65,Default,,0000,0000,0000,,piece of software we can ask: overall, how\Nsecure is it? Kind of a vague question but Dialogue: 0,0:21:58.65,0:22:03.00,Default,,0000,0000,0000,,you could imagine you know there's\Nversions of that question. And two: what Dialogue: 0,0:22:03.00,0:22:07.82,Default,,0000,0000,0000,,are its vulnerabilities. Maybe you want to\Nnitpick with me about what the word Dialogue: 0,0:22:07.82,0:22:11.24,Default,,0000,0000,0000,,vulnerability means but broadly you know\Nthis is a much more specific question Dialogue: 0,0:22:11.24,0:22:18.85,Default,,0000,0000,0000,,right. And here's here's the enticing\Nthing: the first question appears to ask Dialogue: 0,0:22:18.85,0:22:24.93,Default,,0000,0000,0000,,for less information than the second\Nquestion. And maybe if we were taking bets Dialogue: 0,0:22:24.93,0:22:28.52,Default,,0000,0000,0000,,I would put my money on, yes, it actually\Ndoes ask for less information. What do I Dialogue: 0,0:22:28.52,0:22:33.24,Default,,0000,0000,0000,,mean by that what do I mean by that? Well,\Nlet's say that someone told you all of the Dialogue: 0,0:22:33.24,0:22:38.39,Default,,0000,0000,0000,,vulnerabilities in a system right? They\Nsaid, "Hey, I got them all", right? You're Dialogue: 0,0:22:38.39,0:22:41.58,Default,,0000,0000,0000,,like all right that's cool, that's cool.\NAnd if someone asks you hey how secure is Dialogue: 0,0:22:41.58,0:22:45.44,Default,,0000,0000,0000,,this system you can give them a very\Nprecise answer. You can say it has N Dialogue: 0,0:22:45.44,0:22:48.62,Default,,0000,0000,0000,,vulnerabilities, and they're of this kind\Nand like all this stuff right so certainly Dialogue: 0,0:22:48.62,0:22:54.63,Default,,0000,0000,0000,,the second question is enough to answer\Nthe first. But, is the reverse true? Dialogue: 0,0:22:54.63,0:22:58.47,Default,,0000,0000,0000,,Namely, if someone were to tell you, for\Nexample, "hey, this piece of software has Dialogue: 0,0:22:58.47,0:23:06.21,Default,,0000,0000,0000,,exactly 32 vulnerabilities in it." Does\Nthat make it easier to find any of them? Dialogue: 0,0:23:06.21,0:23:12.32,Default,,0000,0000,0000,,Right, there's room for to maybe do that\Nusing some algorithms that are not yet in Dialogue: 0,0:23:12.32,0:23:15.84,Default,,0000,0000,0000,,existence.\NCertainly the computer scientists in here Dialogue: 0,0:23:15.84,0:23:19.45,Default,,0000,0000,0000,,are saying, "well, you know, yeah maybe\Ncounting the number of SAT solutions Dialogue: 0,0:23:19.45,0:23:22.70,Default,,0000,0000,0000,,doesn't help you practically find\Nsolutions. But it might and we just don't Dialogue: 0,0:23:22.70,0:23:27.12,Default,,0000,0000,0000,,know." Okay fine fine fine. Maybe these\Nthings are the same, but the my experience Dialogue: 0,0:23:27.12,0:23:30.41,Default,,0000,0000,0000,,in security, and the experience of many\Nothers perhaps is that they probably Dialogue: 0,0:23:30.41,0:23:36.51,Default,,0000,0000,0000,,aren't the same question. And this\Nmotivates what I'm calling here is Zatko's Dialogue: 0,0:23:36.51,0:23:40.87,Default,,0000,0000,0000,,question, which is basically asking for an\Nalgorithm that demonstrates that the first Dialogue: 0,0:23:40.87,0:23:45.97,Default,,0000,0000,0000,,question is easier than the second\Nquestion, right. So Zatko's question: Dialogue: 0,0:23:45.97,0:23:49.36,Default,,0000,0000,0000,,develop a heuristic which can to\Nefficiently answer one, but not Dialogue: 0,0:23:49.36,0:23:53.55,Default,,0000,0000,0000,,necessarily two. If you're looking for a\Nmetaphor, if you want to know why I care Dialogue: 0,0:23:53.55,0:23:56.64,Default,,0000,0000,0000,,about this distinction, I want you to\Nthink about some certain controversial Dialogue: 0,0:23:56.64,0:24:00.99,Default,,0000,0000,0000,,technologies: maybe think about say\Nnuclear technology, right. An algorithm Dialogue: 0,0:24:00.99,0:24:04.53,Default,,0000,0000,0000,,that answers one, but not two, it's a very\Nsafe algorithm to publish. Very safe Dialogue: 0,0:24:04.53,0:24:11.37,Default,,0000,0000,0000,,algorithm publish indeed. Okay, Claude\NShannon would like more information. happy Dialogue: 0,0:24:11.37,0:24:16.04,Default,,0000,0000,0000,,to oblige. Let's take a look at this\Nquestion from a different perspective Dialogue: 0,0:24:16.04,0:24:19.38,Default,,0000,0000,0000,,maybe a more hands-on perspective: the\Nhacker perspective, right? If you're a Dialogue: 0,0:24:19.38,0:24:22.39,Default,,0000,0000,0000,,hacker and you're watching me up here and\NI'm waving my hands around and I'm showing Dialogue: 0,0:24:22.39,0:24:25.93,Default,,0000,0000,0000,,you charts maybe you're thinking to\Nyourself yeah boy, what do you got? Right, Dialogue: 0,0:24:25.93,0:24:29.73,Default,,0000,0000,0000,,how does this actually go. And maybe what\Nyou're thinking to yourself is that, you Dialogue: 0,0:24:29.73,0:24:34.35,Default,,0000,0000,0000,,know, finding good vulns: that's an\Nartisan craft right? You're in IDA, you Dialogue: 0,0:24:34.35,0:24:37.25,Default,,0000,0000,0000,,know you're reversing old way you're doing\Nall these things or hit and Comm, I don't Dialogue: 0,0:24:37.25,0:24:41.43,Default,,0000,0000,0000,,know all that stuff. And like, you know,\Nthis kind of clever game; cleverness is Dialogue: 0,0:24:41.43,0:24:47.21,Default,,0000,0000,0000,,not like this thing that feels very\Nautomatable. But you know on the other Dialogue: 0,0:24:47.21,0:24:51.36,Default,,0000,0000,0000,,hand there are a lot of tools that do\Nautomate things and so it's not completely Dialogue: 0,0:24:51.36,0:24:57.11,Default,,0000,0000,0000,,not automatable.\NAnd if you're into fuzzing then perhaps Dialogue: 0,0:24:57.11,0:25:01.50,Default,,0000,0000,0000,,you are aware of this very simple\Nobservation, which is that if your harness Dialogue: 0,0:25:01.50,0:25:04.94,Default,,0000,0000,0000,,is perfect if you really know what you're\Ndoing if you have a decent fuzzer then in Dialogue: 0,0:25:04.94,0:25:08.84,Default,,0000,0000,0000,,principle fuzzing can find every single\Nproblem. You have to be able to look for Dialogue: 0,0:25:08.84,0:25:13.87,Default,,0000,0000,0000,,it you have to be able harness for it but\Nin principle it will, right. So the hacker Dialogue: 0,0:25:13.87,0:25:19.21,Default,,0000,0000,0000,,perspective on Zatko's question is maybe\Nof two minds on the one hand assessing Dialogue: 0,0:25:19.21,0:25:22.40,Default,,0000,0000,0000,,security is a game of cleverness, but on\Nthe other hand we're kind of right now at Dialogue: 0,0:25:22.40,0:25:25.88,Default,,0000,0000,0000,,the cusp of having some game-changing tech\Nreally go - maybe you're saying like Dialogue: 0,0:25:25.88,0:25:29.58,Default,,0000,0000,0000,,fuzzing is not at the cusp, I promise it's\Njust at the cusp. We haven't seen all the Dialogue: 0,0:25:29.58,0:25:33.69,Default,,0000,0000,0000,,fuzzing has to offer right and so maybe\Nthere's room maybe there's room for some Dialogue: 0,0:25:33.69,0:25:41.20,Default,,0000,0000,0000,,automation to be possible in pursuit of\NZatko's question. Of course, there are Dialogue: 0,0:25:41.20,0:25:45.92,Default,,0000,0000,0000,,many challenges still in, you know, using\Nexisting hacker technology. Mostly of the Dialogue: 0,0:25:45.92,0:25:49.57,Default,,0000,0000,0000,,form of various open questions. For\Nexample if you're into fuzzing, you know, Dialogue: 0,0:25:49.57,0:25:53.04,Default,,0000,0000,0000,,hey: identifying unique crashes. There's\Nan open question. We'll talk about some of Dialogue: 0,0:25:53.04,0:25:57.06,Default,,0000,0000,0000,,those, we'll talk about some of those. But\NI'm going to offer another perspective Dialogue: 0,0:25:57.06,0:26:01.49,Default,,0000,0000,0000,,here: so maybe you're not in the business\Nof doing software reviews but you know a Dialogue: 0,0:26:01.49,0:26:05.93,Default,,0000,0000,0000,,little computer science. And maybe that\Ncomputer science has you wondering what's Dialogue: 0,0:26:05.93,0:26:12.68,Default,,0000,0000,0000,,this guy talking about, right. I'm here to\Nacknowledge that. So whatever you think Dialogue: 0,0:26:12.68,0:26:16.61,Default,,0000,0000,0000,,the word security means: I've got a list\Nof questions up here. Whatever you think Dialogue: 0,0:26:16.61,0:26:19.50,Default,,0000,0000,0000,,the word security means, probably, some of\Nthese questions are relevant to your Dialogue: 0,0:26:19.50,0:26:23.30,Default,,0000,0000,0000,,definition. Right.\NDoes the software have a hidden backdoor Dialogue: 0,0:26:23.30,0:26:26.60,Default,,0000,0000,0000,,or any kind of hidden functionality, does\Nit handle crypto material correctly, etc, Dialogue: 0,0:26:26.60,0:26:30.43,Default,,0000,0000,0000,,so forth. Anyone in here who knows some\Ncomputers abilities theory knows that Dialogue: 0,0:26:30.43,0:26:34.24,Default,,0000,0000,0000,,every single one of these questions and\Nmany others like them are undecidable due Dialogue: 0,0:26:34.24,0:26:37.96,Default,,0000,0000,0000,,to reasons essentially no different than\Nthe reason the halting problem is Dialogue: 0,0:26:37.96,0:26:41.33,Default,,0000,0000,0000,,undecidable,\ which is to say due to\Nreasons essentially first identified and Dialogue: 0,0:26:41.33,0:26:46.02,Default,,0000,0000,0000,,studied by Alan Turing a long time before\Nwe had microarchitectures and all these Dialogue: 0,0:26:46.02,0:26:50.35,Default,,0000,0000,0000,,other things. And so, the computability\Nperspective says that, you know, whatever Dialogue: 0,0:26:50.35,0:26:54.64,Default,,0000,0000,0000,,your definition of security is ultimately\Nyou have this recognizability problem: Dialogue: 0,0:26:54.64,0:26:57.90,Default,,0000,0000,0000,,fancy way of saying that algorithms won't\Nbe able to recognize secure software Dialogue: 0,0:26:57.90,0:27:02.69,Default,,0000,0000,0000,,because of the undecidability these\Nissues. The takeaway, the takeaway is that Dialogue: 0,0:27:02.69,0:27:07.09,Default,,0000,0000,0000,,the computability angle on all of this\Nsays: anyone who's in the business that Dialogue: 0,0:27:07.09,0:27:12.39,Default,,0000,0000,0000,,we're in has to use heuristics. You have\Nto, you have to. Dialogue: 0,0:27:15.01,0:27:24.85,Default,,0000,0000,0000,,All right, this guy gets it. All right, so\Non the tech side our last technical Dialogue: 0,0:27:24.85,0:27:28.38,Default,,0000,0000,0000,,perspective that we're going to take now\Nis certainly the most abstract which is Dialogue: 0,0:27:28.38,0:27:32.22,Default,,0000,0000,0000,,the Bayesian perspective, right. So if\Nyou're a frequentist, you need to get with Dialogue: 0,0:27:32.22,0:27:37.49,Default,,0000,0000,0000,,the times you know it's everything\NBayesian now. So, let's talk about this Dialogue: 0,0:27:37.49,0:27:43.90,Default,,0000,0000,0000,,for a bit. Only two slides of math, I\Npromise, only two! So, let's say that I Dialogue: 0,0:27:43.90,0:27:47.12,Default,,0000,0000,0000,,have some corpus of software. Perhaps it's\Na collection of all modern browsers, Dialogue: 0,0:27:47.12,0:27:50.51,Default,,0000,0000,0000,,perhaps it's the collection of all the\Npackages in the Debian repository, perhaps Dialogue: 0,0:27:50.51,0:27:53.99,Default,,0000,0000,0000,,it's everything on github that builds on\Nthis system, perhaps it's a hard drive Dialogue: 0,0:27:53.99,0:27:58.16,Default,,0000,0000,0000,,full of warez that some guy mailed you,\Nright? You have some corpus of software Dialogue: 0,0:27:58.16,0:28:02.98,Default,,0000,0000,0000,,and for a random program in that corpus we\Ncan consider this probability: the Dialogue: 0,0:28:02.98,0:28:07.18,Default,,0000,0000,0000,,probability distribution of which software\Nis secure versus which is not. For reasons Dialogue: 0,0:28:07.18,0:28:11.08,Default,,0000,0000,0000,,described on the computability\Nperspective, this number is not a Dialogue: 0,0:28:11.08,0:28:17.13,Default,,0000,0000,0000,,computable number for any reasonable\Ndefinition of security. So that's a neat Dialogue: 0,0:28:17.13,0:28:21.22,Default,,0000,0000,0000,,and so, for practical terms, if you want\Nto do some probabilistic reasoning, you Dialogue: 0,0:28:21.22,0:28:27.51,Default,,0000,0000,0000,,need some surrogate for that and so we\Nconsider this here. So, instead of Dialogue: 0,0:28:27.51,0:28:31.00,Default,,0000,0000,0000,,considering the probability that a piece\Nof software is secure, a non computable Dialogue: 0,0:28:31.00,0:28:35.96,Default,,0000,0000,0000,,non verifiable claim, we take a look here\Nat this indexed collection of Dialogue: 0,0:28:35.96,0:28:38.84,Default,,0000,0000,0000,,probabilities. This is an infinite\Ncountable family of probability Dialogue: 0,0:28:38.84,0:28:44.33,Default,,0000,0000,0000,,distributions, basically P sub h,k is just\Nthe probability that for a random piece of Dialogue: 0,0:28:44.33,0:28:50.33,Default,,0000,0000,0000,,software in the corpus, h work units of\Nfuzzing will find no more than k unique Dialogue: 0,0:28:50.33,0:28:56.13,Default,,0000,0000,0000,,crashes, right. And why is this relevant?\NWell, at the bottom we have this analytic Dialogue: 0,0:28:56.13,0:28:59.39,Default,,0000,0000,0000,,observation, which is that in the limit as\Nh goes to infinity you're basically Dialogue: 0,0:28:59.39,0:29:03.68,Default,,0000,0000,0000,,saying: "Hey, you know, if I fuzz this\Nthing for infinity times, you know, what's Dialogue: 0,0:29:03.68,0:29:07.55,Default,,0000,0000,0000,,that look like?" And, essentially, here we\Nhave analytically that this should Dialogue: 0,0:29:07.55,0:29:12.97,Default,,0000,0000,0000,,converge. The P sub h,1 should converge to\Nthe probability that a piece of software Dialogue: 0,0:29:12.97,0:29:16.33,Default,,0000,0000,0000,,just simply cannot be made to crash. Not\Nthe same thing as being secure, but Dialogue: 0,0:29:16.33,0:29:23.73,Default,,0000,0000,0000,,certainly not a small concern relevant to\Nsecurity. So, none of that stuff actually Dialogue: 0,0:29:23.73,0:29:30.62,Default,,0000,0000,0000,,was Bayesian yet, so we need to get there.\NAnd so here we go, right: so, the previous Dialogue: 0,0:29:30.62,0:29:35.08,Default,,0000,0000,0000,,slide described a probability distribution\Nmeasured based on fuzzing. But fuzzing is Dialogue: 0,0:29:35.08,0:29:39.13,Default,,0000,0000,0000,,expensive and it is also not an answer to\NZatko's question because it finds Dialogue: 0,0:29:39.13,0:29:43.76,Default,,0000,0000,0000,,vulnerabilities, it doesn't measure\Nsecurity in the general sense and so Dialogue: 0,0:29:43.76,0:29:47.11,Default,,0000,0000,0000,,here's where we make the jump to\Nconditional probabilities: Let M be some Dialogue: 0,0:29:47.11,0:29:51.93,Default,,0000,0000,0000,,observable property of software has ASLR,\Nhas RELRO, calls these functions, doesn't Dialogue: 0,0:29:51.93,0:29:56.77,Default,,0000,0000,0000,,call those functions... take your pick.\NFor random s in S we now consider these Dialogue: 0,0:29:56.77,0:30:02.07,Default,,0000,0000,0000,,conditional probability distributions and\Nthis is the same kind of probability as we Dialogue: 0,0:30:02.07,0:30:08.38,Default,,0000,0000,0000,,had on the previous slide but conditioned\Non this observable being true, and this Dialogue: 0,0:30:08.38,0:30:11.48,Default,,0000,0000,0000,,leads to the refined of the Siddall\Nvariant of Zatko's question: Dialogue: 0,0:30:11.48,0:30:17.12,Default,,0000,0000,0000,,Which observable properties of software\Nsatisfy that, when the software has Dialogue: 0,0:30:17.12,0:30:22.59,Default,,0000,0000,0000,,property m, the probability of fuzzing\Nbeing hard is very high? That's what this Dialogue: 0,0:30:22.59,0:30:27.12,Default,,0000,0000,0000,,version of this question phrases, and here\Nwe say, you know, in large log(h)/k, in Dialogue: 0,0:30:27.12,0:30:31.59,Default,,0000,0000,0000,,other words: exponentially more fuzzing\Nthan you expect to find bugs. So this is Dialogue: 0,0:30:31.59,0:30:36.35,Default,,0000,0000,0000,,the technical version of what we're after.\NAll of this can be explored, you can Dialogue: 0,0:30:36.35,0:30:40.34,Default,,0000,0000,0000,,brute-force your way to finding all of\Nthis stuff, and that's exactly what we're Dialogue: 0,0:30:40.34,0:30:48.05,Default,,0000,0000,0000,,doing. So we're looking for all kinds of\Nthings, we're looking for all kinds of Dialogue: 0,0:30:48.05,0:30:53.84,Default,,0000,0000,0000,,things that correlate with fuzzing having\Nlow yield on a piece of software, and Dialogue: 0,0:30:53.84,0:30:57.36,Default,,0000,0000,0000,,there's a lot of ways in which that can\Nhappen. It could be that you are looking Dialogue: 0,0:30:57.36,0:31:01.41,Default,,0000,0000,0000,,at a feature of software that literally\Nprevents crashes. Maybe it's the never Dialogue: 0,0:31:01.41,0:31:08.21,Default,,0000,0000,0000,,crash flag, I don't know. But most of the\Nthings I've talked about, ASLR, RERO, etc. Dialogue: 0,0:31:08.21,0:31:12.17,Default,,0000,0000,0000,,don't prevent crashes. In fact a ASLR can\Ntake non-crashing programs and make them Dialogue: 0,0:31:12.17,0:31:16.85,Default,,0000,0000,0000,,crashing. It's the number one reason\Nvendors don't enable it, right? So why am Dialogue: 0,0:31:16.85,0:31:20.08,Default,,0000,0000,0000,,I talking about ASLR? Why am I talking\Nabout RELRO? Why am i talking about all Dialogue: 0,0:31:20.08,0:31:22.90,Default,,0000,0000,0000,,these things that have nothing to do with\Nstopping crashes and I'm claiming I'm Dialogue: 0,0:31:22.90,0:31:27.40,Default,,0000,0000,0000,,measuring crashes? This is because, in the\NBayesian perspective, correlation is not Dialogue: 0,0:31:27.40,0:31:31.73,Default,,0000,0000,0000,,the same thing as causation, right?\NCorrelation is not the same thing as Dialogue: 0,0:31:31.73,0:31:35.34,Default,,0000,0000,0000,,causation. It could be that M's presence\Nliterally prevents crashes, but it could Dialogue: 0,0:31:35.34,0:31:39.75,Default,,0000,0000,0000,,also be that, by some underlying\Ncoincidence, the things we're looking for Dialogue: 0,0:31:39.75,0:31:43.60,Default,,0000,0000,0000,,are mostly only found in software that's\Nrobust against crashing. Dialogue: 0,0:31:43.60,0:31:48.80,Default,,0000,0000,0000,,If you're looking for security, I submit\Nto you that the difference doesn't matter. Dialogue: 0,0:31:48.80,0:31:54.93,Default,,0000,0000,0000,,Okay, end of my math, danke. I will now go\Nahead and do this like really nice analogy Dialogue: 0,0:31:54.93,0:31:59.28,Default,,0000,0000,0000,,of all those things that I just described,\Nright. So we're looking for indicators of Dialogue: 0,0:31:59.28,0:32:03.64,Default,,0000,0000,0000,,a piece of software being secure enough to\Nbe good for consumers, right. So here's an Dialogue: 0,0:32:03.64,0:32:08.13,Default,,0000,0000,0000,,analogy. Let's say you're a geologist, you\Nstudy minerals and all of that and you're Dialogue: 0,0:32:08.13,0:32:13.56,Default,,0000,0000,0000,,looking for diamonds. Who isn't, right?\NWant those diamonds! And like how do you Dialogue: 0,0:32:13.56,0:32:18.27,Default,,0000,0000,0000,,find diamonds? Even in places that are\Nrich in diamonds, diamonds are not common. Dialogue: 0,0:32:18.27,0:32:21.28,Default,,0000,0000,0000,,You don't just go walking around in your\Nboots, kicking until your toe stubs on a Dialogue: 0,0:32:21.28,0:32:27.05,Default,,0000,0000,0000,,diamond? You don't do that. Instead you\Nlook for other minerals that are mostly Dialogue: 0,0:32:27.05,0:32:31.86,Default,,0000,0000,0000,,only found near diamonds but are much more\Nabundant in those locations than the Dialogue: 0,0:32:31.86,0:32:37.96,Default,,0000,0000,0000,,diamonds. So, this is mineral science 101,\NI guess, I don't know. So, for example, Dialogue: 0,0:32:37.96,0:32:41.39,Default,,0000,0000,0000,,you want to go find diamond: put on your\Nboots and go kicking until you find some Dialogue: 0,0:32:41.39,0:32:46.11,Default,,0000,0000,0000,,chromite, look for some diopside, you\Nknow, look for some garnet. None of these Dialogue: 0,0:32:46.11,0:32:50.34,Default,,0000,0000,0000,,things turn into diamonds, none of these\Nthings cause diamonds but if you're Dialogue: 0,0:32:50.34,0:32:54.02,Default,,0000,0000,0000,,finding good concentrations of these\Nthings, then, statistically, there's Dialogue: 0,0:32:54.02,0:32:58.25,Default,,0000,0000,0000,,probably diamonds nearby. That's what\Nwe're doing. We're not looking for the Dialogue: 0,0:32:58.25,0:33:02.57,Default,,0000,0000,0000,,things that cause good security per se.\NRather, we're looking for the indicators Dialogue: 0,0:33:02.57,0:33:08.35,Default,,0000,0000,0000,,that you have put the effort into your\Nsoftware, right? How's that working out Dialogue: 0,0:33:08.35,0:33:15.07,Default,,0000,0000,0000,,for us? How's that working out for us?\NWell, we're still doing studies. It's, you Dialogue: 0,0:33:15.07,0:33:18.49,Default,,0000,0000,0000,,know, early to say exactly but we do have\Nthe following interesting coincidence: and Dialogue: 0,0:33:18.49,0:33:24.79,Default,,0000,0000,0000,,so, here presented I have a collection of\Nprices that somebody gave much for so- Dialogue: 0,0:33:24.79,0:33:30.37,Default,,0000,0000,0000,,called the underground exploits. And I can\Ntell you these prices are maybe a little Dialogue: 0,0:33:30.37,0:33:34.45,Default,,0000,0000,0000,,low these days but if you work in that\Nbusiness, if you go to Cyscin, if you do Dialogue: 0,0:33:34.45,0:33:39.01,Default,,0000,0000,0000,,that kind of stuff, maybe you know that\Nthis is ballpark, it's ballpark. Dialogue: 0,0:33:39.01,0:33:44.08,Default,,0000,0000,0000,,Alright, and, just a coincidence, maybe it\Nmeans we're on the right track, I don't Dialogue: 0,0:33:44.08,0:33:48.74,Default,,0000,0000,0000,,know, but it's an encouraging sign: When\Nwe run these programs through our Dialogue: 0,0:33:48.74,0:33:53.06,Default,,0000,0000,0000,,analysis, our rankings more or less\Ncorrespond to the actual prices that you Dialogue: 0,0:33:53.06,0:33:58.28,Default,,0000,0000,0000,,encounter in the wild for access via these\Napplications. Up above, I have one of our Dialogue: 0,0:33:58.28,0:34:02.06,Default,,0000,0000,0000,,histogram charts. You can see here that\NChrome and Edge in this particular model Dialogue: 0,0:34:02.06,0:34:06.15,Default,,0000,0000,0000,,scored very close to the same and it's a\Ntest model, so, let's say they're Dialogue: 0,0:34:06.15,0:34:11.48,Default,,0000,0000,0000,,basically the same.\NFirefox, you know, behind there a little Dialogue: 0,0:34:11.48,0:34:15.04,Default,,0000,0000,0000,,bit. I don't have Safari on this chart\Nbecause - this or all Windows applications Dialogue: 0,0:34:15.04,0:34:21.09,Default,,0000,0000,0000,,- but the Safari score falls in between.\NSo, lots of theory, lots of theory, lots Dialogue: 0,0:34:21.09,0:34:27.92,Default,,0000,0000,0000,,of theory and then we have this. So, we're\Ngoing to go ahead now and hand off to our Dialogue: 0,0:34:27.92,0:34:31.68,Default,,0000,0000,0000,,lead engineer, Parker, who is going to\Ntalk about some of the concrete stuff, the Dialogue: 0,0:34:31.68,0:34:35.21,Default,,0000,0000,0000,,non-chalkboard stuff, the software stuff\Nthat actually makes this work. Dialogue: 0,0:34:35.96,0:34:40.98,Default,,0000,0000,0000,,Thompson: Yeah, so I want to talk about\Nthe process of actually doing it. Building Dialogue: 0,0:34:40.98,0:34:45.05,Default,,0000,0000,0000,,the tooling that's required to collect\Nthese observables. Effectively, how do you Dialogue: 0,0:34:45.05,0:34:50.56,Default,,0000,0000,0000,,go mining for indicator indicator\Nminerals? But first the progression of Dialogue: 0,0:34:50.56,0:34:55.81,Default,,0000,0000,0000,,where we are and where we're going. We\Ninitially broke this out into three major Dialogue: 0,0:34:55.81,0:35:00.36,Default,,0000,0000,0000,,tracks of our technology. We have our\Nstatic analysis engine, which started as a Dialogue: 0,0:35:00.36,0:35:05.79,Default,,0000,0000,0000,,prototype, and we have now recently\Ncompleted a much more mature and solid Dialogue: 0,0:35:05.79,0:35:09.93,Default,,0000,0000,0000,,engine that's allowing us to be much more\Nextensible and digging deeper into Dialogue: 0,0:35:09.93,0:35:16.32,Default,,0000,0000,0000,,programs, and provide a much more deep\Nobservables. Then, we have the data Dialogue: 0,0:35:16.32,0:35:21.51,Default,,0000,0000,0000,,collection and data reporting. Tim showed\Nsome of our early stabs at this. We're Dialogue: 0,0:35:21.51,0:35:25.45,Default,,0000,0000,0000,,right now in the process of building new\Nengines to make the data more accessible Dialogue: 0,0:35:25.45,0:35:30.15,Default,,0000,0000,0000,,and easy to work with and hopefully more\Nof that will be available soon. Finally, Dialogue: 0,0:35:30.15,0:35:35.91,Default,,0000,0000,0000,,we have our fuzzer track. We needed to get\Nsome early data, so we played with some Dialogue: 0,0:35:35.91,0:35:40.68,Default,,0000,0000,0000,,existing off-the-shelf fuzzers, including\NAFL, and, while that was fun, Dialogue: 0,0:35:40.68,0:35:44.19,Default,,0000,0000,0000,,unfortunately it's a lot of work to\Nmanually instrument a lot of fuzzers for Dialogue: 0,0:35:44.19,0:35:48.83,Default,,0000,0000,0000,,hundreds of binaries.\NSo, we then built an automated solution Dialogue: 0,0:35:48.83,0:35:52.93,Default,,0000,0000,0000,,that started to get us closer to having a\Nfuzzing harness that could autogenerate Dialogue: 0,0:35:52.93,0:35:57.84,Default,,0000,0000,0000,,itself, depending on the software, the\Nsoftware's behavior. But, right now, Dialogue: 0,0:35:57.84,0:36:01.65,Default,,0000,0000,0000,,unfortunately that technology showed us\Nmore deficiencies than it showed Dialogue: 0,0:36:01.65,0:36:07.36,Default,,0000,0000,0000,,successes. So, we are now working on a\Nmuch more mature fuzzer that will allow us Dialogue: 0,0:36:07.36,0:36:12.78,Default,,0000,0000,0000,,to dig deeper into programs as we're\Nrunning and collect very specific things Dialogue: 0,0:36:12.78,0:36:21.26,Default,,0000,0000,0000,,that we need for our model and our\Nanalysis. But on to our analytic pipeline Dialogue: 0,0:36:21.26,0:36:25.83,Default,,0000,0000,0000,,today. This is one of the most concrete\Ncomponents of our engine and one of the Dialogue: 0,0:36:25.83,0:36:29.00,Default,,0000,0000,0000,,most fun!\NWe effectively wanted some type of Dialogue: 0,0:36:29.00,0:36:34.55,Default,,0000,0000,0000,,software hopper, where you could just pour\Nprograms in, installers and then, on the Dialogue: 0,0:36:34.55,0:36:39.56,Default,,0000,0000,0000,,other end, come reports: Fully annotated\Nand actionable information that we can Dialogue: 0,0:36:39.56,0:36:45.32,Default,,0000,0000,0000,,present to people. So, we went about the\Nprocess of building a large-scale engine. Dialogue: 0,0:36:45.32,0:36:50.50,Default,,0000,0000,0000,,It starts off with a simple REST API,\Nwhere we can push software in, which then Dialogue: 0,0:36:50.50,0:36:55.60,Default,,0000,0000,0000,,gets moved over to our computation cluster\Nthat effectively provides us a fabric to Dialogue: 0,0:36:55.60,0:37:00.31,Default,,0000,0000,0000,,work with. It makes is made up of a lot of\Ndifferent software suites, starting off Dialogue: 0,0:37:00.31,0:37:06.73,Default,,0000,0000,0000,,with our data processing, which is done by\Napache spark and then moves over into data Dialogue: 0,0:37:06.73,0:37:12.91,Default,,0000,0000,0000,,data handling and data analysis in spark,\Nand then we have a common HDFS layer to Dialogue: 0,0:37:12.91,0:37:17.53,Default,,0000,0000,0000,,provide a place for the data to be stored\Nand then a resource manager and Yarn. All Dialogue: 0,0:37:17.53,0:37:22.50,Default,,0000,0000,0000,,of that is backed by our compute and data\Nnodes, which scale out linearly. That then Dialogue: 0,0:37:22.50,0:37:27.59,Default,,0000,0000,0000,,moves into our data science engine, which\Nis effectively spark with Apache Zeppelin, Dialogue: 0,0:37:27.59,0:37:30.48,Default,,0000,0000,0000,,which provides us a really fun interface\Nwhere we can work with the data in an Dialogue: 0,0:37:30.48,0:37:35.83,Default,,0000,0000,0000,,interactive manner but be kicking off\Nlarge-scale jobs into the cluster. And Dialogue: 0,0:37:35.83,0:37:40.11,Default,,0000,0000,0000,,finally, this goes into our report\Ngeneration engine. What this bought us, Dialogue: 0,0:37:40.11,0:37:46.03,Default,,0000,0000,0000,,was the ability to linearly scale and make\Nthat hopper bigger and bigger as we need, Dialogue: 0,0:37:46.03,0:37:50.74,Default,,0000,0000,0000,,but also provide us a way to process data\Nthat doesn't fit in a single machine's Dialogue: 0,0:37:50.74,0:37:54.11,Default,,0000,0000,0000,,RAM. You can push the instance sizes as\Nyou large as you want, but we have Dialogue: 0,0:37:54.11,0:38:00.30,Default,,0000,0000,0000,,datasets that blow away any single host\NRAM set. So this allows us to work with Dialogue: 0,0:38:00.30,0:38:08.69,Default,,0000,0000,0000,,really large collections of observables.\NI want to dive down now into our actual Dialogue: 0,0:38:08.69,0:38:13.16,Default,,0000,0000,0000,,static analysis. But first we have to\Nexplore the problem space, because it's a Dialogue: 0,0:38:13.16,0:38:19.49,Default,,0000,0000,0000,,nasty one. Effectively in settles mission\Nis to process as much software as Dialogue: 0,0:38:19.49,0:38:25.79,Default,,0000,0000,0000,,possible. Hopefully all of it, but it's\Nhard to get your hand on all the binaries Dialogue: 0,0:38:25.79,0:38:29.26,Default,,0000,0000,0000,,that are out there. When you start to look\Nat that problem you understand there's a Dialogue: 0,0:38:29.26,0:38:34.83,Default,,0000,0000,0000,,lot of combinations: there's a lot of CPU\Narchitectures, there's a lot of operating Dialogue: 0,0:38:34.83,0:38:38.61,Default,,0000,0000,0000,,systems, there's a lot of file formats,\Nthere's a lot of environments the software Dialogue: 0,0:38:38.61,0:38:43.16,Default,,0000,0000,0000,,gets deployed into, and every single one\Nof them has their own app Archer app Dialogue: 0,0:38:43.16,0:38:47.32,Default,,0000,0000,0000,,armory features. And it can be\Nspecifically set for one combination Dialogue: 0,0:38:47.32,0:38:51.67,Default,,0000,0000,0000,,button on another and you don't want to\Npenalize a developer for not turning on a Dialogue: 0,0:38:51.67,0:38:56.29,Default,,0000,0000,0000,,feature they had no access to ever turn\Non. So effectively we need to solve this Dialogue: 0,0:38:56.29,0:39:01.05,Default,,0000,0000,0000,,in a much more generic way. And so what we\Ndid is our static analysis engine Dialogue: 0,0:39:01.05,0:39:04.63,Default,,0000,0000,0000,,effectively looks like a gigantic\Ncollection of abstraction libraries to Dialogue: 0,0:39:04.63,0:39:12.39,Default,,0000,0000,0000,,handle binary programs. You take in some\Ntype of input file be it ELF, PE, MachO Dialogue: 0,0:39:12.39,0:39:17.73,Default,,0000,0000,0000,,and then the pipeline splits. It goes off\Ninto two major analyzer classes, our Dialogue: 0,0:39:17.73,0:39:22.36,Default,,0000,0000,0000,,format analyzers, which look at the\Nsoftware much like how a linker or loader Dialogue: 0,0:39:22.36,0:39:26.60,Default,,0000,0000,0000,,would look at it. I want to understand how\Nit's going to be loaded up, what type of Dialogue: 0,0:39:26.60,0:39:30.68,Default,,0000,0000,0000,,armory feature is going to be applied and\Nthen we can run analyzers over that. In Dialogue: 0,0:39:30.68,0:39:34.52,Default,,0000,0000,0000,,order to achieve that we need abstraction\Nlibraries that can provide us an abstract Dialogue: 0,0:39:34.52,0:39:40.90,Default,,0000,0000,0000,,memory map, a symbol resolver, generic\Nsection properties. So all that feeds in Dialogue: 0,0:39:40.90,0:39:46.06,Default,,0000,0000,0000,,and then we run over a collection of\Nanalyzers to collect data and observables. Dialogue: 0,0:39:46.06,0:39:49.65,Default,,0000,0000,0000,,Next we have our code analyzers, these are\Nthe analyzers that run over the code Dialogue: 0,0:39:49.65,0:39:57.60,Default,,0000,0000,0000,,itself. I need to be able to look at every\Npossible executable path. In order to do Dialogue: 0,0:39:57.60,0:40:02.40,Default,,0000,0000,0000,,that we need to do function discovery,\Nfeed that into a control flow recovery Dialogue: 0,0:40:02.40,0:40:07.88,Default,,0000,0000,0000,,engine, and then as a post-processing step\Ndig through all of the possible metadata Dialogue: 0,0:40:07.88,0:40:12.82,Default,,0000,0000,0000,,in the software, such as like a switch\Ntable, or something like that to get even Dialogue: 0,0:40:12.82,0:40:20.77,Default,,0000,0000,0000,,deeper into the software. Then this\Nprovides us a basic list of basic blocks, Dialogue: 0,0:40:20.77,0:40:24.47,Default,,0000,0000,0000,,functions, instruction ranges. And does so\Nin an efficient manner so we can process a Dialogue: 0,0:40:24.47,0:40:30.55,Default,,0000,0000,0000,,lot of software as it goes. Then all that\Ngets fed over into the main modular Dialogue: 0,0:40:30.55,0:40:36.57,Default,,0000,0000,0000,,analyzers. Finally, all of this comes\Ntogether and gets put into a gigantic blob Dialogue: 0,0:40:36.57,0:40:41.85,Default,,0000,0000,0000,,of observables and fed up to the pipeline.\NWe really want to thank the Ford Dialogue: 0,0:40:41.85,0:40:46.92,Default,,0000,0000,0000,,Foundation for supporting our work in\Nthis, because the pipeline and the static Dialogue: 0,0:40:46.92,0:40:51.84,Default,,0000,0000,0000,,analysis has been a massive boon for our\Nproject and we're only beginning now to Dialogue: 0,0:40:51.84,0:40:58.92,Default,,0000,0000,0000,,really get our engine running and we're\Nhaving a great time with it. So digging Dialogue: 0,0:40:58.92,0:41:03.76,Default,,0000,0000,0000,,into the observables themselves, what are\Nwe looking at and let's break them apart. Dialogue: 0,0:41:03.76,0:41:08.98,Default,,0000,0000,0000,,So the format structure components, things\Nlike ASLR, DEP, RELRO. Dialogue: 0,0:41:08.98,0:41:13.37,Default,,0000,0000,0000,,basic app armory, that's going to go into\Nthe feature and gonna be enabled at the OS Dialogue: 0,0:41:13.37,0:41:17.83,Default,,0000,0000,0000,,layer when it gets loaded up or linked.\NAnd we also collect other metadata about Dialogue: 0,0:41:17.83,0:41:22.00,Default,,0000,0000,0000,,the program such as like: "What libraries\Nare linked in?", "What's its dependency Dialogue: 0,0:41:22.00,0:41:26.40,Default,,0000,0000,0000,,tree look like – completely?", "How did\Nthose software, how did those library Dialogue: 0,0:41:26.40,0:41:32.04,Default,,0000,0000,0000,,score?", because that can affect your main\Nsoftware. Interesting example on Linux, if Dialogue: 0,0:41:32.04,0:41:35.84,Default,,0000,0000,0000,,you link a library that requires an\Nexecutable stack, guess what your software Dialogue: 0,0:41:35.84,0:41:39.99,Default,,0000,0000,0000,,now has an executable stack, even if you\Ndidn't mark that. So we need to be owners Dialogue: 0,0:41:39.99,0:41:44.70,Default,,0000,0000,0000,,to understand what ecosystem the software\Nis gonna live in. And the code structure Dialogue: 0,0:41:44.70,0:41:47.59,Default,,0000,0000,0000,,analyzers look at things like\Nfunctionality: "What's the software Dialogue: 0,0:41:47.59,0:41:52.60,Default,,0000,0000,0000,,doing?", "What type of app armory is\Ngetting injected into the code?". A great Dialogue: 0,0:41:52.60,0:41:55.85,Default,,0000,0000,0000,,example of that is something like stack\Nguards or fortify source. These are our Dialogue: 0,0:41:55.85,0:42:01.55,Default,,0000,0000,0000,,main features that only really apply and\Ncan be observed inside of the control flow Dialogue: 0,0:42:01.55,0:42:08.24,Default,,0000,0000,0000,,or inside of the actual instructions\Nthemselves. This is why control Dialogue: 0,0:42:08.24,0:42:10.88,Default,,0000,0000,0000,,photographs are key.\NWe played around with a number of Dialogue: 0,0:42:10.88,0:42:15.98,Default,,0000,0000,0000,,different ways of analyzing software that\Nwe could scale out and ultimately we had Dialogue: 0,0:42:15.98,0:42:20.17,Default,,0000,0000,0000,,to come down to working with control\Nphotographs. Provided here is a basic Dialogue: 0,0:42:20.17,0:42:23.40,Default,,0000,0000,0000,,visualization of what I'm talking about\Nwith a control photograph, provided by Dialogue: 0,0:42:23.40,0:42:28.69,Default,,0000,0000,0000,,Benja, which has wonderful visualization\Ntools, hence this photo, and not our Dialogue: 0,0:42:28.69,0:42:33.17,Default,,0000,0000,0000,,engine because we don't build their very\Nmany visualization engines. But you Dialogue: 0,0:42:33.17,0:42:38.47,Default,,0000,0000,0000,,basically have a function that's broken up\Ninto basic blocks, which is broken up into Dialogue: 0,0:42:38.47,0:42:42.91,Default,,0000,0000,0000,,instructions, and then you have basic flow\Nbetween them. Having this as an iterable Dialogue: 0,0:42:42.91,0:42:47.65,Default,,0000,0000,0000,,structure that we can work with, allows us\Nto walk over that and walk every single Dialogue: 0,0:42:47.65,0:42:50.79,Default,,0000,0000,0000,,instruction, understand the references,\Nunderstand where code and data is being Dialogue: 0,0:42:50.79,0:42:54.50,Default,,0000,0000,0000,,referenced, and how is it being\Nreferenced. Dialogue: 0,0:42:54.50,0:42:57.64,Default,,0000,0000,0000,,And then what type of functionalities\Nbeing used, so this is a great way to find Dialogue: 0,0:42:57.64,0:43:02.53,Default,,0000,0000,0000,,something, like whether or not your stack\Nguards are being applied on every function Dialogue: 0,0:43:02.53,0:43:08.34,Default,,0000,0000,0000,,that needs them, how deep are they being\Napplied, and is the compiler possibly Dialogue: 0,0:43:08.34,0:43:11.85,Default,,0000,0000,0000,,introducing errors into your armory\Nfeatures. which are interesting side Dialogue: 0,0:43:11.85,0:43:19.59,Default,,0000,0000,0000,,studies. Also why we did this is because\Nwe want to push the concept of what type Dialogue: 0,0:43:19.59,0:43:28.34,Default,,0000,0000,0000,,of observables even farther. Let's say\Ntake this example you want to be able to Dialogue: 0,0:43:28.34,0:43:34.34,Default,,0000,0000,0000,,take instruction abstractions. Let's say\Nfor all major architectures you can break Dialogue: 0,0:43:34.34,0:43:38.69,Default,,0000,0000,0000,,them up into major categories. Be it\Narithmetic instructions, data manipulation Dialogue: 0,0:43:38.69,0:43:45.85,Default,,0000,0000,0000,,instructions, like load stores and then\Ncontrol flow instructions. Then with these Dialogue: 0,0:43:45.85,0:43:52.83,Default,,0000,0000,0000,,basic fundamental building blocks you can\Nmake artifacts. Think of them like a unit Dialogue: 0,0:43:52.83,0:43:56.40,Default,,0000,0000,0000,,of functionality: has some type of input,\Nsome type of output, it provides some type Dialogue: 0,0:43:56.40,0:44:01.28,Default,,0000,0000,0000,,of operation on it. And then with these\Nlittle units of functionality, you can Dialogue: 0,0:44:01.28,0:44:05.21,Default,,0000,0000,0000,,link them together and think of these\Nartifacts as may be sub-basic block or Dialogue: 0,0:44:05.21,0:44:09.44,Default,,0000,0000,0000,,crossing a few basic blocks, but a\Ndifferent way to break up the software. Dialogue: 0,0:44:09.44,0:44:13.13,Default,,0000,0000,0000,,Because a basic block is just a branch\Nbreak, but we want to look at Dialogue: 0,0:44:13.13,0:44:18.68,Default,,0000,0000,0000,,functionality brakes, because these\Nartifacts can provide the basic Dialogue: 0,0:44:18.68,0:44:24.89,Default,,0000,0000,0000,,fundamental building blocks of the\Nsoftware itself. It's more important, when Dialogue: 0,0:44:24.89,0:44:28.84,Default,,0000,0000,0000,,we want to start doing symbolic lifting.\NSo that we can lift the entire software up Dialogue: 0,0:44:28.84,0:44:35.25,Default,,0000,0000,0000,,into a generic representation, that we can\Nslice and dice as needed. Dialogue: 0,0:44:38.64,0:44:42.76,Default,,0000,0000,0000,,Moving from there, I want to talk about\Nfuzzing a little bit more. Fuzzing is Dialogue: 0,0:44:42.76,0:44:47.37,Default,,0000,0000,0000,,effectively at the heart of our project.\NIt provides us the rich dataset that we Dialogue: 0,0:44:47.37,0:44:52.04,Default,,0000,0000,0000,,can use to derive a model. It also\Nprovides us awesome other metadata on the Dialogue: 0,0:44:52.04,0:44:58.06,Default,,0000,0000,0000,,side. But why? Why do we care about\Nfuzzing? Why is fuzzing the metric, that Dialogue: 0,0:44:58.06,0:45:04.68,Default,,0000,0000,0000,,you build an engine, that you build a\Nmodel that you drive some type of reason Dialogue: 0,0:45:04.68,0:45:11.56,Default,,0000,0000,0000,,from? So think of the set of bugs,\Nvulnerabilities, and exploitable Dialogue: 0,0:45:11.56,0:45:16.93,Default,,0000,0000,0000,,vulnerabilities. In an ideal world you'd\Nwant to just have a machine that pulls out Dialogue: 0,0:45:16.93,0:45:20.25,Default,,0000,0000,0000,,exploitable vulnerabilities.\NUnfortunately, this is exceedingly costly Dialogue: 0,0:45:20.25,0:45:25.69,Default,,0000,0000,0000,,for a series of decision problems, that go\Nbetween these sets. So now consider the Dialogue: 0,0:45:25.69,0:45:31.90,Default,,0000,0000,0000,,superset of bugs or faults. A fuzzer can\Neasily recognize, or other software can Dialogue: 0,0:45:31.90,0:45:37.40,Default,,0000,0000,0000,,easily recognize faults, but if you want\Nto move down the sets you unfortunately Dialogue: 0,0:45:37.40,0:45:42.77,Default,,0000,0000,0000,,need to jump through a lot of decision\Nhoops. For example, if you want to move to Dialogue: 0,0:45:42.77,0:45:45.76,Default,,0000,0000,0000,,a vulnerability you have to understand:\NDoes the attacker have some type of Dialogue: 0,0:45:45.76,0:45:51.15,Default,,0000,0000,0000,,control? Is there a trust boundary being\Ncrossed? Is this software configured in Dialogue: 0,0:45:51.15,0:45:55.00,Default,,0000,0000,0000,,the right way for this to be vulnerable\Nright now? So they're human factors that Dialogue: 0,0:45:55.00,0:45:59.28,Default,,0000,0000,0000,,are not deducible from the outside. You\Nthen amplify this decision problem even Dialogue: 0,0:45:59.28,0:46:05.32,Default,,0000,0000,0000,,worse going to exploitable\Nvulnerabilities. So if we collect the Dialogue: 0,0:46:05.32,0:46:11.36,Default,,0000,0000,0000,,superset of bugs, we will know that there\Nis some proportion of subsets in there. Dialogue: 0,0:46:11.36,0:46:15.83,Default,,0000,0000,0000,,And this provides us a datasets easily\Nrecognizable and we can collect in a cost- Dialogue: 0,0:46:15.83,0:46:22.17,Default,,0000,0000,0000,,efficient manner. Finally, fuzzing is key\Nand we're investing a lot of our time Dialogue: 0,0:46:22.17,0:46:26.57,Default,,0000,0000,0000,,right now and working on a new fuzzing\Nengine, because there are some key things Dialogue: 0,0:46:26.57,0:46:32.29,Default,,0000,0000,0000,,we want to do.\NWe want to be able to understand all of Dialogue: 0,0:46:32.29,0:46:35.34,Default,,0000,0000,0000,,the different paths the software could be\Ntaking, and as you're fuzzing you're Dialogue: 0,0:46:35.34,0:46:40.01,Default,,0000,0000,0000,,effectively driving the software down as\Nmany unique paths while referencing as Dialogue: 0,0:46:40.01,0:46:47.76,Default,,0000,0000,0000,,many unique data manipulations as\Npossible. So if we save off every path, Dialogue: 0,0:46:47.76,0:46:51.84,Default,,0000,0000,0000,,annotate the ones that are faulting, we\Nnow have this beautiful rich data set of Dialogue: 0,0:46:51.84,0:46:57.06,Default,,0000,0000,0000,,exactly where the software went as we were\Ndriving it in specific ways. Then we feed Dialogue: 0,0:46:57.06,0:47:02.01,Default,,0000,0000,0000,,that back into our static analysis engine\Nand begin to generate those instruction Dialogue: 0,0:47:02.01,0:47:07.68,Default,,0000,0000,0000,,out of those instruction abstractions,\Nthose artifacts. And with that, imagine we Dialogue: 0,0:47:07.68,0:47:14.56,Default,,0000,0000,0000,,have these gigantic traces of instruction\Nabstractions. From there we can then begin Dialogue: 0,0:47:14.56,0:47:20.99,Default,,0000,0000,0000,,to train the model to explore around the\Nfault location and begin to understand and Dialogue: 0,0:47:20.99,0:47:27.30,Default,,0000,0000,0000,,try and study the fundamental building\Nblocks of what a bug looks like in an Dialogue: 0,0:47:27.30,0:47:32.99,Default,,0000,0000,0000,,abstract instruction agnostic way. This is\Nwhy we're spending a lot of time on our Dialogue: 0,0:47:32.99,0:47:36.98,Default,,0000,0000,0000,,Fuzzing engine right now. But hopefully\Nsoon we'll be able to talk about that more Dialogue: 0,0:47:36.98,0:47:40.38,Default,,0000,0000,0000,,and maybe a tech track and not the policy\Ntrack. Dialogue: 0,0:47:44.75,0:47:49.17,Default,,0000,0000,0000,,C: Yeah, so from then on when anything\Nwent wrong with the computer we said it Dialogue: 0,0:47:49.17,0:47:55.70,Default,,0000,0000,0000,,had bugs in it. {\i1}laughs{\i0} All right, I\Npromised you a technical journey, I Dialogue: 0,0:47:55.70,0:47:59.46,Default,,0000,0000,0000,,promised you a technical journey into the\Ndark abyss of as deep as you want to get Dialogue: 0,0:47:59.46,0:48:03.46,Default,,0000,0000,0000,,with it. So let's go ahead and bring it\Nup. Let's wrap it up and bring it up a Dialogue: 0,0:48:03.46,0:48:07.34,Default,,0000,0000,0000,,little bit here. We've talked a great deal\Ntoday about some theory. We've talked Dialogue: 0,0:48:07.34,0:48:09.97,Default,,0000,0000,0000,,about development in our tooling and\Neverything else and so I figured I should Dialogue: 0,0:48:09.97,0:48:14.01,Default,,0000,0000,0000,,end with some things that are not in\Nprogress, but in fact which are done in Dialogue: 0,0:48:14.01,0:48:20.63,Default,,0000,0000,0000,,yesterday's news. Just to go ahead and\Nmake that shared here with Europe. So in Dialogue: 0,0:48:20.63,0:48:24.14,Default,,0000,0000,0000,,the midst of all of our development we\Nhave been discovering and reporting bugs, Dialogue: 0,0:48:24.14,0:48:28.68,Default,,0000,0000,0000,,again this not our primary purpose really.\NBut you know you can't help but do it. You Dialogue: 0,0:48:28.68,0:48:32.17,Default,,0000,0000,0000,,know how computers are these days. You\Nfind bugs just for turning them on, right? Dialogue: 0,0:48:32.17,0:48:38.61,Default,,0000,0000,0000,,So we've been disclosing all of that a\Nlittle while ago. At DEFCON and Black Hat Dialogue: 0,0:48:38.61,0:48:43.03,Default,,0000,0000,0000,,our chief scientist Sarah together with\NMudge went ahead and dropped this Dialogue: 0,0:48:43.03,0:48:47.84,Default,,0000,0000,0000,,bombshell on the Firefox team which is\Nthat for some period of time they had ASLR Dialogue: 0,0:48:47.84,0:48:54.31,Default,,0000,0000,0000,,disabled on OS X. When we first found it\Nwe assumed it was a bug in our tools. When Dialogue: 0,0:48:54.31,0:48:57.72,Default,,0000,0000,0000,,we first mentioned it in a talk they came\Nto us and said it's definitely a bug on Dialogue: 0,0:48:57.72,0:49:03.14,Default,,0000,0000,0000,,our tools or might be or some level of\Nsurprise and then people started looking Dialogue: 0,0:49:03.14,0:49:08.84,Default,,0000,0000,0000,,into it and in fact at one point it had\Nbeen enabled and then temporarily Dialogue: 0,0:49:08.84,0:49:12.96,Default,,0000,0000,0000,,disabled. No one knew, everyone thought it\Nwas on. It takes someone looking to notice Dialogue: 0,0:49:12.96,0:49:18.01,Default,,0000,0000,0000,,that kind of stuff, right. Major shout out\Nthough, they fixed it immediately despite Dialogue: 0,0:49:18.01,0:49:23.95,Default,,0000,0000,0000,,our full disclosure on stage and\Neverything. So very impressed, but in Dialogue: 0,0:49:23.95,0:49:27.87,Default,,0000,0000,0000,,addition to popping surprises on people\Nwe've also been doing the usual process of Dialogue: 0,0:49:27.87,0:49:32.89,Default,,0000,0000,0000,,submitting patches and bugs, particularly\Nto LLVM and Qemu and if you work in Dialogue: 0,0:49:32.89,0:49:35.81,Default,,0000,0000,0000,,software analysis you could probably guess\Nwhy. Dialogue: 0,0:49:36.51,0:49:39.28,Default,,0000,0000,0000,,Incidentally, if you're looking for a\Ntarget to fuzz if you want to go home from Dialogue: 0,0:49:39.28,0:49:45.87,Default,,0000,0000,0000,,CCC and you want to find a ton of findings\NLLVM comes with a bunch of parsers. You Dialogue: 0,0:49:45.87,0:49:50.06,Default,,0000,0000,0000,,should fuzz them, you should fuzz them and\NI say that because I know for a fact you Dialogue: 0,0:49:50.06,0:49:53.17,Default,,0000,0000,0000,,are gonna get a bunch of findings and it'd\Nbe really nice. I would appreciate it if I Dialogue: 0,0:49:53.17,0:49:56.36,Default,,0000,0000,0000,,didn't have to pay the people to fix them.\NSo if you wouldn't mind disclosing them Dialogue: 0,0:49:56.36,0:50:00.24,Default,,0000,0000,0000,,that would help. But besides these bug\Nreports and all these other things we've Dialogue: 0,0:50:00.24,0:50:04.21,Default,,0000,0000,0000,,also been working with lots of others. You\Nknow we gave a talk earlier this summer, Dialogue: 0,0:50:04.21,0:50:06.91,Default,,0000,0000,0000,,Sarah gave a talk earlier this summer,\Nabout these things and she presented Dialogue: 0,0:50:06.91,0:50:11.83,Default,,0000,0000,0000,,findings on comparing some of these base\Nscores of different Linux distributions. Dialogue: 0,0:50:11.83,0:50:16.32,Default,,0000,0000,0000,,And based on those findings there was a\Nperson on the fedora red team, Jason Dialogue: 0,0:50:16.32,0:50:20.47,Default,,0000,0000,0000,,Calloway, who sat there and well I can't\Nread his mind but I'm sure that he was Dialogue: 0,0:50:20.47,0:50:24.70,Default,,0000,0000,0000,,thinking to himself: golly it would be\Nnice to not, you know, be surprised at the Dialogue: 0,0:50:24.70,0:50:28.56,Default,,0000,0000,0000,,next one of these talks. They score very\Nwell by the way. They were leading in Dialogue: 0,0:50:28.56,0:50:33.66,Default,,0000,0000,0000,,many, many of our metrics. Well, in any\Ncase, he left Vegas and he went back home Dialogue: 0,0:50:33.66,0:50:36.85,Default,,0000,0000,0000,,and him and his colleagues have been\Nworking on essentially re-implementing Dialogue: 0,0:50:36.85,0:50:41.57,Default,,0000,0000,0000,,much of our tooling so that they can check\Nthe stuff that we check before they Dialogue: 0,0:50:41.57,0:50:47.53,Default,,0000,0000,0000,,release. Before they release. Looking for\Nsecurity before you release. So that would Dialogue: 0,0:50:47.53,0:50:51.52,Default,,0000,0000,0000,,be a good thing for others to do and I'm\Nhoping that that idea really catches on. Dialogue: 0,0:50:51.52,0:50:58.99,Default,,0000,0000,0000,,{\i1}laughs{\i0} Yeah, yeah right, that would be\Nnice. That would be nice. Dialogue: 0,0:50:58.99,0:51:04.31,Default,,0000,0000,0000,,But in addition to that, in addition to\Nthat our mission really is to get results Dialogue: 0,0:51:04.31,0:51:08.22,Default,,0000,0000,0000,,out to the public and so in order to\Nachieve that, we have broad partnerships Dialogue: 0,0:51:08.22,0:51:12.34,Default,,0000,0000,0000,,with Consumer Reports and the digital\Nstandard. Especially if you're into cyber Dialogue: 0,0:51:12.34,0:51:16.41,Default,,0000,0000,0000,,policy, I really encourage you to take a\Nlook at the proposed digital standard, Dialogue: 0,0:51:16.41,0:51:21.22,Default,,0000,0000,0000,,which is encompassing of the things we\Nlook for and and and so much more. URLs, Dialogue: 0,0:51:21.22,0:51:25.72,Default,,0000,0000,0000,,data, traffic, motion and cryptography and\Nupdate mechanisms and all that good stuff. Dialogue: 0,0:51:25.72,0:51:31.95,Default,,0000,0000,0000,,So, where we are and where we're going,\Nthe big takeaways here for if you're Dialogue: 0,0:51:31.95,0:51:36.31,Default,,0000,0000,0000,,looking for that, so what, three points\Nfor you: one we are building a tooling Dialogue: 0,0:51:36.31,0:51:39.75,Default,,0000,0000,0000,,necessary to do larger and larger and\Nlarger studies regarding these surrogate Dialogue: 0,0:51:39.75,0:51:44.98,Default,,0000,0000,0000,,security stores. My hope is that in some\Nperiod of the not-too-distant future, I Dialogue: 0,0:51:44.98,0:51:48.60,Default,,0000,0000,0000,,would like to be able to, with my\Ncolleagues, publish some really nice Dialogue: 0,0:51:48.60,0:51:51.64,Default,,0000,0000,0000,,findings about what are the things that\Nyou can observe in software, which have a Dialogue: 0,0:51:51.64,0:51:57.39,Default,,0000,0000,0000,,suspiciously high correlation with the\Nsoftware being good. Right, nobody really Dialogue: 0,0:51:57.39,0:52:00.39,Default,,0000,0000,0000,,knows right now. It's an empirical\Nquestion. As far as I know, the study Dialogue: 0,0:52:00.39,0:52:03.08,Default,,0000,0000,0000,,hasn't been done. We've been running it on\Nthe small scale. We're building the Dialogue: 0,0:52:03.08,0:52:06.62,Default,,0000,0000,0000,,tooling to do it on a much larger scale.\NWe are hoping that this winds up being a Dialogue: 0,0:52:06.62,0:52:11.48,Default,,0000,0000,0000,,useful field in security as that\Ntechnology develops. In the meantime our Dialogue: 0,0:52:11.48,0:52:15.56,Default,,0000,0000,0000,,static analyzers are already making\Nsurprising discoveries: hit YouTube and Dialogue: 0,0:52:15.56,0:52:21.30,Default,,0000,0000,0000,,take a look for Sara Zatko's recent talks\Nat DEFCON/Blackhat. Lots of fun findings Dialogue: 0,0:52:21.30,0:52:25.91,Default,,0000,0000,0000,,in there. Lots of things that anyone who\Nlooks would have found it. Lots of that. Dialogue: 0,0:52:25.91,0:52:29.08,Default,,0000,0000,0000,,And then lastly, if you were in the\Nbusiness of shipping software and you are Dialogue: 0,0:52:29.08,0:52:32.62,Default,,0000,0000,0000,,thinking to yourself.. okay so these guys,\Nsomeone gave them some money to mess up my Dialogue: 0,0:52:32.62,0:52:36.84,Default,,0000,0000,0000,,day and you're wondering: what can I do to\Nnot have my day messed up? One simple Dialogue: 0,0:52:36.84,0:52:40.87,Default,,0000,0000,0000,,piece of advice, one simple piece of\Nadvice: make sure your software employs Dialogue: 0,0:52:40.87,0:52:45.92,Default,,0000,0000,0000,,every exploit mitigation technique Mudge\Nhas ever or will ever hear of. And he's Dialogue: 0,0:52:45.92,0:52:49.50,Default,,0000,0000,0000,,heard of a lot of them. He's only gonna,\Nyou know all that, turn all those things Dialogue: 0,0:52:49.50,0:52:52.28,Default,,0000,0000,0000,,on and if you don't know anything about\Nthat stuff, if nobody on your team knows Dialogue: 0,0:52:52.28,0:52:57.37,Default,,0000,0000,0000,,anything about that stuff didn't I don't\Neven know I'm saying this if you hear you Dialogue: 0,0:52:57.37,0:53:00.97,Default,,0000,0000,0000,,know about that stuff so do that. If\Nyou're not here, then you should be here. Dialogue: 0,0:53:04.43,0:53:16.33,Default,,0000,0000,0000,,Danke, Danke.\NHerald Angel: Thank you, Tim and Parker. Dialogue: 0,0:53:17.50,0:53:23.63,Default,,0000,0000,0000,,Do we have any questions from the\Naudience? It's really hard to see you with Dialogue: 0,0:53:23.63,0:53:30.12,Default,,0000,0000,0000,,that bright light in my face. I think the\Nsignal angel has a question. Signal Angel: Dialogue: 0,0:53:30.12,0:53:34.55,Default,,0000,0000,0000,,So the IRC channel was impressed by your\Ntools and your models that you wrote. And Dialogue: 0,0:53:34.55,0:53:38.05,Default,,0000,0000,0000,,they are wondering what's going to happen\Nto that, because you do have funding from Dialogue: 0,0:53:38.05,0:53:42.04,Default,,0000,0000,0000,,the Ford foundation now and so what are\Nyour plans with this? Do you plan on Dialogue: 0,0:53:42.04,0:53:46.08,Default,,0000,0000,0000,,commercializing this or is it going to be\Nopen source or how do we get our hands on Dialogue: 0,0:53:46.08,0:53:49.15,Default,,0000,0000,0000,,this?\NC: It's an excellent question. So for the Dialogue: 0,0:53:49.15,0:53:53.55,Default,,0000,0000,0000,,time being the money that we are receiving\Nis to develop the tooling, pay for the AWS Dialogue: 0,0:53:53.55,0:53:57.79,Default,,0000,0000,0000,,instances, pay for the engineers and all\Nthat stuff. The direction as an Dialogue: 0,0:53:57.79,0:54:01.41,Default,,0000,0000,0000,,organization that we would like to take\Nthings I have no interest in running a Dialogue: 0,0:54:01.41,0:54:05.41,Default,,0000,0000,0000,,monopoly. That sounds like a fantastic\Namount of work and I really don't want to Dialogue: 0,0:54:05.41,0:54:09.43,Default,,0000,0000,0000,,do it. However, I have a great deal of\Ninterest in taking the gains that we are Dialogue: 0,0:54:09.43,0:54:13.86,Default,,0000,0000,0000,,making in the technology and releasing the\Ndata so that other competent researchers Dialogue: 0,0:54:13.86,0:54:19.02,Default,,0000,0000,0000,,can go through and find useful things that\Nwe may not have noticed ourselves. So Dialogue: 0,0:54:19.02,0:54:22.15,Default,,0000,0000,0000,,we're not at a point where we are\Nreleasing data in bulk just yet, but that Dialogue: 0,0:54:22.15,0:54:26.43,Default,,0000,0000,0000,,is simply a matter of engineering our\Ntools, are still in flux as we, you know. Dialogue: 0,0:54:26.43,0:54:29.23,Default,,0000,0000,0000,,When we do that, we want to make sure the\Ndata is correct and so our software has to Dialogue: 0,0:54:29.23,0:54:33.64,Default,,0000,0000,0000,,have its own low bug counts and all these\Nother things. But ultimately for the Dialogue: 0,0:54:33.64,0:54:37.95,Default,,0000,0000,0000,,scientific aspect of our mission. Though\Nthe science is not our primary mission. Dialogue: 0,0:54:37.95,0:54:41.92,Default,,0000,0000,0000,,Our primary mission is to apply it to help\Nconsumers. At the same time, it is our Dialogue: 0,0:54:41.92,0:54:47.59,Default,,0000,0000,0000,,belief that an opaque model is as good as\Ncrap, no one should trust an opaque model, Dialogue: 0,0:54:47.59,0:54:50.94,Default,,0000,0000,0000,,if somebody is telling you that they have\Nsome statistics and they do not provide Dialogue: 0,0:54:50.94,0:54:54.54,Default,,0000,0000,0000,,you with any underlying data and it is not\Nreproducible you should ignore them. Dialogue: 0,0:54:54.54,0:54:58.36,Default,,0000,0000,0000,,Consequently what we are working towards\Nright now is getting to a point where we Dialogue: 0,0:54:58.36,0:55:02.73,Default,,0000,0000,0000,,will be able to share all of those\Nfindings. The surrogate scores, the Dialogue: 0,0:55:02.73,0:55:06.00,Default,,0000,0000,0000,,interesting correlations between\Nobservables and fuzzing. All that will be Dialogue: 0,0:55:06.00,0:55:09.20,Default,,0000,0000,0000,,public as the material comes online.\NSignal Angel: Thank you. Dialogue: 0,0:55:09.20,0:55:11.87,Default,,0000,0000,0000,,C: Thank you.\NHerald Angel: Thank you. And microphone Dialogue: 0,0:55:11.87,0:55:14.86,Default,,0000,0000,0000,,number three please.\NMic3: Hi, thanks so some really Dialogue: 0,0:55:14.86,0:55:18.45,Default,,0000,0000,0000,,interesting work you presented here. So\Nthere's something I'm not sure I Dialogue: 0,0:55:18.45,0:55:22.91,Default,,0000,0000,0000,,understand about the approach that you're\Ntaking. If you are evaluating the security Dialogue: 0,0:55:22.91,0:55:26.32,Default,,0000,0000,0000,,of say a library function or the\Nimplementation of a network protocol for Dialogue: 0,0:55:26.32,0:55:29.78,Default,,0000,0000,0000,,example you know there'd be a precise\Nspecification you could check that Dialogue: 0,0:55:29.78,0:55:35.19,Default,,0000,0000,0000,,against. And the techniques you're using\Nwould make sense to me. But it's not so Dialogue: 0,0:55:35.19,0:55:37.97,Default,,0000,0000,0000,,clear since you've set the goal that\Nyou've set for yourself is to evaluate Dialogue: 0,0:55:37.97,0:55:43.58,Default,,0000,0000,0000,,security of consumer software. It's not\Nclear to me whether it's fair to call Dialogue: 0,0:55:43.58,0:55:47.43,Default,,0000,0000,0000,,these results security scores in the\Nabsence of a threat model so. So my Dialogue: 0,0:55:47.43,0:55:50.35,Default,,0000,0000,0000,,question is, you know, how is it\Nmeaningful to make a claim that a piece of Dialogue: 0,0:55:50.35,0:55:52.24,Default,,0000,0000,0000,,software is secure if you don't have a\Nthreat model for it? Dialogue: 0,0:55:52.24,0:55:56.09,Default,,0000,0000,0000,,C: This is an excellent question and I\Nanyone who disagrees is they should the Dialogue: 0,0:55:56.09,0:56:01.33,Default,,0000,0000,0000,,wrong. Security without a threat model is\Nnot security at all. It's absolutely a Dialogue: 0,0:56:01.33,0:56:05.56,Default,,0000,0000,0000,,true point. So the things that we are\Nlooking for, most of them are things that Dialogue: 0,0:56:05.56,0:56:08.80,Default,,0000,0000,0000,,you will already find present in your\Nthreat model. And so for example we were Dialogue: 0,0:56:08.80,0:56:12.39,Default,,0000,0000,0000,,reporting on the presence of things like a\NASLR and lots of other things that get to Dialogue: 0,0:56:12.39,0:56:17.03,Default,,0000,0000,0000,,the heart of exploitability of a piece of\Nsoftware. So for example if we are Dialogue: 0,0:56:17.03,0:56:19.87,Default,,0000,0000,0000,,reviewing a piece of software, that has no\Nattack surface Dialogue: 0,0:56:19.87,0:56:24.16,Default,,0000,0000,0000,,then it is canonically not in the threat\Nmodel and in that sense it makes no sense Dialogue: 0,0:56:24.16,0:56:29.27,Default,,0000,0000,0000,,to report on its overall security. On the\Nother hand, if we're talking about Dialogue: 0,0:56:29.27,0:56:33.47,Default,,0000,0000,0000,,software like say a word processor, a\Nbrowser, anything on your phone, anything Dialogue: 0,0:56:33.47,0:56:36.12,Default,,0000,0000,0000,,that talks on the network, we're talking\Nabout those kinds of applications then I Dialogue: 0,0:56:36.12,0:56:39.28,Default,,0000,0000,0000,,would argue that exploit mitigations and\Nthe other things that we are measuring are Dialogue: 0,0:56:39.28,0:56:44.33,Default,,0000,0000,0000,,almost certainly very relevant. So there's\Na sense in which what we are measuring is Dialogue: 0,0:56:44.33,0:56:48.41,Default,,0000,0000,0000,,the lowest common denominator among what\Nwe imagine or the dominant threat models Dialogue: 0,0:56:48.41,0:56:53.18,Default,,0000,0000,0000,,for the applications. The hand-wavy\Nanswer, but I promised heuristics so there Dialogue: 0,0:56:53.18,0:56:55.18,Default,,0000,0000,0000,,you go.\NMic3: Thanks. Dialogue: 0,0:56:55.18,0:57:01.62,Default,,0000,0000,0000,,C: Thank you.\NHerald Angel: Any questions? No raising Dialogue: 0,0:57:01.62,0:57:07.06,Default,,0000,0000,0000,,hands, okay. And then the herald can ask a\Nquestion, because I never can. So the Dialogue: 0,0:57:07.06,0:57:11.92,Default,,0000,0000,0000,,question is: you mentioned earlier these\Nsecurity labels and for example what Dialogue: 0,0:57:11.92,0:57:15.88,Default,,0000,0000,0000,,institution could give out the security\Nlabels? Because as obviously the vendor Dialogue: 0,0:57:15.88,0:57:21.74,Default,,0000,0000,0000,,has no interest in IT security?\NC: Yes it's a very good question. So our Dialogue: 0,0:57:21.74,0:57:25.58,Default,,0000,0000,0000,,partnership with Consumer Reports. I don't\Nknow if you're familiar with them, but in Dialogue: 0,0:57:25.58,0:57:31.34,Default,,0000,0000,0000,,the United States Consumer Reports is a\Nmajor huge consumer watchdog organization. Dialogue: 0,0:57:31.34,0:57:36.55,Default,,0000,0000,0000,,They test the safety of automobiles, they\Ntest you know lots of consumer appliances. Dialogue: 0,0:57:36.55,0:57:40.07,Default,,0000,0000,0000,,All kinds of things both to see if they\Nfunction more or less as advertised but Dialogue: 0,0:57:40.07,0:57:45.21,Default,,0000,0000,0000,,most importantly they're checking for\Nquality, reliability and safety. So our Dialogue: 0,0:57:45.21,0:57:49.84,Default,,0000,0000,0000,,partnership with Consumer Reports is all\Nabout us doing our work and then Dialogue: 0,0:57:49.84,0:57:54.06,Default,,0000,0000,0000,,publishing that. And so for example the\Ntelevisions that we presented the data on Dialogue: 0,0:57:54.06,0:57:58.29,Default,,0000,0000,0000,,all of that was collected and published in\Npartnership with Consumer Reports. Dialogue: 0,0:57:58.29,0:58:00.97,Default,,0000,0000,0000,,Herald: Thank you.\NC: Thank you. Dialogue: 0,0:58:02.63,0:58:12.43,Default,,0000,0000,0000,,Herald: Any other questions for stream. I\Nhear a no. Well in this case people thank Dialogue: 0,0:58:12.43,0:58:16.44,Default,,0000,0000,0000,,you.\NThank Tim and Parker for their nice talk Dialogue: 0,0:58:16.44,0:58:19.96,Default,,0000,0000,0000,,and please give them a very very warm hall\Nround of applause. Dialogue: 0,0:58:19.96,0:58:24.69,Default,,0000,0000,0000,,{\i1}applause{\i0}\NC: Thank you. T: Thank you. Dialogue: 0,0:58:24.69,0:58:51.00,Default,,0000,0000,0000,,subtitles created by c3subtitles.de\Nin the year 2017. Join, and help us!