35C3 - The Layman's Guide to Zero-Day Engineering

Herald-Angel: The Layman's Guide to Zero-
Day Engineering is our next talk by, and

my colleagues out in Austin who run the
Pown2Own contest, assure me that our next

speakers are really very much top of their
class, and I'm really looking forward to

this talk for that. A capture the flag
contest like that requires having done a

lot of your homework upfront so that you
have the tools at your disposal, at the

time, so that you can win. And Marcus and
Amy are here to tell us something way more

valuable about the actual tools they found,
but how they actually arrived at those

tools, and you know, the process of going
to that. And I think that is going to be a

very valuable recipe or lesson for us. So,
please help me welcome Marcus and Amy to a

very much anticipated talk.
applause

Marcus: All right. Hi everyone. Thank you
for making out to our talk this evening.

So, I'd like to start by thanking the CCC
organizers for inviting us out here to

give this talk. This was a unique
opportunity for us to share some of our

experience with the community, and we're
really happy to be here. So yeah, I hope

you guys enjoy. OK, so who are we? Well,
my name is Marcus Gaasedelen. I sometimes

go by the handle @gaasedelen which is my
last name. And I'm joined here by my co-

worker Amy who was also a good friend and
longtime collaborator. We worked for a

company called Ret2 Systems. Ret2 is best
known publicly for its security research

and development. Behind the scenes we do
consulting and have been pushing to

improve the availability of security
education, and specialized security

training, as well as, raising awareness
and sharing information like you're going

to see today. So this talk has been
structured roughly to show our approach in

breaking some of the world's most hardened
consumer software. In particular, we're

going to talk about one of the Zero-Days
that we produce at Ret2 in 2018. And over

the course of the talk, we hope to break
some common misconceptions about the

process of Zero-Day Engineering. We're
going to highlight some of the

observations that we've gathered and built
up about this industry and this trade over

the course of many years now. And we're
going to try to offer some advice on how

to get started doing this kind of work as
an individual. So, we're calling this talk

a non-technical commentary about the
process of Zero-Day Engineering. At times,

it may seem like we're stating the
obvious. But the point is to show that

there's less magic behind the curtain than
most of the spectators probably realize.

So let's talk about PWN2OWN 2018. For
those that don't know, PWN2OWN is an

industry level security competition,
organized annually by Trend Micro's Zero-

Day Initiative. PWN2OWN invites the top
security researchers from around the world

to showcase Zero-Day exploits against high
value software targets such as premier web

browsers, operating systems and
virtualization solutions, such as Hyper-V,

VMware, VirtualBox, Xen, whatever.
So at Ret2, we thought it would be fun to

play on PWN2OWN this year. Specifically we
wanted to target the competition's browser

category. We chose to attack Apple's
Safari web browser on MacOS because it was

new, it was mysterious. But also to avoid
any prior conflicts of interest. And so

for this competition, we ended up
developing a type of Zero-Day, known

typically as a single click RCE or Safari
remote, kind of as some industry language.

So what this means is that we could gain
remote, root level access to your Macbook,

should you click a single malicious link
of ours. That's kind of terrifying. You

know, a lot of you might feel like you're
very prone to not clicking malicious

links, or not getting spearfished. But it's
so easy. Maybe you're in a coffee shop, maybe

I just man-in-the-middle your connection.
It's pretty, yeah, it's a pretty scary

world. So this is actually a picture that
you took on stage at PWN2OWN 2018,

directly following our exploit attempt.
This is actually Joshua Smith from ZDI

holding the competition machine after our
exploit had landed, unfortunately, a

little bit too late. But the payload at
the end of our exploit would pop Apple's

calculator app and a root shell on the
victim machine. This is usually used to

demonstrate code execution. So, for fun we
also made the payload change the desktop's

background to the Ret2 logo. So that's
what you're seeing there. So, what makes a

Zero-Day a fun case study, is that we had
virtually no prior experience with Safari

or MacOS going into this event. We
literally didn't even have a single

Macbook in the office. We actually had to
go out and buy one. And, so as a result

you get to see, how we as expert
researchers approach new and unknown

software targets. So I promised that this
was a non-technical talk which is mostly

true. That's because we actually publish
all the nitty gritty details for the

entire exploit chain as a verbose six part
blog series on our blog this past summer.

It's hard to make highly tactical talks
fun and accessible to all audiences. So

we've reserved much of the truly technical
stuff for you to read at your own leisure.

It's not a prerequisite for this talk, so
don't feel bad if you haven't read those.

So with that in mind, we're ready to
introduce you to the very 1st step of what

we're calling, The Layman's Guide to Zero-
Day Engineering. So, at the start of this

talk, I said we'd be attacking some of the
most high value and well protected

consumer software. This is no joke, right?
This is a high stakes game. So before any

of you even think about looking at code,
or searching for vulnerabilities in these

products, you need to set some
expectations about what you're going to be

up against. So this is a picture of you.
You might be a security expert, a software

engineer, or even just an enthusiast. But
there are some odd twists of self-

loathing, you find yourself interested in
Zero-Days, and the desire to break some

high impact software, like a web browser.
But it's important to recognize that

you're looking to devise some of the
largest, most successful organizations of

our generation. These types of companies have
every interest in securing their products

and building trust with consumers. These
vendors have steadily been growing their

investments in software and device
security, and that trend will only

continue. You see cyber security in
headlines every day, hacking, you know,

these systems compromised. It's only
getting more popular. You know, there's

more money than ever in this space. This
is a beautiful mountain peak that

represents your mission of, I want to
craft a Zero-Day. But you're cent up this

mountain is not going to be an easy task.
As an individual, the odds are not really

in your favor. This game is sort of a free
for all, and everyone is at each other's

throats. So, in one corner is the vendor,
might as well have infinite money and

infinite experience. In another corner, is
the rest of the security research

community, fellow enthusiasts, other
threat actors. So, all of you are going to

be fighting over the same terrain, which
is the code. This is unforgiving terrain

in and of itself. But the vendor has home
field advantage. So these obstacles are

not fun, but it's only going to get worse
for you. Newcomers often don't prepare

themselves for understanding what kind of
time scale they should expect when working

on these types of projects. So, for those
of you who are familiar with the Capture

The Flag circuit. These competitions
usually are time boxed from 36 to 48

hours. Normally, they're over a weekend. We
came out of that circuit. We love the

sport. We still play. But how long does it
take to develop a Zero-Day? Well, it can

vary a lot. Sometimes, you get really
lucky. I've seen someone produce a

Chrome-/V8-bug in 2 days. Other times,
it's taken two weeks. Sometimes, it takes

a month. But sometimes, it can actually
take a lot longer to study and exploit new

targets. You need to be thinking, you
know, you need to be looking at time in

these kind of scales. And so it could take
3.5 months. It could take maybe even 6

months for some targets. The fact of the
matter is that it's almost impossible to

tell how long the process is going take
you. And so unlike a CTF challenge,

there's no upper bound to this process of
Zero-Day Engineering. There's no guarantee

that the exploitable bugs you need to make
a Zero-Day, even exist in the software

you're targeting. You also don't always
know what you're looking for, and you're

working on projects that are many order
magnitudes larger than any sort of

educational resource. We're talking
millions of lines of code where your

average CTF challenge might be a couple
hundred lines to see at most. So I can

already see the tear and self-doubt in
some of your eyes. But I really want to

stress that you shouldn't be too hard on
yourself about this stuff. As a novice,

you need to keep these caveats in mind and
accept that failure is not unlikely in the

journey. All right? So please check this
box before watching the rest of the talk.

So having built some psychological
foundation for the task at hand, the next

step in the Layman's Guide is what we call
reconnaissance. So this is kind of a goofy

slide, but yes, even Metasploit reminds
you to start out doing recon. So with

regard to Zero-Day Engineering,
discovering vulnerabilities against large

scale software can be an absolutely
overwhelming experience. Like that

mountain, it's like, where do I start?
What hill do I go up? Like, where do I

even go from there? So to overcome this,
it's vital to build foundational knowledge

about the target. It's also one of the
least glamorous parts of the Zero-Day

development process. And it's often
skipped by many. You don't see any of the

other speakers really talking about this
so much. You don't see blog posts where

people are like, I googled for eight hours
about Apple Safari before writing a Zero-

Day for it. So you want to aggregate and
review all existing research related to

your target. This is super, super
important. So how do we do our recon? Well

the simple answer is Google everything.
This is literally us Googling something,

and what we do is we go through and we
click, and we download, and we bookmark

every single thing for about five pages.
And you see all those buttons that you

never click at the bottom of Google. All
the things are related searches that you

might want to look at. You should
definitely click all of those. You should

also go through at least four or five
pages and keep downloading and saving

everything that looks remotely relevant.
So you just keep doing this over, and

over, and over again. And you just Google,
and Google, and Google everything that you

think could possibly be related. And the
idea is, you know, you just want to grab

all this information, you want to
understand everything you can about this

target. Even if it's not Apple Safari
specific. I mean, look into V8, look into

Chrome, look into Opera, look into Chakra,
look into whatever you want. So the goal

is to build up a library of security
literature related to your target and its

ecosystem. And then, I want you to read
all of it. But I don't want you, don't,

don't force yourself to understand
everything in your sack in your

literature. The point of this exercise is
to build additional concepts about the

software, its architecture and its
security track record. By the end of the

reconnaissance phase, you should aim to be
able to answer these kinds of questions

about your target. What is the purpose of
the software? How is it architected? Can

anyone describe what WebKit's architecture
is to me? What are its major components?

Is there a sandbox around it? How do you
debug it? How did the developers debug it?

Are there any tips and tricks, are there
special flags? What does its security

track record look like? Does it have
historically vulnerable components? Is

there existing writeups, exploits, or
research in it? etc. All right, we're

through reconnaissance. Step 2 is going to
be target selection. So, there's actually

a few different names that you could maybe
call this. Technically, we're targeting

Apple's Safari, but you want to try and
narrow your scope. So what we're looking

at here is a TreeMap visualization of the
the Web Kit source. So Apple's Safari web

browser is actually built on top of the
Web Kit framework which is essentially a

browser engine. This is Open Source. So
yeah, this is a TreeMap visualization of

the source directory where files are
sorted in by size. So each of those boxes

is essentially a file. While all the grey
boxes, the big gray boxes are directories.

All the sub squares are files. In each
file is sized, based on its lines of code.

Hue, the blue hues represent approximate
maximum cyclomatic complexity detected in

each source file. And you might be
getting, anyway, you might be getting

flashbacks back to that picture of that
mountain peak. How do you even start to

hunt for security vulnerabilities in a
product or codebase of this size?

3 million lines of code. You know, I've
maybe written like, I don't know, like a

100,000 lines of C or C++ in my life, let
alone read or reviewed 3 million. So the

short answer to breaking this problem down
is that you need to reduce your scope of

valuation, and focus on depth over
breadth. And this is most critical when

attacking extremely well picked over
targets. Maybe you're probing an IoT

device? You can probably just sneeze at
that thing and you are going to find

vulnerabilities. But you know, you're
fighting on a very different landscape here.

And so you need to be very detailed
with your review. So reduce your scope.

Our reconnaissance and past experience
with exploiting browsers had lead us to

focus on WebKit's JavaScript engine,
highlighted up here in orange. So, bugs in

JS engines, when it comes to browsers, are
generally regarded as extremely powerful

bugs. But they're also few and far
between, and they're kind of becoming more

rare, as more of you are looking for bugs.
More people are colliding, they're dying

quicker. And so, anyway, let's try to
reduce our scope. So we reduce our scope

from 3 million down in 350,000 lines of
code. Here, we'll zoom into that orange.

So now we're looking at the JavaScript
directory, specifically the JavaScript

core directory. So this is a JavaScript
engine within WebKit, as used by Safari,

on MacOS. And specifically, to further
reduce our scope, we chose to focus on the

highest level interface of the JavaScript
core which is the runtime folder. So this

contains code that's almost 1 for 1
mappings to JavaScript objects and methods

in the interpreter. So, for example,
Array.reverse, or concat, or whatever.

It's very close to what you JavaScript
authors are familiar with. And so this is

what the runtime folder looks like, at
approximately 70,000 lines of code. When

we were spinning up for PWN2OWN, we said,
okay, we are going to find a bug in this

directory in one of these files, and we're
not going to leave it until we have, you

know, walked away with something. So if we
take a step back now. This is what we

started with, and this is what we've done.
We've reduced our scope. So it helps

illustrate this, you know, whittling
process. It was almost a little bit

arbitrary. There's a lot, previously,
there's been a lot of bugs in the runtime

directory. But it's really been cleaned up
the past few years. So anyway, this is

what we chose for our RCE. So having spent
a number of years going back and forth

between attacking and defending, I've come
to recognize that bad components do not

get good fast. Usually researchers are
able to hammer away at these components

for years before they reach some level of
acceptable security. So to escape Safari's

sandbox, we simply looked at the security
trends covered during the reconnaissance phase.

So, this observation, historically
bad components will often take years to

improve, means that we chose to look at
WindowServer. And for those that don't

know, WindowServer is a root level system
service that runs on MacOS. Our research

turned up a trail of ugly bugs from a
MacOS, from essentially the WindowServer

which is accessible to the Safari sandbox.
And in particular, when we're doing our

research, we're looking at ZDI's website,
and you can just search all their

advisories that they've disclosed. And in
particular in 2016, there is over 10 plus

vulnerabilities report to ZDI that were
used as sandbox escapes or privilege

escalation style issues. And so, these are
only vulnerabilities that are reported to

ZDI. If you look in 2017, there is 4 all,
again, used for the same purpose. I think,

all of these were actually, probably used
at PWN2OWN both years. And then in 2018,

there is just one. And so, this is 3
years. Over the span of 3 years where

people were hitting the same exact
component, and Apple or researchers around

the world could have been watching, or
listening and finding bugs, and fighting

over this land right here. And so, it's
pretty interesting. I mean, they give some

perspective. The fact of the matter is
that it's hard to write, it's really hard

for bad components to improve quickly.
Nobody wants to try and sit down and

rewrite bad code. And vendors are
terrified, absolutely terrified of

shipping regressions. Most vendors will
only patch or modify old bad code only

when they absolutely must. For example,
when a vulnerability is reported to them.

And so, as listed on this slide, there's a
number of reasons why a certain module or

component has a terrible security track
record. Just try to keep in mind, that's

usually a good place to look for more
bugs. So if you see a waterfall of bugs

this year in some component, like, wasm or
JIT, maybe you should be looking there,

right? Because that might be good for a
few more years. All right.

Step three. So after all this talk, we are finally
getting to a point where we can start

probing and exploring the codebase in
greater depth. This step is all about bug hunting.

So as an individual researcher or
small organization, the hardest part of

the Zero-Day engineering process is
usually discovering and exploiting a

vulnerability. That's just kind of from
our perspective. This can maybe vary from

person to person. But you know, we don't
have a hundred million dollars to spend on

fuzzers, for example. And so we literally
have one Macbook, right? So it's kind of

like looking for a needle in a haystack.
We're also well versed in the exploitation

process itself. And so those end up being
a little bit more formulaic for ourselves.

So there are two core strategies for
finding exploitable vulnerabilities.

There's a lot of pros and cons to both of
these approaches. But I don't want to

spend too much time talking about their
strengths or weaknesses. So they're all

listed here, the short summary is that
fuzzing is the main go-to strategy for

many security enthusiasts. Some of the key
perks is that it's scalable and almost

always yields result. And so, spoiler
alert, but later in the software industry,

we fuzzed both of our bugs. Both the bugs
that we use for a full chain. And we know

it's 2018. These things are still falling
out with some very trivial means. OK. So,

source review is the other main strategy.
Source review is often much harder for

novices, but it can produce some high
quality bugs when performed diligently.

You know if you're looking to just get
into this stuff, I would say, start real

simple, start with fuzzing and see how
fare you get. So, yeah, for the purpose of

this talk, we are mostly going to focus on
fuzzing. This is a picture from the

dashboard of a simple, scalable fuzzing
harness we built for JavaScript core. This

is when we were ramping up for PWN2OWN and
trying to build our chain. It was a

grammar based JavaScript fuzzer, based on
Mozilla's Darma. There is nothing fancy

about it. This is a snippet of some of
what some of its output looked like. We

had only started building it out when we
actually found the exploitable

vulnerability that we ended up using. So
we haven't really played with this much

since then, but it's, I mean, it shows
kind of how easy it was to get where we

needed to go. So, something like we'd like
to stress heavily to the folks who fuzz,

is that it really must be treated as a
science for these competitive targets.

Guys, I know code coverage isn't the best
metric, but you absolutely must use some

form of introspection to quantify the
progress and reach of your fuzzing. Please

don't just fuzz blindly. So our fuzzer
would generate web based code coverage

reports of our grammars every 15 minutes,
or so. This allowed us to quickly iterate

upon our fuzzer, helping generate more
interesting complex test cases. A good

target is 60 percent code coverage. So you
can see that in the upper right hand

corner. That's kind of what we were
shooting for. Again, it really varies from

target to target. This was also just us
focusing on the runtime folder. If you see

in the upper left hand corner. And so,
something that we have observed, again

over many targets and exotic, exotic
targets, is that bugs almost always fall

out of what we call the hard fought final
coverage percentages. And so, what this

means is, you might work for a while,
trying to build up your coverage, trying

to, you know, build a good set of test
cases, or Grammar's for fuzzing, and then

you'll hit that 60 percent, and you'll be,
okay, what am I missing now? Like everyone

gets that 60 percent, let's say. But then,
once you start inching a little bit

further is when you start fining a lot of
bugs. So, for example, we will pull up

code, and we'll be like, why did we not
hit those blocks up there? Why are those

grey box? Why did we never hit those in
our millions of test cases? And we'll go

find that that's some weird edge case, or
some unoptimized condition, or something

like that, and we will modify our test
cases to hit that code. Other times we'll

actually sit down pull it up on our
projector and talk through some of that

and we'll be like: What the hell is going
on there? This is actually, it's funny,

this is actually a live photo that I took
during our pawn2own hunt. You know as

cliche as this picture is of hackers
standing in front of like a dark screen in

a dark room, this was absolutely real. You
know we were we were just reading some

code. And so it's good to rubber ducky
among co-workers and to hash out ideas to

help confirm theories or discard them. And
so yeah this kinda leads us to the next

piece of advice is when you're doing
source reviews so this applies to both

debugging or assessing those corner cases
and whatnot. If you're ever unsure about

the code that you're reading you
absolutely should be using debuggers and

dynamic analysis. So as painful as it can
maybe be to set up a JavaScript core or

debug this massive C++ application that's
dumping these massive call stacks that are

100 [steps] deep you need to learn those
tools or you are never going to be able to

understand the amount of context necessary
for some of these bugs and complex code.

So for example one of our blog posts makes
extensive use of rr to reverse or to root

cause the vulnerability that we end up
exploiting. It was a race condition in the

garbage collector - totally wild bug.
There's probably, I said there's probably

3 people on earth that could have spotted
this bug through source review. It

required immense knowledge of code base in
my opinion to be able to recognize this as

a vulnerability. We found it through
fuzzing, we had a root cause it using time

travel debugging. Mozilla's rr which is an
amazing project. And so, yeah. Absolutely

use debuggers. This is an example of a
call stack again, just using a debugger to

dump the callstack from a function that
you are auditing can give you an insane

amount of context as to how that function
is used, what kind of data it's operating

on. Maybe, you know, what kind of areas of
the codebase it's called from. You're not

actually supposed to be able to read the
size or read the slide but it's a

backtrace from GDB that is 40 or 50 call
stacks deep. All right. So there is this

huge misconception by novices that new
code is inherently more secure and that

vulnerabilities are only being removed
from code bases, not added. This is almost

patently false and this is something that
I've observed over the course of several

years. Countless targets you know, code
from all sorts of vendors. And there's

this really great blog post put out by
Ivan from GPZ this past fall and in his

blog post he basically ... so one year ago
he fuzzed WebKit using his fuzzer

called Domato. He found a bunch of
vulnerabilities, he reported them and then

he open sourced the fuzzer. But then this
year, this fall, he downloads his fuzzer,

ran it again with little to no changes,
just to get things up and running. And

then he found another eight plus
exploitable use after free vulnerabilities.

So what's really amazing
about this, is when you look at these last

two columns that I have highlighted in
red, virtually all the bugs he found had

been introduced or regressed in the past
12 months. So yes, new vulnerabilities get

introduced every single day. The biggest
reason new code is considered harmful, is

simply that it's not had years to sit in
market. This means it hasn't had time to

mature, it hasn't been tested exhaustively
like the rest of the code base. As soon as

that developer pushes it, whenever it hits
release, whenever it hits stable that's

when you have a billion users pounding at
- it let's say on Chrome. I don't know how

big that user base is but it's massive and
that's a thousand users around the world

just using the browser who are effectively
fuzzing it just by browsing the web. And

so of course you're going to manifest
interesting conditions that will cover

things that are not in your test cases and
unit testing. So yeah, it's not uncommon.

The second point down here is that it's
not uncommon for new code to break

assumptions made elsewhere in the code
base. And this is also actually extremely

common. The complexity of these code bases
can be absolutely insane and it can be

extremely hard to tell if let's say some
new code that Joe Schmoe, the new

developer, adds breaks some paradigm held
by let's say the previous owner of the

codebase. He maybe doesn't understand it
as well - you know, maybe it could be an

expert developer who just made a mistake.
It's super common. Now a piece of advice.

This should be a no brainer for bug
hunting but novices often grow impatient

and start hopping around between code and
functions and getting lost or trying to

chase use-after-frees or bug classes
without really truly understanding what

they're looking for. So a great starting
point is always identify the sources of

user input or the way that you can
interface with the program and then just

follow the data, follow it down. You know
what functions parse it, what manipulates

your data, what reads it, what writes to
it. You know just keep it simple. And so

when we're looking for our sandbox escapes
when you're looking at window server and

our research had showed that there's all
of these functions we don't know anything

about Mac but we read this blog post from
Keine that was like "Oh there's all these

functions that you can send data to in
window server" and apparently there's

about six hundred and there are all these
functions prefixed with underscore

underscore x. And so the 600 end points
will parse and operate upon data that we

send to them. And so to draw a rough
diagram, there is essentially this big red

data tube from the safari sandbox to the
windows server system service. This tube

can deliver arbitrary data that we control
to all those six hundred end points. We

immediately thought let's just try to man-
in-the-middle this data pipe, so that we

can see what's going on. And so that's
exactly what we did. We just hooked up

FRIDA to it, another open source DBI. It's
on GitHub. It's pretty cool. And we're

able to stream all of the messages flowing
over this pipe so we can see all this data

just being sent into the window server
from all sorts of applications - actually

everything on Mac OS talks to this. The
window server is responsible for drawing

all your windows on the desktop, your
mouse clicks, your whatever. It's kind of

like explorer.exe on Windows. So you know
we see all this data coming through, we

see all these crazy messages, all these
unique message formats, all these data

buffers that it's sending in and this is
just begging to be fuzzed. And so we said

"OK, let's fuzz it" and we're getting all
hyped and I distinctly remember thinking

maybe we can jerry-rig AFL into the
window server or let's mutate these

buffers with Radamsa or why don't we just
try flipping some bits. So that's what we

did. So I actually had a very timely tweet
just a few weeks back that echoed this

exact experience. He said that "Looking at
my Security / Vulnerability research

career, my biggest mistakes were almost
always trying to be too clever. Success

hides behind what is the dumbest thing
that could possibly work." The takeaway

here is that you should always start
simple and iterate. So this is our Fuzz

farm. It's a single 13 inch MacBook Pro. I
don't know if this is actually going to

work, it's not a big deal if it doesn't.
I'm only gonna play a few seconds of it.

This is me literally placing my wallet on
the enter key and you can see this box

popping up and we're fuzzing - our fuzzer
is running now and flipping bits in the

messages. And the screen is changing
colors. You're going to start seeing the

boxes freaking out. It's going all over
the place. This is because the bits are

being flipped, it's corrupting stuff, it's
changing the messages. Normally, this

little box is supposed to show your
password hint. But the thing is by holding

the enter key on the locked screen. All
this traffic was being generated to the

window server, and every time the window
server crashed - you know where it brings

you? It brings you right back to your lock
screen. So we had this awesome fuzzing

setup by just holding the enter key.
Applause

And we you know we lovingly titled that
picture "Advanced Persistent Threat" in

our blog. So this is a crash that we got
out of the fuzzer. This occurred very

quickly after ... this was probably within
the first 24 hours. So we found a ton of

crashes, we didn't even explore all of
them. There is probably a few still

sitting on our server. But there's lots
and all the rest ... lots of garbage. But

then this one stands out in particular:
Anytime you see this thing up here that says

"EXC_BAD_ACCESS" with a big number up
there with address equals blah blah blah.

That's a really bad place to be. And so
this is a bug that we end up using at

pwn2own to perform our sandbox escape if
you want to read about it again, it's on

the blog, we're not going to go too deep
into it here. So maybe some of you have

seen the infosec comic. You know it's all
about how people try to do these really

cool clever things. They get ... People
can get too caught up trying to inject so

much science and technology into these
problems that they often miss the forest

for the trees. And so here we are in the
second panel. We just wrote this really

crappy little fuzzer and we found our bug
pretty quickly. And this guy's really

upset. Which brings us to the
misconception that only expert researchers

with blank tools can find bugs. And so you
can fill in the blank with whatever you

want. It can be cutting edge tools, state
of the art, state sponsored, magic bullet.

This is not true. There are very few
secrets. So the next observation: you

should be very wary of any bugs that you
find quickly. A good mantra is that an

easy to find bug is just as easily found
by others. And so what this means is that

soon after our blog post went out ...
actually at pwn2own 2018 we actually knew

we had collided with fluorescence one of
the other competitors. We both struggled

with exploiting this issue ... is a
difficult bug to exploit. And we were ...

we had some very creative exploit, it was
very strange. But there was some

discussion after the fact on Twitter by
nadge - started by Neddy -he's probably

out here, actually speaking tomorrow. You
guys should go see this talk about the

Chrome IPC. That should be really good.
But there is some discussion on Twitter,

that Ned had started, and Nicholas, who is
also here, said "well, at least three

teams found it separately". So at least
us, fluorescence and Nicholas had found

this bug. And we were all at pwn2own, so
you can think how many people out there

might have also found this. There's
probably at least a few. How many people

actually tried to weaponize this thing?
Maybe not many. It is kind of a difficult

bug. And so there are probably at least a
few other researchers who are aware of

this bug. So yeah, that kinda closes the,
you know, if you found a bug very quickly

especially with fuzzing, you can almost
guarantee that someone else has found it.

So I want to pass over the next section to
Amy to continue.

Amy: So we just talked a bunch about, you
know, techniques and expectations when

you're actually looking for the bug. Let
me take over here and talk a little bit

about what to expect when trying to
exploit whatever bug you end up finding.

Yeah and so we have the exploit
development as the next step. So OK, you

found a bug right, you've done the hard
part. You were looking at whatever your

target is, maybe it's a browser maybe it's
the window server or the kernel or

whatever you're trying to target. But the
question is how do you actually do the rest?

How do you go from the bug to
actually popping a calculator onto the

screen? The systems that you're working
with have such a high level of complexity

that even just like understanding you know
enough to know how your bug works it might

not be enough to actually know how to
exploit it. So we try to like brute force

our way to an exploit, is that a good
idea? Well all right before we try to

tackle your bug let's take a step back and
ask a slightly different question. How do

we actually write an exploit like this in
general? Now I feel like a lot of people

consider these kind of exploits maybe be
in their own league at least when you

compare them to something like maybe what
you'd do at a CTF competition or something

simpler like that. And if you were for
example to be given a browser exploit

challenge at a CTF competition it may seem
like an impossibly daunting task has just

been laid in front of you if you've never
done this stuff before. So how can we work

to sort of change that view? And you know
it might be kind of cliche but I actually

think the best way to do it is through
practice. And I know everyone says "oh how

do you get good", "oh, practice". But I
think that this is actually very valuable

for this and the way that practicing
actually comes out is that, well, before we

talked a lot about consuming everything
you could about your targets, like

searching for everything you could that's
public, downloading it, trying to read it

even if you don't understand it, because
you'll hopefully gleam something from it

it doesn't hurt but maybe your goal now
could be actually trying to understand it

as at least as much as you can. You know
it's going to be... it's not going to be

easy. These are very intricate systems
that we're attacking here. And so it will

be a lot of work to understand this stuff.
But for every old exploit you can work

your way through, the path will become
clearer for actually exploiting these

targets. So as because I focused mostly on
browser work and I did that browser part

of our chain, at least the exploitation
part. I have done a lot of exploits and

read a ton of browser exploits and one
thing that I have found is that a lot of

them have very very similar structure. And
they'll have similar techniques in them

they'll have similar sort of primitives
that are being used to build up the

exploit. And so that's one observation.
And to actually illustrate that I have an

example. So alongside us at this at
PWN2OWN this spring we had Samuel Groffs

of Phoenix. He's probably here right now.
So he was targeting Safari just like we

were. But his bug was in the just in time
compiler, the JIT, which converts

JavaScript to the machine code. Our bug
was nowhere near that. It was over in the

garbage collector so a completely
different kind of bug. But the bug here is

super reliable. It was very very clean. I
recommend you go look at it online. It's a

very good resource. And then, a few months
later, at PWN2OWN Mobile, so another pwning

event, we have Fluoroacetate, which was an
amazing team who managed to pretty much

pwn everything they could get their hands
on at that competition, including an

iPhone which of course iPhone uses Safari
so they needed a Safari bug. The safari

bug that they had was very similar in
structure to the previous bug earlier that

year, at least in terms of how the bug
worked and what you could do with it. So

now you could exploit both of these bugs
with very similar exploit code almost in

the same way. There were a few tweaks you
had to do because Apple added a few things

since then. But the path between bug and
code execution was very similar. Then,

even a few months after that, there is a
CTF called "Realworld CTF" which took

place in China and as the title suggests
they had a lot of realistic challenges

including Safari. So of course my team
RPISEC was there and they woke me up in

the middle of the night and tasked me with
solving it. And so I was like "Okay, okay

I'll look at this". And I looked at it and
it was a JIT bug and I've never actually

before that looked at the Safari JIT. And
so, you know, I didn't have much previous

experience doing that, but because I had
taken the time to read all the public

exploits. So I read all the other PWN2OWN
competitors exploits and I read all the

other things that people were releasing
for different CVEs. I had seen a bug like

this before very similar and I knew how to
exploit it, so I could... I was able to

quickly build the path from bug to code
exec and we actually managed to get first

blood on the challenge which was really
really cool.

Applaus
So... So what does this actually mean?

Well I think not every bug is going to be
that easily to swap into an exploit but I

do think that understanding old exploits
is extremely valuable if you're trying to

exploit new bugs. A good place to start if
you're interested in looking at old bugs

is on places like this with the js-vuln-db,
which is a basically a repository of a

whole bunch of JavaScript bugs and proof
of concepts and sometimes even exploits

for them. And so if you were to go through
all of those, I guarantee by the end you'd

have a great understanding of the types of
bugs that are showing up these days and

probably how to exploit most of them.
And... But there aren't that many bugs

that get published that are full exploits.
There's only a couple a year maybe. So

what do you do from there once you've read
all those in and you need to learn more?

Well maybe start trying to exploit other
bugs yourself so you can go... For

example, I like Chrome because they have a
very nice list of all their

vulnerabilities that they post every time
they have an update and they even link you

to the issue, so you can go and see
exactly what was wrong and so take some of

these for example, at the very top you
have out-of-bounds write in V8. So we

could click on that and go and see what
the bug was and then could try to write an

exploit for it. And then by the end we all
have had a much better idea of how to

exploit an out-of-bounds write in V8 and
we've now done it ourselves too. So this

is a chance to sort of apply what you've
learned. But you say OK that's a lot of

work. You know I have to do all kinds of
other stuff, I'm still in school or I have

a full time job. Can't I just play CTFs?
Well it's a good question. The question is

how much do CTFs actually help you with
these kind of exploits. I do think that

you can build a very good mindset for this
because you need a very adversarial

mindset to do this sort of work. But a lot
of the times the challenges don't really

represent the real world exploitation.
There was a good tweet just the other day,

like a few days ago, where we were saying
that yeah libc is... like random libc

challenges - Actually I don't think
it's... Yes. It's libc here. Yeah. - are

often very artificial and don't carry much
value to real world because they're very

specific. Some people love these sort of
very specific CTF challenges, but I don't

think that there's as much value as there
could be. However a lot of... There's been

a couple CTFs recently and historically as
well that have had pretty realistic

challenges in them. In fact right now is a
CTF, 35c3 CTF is running and they have 3

browser exploit challenges, they have a
full chain safari challenge, they have a

virtual box challenge, It's like it's
pretty crazy and it's crazy to see people

solve those challenges in such a short
time span too. But I think it's definitely

something that you can look at afterwards
even if you don't manage to get through

one of those challenges today. But
something to try to work on. And so these

are the sort of new CTFs are actually
pretty good for people who want to jump

off to this kind of real estate or a real
exploit development work. However it can

be kind of scary for newer newcomers to
the CTF scene because suddenly you know

it's your first CTF and they're asking you
to exploit Chrome and you're like what...

what is going on here. So there, it is a
double edged sword sometimes. All right so

now we found the bug and we have
experience, so what do we actually do?

Well you have to kind of have to get lucky
though because even if you've had a ton of

experience that doesn't necessarily mean
that you can instantly write an exploit

for a bug. Our javascript exploit was kind
of like that, it was kind of nice, we knew

what to do very right away but the
brows... or our sandbox exploit did not

fit into a nice box of a previous exploit
that we had seen. So it took a lot of

effort. Quickly I'll show... So this was
the actual bug that we exploited for the

Sandbox. It's a pretty simple bug. It's a
integer issue where index is signed which

means it can be negative. So normally it
expects a value like 4 but we could give

it a value like negative 3 and that would
make it go out of bounds and we could

corrupt memory. So very simple bug not
like a crazy complex one like some of the

other ones we've seen. But does that mean
that this exploit is going to be really

simple? Well let's see... That's a lot of
code. So our exploit for this bug ended up

being about 1300 lines. And so that's
pretty crazy. And you're probably

wondering how it gets there but I do want
to say just be aware that when you do find

it simple looking bug, it might not be
that easy to solve or to exploit. And it

might take a lot of effort but don't get
discouraged if it happens to you. It just

means it's time to ride the exploit
development rollercoaster. And basically

what that means is there's a lot of ups
and downs to an exploit and we have to

basically ride the rollercoaster until
hopefully we have it, the exploit,

finished and we had to do that for our
sandbox escape. And so to say we found the

bug and we had a bunch of great ideas we'd
previously seen a bug exploited like this

by Kiehne and we read their papers and we
had a great idea but then we were like OK

OK it's going to work we just have to make
sure this one bit is not set and it was

like in a random looking value, so we
assumed it would be fine. But turns out

that bit is always set and we have no idea
why and no one else knows why, so thank

you Apple for that. And so OK maybe we can
work around it, maybe we can figure out a

way to unset it and we're like oh yes we
can delete it. It's going to work again!

Everything will be great! Until we realize
that actually breaks the rest of the

exploit. So it's this back and forth, it's
an up and down. And you know sometimes

when you solve one issue you think you've
got what you need and then another issue

shows up.
So it's all about making incremental

progress towards removing all the issues
that are in your way, getting at least

something that works.
Marcus: And so just as a quick aside, this

all happened within like 60 minutes one
night.

Amy: Yeah.
Marcus: Amy saw me just like I was

walking, out of breath, I was like Are you
kidding me? There's two bugs that tripped

us up that made this bug much more
difficult to exploit. And there is no good

reason for why those issues were there and
that it was a horrible experience.

Amy: But it's still one I'd recommend. And
so this rollercoaster it actually applies

to the entire process not just for, you
know, the exploit development because

you'll have it when you find crashes that
don't actually lead to vulnerabilities or

unexplainable crashes or super unreliable
exploits. You just have to keep pushing

your way through until eventually you
hopefully you get to the end of the ride

and you've got yourself a nice exploit.
And so now. OK. So we assume, OK, we've

written an exploit at this point. Maybe
it's not the most reliable thing but it

works like I can get to my code exec every
now and then. So we have to start talking

about the payload. So what is the payload
exactly? A payload is whatever your

exploits trying to actually do. It could
be trying to open up a calculator on the

screen, it could be trying to launch your
sandbox escape exploit, it could be trying

to clean up your system after your
exploit. And by that, I mean fix the

program that you're actually exploiting.
So in CTF we don't get a lot of practice

with this because we're so used to doing
'system', you know, 'cat flag', and then

it doesn't matter if the entire program is
crashing down in flames around us because

we got the flag. So in this case yeah you
cat the flag and then it crashes right

away because you didn't have anything
after your action. But in the real world

it kind of matters all the more, so here's
an example like what would happen if your

exploit didn't clean up after itself, and
just crashes and goes back to the login

screen. This doesn't look very good. If
you're at a conference like Pwn2Own this

won't work, I don't think that they would
let you win if this happened. And so, it's

very important to try to go back and fix
up any damage that you've done to the

system, before it crashes, right after you
finished. Right. And so actually running

your payload, so, a lot of times we see or
in the exploits we see that you'll get to

the code exec here which is just CCs
which means INT3 which just tells the

program to stop or trap to break point and
all the exploits you see most of the time

they just stop here. They don't tell you
anymore and to be fair you know they've

gotten a new code exec they're just
talking about the exploit but you, you

still have to figure out how to do your
payload because unless you want to write

those 1300 lines of code in handwritten
assembly and then make it into shellcode,

you're not going to have a good time. And
so, we had to figure out a way to actually

take our payload, write it to the file
system in the only place that the sandbox

would let us, and then we could run it
again, as a library, and then it would go

and actually do our exploit. And so yeah.
And so now that you've like assembled

everything you're almost done here. You
have your exploit working. You get a

calculator pops up. This was actually our
sandbox escape of running and popping

calculator and proving that we had brute
code exec, but we're not completely done

yet because we need to do a little bit
more, which is exploit reliability. We

need to make sure that our exploit is
actually as reliable as we want it to

because if it only works 1 in 100 times
that's not going to be very good. For

Pwn2own, we ended up building a harness
for our Mac which would let us run the

exploit multiple times, and then collect
information about it so we could look

here, and we could see, very easily how
often it would fail and how often it would

succeed, and then we could go and get more
information, maybe a log, and other stuff

like how long it ran. And this made it
very easy to iterate over our exploit, and

try to correct issues and, make it better
and more reliable. We found that most of

our failures were coming from our heap
groom, which is where we try to line all

your memory in certain ways, but there's
not much that you can do there in our

situation, so we tried to make it as best
as we could and then accepted the

reliability that we got. You, something
else might want to test on is a bunch of

multiple devices. For example our
javascript exploit was a race condition,

so that means the number of cpus on the
device and the speed of the cpu actually

might matter when you're running your
exploit. You might want to try different

operating systems or different operating
system versions, because even if they're

all vulnerable, they may have different
quirks, or tweaks, that you have to do to

actually make your exploit work reliably
on all of them. We had to, we wanted to

test on the MacOS beta as well as the
normal MacOS at least, so that we could

make sure it worked in case Apple pushed
an update right before the competition. So

we did make sure that some parts of our
code, and our exploit, could be

interchanged. So for example, we have
addresses here that are specific to the

operating system version, and we could
swap those out very easily by changing

what part of the code is done here. Yeah.
And then also if you're targeting some

browsers you might be interested in
testing them on mobile too even if you're

not targeting the mobile device. Because a
lot of times the bugs might also work on a

phone, or at least the initial bug will.
And so that's another interesting target

you might be interested in if you weren't
thinking about it originally. So in

general what I'm trying to say is try
throwing your exploit at everything you

can, and hopefully you will be able to
recover some reliability percentages or

figure out things that you overlooked in
your initial testing. Alright. I'm gonna

throw it back over, for the final section.
Marcus: You're in the final section here.

So I didn't get to spend as much time as I
would have liked on this section, but I

think it's an important discussion to have
on here. And so the very last step of our

layman's guide is about responsibilities.
And so this is critical. And so you've

listened to our talk. You've seen us
develop the skeleton keys to computers and

systems and devices. We can create doors
into computers and servers and people's

machines, you can invade privacy, you can
deal damage to people's lives and

companies and systems and countries and so
there is a lot of you have to be very

careful with these. And so everyone in
this room, you know if you take any of our

advice going into this stuff, you know,
please acknowledge what you're getting

into and what can be done to people. And
so, there's at least one example that's

kind of related, that, you know I pulled
out quickly that, you know, quickly came

to mind. It was in 2016. I have to say i
remember this day actually sitting at

work. And there is this massive ddos, that
plagued the Internet, at least in the

U.S., and it took down all your favorite
sites:Twitter, Amazon, Netflix, Etsy,

Github, Spotify, Reddit. I remember the
whole Internet came to a crawl in the U.S.

This is the L3 outage map. This was
absolutely insane. And I remember people

were bouncing off the walls like crazy,
you know. After the fact and they were all

referencing Bruce Schneier's blog, and
there on Twitter there's all this

discussion popping up that "this was
likely a state actor". This was a newly

sophisticated ddos attack. Bruce suggested
it was China or Russia, or you know, some

nation state and the blog post was
specifically titled someone is learning

how to take down the Internet. But then a
few months later we figured out that this

was called the mirai botnet and it was
actually just a bunch of kids trying to

ddos each other minecraft servers. And so
it's a... it's scary because you know I

have a lot of respect for the young people
and how talented they are. And it's a... But

people need to be very conscious about the
damage that can be caused by these things.

Mirai, they weren't using 0days per se.
Later. Now nowadays they are using 0days

but back then they weren't, they were just
using an IoT based botnet. One of the

biggest in the world, our highest
throughput. But it was incredibly

damaging. And you know, so when you you,
it's hard to recognize the power of an

0day until you are wielding it. And so
that's why it's not the first step of the

layman's guide. Once you finish this
process you will come to realize the

danger that you can cause, but also the
danger that you might be putting yourself

in. And so I kind of want to close on that
please be very careful. All right. And so,

that's all we have. This is the
conclusion. The layman's guide, that is

the summary. And if you have any questions
we'll take them now. Otherwise if we run

out of time you can catch us after the talk,
and we'll have some cool stickers too,

so...
Applause

Herald-Angel: Wow, great talk. Thank you.
We have very very little time for

questions. If somebody is very quick they
can come up to one of the microphones in

the front, we will handle one but
otherwise, will you guys be available

after the talk?
Marcus: We will be available after the

talk, if ou want to come up and chat. We
might get swarmed but we also have some

cool Rekto stickers, so, come grab them if
you want them.

Herald-angel: And where can we find you.
Marcus: We'll be over here. We'll try to

head out to the back
Herald-angel: Yeah yeah because we have

another talk coming up in a moment or so.
Marcus: OK.

Herald-angel: I don't see any questions.
So I'm going to wrap it up at this point,

but as I said the speakers will be
available. Let's give this great speech

another round of applause.

Applause

Outro

subtitles created by c3subtitles.de
in the year 2020. Join, and help us!