Herald-Angel: The Layman's Guide to Zero-
Day Engineering is our next talk by, and
my colleagues out in Austin who run the
Pown2Own contest, assure me that our next
speakers are really very much top of their
class, and I'm really looking forward to
this talk for that. A capture the flag
contest like that requires having done a
lot of your homework upfront so that you
have the tools at your disposal, at the
time, so that you can win. And Marcus and
Amy are here to tell us something way more
valuable about the actual tools they found,
but how they actually arrived at those
tools, and you know, the process of going
to that. And I think that is going to be a
very valuable recipe or lesson for us. So,
please help me welcome Marcus and Amy to a
very much anticipated talk.
applause
Marcus: All right. Hi everyone. Thank you
for making out to our talk this evening.
So, I'd like to start by thanking the CCC
organizers for inviting us out here to
give this talk. This was a unique
opportunity for us to share some of our
experience with the community, and we're
really happy to be here. So yeah, I hope
you guys enjoy. OK, so who are we? Well,
my name is Marcus Gaasedelen. I sometimes
go by the handle @gaasedelen which is my
last name. And I'm joined here by my co-
worker Amy who was also a good friend and
longtime collaborator. We worked for a
company called Ret2 Systems. Ret2 is best
known publicly for its security research
and development. Behind the scenes we do
consulting and have been pushing to
improve the availability of security
education, and specialized security
training, as well as, raising awareness
and sharing information like you're going
to see today. So this talk has been
structured roughly to show our approach in
breaking some of the world's most hardened
consumer software. In particular, we're
going to talk about one of the Zero-Days
that we produce at Ret2 in 2018. And over
the course of the talk, we hope to break
some common misconceptions about the
process of Zero-Day Engineering. We're
going to highlight some of the
observations that we've gathered and built
up about this industry and this trade over
the course of many years now. And we're
going to try to offer some advice on how
to get started doing this kind of work as
an individual. So, we're calling this talk
a non-technical commentary about the
process of Zero-Day Engineering. At times,
it may seem like we're stating the
obvious. But the point is to show that
there's less magic behind the curtain than
most of the spectators probably realize.
So let's talk about PWN2OWN 2018. For
those that don't know, PWN2OWN is an
industry level security competition,
organized annually by Trend Micro's Zero-
Day Initiative. PWN2OWN invites the top
security researchers from around the world
to showcase Zero-Day exploits against high
value software targets such as premier web
browsers, operating systems and
virtualization solutions, such as Hyper-V,
VMware, VirtualBox, Xen, whatever.
So at Ret2, we thought it would be fun to
play on PWN2OWN this year. Specifically we
wanted to target the competition's browser
category. We chose to attack Apple's
Safari web browser on MacOS because it was
new, it was mysterious. But also to avoid
any prior conflicts of interest. And so
for this competition, we ended up
developing a type of Zero-Day, known
typically as a single click RCE or Safari
remote, kind of as some industry language.
So what this means is that we could gain
remote, root level access to your Macbook,
should you click a single malicious link
of ours. That's kind of terrifying. You
know, a lot of you might feel like you're
very prone to not clicking malicious
links, or not getting spearfished. But it's
so easy. Maybe you're in a coffee shop, maybe
I just man-in-the-middle your connection.
It's pretty, yeah, it's a pretty scary
world. So this is actually a picture that
you took on stage at PWN2OWN 2018,
directly following our exploit attempt.
This is actually Joshua Smith from ZDI
holding the competition machine after our
exploit had landed, unfortunately, a
little bit too late. But the payload at
the end of our exploit would pop Apple's
calculator app and a root shell on the
victim machine. This is usually used to
demonstrate code execution. So, for fun we
also made the payload change the desktop's
background to the Ret2 logo. So that's
what you're seeing there. So, what makes a
Zero-Day a fun case study, is that we had
virtually no prior experience with Safari
or MacOS going into this event. We
literally didn't even have a single
Macbook in the office. We actually had to
go out and buy one. And, so as a result
you get to see, how we as expert
researchers approach new and unknown
software targets. So I promised that this
was a non-technical talk which is mostly
true. That's because we actually publish
all the nitty gritty details for the
entire exploit chain as a verbose six part
blog series on our blog this past summer.
It's hard to make highly tactical talks
fun and accessible to all audiences. So
we've reserved much of the truly technical
stuff for you to read at your own leisure.
It's not a prerequisite for this talk, so
don't feel bad if you haven't read those.
So with that in mind, we're ready to
introduce you to the very 1st step of what
we're calling, The Layman's Guide to Zero-
Day Engineering. So, at the start of this
talk, I said we'd be attacking some of the
most high value and well protected
consumer software. This is no joke, right?
This is a high stakes game. So before any
of you even think about looking at code,
or searching for vulnerabilities in these
products, you need to set some
expectations about what you're going to be
up against. So this is a picture of you.
You might be a security expert, a software
engineer, or even just an enthusiast. But
there are some odd twists of self-
loathing, you find yourself interested in
Zero-Days, and the desire to break some
high impact software, like a web browser.
But it's important to recognize that
you're looking to devise some of the
largest, most successful organizations of
our generation. These types of companies have
every interest in securing their products
and building trust with consumers. These
vendors have steadily been growing their
investments in software and device
security, and that trend will only
continue. You see cyber security in
headlines every day, hacking, you know,
these systems compromised. It's only
getting more popular. You know, there's
more money than ever in this space. This
is a beautiful mountain peak that
represents your mission of, I want to
craft a Zero-Day. But you're cent up this
mountain is not going to be an easy task.
As an individual, the odds are not really
in your favor. This game is sort of a free
for all, and everyone is at each other's
throats. So, in one corner is the vendor,
might as well have infinite money and
infinite experience. In another corner, is
the rest of the security research
community, fellow enthusiasts, other
threat actors. So, all of you are going to
be fighting over the same terrain, which
is the code. This is unforgiving terrain
in and of itself. But the vendor has home
field advantage. So these obstacles are
not fun, but it's only going to get worse
for you. Newcomers often don't prepare
themselves for understanding what kind of
time scale they should expect when working
on these types of projects. So, for those
of you who are familiar with the Capture
The Flag circuit. These competitions
usually are time boxed from 36 to 48
hours. Normally, they're over a weekend. We
came out of that circuit. We love the
sport. We still play. But how long does it
take to develop a Zero-Day? Well, it can
vary a lot. Sometimes, you get really
lucky. I've seen someone produce a
Chrome-/V8-bug in 2 days. Other times,
it's taken two weeks. Sometimes, it takes
a month. But sometimes, it can actually
take a lot longer to study and exploit new
targets. You need to be thinking, you
know, you need to be looking at time in
these kind of scales. And so it could take
3.5 months. It could take maybe even 6
months for some targets. The fact of the
matter is that it's almost impossible to
tell how long the process is going take
you. And so unlike a CTF challenge,
there's no upper bound to this process of
Zero-Day Engineering. There's no guarantee
that the exploitable bugs you need to make
a Zero-Day, even exist in the software
you're targeting. You also don't always
know what you're looking for, and you're
working on projects that are many order
magnitudes larger than any sort of
educational resource. We're talking
millions of lines of code where your
average CTF challenge might be a couple
hundred lines to see at most. So I can
already see the tear and self-doubt in
some of your eyes. But I really want to
stress that you shouldn't be too hard on
yourself about this stuff. As a novice,
you need to keep these caveats in mind and
accept that failure is not unlikely in the
journey. All right? So please check this
box before watching the rest of the talk.
So having built some psychological
foundation for the task at hand, the next
step in the Layman's Guide is what we call
reconnaissance. So this is kind of a goofy
slide, but yes, even Metasploit reminds
you to start out doing recon. So with
regard to Zero-Day Engineering,
discovering vulnerabilities against large
scale software can be an absolutely
overwhelming experience. Like that
mountain, it's like, where do I start?
What hill do I go up? Like, where do I
even go from there? So to overcome this,
it's vital to build foundational knowledge
about the target. It's also one of the
least glamorous parts of the Zero-Day
development process. And it's often
skipped by many. You don't see any of the
other speakers really talking about this
so much. You don't see blog posts where
people are like, I googled for eight hours
about Apple Safari before writing a Zero-
Day for it. So you want to aggregate and
review all existing research related to
your target. This is super, super
important. So how do we do our recon? Well
the simple answer is Google everything.
This is literally us Googling something,
and what we do is we go through and we
click, and we download, and we bookmark
every single thing for about five pages.
And you see all those buttons that you
never click at the bottom of Google. All
the things are related searches that you
might want to look at. You should
definitely click all of those. You should
also go through at least four or five
pages and keep downloading and saving
everything that looks remotely relevant.
So you just keep doing this over, and
over, and over again. And you just Google,
and Google, and Google everything that you
think could possibly be related. And the
idea is, you know, you just want to grab
all this information, you want to
understand everything you can about this
target. Even if it's not Apple Safari
specific. I mean, look into V8, look into
Chrome, look into Opera, look into Chakra,
look into whatever you want. So the goal
is to build up a library of security
literature related to your target and its
ecosystem. And then, I want you to read
all of it. But I don't want you, don't,
don't force yourself to understand
everything in your sack in your
literature. The point of this exercise is
to build additional concepts about the
software, its architecture and its
security track record. By the end of the
reconnaissance phase, you should aim to be
able to answer these kinds of questions
about your target. What is the purpose of
the software? How is it architected? Can
anyone describe what WebKit's architecture
is to me? What are its major components?
Is there a sandbox around it? How do you
debug it? How did the developers debug it?
Are there any tips and tricks, are there
special flags? What does its security
track record look like? Does it have
historically vulnerable components? Is
there existing writeups, exploits, or
research in it? etc. All right, we're
through reconnaissance. Step 2 is going to
be target selection. So, there's actually
a few different names that you could maybe
call this. Technically, we're targeting
Apple's Safari, but you want to try and
narrow your scope. So what we're looking
at here is a TreeMap visualization of the
the Web Kit source. So Apple's Safari web
browser is actually built on top of the
Web Kit framework which is essentially a
browser engine. This is Open Source. So
yeah, this is a TreeMap visualization of
the source directory where files are
sorted in by size. So each of those boxes
is essentially a file. While all the grey
boxes, the big gray boxes are directories.
All the sub squares are files. In each
file is sized, based on its lines of code.
Hue, the blue hues represent approximate
maximum cyclomatic complexity detected in
each source file. And you might be
getting, anyway, you might be getting
flashbacks back to that picture of that
mountain peak. How do you even start to
hunt for security vulnerabilities in a
product or codebase of this size?
3 million lines of code. You know, I've
maybe written like, I don't know, like a
100,000 lines of C or C++ in my life, let
alone read or reviewed 3 million. So the
short answer to breaking this problem down
is that you need to reduce your scope of
valuation, and focus on depth over
breadth. And this is most critical when
attacking extremely well picked over
targets. Maybe you're probing an IoT
device? You can probably just sneeze at
that thing and you are going to find
vulnerabilities. But you know, you're
fighting on a very different landscape here.
And so you need to be very detailed
with your review. So reduce your scope.
Our reconnaissance and past experience
with exploiting browsers had lead us to
focus on WebKit's JavaScript engine,
highlighted up here in orange. So, bugs in
JS engines, when it comes to browsers, are
generally regarded as extremely powerful
bugs. But they're also few and far
between, and they're kind of becoming more
rare, as more of you are looking for bugs.
More people are colliding, they're dying
quicker. And so, anyway, let's try to
reduce our scope. So we reduce our scope
from 3 million down in 350,000 lines of
code. Here, we'll zoom into that orange.
So now we're looking at the JavaScript
directory, specifically the JavaScript
core directory. So this is a JavaScript
engine within WebKit, as used by Safari,
on MacOS. And specifically, to further
reduce our scope, we chose to focus on the
highest level interface of the JavaScript
core which is the runtime folder. So this
contains code that's almost 1 for 1
mappings to JavaScript objects and methods
in the interpreter. So, for example,
Array.reverse, or concat, or whatever.
It's very close to what you JavaScript
authors are familiar with. And so this is
what the runtime folder looks like, at
approximately 70,000 lines of code. When
we were spinning up for PWN2OWN, we said,
okay, we are going to find a bug in this
directory in one of these files, and we're
not going to leave it until we have, you
know, walked away with something. So if we
take a step back now. This is what we
started with, and this is what we've done.
We've reduced our scope. So it helps
illustrate this, you know, whittling
process. It was almost a little bit
arbitrary. There's a lot, previously,
there's been a lot of bugs in the runtime
directory. But it's really been cleaned up
the past few years. So anyway, this is
what we chose for our RCE. So having spent
a number of years going back and forth
between attacking and defending, I've come
to recognize that bad components do not
get good fast. Usually researchers are
able to hammer away at these components
for years before they reach some level of
acceptable security. So to escape Safari's
sandbox, we simply looked at the security
trends covered during the reconnaissance phase.
So, this observation, historically
bad components will often take years to
improve, means that we chose to look at
WindowServer. And for those that don't
know, WindowServer is a root level system
service that runs on MacOS. Our research
turned up a trail of ugly bugs from a
MacOS, from essentially the WindowServer
which is accessible to the Safari sandbox.
And in particular, when we're doing our
research, we're looking at ZDI's website,
and you can just search all their
advisories that they've disclosed. And in
particular in 2016, there is over 10 plus
vulnerabilities report to ZDI that were
used as sandbox escapes or privilege
escalation style issues. And so, these are
only vulnerabilities that are reported to
ZDI. If you look in 2017, there is 4 all,
again, used for the same purpose. I think,
all of these were actually, probably used
at PWN2OWN both years. And then in 2018,
there is just one. And so, this is 3
years. Over the span of 3 years where
people were hitting the same exact
component, and Apple or researchers around
the world could have been watching, or
listening and finding bugs, and fighting
over this land right here. And so, it's
pretty interesting. I mean, they give some
perspective. The fact of the matter is
that it's hard to write, it's really hard
for bad components to improve quickly.
Nobody wants to try and sit down and
rewrite bad code. And vendors are
terrified, absolutely terrified of
shipping regressions. Most vendors will
only patch or modify old bad code only
when they absolutely must. For example,
when a vulnerability is reported to them.
And so, as listed on this slide, there's a
number of reasons why a certain module or
component has a terrible security track
record. Just try to keep in mind, that's
usually a good place to look for more
bugs. So if you see a waterfall of bugs
this year in some component, like, wasm or
JIT, maybe you should be looking there,
right? Because that might be good for a
few more years. All right.
Step three. So after all this talk, we are finally
getting to a point where we can start
probing and exploring the codebase in
greater depth. This step is all about bug hunting.
So as an individual researcher or
small organization, the hardest part of
the Zero-Day engineering process is
usually discovering and exploiting a
vulnerability. That's just kind of from
our perspective. This can maybe vary from
person to person. But you know, we don't
have a hundred million dollars to spend on
fuzzers, for example. And so we literally
have one Macbook, right? So it's kind of
like looking for a needle in a haystack.
We're also well versed in the exploitation
process itself. And so those end up being
a little bit more formulaic for ourselves.
So there are two core strategies for
finding exploitable vulnerabilities.
There's a lot of pros and cons to both of
these approaches. But I don't want to
spend too much time talking about their
strengths or weaknesses. So they're all
listed here, the short summary is that
fuzzing is the main go-to strategy for
many security enthusiasts. Some of the key
perks is that it's scalable and almost
always yields result. And so, spoiler
alert, but later in the software industry,
we fuzzed both of our bugs. Both the bugs
that we use for a full chain. And we know
it's 2018. These things are still falling
out with some very trivial means. OK. So,
source review is the other main strategy.
Source review is often much harder for
novices, but it can produce some high
quality bugs when performed diligently.
You know if you're looking to just get
into this stuff, I would say, start real
simple, start with fuzzing and see how
fare you get. So, yeah, for the purpose of
this talk, we are mostly going to focus on
fuzzing. This is a picture from the
dashboard of a simple, scalable fuzzing
harness we built for JavaScript core. This
is when we were ramping up for PWN2OWN and
trying to build our chain. It was a
grammar based JavaScript fuzzer, based on
Mozilla's Darma. There is nothing fancy
about it. This is a snippet of some of
what some of its output looked like. We
had only started building it out when we
actually found the exploitable
vulnerability that we ended up using. So
we haven't really played with this much
since then, but it's, I mean, it shows
kind of how easy it was to get where we
needed to go. So, something like we'd like
to stress heavily to the folks who fuzz,
is that it really must be treated as a
science for these competitive targets.
Guys, I know code coverage isn't the best
metric, but you absolutely must use some
form of introspection to quantify the
progress and reach of your fuzzing. Please
don't just fuzz blindly. So our fuzzer
would generate web based code coverage
reports of our grammars every 15 minutes,
or so. This allowed us to quickly iterate
upon our fuzzer, helping generate more
interesting complex test cases. A good
target is 60 percent code coverage. So you
can see that in the upper right hand
corner. That's kind of what we were
shooting for. Again, it really varies from
target to target. This was also just us
focusing on the runtime folder. If you see
in the upper left hand corner. And so,
something that we have observed, again
over many targets and exotic, exotic
targets, is that bugs almost always fall
out of what we call the hard fought final
coverage percentages. And so, what this
means is, you might work for a while,
trying to build up your coverage, trying
to, you know, build a good set of test
cases, or Grammar's for fuzzing, and then
you'll hit that 60 percent, and you'll be,
okay, what am I missing now? Like everyone
gets that 60 percent, let's say. But then,
once you start inching a little bit
further is when you start fining a lot of
bugs. So, for example, we will pull up
code, and we'll be like, why did we not
hit those blocks up there? Why are those
grey box? Why did we never hit those in
our millions of test cases? And we'll go
find that that's some weird edge case, or
some unoptimized condition, or something
like that, and we will modify our test
cases to hit that code. Other times we'll
actually sit down pull it up on our
projector and talk through some of that
and we'll be like: What the hell is going
on there? This is actually, it's funny,
this is actually a live photo that I took
during our pawn2own hunt. You know as
cliche as this picture is of hackers
standing in front of like a dark screen in
a dark room, this was absolutely real. You
know we were we were just reading some
code. And so it's good to rubber ducky
among co-workers and to hash out ideas to
help confirm theories or discard them. And
so yeah this kinda leads us to the next
piece of advice is when you're doing
source reviews so this applies to both
debugging or assessing those corner cases
and whatnot. If you're ever unsure about
the code that you're reading you
absolutely should be using debuggers and
dynamic analysis. So as painful as it can
maybe be to set up a JavaScript core or
debug this massive C++ application that's
dumping these massive call stacks that are
100 [steps] deep you need to learn those
tools or you are never going to be able to
understand the amount of context necessary
for some of these bugs and complex code.
So for example one of our blog posts makes
extensive use of rr to reverse or to root
cause the vulnerability that we end up
exploiting. It was a race condition in the
garbage collector - totally wild bug.
There's probably, I said there's probably
3 people on earth that could have spotted
this bug through source review. It
required immense knowledge of code base in
my opinion to be able to recognize this as
a vulnerability. We found it through
fuzzing, we had a root cause it using time
travel debugging. Mozilla's rr which is an
amazing project. And so, yeah. Absolutely
use debuggers. This is an example of a
call stack again, just using a debugger to
dump the callstack from a function that
you are auditing can give you an insane
amount of context as to how that function
is used, what kind of data it's operating
on. Maybe, you know, what kind of areas of
the codebase it's called from. You're not
actually supposed to be able to read the
size or read the slide but it's a
backtrace from GDB that is 40 or 50 call
stacks deep. All right. So there is this
huge misconception by novices that new
code is inherently more secure and that
vulnerabilities are only being removed
from code bases, not added. This is almost
patently false and this is something that
I've observed over the course of several
years. Countless targets you know, code
from all sorts of vendors. And there's
this really great blog post put out by
Ivan from GPZ this past fall and in his
blog post he basically ... so one year ago
he fuzzed WebKit using his fuzzer
called Domato. He found a bunch of
vulnerabilities, he reported them and then
he open sourced the fuzzer. But then this
year, this fall, he downloads his fuzzer,
ran it again with little to no changes,
just to get things up and running. And
then he found another eight plus
exploitable use after free vulnerabilities.
So what's really amazing
about this, is when you look at these last
two columns that I have highlighted in
red, virtually all the bugs he found had
been introduced or regressed in the past
12 months. So yes, new vulnerabilities get
introduced every single day. The biggest
reason new code is considered harmful, is
simply that it's not had years to sit in
market. This means it hasn't had time to
mature, it hasn't been tested exhaustively
like the rest of the code base. As soon as
that developer pushes it, whenever it hits
release, whenever it hits stable that's
when you have a billion users pounding at
- it let's say on Chrome. I don't know how
big that user base is but it's massive and
that's a thousand users around the world
just using the browser who are effectively
fuzzing it just by browsing the web. And
so of course you're going to manifest
interesting conditions that will cover
things that are not in your test cases and
unit testing. So yeah, it's not uncommon.
The second point down here is that it's
not uncommon for new code to break
assumptions made elsewhere in the code
base. And this is also actually extremely
common. The complexity of these code bases
can be absolutely insane and it can be
extremely hard to tell if let's say some
new code that Joe Schmoe, the new
developer, adds breaks some paradigm held
by let's say the previous owner of the
codebase. He maybe doesn't understand it
as well - you know, maybe it could be an
expert developer who just made a mistake.
It's super common. Now a piece of advice.
This should be a no brainer for bug
hunting but novices often grow impatient
and start hopping around between code and
functions and getting lost or trying to
chase use-after-frees or bug classes
without really truly understanding what
they're looking for. So a great starting
point is always identify the sources of
user input or the way that you can
interface with the program and then just
follow the data, follow it down. You know
what functions parse it, what manipulates
your data, what reads it, what writes to
it. You know just keep it simple. And so
when we're looking for our sandbox escapes
when you're looking at window server and
our research had showed that there's all
of these functions we don't know anything
about Mac but we read this blog post from
Keine that was like "Oh there's all these
functions that you can send data to in
window server" and apparently there's
about six hundred and there are all these
functions prefixed with underscore
underscore x. And so the 600 end points
will parse and operate upon data that we
send to them. And so to draw a rough
diagram, there is essentially this big red
data tube from the safari sandbox to the
windows server system service. This tube
can deliver arbitrary data that we control
to all those six hundred end points. We
immediately thought let's just try to man-
in-the-middle this data pipe, so that we
can see what's going on. And so that's
exactly what we did. We just hooked up
FRIDA to it, another open source DBI. It's
on GitHub. It's pretty cool. And we're
able to stream all of the messages flowing
over this pipe so we can see all this data
just being sent into the window server
from all sorts of applications - actually
everything on Mac OS talks to this. The
window server is responsible for drawing
all your windows on the desktop, your
mouse clicks, your whatever. It's kind of
like explorer.exe on Windows. So you know
we see all this data coming through, we
see all these crazy messages, all these
unique message formats, all these data
buffers that it's sending in and this is
just begging to be fuzzed. And so we said
"OK, let's fuzz it" and we're getting all
hyped and I distinctly remember thinking
maybe we can jerry-rig AFL into the
window server or let's mutate these
buffers with Radamsa or why don't we just
try flipping some bits. So that's what we
did. So I actually had a very timely tweet
just a few weeks back that echoed this
exact experience. He said that "Looking at
my Security / Vulnerability research
career, my biggest mistakes were almost
always trying to be too clever. Success
hides behind what is the dumbest thing
that could possibly work." The takeaway
here is that you should always start
simple and iterate. So this is our Fuzz
farm. It's a single 13 inch MacBook Pro. I
don't know if this is actually going to
work, it's not a big deal if it doesn't.
I'm only gonna play a few seconds of it.
This is me literally placing my wallet on
the enter key and you can see this box
popping up and we're fuzzing - our fuzzer
is running now and flipping bits in the
messages. And the screen is changing
colors. You're going to start seeing the
boxes freaking out. It's going all over
the place. This is because the bits are
being flipped, it's corrupting stuff, it's
changing the messages. Normally, this
little box is supposed to show your
password hint. But the thing is by holding
the enter key on the locked screen. All
this traffic was being generated to the
window server, and every time the window
server crashed - you know where it brings
you? It brings you right back to your lock
screen. So we had this awesome fuzzing
setup by just holding the enter key.
Applause
And we you know we lovingly titled that
picture "Advanced Persistent Threat" in
our blog. So this is a crash that we got
out of the fuzzer. This occurred very
quickly after ... this was probably within
the first 24 hours. So we found a ton of
crashes, we didn't even explore all of
them. There is probably a few still
sitting on our server. But there's lots
and all the rest ... lots of garbage. But
then this one stands out in particular:
Anytime you see this thing up here that says
"EXC_BAD_ACCESS" with a big number up
there with address equals blah blah blah.
That's a really bad place to be. And so
this is a bug that we end up using at
pwn2own to perform our sandbox escape if
you want to read about it again, it's on
the blog, we're not going to go too deep
into it here. So maybe some of you have
seen the infosec comic. You know it's all
about how people try to do these really
cool clever things. They get ... People
can get too caught up trying to inject so
much science and technology into these
problems that they often miss the forest
for the trees. And so here we are in the
second panel. We just wrote this really
crappy little fuzzer and we found our bug
pretty quickly. And this guy's really
upset. Which brings us to the
misconception that only expert researchers
with blank tools can find bugs. And so you
can fill in the blank with whatever you
want. It can be cutting edge tools, state
of the art, state sponsored, magic bullet.
This is not true. There are very few
secrets. So the next observation: you
should be very wary of any bugs that you
find quickly. A good mantra is that an
easy to find bug is just as easily found
by others. And so what this means is that
soon after our blog post went out ...
actually at pwn2own 2018 we actually knew
we had collided with fluorescence one of
the other competitors. We both struggled
with exploiting this issue ... is a
difficult bug to exploit. And we were ...
we had some very creative exploit, it was
very strange. But there was some
discussion after the fact on Twitter by
nadge - started by Neddy -he's probably
out here, actually speaking tomorrow. You
guys should go see this talk about the
Chrome IPC. That should be really good.
But there is some discussion on Twitter,
that Ned had started, and Nicholas, who is
also here, said "well, at least three
teams found it separately". So at least
us, fluorescence and Nicholas had found
this bug. And we were all at pwn2own, so
you can think how many people out there
might have also found this. There's
probably at least a few. How many people
actually tried to weaponize this thing?
Maybe not many. It is kind of a difficult
bug. And so there are probably at least a
few other researchers who are aware of
this bug. So yeah, that kinda closes the,
you know, if you found a bug very quickly
especially with fuzzing, you can almost
guarantee that someone else has found it.
So I want to pass over the next section to
Amy to continue.
Amy: So we just talked a bunch about, you
know, techniques and expectations when
you're actually looking for the bug. Let
me take over here and talk a little bit
about what to expect when trying to
exploit whatever bug you end up finding.
Yeah and so we have the exploit
development as the next step. So OK, you
found a bug right, you've done the hard
part. You were looking at whatever your
target is, maybe it's a browser maybe it's
the window server or the kernel or
whatever you're trying to target. But the
question is how do you actually do the rest?
How do you go from the bug to
actually popping a calculator onto the
screen? The systems that you're working
with have such a high level of complexity
that even just like understanding you know
enough to know how your bug works it might
not be enough to actually know how to
exploit it. So we try to like brute force
our way to an exploit, is that a good
idea? Well all right before we try to
tackle your bug let's take a step back and
ask a slightly different question. How do
we actually write an exploit like this in
general? Now I feel like a lot of people
consider these kind of exploits maybe be
in their own league at least when you
compare them to something like maybe what
you'd do at a CTF competition or something
simpler like that. And if you were for
example to be given a browser exploit
challenge at a CTF competition it may seem
like an impossibly daunting task has just
been laid in front of you if you've never
done this stuff before. So how can we work
to sort of change that view? And you know
it might be kind of cliche but I actually
think the best way to do it is through
practice. And I know everyone says "oh how
do you get good", "oh, practice". But I
think that this is actually very valuable
for this and the way that practicing
actually comes out is that, well, before we
talked a lot about consuming everything
you could about your targets, like
searching for everything you could that's
public, downloading it, trying to read it
even if you don't understand it, because
you'll hopefully gleam something from it
it doesn't hurt but maybe your goal now
could be actually trying to understand it
as at least as much as you can. You know
it's going to be... it's not going to be
easy. These are very intricate systems
that we're attacking here. And so it will
be a lot of work to understand this stuff.
But for every old exploit you can work
your way through, the path will become
clearer for actually exploiting these
targets. So as because I focused mostly on
browser work and I did that browser part
of our chain, at least the exploitation
part. I have done a lot of exploits and
read a ton of browser exploits and one
thing that I have found is that a lot of
them have very very similar structure. And
they'll have similar techniques in them
they'll have similar sort of primitives
that are being used to build up the
exploit. And so that's one observation.
And to actually illustrate that I have an
example. So alongside us at this at
PWN2OWN this spring we had Samuel Groffs
of Phoenix. He's probably here right now.
So he was targeting Safari just like we
were. But his bug was in the just in time
compiler, the JIT, which converts
JavaScript to the machine code. Our bug
was nowhere near that. It was over in the
garbage collector so a completely
different kind of bug. But the bug here is
super reliable. It was very very clean. I
recommend you go look at it online. It's a
very good resource. And then, a few months
later, at PWN2OWN Mobile, so another pwning
event, we have Fluoroacetate, which was an
amazing team who managed to pretty much
pwn everything they could get their hands
on at that competition, including an
iPhone which of course iPhone uses Safari
so they needed a Safari bug. The safari
bug that they had was very similar in
structure to the previous bug earlier that
year, at least in terms of how the bug
worked and what you could do with it. So
now you could exploit both of these bugs
with very similar exploit code almost in
the same way. There were a few tweaks you
had to do because Apple added a few things
since then. But the path between bug and
code execution was very similar. Then,
even a few months after that, there is a
CTF called "Realworld CTF" which took
place in China and as the title suggests
they had a lot of realistic challenges
including Safari. So of course my team
RPISEC was there and they woke me up in
the middle of the night and tasked me with
solving it. And so I was like "Okay, okay
I'll look at this". And I looked at it and
it was a JIT bug and I've never actually
before that looked at the Safari JIT. And
so, you know, I didn't have much previous
experience doing that, but because I had
taken the time to read all the public
exploits. So I read all the other PWN2OWN
competitors exploits and I read all the
other things that people were releasing
for different CVEs. I had seen a bug like
this before very similar and I knew how to
exploit it, so I could... I was able to
quickly build the path from bug to code
exec and we actually managed to get first
blood on the challenge which was really
really cool.
Applaus
So... So what does this actually mean?
Well I think not every bug is going to be
that easily to swap into an exploit but I
do think that understanding old exploits
is extremely valuable if you're trying to
exploit new bugs. A good place to start if
you're interested in looking at old bugs
is on places like this with the js-vuln-db,
which is a basically a repository of a
whole bunch of JavaScript bugs and proof
of concepts and sometimes even exploits
for them. And so if you were to go through
all of those, I guarantee by the end you'd
have a great understanding of the types of
bugs that are showing up these days and
probably how to exploit most of them.
And... But there aren't that many bugs
that get published that are full exploits.
There's only a couple a year maybe. So
what do you do from there once you've read
all those in and you need to learn more?
Well maybe start trying to exploit other
bugs yourself so you can go... For
example, I like Chrome because they have a
very nice list of all their
vulnerabilities that they post every time
they have an update and they even link you
to the issue, so you can go and see
exactly what was wrong and so take some of
these for example, at the very top you
have out-of-bounds write in V8. So we
could click on that and go and see what
the bug was and then could try to write an
exploit for it. And then by the end we all
have had a much better idea of how to
exploit an out-of-bounds write in V8 and
we've now done it ourselves too. So this
is a chance to sort of apply what you've
learned. But you say OK that's a lot of
work. You know I have to do all kinds of
other stuff, I'm still in school or I have
a full time job. Can't I just play CTFs?
Well it's a good question. The question is
how much do CTFs actually help you with
these kind of exploits. I do think that
you can build a very good mindset for this
because you need a very adversarial
mindset to do this sort of work. But a lot
of the times the challenges don't really
represent the real world exploitation.
There was a good tweet just the other day,
like a few days ago, where we were saying
that yeah libc is... like random libc
challenges - Actually I don't think
it's... Yes. It's libc here. Yeah. - are
often very artificial and don't carry much
value to real world because they're very
specific. Some people love these sort of
very specific CTF challenges, but I don't
think that there's as much value as there
could be. However a lot of... There's been
a couple CTFs recently and historically as
well that have had pretty realistic
challenges in them. In fact right now is a
CTF, 35c3 CTF is running and they have 3
browser exploit challenges, they have a
full chain safari challenge, they have a
virtual box challenge, It's like it's
pretty crazy and it's crazy to see people
solve those challenges in such a short
time span too. But I think it's definitely
something that you can look at afterwards
even if you don't manage to get through
one of those challenges today. But
something to try to work on. And so these
are the sort of new CTFs are actually
pretty good for people who want to jump
off to this kind of real estate or a real
exploit development work. However it can
be kind of scary for newer newcomers to
the CTF scene because suddenly you know
it's your first CTF and they're asking you
to exploit Chrome and you're like what...
what is going on here. So there, it is a
double edged sword sometimes. All right so
now we found the bug and we have
experience, so what do we actually do?
Well you have to kind of have to get lucky
though because even if you've had a ton of
experience that doesn't necessarily mean
that you can instantly write an exploit
for a bug. Our javascript exploit was kind
of like that, it was kind of nice, we knew
what to do very right away but the
brows... or our sandbox exploit did not
fit into a nice box of a previous exploit
that we had seen. So it took a lot of
effort. Quickly I'll show... So this was
the actual bug that we exploited for the
Sandbox. It's a pretty simple bug. It's a
integer issue where index is signed which
means it can be negative. So normally it
expects a value like 4 but we could give
it a value like negative 3 and that would
make it go out of bounds and we could
corrupt memory. So very simple bug not
like a crazy complex one like some of the
other ones we've seen. But does that mean
that this exploit is going to be really
simple? Well let's see... That's a lot of
code. So our exploit for this bug ended up
being about 1300 lines. And so that's
pretty crazy. And you're probably
wondering how it gets there but I do want
to say just be aware that when you do find
it simple looking bug, it might not be
that easy to solve or to exploit. And it
might take a lot of effort but don't get
discouraged if it happens to you. It just
means it's time to ride the exploit
development rollercoaster. And basically
what that means is there's a lot of ups
and downs to an exploit and we have to
basically ride the rollercoaster until
hopefully we have it, the exploit,
finished and we had to do that for our
sandbox escape. And so to say we found the
bug and we had a bunch of great ideas we'd
previously seen a bug exploited like this
by Kiehne and we read their papers and we
had a great idea but then we were like OK
OK it's going to work we just have to make
sure this one bit is not set and it was
like in a random looking value, so we
assumed it would be fine. But turns out
that bit is always set and we have no idea
why and no one else knows why, so thank
you Apple for that. And so OK maybe we can
work around it, maybe we can figure out a
way to unset it and we're like oh yes we
can delete it. It's going to work again!
Everything will be great! Until we realize
that actually breaks the rest of the
exploit. So it's this back and forth, it's
an up and down. And you know sometimes
when you solve one issue you think you've
got what you need and then another issue
shows up.
So it's all about making incremental
progress towards removing all the issues
that are in your way, getting at least
something that works.
Marcus: And so just as a quick aside, this
all happened within like 60 minutes one
night.
Amy: Yeah.
Marcus: Amy saw me just like I was
walking, out of breath, I was like Are you
kidding me? There's two bugs that tripped
us up that made this bug much more
difficult to exploit. And there is no good
reason for why those issues were there and
that it was a horrible experience.
Amy: But it's still one I'd recommend. And
so this rollercoaster it actually applies
to the entire process not just for, you
know, the exploit development because
you'll have it when you find crashes that
don't actually lead to vulnerabilities or
unexplainable crashes or super unreliable
exploits. You just have to keep pushing
your way through until eventually you
hopefully you get to the end of the ride
and you've got yourself a nice exploit.
And so now. OK. So we assume, OK, we've
written an exploit at this point. Maybe
it's not the most reliable thing but it
works like I can get to my code exec every
now and then. So we have to start talking
about the payload. So what is the payload
exactly? A payload is whatever your
exploits trying to actually do. It could
be trying to open up a calculator on the
screen, it could be trying to launch your
sandbox escape exploit, it could be trying
to clean up your system after your
exploit. And by that, I mean fix the
program that you're actually exploiting.
So in CTF we don't get a lot of practice
with this because we're so used to doing
'system', you know, 'cat flag', and then
it doesn't matter if the entire program is
crashing down in flames around us because
we got the flag. So in this case yeah you
cat the flag and then it crashes right
away because you didn't have anything
after your action. But in the real world
it kind of matters all the more, so here's
an example like what would happen if your
exploit didn't clean up after itself, and
just crashes and goes back to the login
screen. This doesn't look very good. If
you're at a conference like Pwn2Own this
won't work, I don't think that they would
let you win if this happened. And so, it's
very important to try to go back and fix
up any damage that you've done to the
system, before it crashes, right after you
finished. Right. And so actually running
your payload, so, a lot of times we see or
in the exploits we see that you'll get to
the code exec here which is just CCs
which means INT3 which just tells the
program to stop or trap to break point and
all the exploits you see most of the time
they just stop here. They don't tell you
anymore and to be fair you know they've
gotten a new code exec they're just
talking about the exploit but you, you
still have to figure out how to do your
payload because unless you want to write
those 1300 lines of code in handwritten
assembly and then make it into shellcode,
you're not going to have a good time. And
so, we had to figure out a way to actually
take our payload, write it to the file
system in the only place that the sandbox
would let us, and then we could run it
again, as a library, and then it would go
and actually do our exploit. And so yeah.
And so now that you've like assembled
everything you're almost done here. You
have your exploit working. You get a
calculator pops up. This was actually our
sandbox escape of running and popping
calculator and proving that we had brute
code exec, but we're not completely done
yet because we need to do a little bit
more, which is exploit reliability. We
need to make sure that our exploit is
actually as reliable as we want it to
because if it only works 1 in 100 times
that's not going to be very good. For
Pwn2own, we ended up building a harness
for our Mac which would let us run the
exploit multiple times, and then collect
information about it so we could look
here, and we could see, very easily how
often it would fail and how often it would
succeed, and then we could go and get more
information, maybe a log, and other stuff
like how long it ran. And this made it
very easy to iterate over our exploit, and
try to correct issues and, make it better
and more reliable. We found that most of
our failures were coming from our heap
groom, which is where we try to line all
your memory in certain ways, but there's
not much that you can do there in our
situation, so we tried to make it as best
as we could and then accepted the
reliability that we got. You, something
else might want to test on is a bunch of
multiple devices. For example our
javascript exploit was a race condition,
so that means the number of cpus on the
device and the speed of the cpu actually
might matter when you're running your
exploit. You might want to try different
operating systems or different operating
system versions, because even if they're
all vulnerable, they may have different
quirks, or tweaks, that you have to do to
actually make your exploit work reliably
on all of them. We had to, we wanted to
test on the MacOS beta as well as the
normal MacOS at least, so that we could
make sure it worked in case Apple pushed
an update right before the competition. So
we did make sure that some parts of our
code, and our exploit, could be
interchanged. So for example, we have
addresses here that are specific to the
operating system version, and we could
swap those out very easily by changing
what part of the code is done here. Yeah.
And then also if you're targeting some
browsers you might be interested in
testing them on mobile too even if you're
not targeting the mobile device. Because a
lot of times the bugs might also work on a
phone, or at least the initial bug will.
And so that's another interesting target
you might be interested in if you weren't
thinking about it originally. So in
general what I'm trying to say is try
throwing your exploit at everything you
can, and hopefully you will be able to
recover some reliability percentages or
figure out things that you overlooked in
your initial testing. Alright. I'm gonna
throw it back over, for the final section.
Marcus: You're in the final section here.
So I didn't get to spend as much time as I
would have liked on this section, but I
think it's an important discussion to have
on here. And so the very last step of our
layman's guide is about responsibilities.
And so this is critical. And so you've
listened to our talk. You've seen us
develop the skeleton keys to computers and
systems and devices. We can create doors
into computers and servers and people's
machines, you can invade privacy, you can
deal damage to people's lives and
companies and systems and countries and so
there is a lot of you have to be very
careful with these. And so everyone in
this room, you know if you take any of our
advice going into this stuff, you know,
please acknowledge what you're getting
into and what can be done to people. And
so, there's at least one example that's
kind of related, that, you know I pulled
out quickly that, you know, quickly came
to mind. It was in 2016. I have to say i
remember this day actually sitting at
work. And there is this massive ddos, that
plagued the Internet, at least in the
U.S., and it took down all your favorite
sites:Twitter, Amazon, Netflix, Etsy,
Github, Spotify, Reddit. I remember the
whole Internet came to a crawl in the U.S.
This is the L3 outage map. This was
absolutely insane. And I remember people
were bouncing off the walls like crazy,
you know. After the fact and they were all
referencing Bruce Schneier's blog, and
there on Twitter there's all this
discussion popping up that "this was
likely a state actor". This was a newly
sophisticated ddos attack. Bruce suggested
it was China or Russia, or you know, some
nation state and the blog post was
specifically titled someone is learning
how to take down the Internet. But then a
few months later we figured out that this
was called the mirai botnet and it was
actually just a bunch of kids trying to
ddos each other minecraft servers. And so
it's a... it's scary because you know I
have a lot of respect for the young people
and how talented they are. And it's a... But
people need to be very conscious about the
damage that can be caused by these things.
Mirai, they weren't using 0days per se.
Later. Now nowadays they are using 0days
but back then they weren't, they were just
using an IoT based botnet. One of the
biggest in the world, our highest
throughput. But it was incredibly
damaging. And you know, so when you you,
it's hard to recognize the power of an
0day until you are wielding it. And so
that's why it's not the first step of the
layman's guide. Once you finish this
process you will come to realize the
danger that you can cause, but also the
danger that you might be putting yourself
in. And so I kind of want to close on that
please be very careful. All right. And so,
that's all we have. This is the
conclusion. The layman's guide, that is
the summary. And if you have any questions
we'll take them now. Otherwise if we run
out of time you can catch us after the talk,
and we'll have some cool stickers too,
so...
Applause
Herald-Angel: Wow, great talk. Thank you.
We have very very little time for
questions. If somebody is very quick they
can come up to one of the microphones in
the front, we will handle one but
otherwise, will you guys be available
after the talk?
Marcus: We will be available after the
talk, if ou want to come up and chat. We
might get swarmed but we also have some
cool Rekto stickers, so, come grab them if
you want them.
Herald-angel: And where can we find you.
Marcus: We'll be over here. We'll try to
head out to the back
Herald-angel: Yeah yeah because we have
another talk coming up in a moment or so.
Marcus: OK.
Herald-angel: I don't see any questions.
So I'm going to wrap it up at this point,
but as I said the speakers will be
available. Let's give this great speech
another round of applause.
Applause
Outro
subtitles created by c3subtitles.de
in the year 2020. Join, and help us!