35C3 - Attacking Chrome IPC

35c3 Preroll music

Herald: And now Ned Williamson's talk:
"Attacking Chrome Inter Process

Communication: reliably finding bugs to
escape the Chrome sandbox". He will be

talking about finding bugs in the Chrome
inter-process communication in order to

escape from the sandbox using a fuzzing
method to enumerate the attack surface of

the Chrome inter-process communication.
Ned is a vulnerability researcher. He

likes C and C++ vulnerabilities. He did
research for consoles and browsers and now

started to work on mobile devices. Please
welcome him with a huge round of applause.

Applause

Ned: All right. Hello everyone. My name is
Ned and today I'll be talking about Chrome

IPC. And actually as I was writing this
talk, I kind of came up with this idea to

make it more usable for everyone and the
way I ended up doing this was by trying to

start really general and then kind of
going more and more specific all the way

down to the Chrome IPC fuzzing. So if
you're really technical the end will be

still interesting and then if you're new
to this stuff hopefully the beginning part

will show some of how to get started. So
just a quick overview about me. I've

mostly been spending the last several
years on low level vulnerability research

and my particular interest is on any kind
of critical bugs. Meaning kind of the more

severe individual bug the more interesting
to me. So I'm trying to kind of solve this

problem of: How do we make the bug finding
process effective enough to bring out

these really rare hidden bugs? And you will
see an example of how to do that by the

end. But just an overview. I've basically
worked on four things the first being CTFs

then went to 3DS and Chrome. Now I'm
starting on XNU, but just a month ago

so, not too much yet. Before we get into it
I just want to give a little recap of what

happened since last time. So I was part of
the Nintendo hacking talk two years ago

here and I presented two exploits called
Soundhax and Fasthax and not to go into it

too much but I did want to share like what
happened here because I was actually

surprised. I put Google analytics on the
Soundhax website and I thought like maybe

a thousand people use it or something but
I just looked at the stats like a couple

weeks ago and then turned out like 100k
people used it or something. And then I

searched YouTube and found like these huge
videos where they were copied so like I

wanted to have a screenshot but it's
copyrighted so didn't do that, but

basically it looks like something on the
order of about a million users which is

crazy because this is one of my intro
projects really so I think like this

should show you that you don't have to be
all the way up onto Chrome or whatever it

is to get into this and do some huge fun
project. And then I just wanted to

publicly talk about the donations because
I had a donation link on the Soundhax

website and we fortunately received about
a thousand dollars in donations and then

half of that went to the emulator people
because they, uh, that's how I eventually

wrote my exploit for Soundhax so it made
sense to repay that, and then the other

half went to buying switches for like the
toolchain developers who couldn't afford

it and so just wanted to thank everyone
who used this or whoever donated. Just a

shout out. So we will get into the actual
meat of the talk. So basically I want to

focus on the bug finding process, not
exploitation necessarily because this

topic is pretty well explored I think and
I think the bug hunting aspect is kind of

what's the most prohibitive for people to
join in. And when I look at the number of

people who I play CTF with who are really
good at exploitation and then a number of

like these prolific bug hunters it just
seems like from what I see from how smart

people are there should be more people
doing the bug hunting and I hope that if I

can talk about it more people can come
over. So with that, the agenda will be

just overall how do you make a process to
achieve any goal. Then next how do you

apply this kind of, some kind of strategy
to bug hunting, then this new fuzzing

style I've been kind of developing, some
other people out in the industry have

been working on. And then finally how does
this all tie back to Chrome IPC. So also

just to mention - I should mention that
the bug I'll be showing in this

presentation was used in a full chain
exploit that I developed with a couple

other people and the details of the
exploitation of that will be discussed at

OffensiveCon, so that's also here in
Germany and hopefully people will check it

out. So how do you become an expert in
anything and I kind of was thinking this

before I even started anything and I was
like in the CTF stage and I was just kind

of curious like if I approach this with
the mindset of there's this arbitrary

skill I want to learn. And if I approach
it strategically like what's going to

happen. So I looked into this expert
research and then there's kind of this

idea of pop psych like you need to study
something deliberately for 10000 hours to

get good at it. And you know there's some
debate about this number it's kind of made

up, I guess, but the essential idea of
deliberate practice I think is very

useful. And it's exactly how I structured
my study. And so what deliberate means is,

when you're learning you want to be
thinking like purposefully like I want to

make sure that the project that I'm doing
is making me get better. I want to be

actually thinking about how I'm
structuring my training and then you want

to make sure that you're kind of always
struggling because that's just how you're

growing. So essentially to do this you
just need to keep picking projects that

have some like success and failure
feedback mechanism that's tied to the real

world and you know with bug hunting like
this is very obvious you know you're

either finding a bug or not. And as I
mentioned you want something difficult but

achievable. And so this kind of order that
I did the different projects I mentioned

in the beginning was specifically chosen
so that each stage would be achievable to

me. But also like really really stretching
what I could do. And there's a funny

anecdote there's this guy named Ben
Franklin from American history. And I read

the story that he used to be really bad at
writing and wanted to get better. So the

way he did it was he took an essay that
looked perfect to him and then he took

notes on it and then a week later he
rewrote the essay from the notes and then

he would just compare the goal versus what
he had done and basically saw all the

shortcomings. And so that kind of just
stuck in my head. And so I'll show like

how do you play this kind of trick to bug
finding practice. And then just another

thing with setting goals for bug hunting.
A lot of it is psychological. I think it's

almost psychological more than
intelligence, for sure. And basically you

want to iteratively pick harder and harder
project so that your tolerance for failure goes

up and up. And so by the time I was working
on Chrome I worked on it every day for six

months, like right home from work until 1:00
AM, sleep, up, and then all day every

weekend and found nothing the whole time.
And then just one day found something and

then from there all that accumulated
struggle and effort, when the bug

precipitated it was just like a sign that
all of these necessary skills were there

and then I was able to repeat it. And so now
we'll talk about what that actually looked

like for bug hunting. So when you think
about how to train the skill I think

there's kind of two constituent skills
that are important and those are knowing

where to look and then recognizing the bug
when you're looking at it. And this first

part is just from my own experience it
seemed like just being a developer it's

pretty easy to get a sense for, you know,
you can look at the Git logs like I'm

mentioning here, other crashes happening
somewhere in the library, are bugs getting

reported publicly? Does the code look bad?
You know it's not hard to tell that

something looks sketchy. But I think
what's really hard is getting the bug to

kind of come out. And so that's where I'll
talk about strategy kind of directly. And

so I kind of have this training idea, where
essentially once you have this kind of

target in mind where it's a little bit of
your skill range but you think it's

doable, you try to enumerate all the
existing bug reports and then look through

each of them and then it's this like Ben
Franklin idea like you take the bug and

then you look at. Usually there will be
like you know this block of text and

they're mentioning like the file where
it's happening and stuff and you can kind

of skim it and sense like, where the bug is.
I went out without actually looking at what it

is. And so then you go over and you try to
find it yourself. And you know it's really

important that you actively try to look
for the bug yourself and kind of strain

yourself and when you've given up
essentially then you look at what was the

bug. And then through that struggle it's
usually pretty clear like what was the

fundamental thing you were missing. And
you know just by repeating this process

constantly this is how you train. And so
this is actually how I first ever started

on bug hunting was you know, some of you
may know j00ru he's this like, really

talented researcher. He's been at it for a
long time and I remember seeing this blog

post from him showing all these IDA pro
bugs and it just kind of blew my mind.

Like wow someone took IDA and found like
security vulnerabilities in it and then

when I looked at the bug reports they're
pretty small so I thought OK how do I

practice and how could I have done this
myself. So basically the first day, you

know they're all like integer overflow
bugs I could barely even, like, I knew

what integer overflow was, but I hadn't
actively looked for it before. And so I

was looking at the function and I couldn't
find it. Basically I went to sleep feeling

like oh god like "I'll never be able to do
this stuff." And then the next day I looked

at again. I was like "Oh yeah that's
actually easy". And then I kind of failed

the second one and then by the third day I
was like able to just see where they were

once you know I knew where to look. So
that kind of made me think "OK, I'll just

keep doing this for a long time and keep
doing harder and harder." So this is

essentially the strategy. Like, I think, you
know, I'm probably the perfect example of

someone who was like an intermediate CTF
player, really like insecure or whatever

like and just wanted to get into this but
I had no idea what I was doing. And I just

kept thinking if I just believe in this
kind of process you know hopefully it

works out. And so here's just like a
little really basic roadmap if you want to

try to replicate what I did which is to
focus on CTF because if you can do CTF

binary problems these are perfect examples
of a kind of training where you try to do

something yourself. There's a write up and
like once you can do these problems you

know all the kind of low level details
that are needed. You know what a bug is.

Things like that. And then from there you
just kind of progressively do harder and

harder targets. And so there's kind of
this component where like you know, I

don't - you can't really assess your own
ability like how much of this is innate or

something and it just seemed to me that
regardless of that, like I'm saying here,

you know, "This isn't chess where you have
people trained from birth with perfect

study and decades of, you know, like we're
barely figuring this stuff out and it's

just kind of a huge mess." And so there's
plenty of room for new people to join in.

And then also there's a lot of these kind
of stories about people who are just

insanely naturally gifted and stuff. And I
tried really hard to like, look into what

these people are actually doing and I
haven't found a case where someone wasn't

working extremely hard. And so you know
just keep that in mind. So just for the

sake of time I won't go into this too much
but if you're looking at the slides later

I just kind of give more detail on like
how I pick the mini projects and got down

to Chrome. So now let's talk about fuzzing
and so before I get into it, I should

emphasize that you should really know how
to do auditing and the first couple of

years like not until into the six months
of failure on Chrome, you know, I was

doing auditing the whole time. And I think
fuzzing gets a bad rap because people

think that these are unrelated strategies
and people are only a fuzzer person or an

auditor person. And really I think these
things are extremely like their work

really well together. But you can't really
know why fuzzing is failing or how does it

even apply it or where to apply it without
being able to audit yourself. And part of

this was like, I noticed on Chrome that I
could audit things but essentially the bug

density was so low on the sandbox attack
surface that I needed a way to kind of

automate what I was looking for in each
subsystem I was looking at. So you know

you have like 20 subsystems that you want
to read, well you know it takes about a

week each minimum to learn. It's a lot
faster to try to fuzz for like a day or

two each thing and then... I don't know
like it's... I can't explain it it just

did random things and then this is what
worked. So. So how would you practice

fuzzing. It's really the same idea that I
had about auditing where you take a bug

and just ask yourself like how would I
have written a fuzzer in the first place

to hit the bug. How could I have known to
write the fuzzer that would have triggered

this. Am I lacking something in auditing
ability? Am I not able to write fuzzers

well enough and actually it took me
probably like a year of fuzzer writing to

get good enough where I could actually act
on my ideas, like, just it's kind of

tricky. And so we'll get back to it later
but this exact idea of practicing fuzzing

on something that looks un-fuzzable is how
I found this real exploitable sandbox

escape. So really quick, just for those of
you who don't know too much about fuzzing,

at least in like the current meta.
Essentially there's this tool called AFL

that came out in 2014 which I think really
shifted how well fuzzing worked. And the

idea is essentially that you have some
corpus of inputs that you want to Fuzz and

then as you're mutating them you're
looking for coverage feedback which is

compiled into your code and then as you're
mutating and running new test cases when

you find new coverage you take that input
and put in your corpus and over time your

corpus kind of grows and grows as more
coverage is hit. And so there's... this

just seems to work really well and then
there's another version of this basically

called libFuzzer, and this is just written
by the LLVM project and the same people

who wrote address sanitiser also wrote
libFuzzer and just in my experience it's

written in a way that's a lot more
extensible and easy to understand and play

with. And so it makes it kind of easier to
audit and fuzz together. And so if you

want to think about what fuzzing is,
essentially you're trying to replicate the

normal testing process, but kind of
parameterizing like what a unit test would

be doing with some input bytes, that you're
just feeding into something and seeing if

it crashes. And so what's interesting is
there's kind of this gap in the middle of

like an end to end test which AFL will
give you just feed [it] a binary or like

the unit test which libFuzzer will give
you where you just keep stuffing bytes

into a parser and real security
vulnerabilities are kind of logical in

nature. And I think that's why people
think that fuzzing isn't applicable and I

think there's actually kind of this part
in the middle where if you see a few

components that look suspicious and then
you can integrate them and fuzz them in

isolation, but have the complexity that
you'd kind of see in the real program,

that's where a lot of bugs come out. And
so how we do this is using a grammar. And

so essentially it's combining generative
fuzzing with coverage guided fuzzing and

so we'll touch on how that works in a
minute, but just for some more evidence on

why does this work well, like I'm not the
only person who is doing this. Kind of

simultaneously myself and two other people
I guess seem to have stumbled across this

idea last year or two years ago, and those
are Syzkaller and Lokihardt. So Syzkaller

is a kind of fully automated Linux kernel
fuzzer. And if you guys haven't seen this

it's kind of hilarious like essentially
they are automatically generating zero day

bugs, like tens per month at least and
they automatically generate the test case,

like submit the report when the commit
comes in it's like automatically tracked.

It's basically this 0-day generator
sitting there. laughter Yeah I know! And

I see this. I'm like OK there's 3000 bugs
that are being found. There's a web app

for it and you can just download it, you
know. And I saw the Linux talk from the

author of Syzkaller and the YouTube videos
has like 100 views and stuff. I'm just

like OK, so people need to reiterate how
important this stuff is. So then there's

Lokihardt as well who's like a famous,
extremely talented, kind of canonical

auditing person and he seems to be doing a
very similar thing with Chakra and V8 and

he's finding like tens of interesting
exploitable bugs. And then there's me who

applied this on the Chrome sandbox and
found over 30 bugs, about half of which

are security relevant, and then five of
which were a sandbox escape without render

code execution. So you know this is just
to emphasize like we're finding really

important things with this technique. And
since I discussed this the first time a

couple of months ago at PoC conference
it's been used by someone in their Chrome

security team to fuzz SQLite, and they're
already finding new bugs in the first

week. So just more of the evidence like
here's the kind of the breakdown of some

of the bugs I found with this strategy. So
just to highlight a couple of them, or

maybe three of them. So the first one was
an out of bounds read - just an integer

overflow in blobs. And this lets you...
you can make a blob and then ask to read

part of it, and then the offset could have
been negative and there's integer overflow

- they got the check wrong so it was a
full memory disclosure from the browser

process. There's also this AppCache use
after free which is what I used in the

exploit this year. And then finally... I
guess the critical bugs are pretty

interesting so, two of these I guess the
first pair are in QUIC and the first

one is a stack buffer overflow with just a
bad packet that comes in over the network.

So you just browse to an attacker site and
they stack buffer overflow Chrome browser

process which is outside the sandbox and
it jumped over the stack cookie so that

was bad. And then then these block file
cache problems. These were in the HTTP

caching mechanism which is also in the
privileged process and these were actually

crashing in the wild for three years and
they didn't know how to... I guess I don't

know if they didn't have resources or they
didn't know how to address the problem or

something, but I sent them the test case
and then they closed like four bug reports

in ancient bugs. So you know it just goes
to show that this kind of technique works

in a variety of really interesting places
that are really important. And so now

let's get to the boring stuff. So what's
Protobuf. Well Protobuf is this data

serialisation format from Google, and it
doesn't really matter that it's Protobuf,

just this idea is you want some kind of...
you want to encode like a little language

for yourself that expresses what you want
to fuzz a kind of a higher abstraction

layer than just fuzzing bytes randomly.
And so if any of you have done functional

programming like, I had been doing stuff
with OCaml and quickcheck for a couple of

years, and then when I saw this I just
immediately recognized the pattern.

Essentially what you can do is, you can
create this little tree structure of just

basic types like enum, you create these
messages, you can just kind of specify

actions you want your fuzzer to take. And
then next what libprotobuf-mutator will do

is, it will take the specification you've
written and link it into libFuzzer so that

it will automatically fuzz and create
these, like, trees that are these kind of

random ASTs from this little language you
wrote and then you can kind of parse this

language which sounds crazy, or more hard
than it is, but you essentially you can

generate this highly structured input
which makes it a lot easier to explore

like, logical type of bugs. So I just
really want to emphasize this strategy can

be used to fuzz anything and so kind of
this same exact idea is being used to find

bugs in caching APIs, encrypted networking
protocols, kernels, sandbox, serialisation

code, stateful systems that have IPC and
network interaction and timing as part of

it, which is what we'll show at the end.
And so like what's what's common here. You

know we just fuzz all of these different
systems in the same way. The idea is, as

an auditor what you do is you kind of
notice like "okay there's some subsystem,

like some caching mechanism with a simple
API" and you look at how it's implemented

and it looks complicated, so you think
"okay, you know if I can write a fuzzer in

like a few hours for this, you know it
seems like high value". So once you kind

of play with the API a bit and understand
like how the API works you know you can

just write this little specification for
the API in Protobuf, and go ahead and

write the fuzzer. So basically I'll show
how this works on Chrome. So just to make

sure I cover all of the background
knowledge, for those of you that don't

really care about fuzzing or don't care
about anything else at least you can get

bootstrapped on Chrome IPC research. The
basic idea of how the Chrome sandboxing

situation works is - when I'm saying I'm
finding bugs in the sandbox, like it's really

finding bugs in the browser process which
are reachable from a sandboxed process and

so the sandbox itself is just constraining
these render tab processes. So they can't

really do much and then what you want to
do is jump from there to the browser

process which can do anything. It's a very
common model. Like almost... on 3DS you

have userland kernel then security code
processor you have a Linux like you might

have a userland process and then in the
sandbox there's some APIs in the kernel

you can hit - syscalls you can hit and
basically everything just keeps boiling

down to "there's some API that you can
look at from the less privileged context".

And then if you can trigger a bug in that
API you escape, and you know, this kind of

applies everywhere. And so this idea of
understanding self-contained chunks of

syscalls in Linux - like hundreds - but
being able to look at and say like okay,

here are 10 related syscalls. This is a
subsystem that I want to fuzz in isolation

- like this is kind of how you think about
it. And so if you just want to get started

on Chrome what you want to do is look at,
"OK what are these endpoints in the

browser process that I can reach from the
render", and then.. you don't really have

to understand how IPC works to do this.
You just have to be able to recognize what

you're allowed to hit from the render to
the browser and what's actually in the

browser. And so fortunately the Chrome
codebase is pretty well organized so they

just tell you if you just don't really go
into any folder that says browser in it.

All of this is outside the sandbox and
prone to sandbox escape. And so most of my

bugs I found were in this content browser
subsystem kind of thing. But you can look

anywhere and I think like all these
results I've had the in last year were

just in one folder. And so you know
there's so many other places where bugs

can manifest that I didn't even look at.
So basically there's plenty of room for

more. So just to go in on what I did is -
in this kind of content stuff - is: you

just want to see where APIs are reachable
from the renderer are enumerated and those

are in this RenderProcessHostImpl::Init
function. So yeah C++ kind of wordy but

you get used to it! Basically there's
there's two places where the APIs are set

up, or the interfaces are exposed. Those
are CreateMessageFilters() and

RegisterMojoInterfaces() and it took me a
while to realize where these were. Like a

year or something. But like those are the
key functions to look at. And so I'll skip

over old style IPC because it's going
away, but it's pretty easy to figure out

what's going on if you look at it. So I'll
talk a bit about Mojo. So essentially this

is a new IPC platform that the Chrome team
has developed and the idea is they want

to, I guess, simplify this process for
developers in terms of defining an

interface that you want to expose to a
render or some other client somewhere

else, and essentially you write these
little interface files called .mojom and

then the build system will generate all
the C++ glue for you - you can just like

subclass something and then it handles all
the mechanics of actually exposing this to

other processes and so on. And so as a
security researcher you know you don't

really care about that. All you care about
is "what can I reach" and "how do I know

what to fuzz" or something. So what I
guess I looked at is just: what are some

of the .mojom files that are subclassed in
this content/browser and you can

just do little grep to check this. So
essentially the AppCache is one of the

bugs I found this year. And here's the API
that the render can... these are all the

messages that the render can send to the
browser and along with the types of

documents. And so you know that's pretty
straightforward. So in the browser process

this is the code that we're trying to
attack which is the actual C++

implementation code for this API. And so
you can see they're subclassing there and

then they just make sure to override all
these virtual functions that actually

implements the API. And so I won't go too
into detail on this part because it's a

little boring, but essentially: how does a
render get from it to all the way over to

this kind of browser C++ code? Well it
essentially goes through this request

mechanism where the render tells the
browser process "Hey I have this kind of

request to access this interface" and then
it'll actually just create that

DispatcherHost implementation object and
just feed in that request over there. So

essentially stuff gets glued together
somehow. And then there's this stuff which

is kind of ugly, but I mean here's where
you're actually exposing the ability to do

this. So here's here's where the request
comes in, and then where this requests

handler function gets fed in as that thing
I mentioned earlier - the

RegisterMojoInterfaces. So it's named
pretty well it's kind of easy to follow.

And they're adding new stuff constantly
all of this stuff is on the attack

surface. Like I think I stopped Chrome a
couple of months ago, I think I looked and

there's like you know five new APIs in
there, they're constantly adding things.

So just a quick point about this.
Essentially you want to do fuzzing

in-process with this LibFuzzer+protobuf-
mutator strategy, and you don't want to be

actually doing IPC - it's just very
brittle and weird. So what you really want

to do is just like here's the C++ object I
want to just instantiate it and call those

functions myself and then this whole thing
is just very lightweight and easy to play

with which is... you know having a
lightweight and very easy to rebuild,

tweak something and play with it, print
things... like the faster you can iterate

the better so anything that's too
complicated, like the success rate goes

way down. So essentially you know the
fuzzer that I made open source is the way

you should do it. But the way I actually
did it was: I just made the object...

commented out the private... I don't know
if you can see it on here... Yeah, so just

commented out the private, created the
object, started calling these things

randomly, it would crash and I would just
hand fix things and you know it's kind of

sloppy but you're testing something in a
very small unit that's not really exposed

to that kind of testing. So now let's kind
of put together everything I've talked

about so far. So this exploitable AppCache
Use-After-Free I found this year was found

using this same idea of deliberate
practice. So I looked at this AppCache

subsystem in the browser process, and I
noticed that there were three old bug

reports that were triggering memory
corruption and they were pretty

interesting because they involved
different kind of ways of attacking and

these things had clearly been audited and
I had actually seen these bugs a couple of

years ago and I kind of used it as
evidence to myself at the time that

fuzzing doesn't work and you need
auditing. But it kind of stuck in my head

and I kept thinking "Someday I'll come
back to this and like I'll overcome it"

you know. So essentially what's
interesting is - I've already talked

about - you know - it's easy to specify
the API and just feed IPC messages into

it. And I think everyone kind of
understands that, who does any IPC

fuzzing. But then there's also this idea
that you've got some remote server that

the AppCache thing like creates a network
request, some server's serving the request

and doing different things, and so on the
second bug it actually matters when things

were - like when the server was returning
data. Because some jobs like stay alive

and then if you send an IPC message to
close your session and then the job is

still alive, there's like a raw pointer
somewhere, and you know something going on

that it matters that the server keeps the
connection open. And then the last thing

is just kind of a logical issue. And if
the server returns these HTTP codes in the

headers of the response in this kind of
weird order, you trigger some logical bug

that actually leads to memory corruption.
And so you know I looked at this and I

said OK, well, so what do we need to test
to cover all this? Basically IPC, Network,

and that timing. And so not only that, but
this is kind of a stateful thing. So we

want to make sure that for each fuzzing
session that we kind of reset the state

completely. And fortunately in C++ this
isn't too hard because you just destroy

the object and if it doesn't exist anymore
what state is there. So you just make sure

that you don't leave things lingering. So
yes I just had this basic idea: we'll call

random IPCs with this fuzzed input, we
return random data from the network, and

then, we reset the state of the cache on
every iteration. And then part of it was

thinking OK, if I can repro these old bugs
if I reintroduce them by editing the

source, this is kind of appealing to this
deliberate practice idea that I could have

written a fuzzer that would trigger these
old things, and this is a kind of the idea

I was pursuing when actually triggered a
new bug. So now what's tricky about this

is if you just return random data from the
network you're not going to make much

progress. And this is kind of where the
auditing background comes in. You want to

think about what is expressive enough
of... Like, how do I make my fuzzer

expressive enough that I can hit
everything, but then not so generic that

it's just spraying... like it's just
noise. And so I'll show he did that in

this specification and so at a high level
I kind of root node in the AST or the tree

of the fuzzier message is this session
message, and then this just contains a

sequence of commands and so... Commands
are something I also made up, and so the

first ten of them are all the different
IPC calls I can do, the eleventh one is

handling any pending requests or pre-
caching like a response to any new

requests that comes in. So that handles
like both the asynchronous case where it

makes a request and it's waiting for the
server and also like the synchronous

version where the response comes
immediately. And then lastly this run

until idle thing which essentially just...
it helps you... like if you place these

RunUntilIdles randomly as you're doing
these IPC messages, you're kind of

flushing the queue of accumulated work.
And so, what this lets you do, is kind of

identify these race condition type things.
Because you can do something like do a

bunch of IPCs that come in and are handled
at the same time without actually

serving... like actually doing the work
yet, and then you do this RunUntilIdle and

then all the work happens. And you know I
didn't think of this a priori in some

smart way, I just looked at the unit tests
and I just tried to think about like "OK

how are these developers already testing
it". And this is what it looked like

they're doing. So these messages are very
easy to write, essentially just provide

for each IPC message that I could have
sent to this thing, just make sure all the

arguments are correct, and then there's a
little bit of cleverness which is like the

HostID also breaks down to just an
enum of like 0,1,2 because just from

looking at the code you know that if I'm
randomly creating hosts, destroying them

and stuff over the whole 4 billion int32
IDs, like it's just going to fall apart

and not find anything interesting. So I
just constrained that for the URL. That is

also a custom message that I constrained
to just return a few premade legit URLs so

that way I'm also not testing the URL
parsing stuff. So then, how do I handle

the network? Well, I just read the source
and looked at "what are all the types of

HTTPS response codes that affect control
flow?". And I just enumerated them and

then for any given request, that comes in
from the AppCache system, I just encode

anything interesting about the response
that I thought of just by reviewing the

source. And it seems like the things that
mattered were those HTTP codes whether or

not the headers asked AppCache to do
caching or just download it once. And then

also the AppCache can request from the
server this manifest file which has some

metadata about what files that it should
be caching. And so essentially just all of

this is encoded in one message. And so how
you go from this like high level

description to actually fuzzing is just
this. So you can see how simple it is.

You're really just... you know I looked at
the unit test code and saw how they set up

this AppCache service. And so they let you
pass in this URLLoaderFactory, and what

this is, is just this kind of unit testable
network request thing so this is how I'm

like, intercepting the network requests
and feeding data. And so I do this little

setup and then here I just create the one
render to browser host. This just kind of

simulating how you would do the Mojo stuff
if it was the real render to browser

interaction, and then I just go through
those commands that I mentioned and just

do these things. So I mean this is all it
is. You just pull the HostID out of this

Protobuf message that we're getting at the
top there - that session that I defined as

the top level TreeNode. And you just go
through and you just call the APIs that

are there. And so, how to get the network
stuff to work, as I mentioned they have

this like mock URLLoaderFactory - also C++-y!
But essentially it's this... well okay

so this is when I basically handle one of
my request messages that I came up with. I

just simulated a response. This is a built
in like unit test function that they have

in their codebase and I just pass in the
relevant bits that came from that message.

So, yes this is what it looks like. I have
some kind of DoRequest helper function and

then I just pass my stuff through to it.
And so it takes that URLFactory and then

serves responses to anything that's
waiting and then what's interesting here

and what's necessary to find the bug is
that - I mentioned that this is

asynchronous - so what will happen is when
you do RegisterHost... and then if I go

back to.. Yeah, you like register host,
select a cache, do some things - like the

AppCache will make a request to the server
and then get this manifest, and then it

will start making requests to download
things and then these things are pending like

responses that it's waiting for from the
server. And so it actually mattered that

you mutate the state further before those
responses come in. And so by doing this

like in between the IPC messages - not
like preloading the network factory with a

bunch of responses - I'm actually serving
things… Like I'm not encoding an assumption

about when I'm serving responses. And I
know this is kind of tedious to go so into

detail, but essentially you run this thing
that's maybe 150 lines or something, and

then trigger this bug with AddressSanitiser.
And so essentially a Use-After-Free

happens and what what's going
on here is you can see the scoped_refptr

pointer destructor. And it turns out that
when... yes so when you go to unregister

the host - like it's an IPC message there
at the bottom that I that I send - and

then it just accidently... This is kind of
inaccurate, this stack trace, but

essentially some ref count goes from one
to zero, and then it starts destroying

this AppCache object. And then in the
destructor one of these requests was

waiting on a response from the server, and
then it essentially gives a reference back

to that other object. And that's kind of
eliding some details but essentially the

refcount went back up to 1 and then, now
you're adding a bunch of references all

over the place to something while it's
being destroyed. And so what happens is

now you have all these pointers to a freed
object and then you can trigger access to

that freed thing again later. And so this
kind of the recipe for an exploitable bug.

And so I just want to point out that all
of this fuzzer is open source and it's

just in the Chrome codebase. So if you
download it or go online to the code

search tool you can search for appcache
fuzzer and it will come up. So then real

quickly, just to cover the exploitation.
You know I guess I have more time and I

thought - I compressed this a lot! But
essentially I did this part in a chain

with two other guys - Saelo and Niklas -
and so Saelo provided the RCE bug. And so

from there we get code execution in the
render, and then this lets us send

arbitrary IPC messages and so it's kind of
annoying to send IPC with Mojo like

arbitrarily, so we kind of piggybacked on
the renderer-sided glue code for sending

these AppCache messages. So we just like
found the C++ object and called into it

and then all in all we end up with this
primitive where we can decref and like

release a reference to this refcounted
thing. Like after it's been freed multiple

times. So, there's two stages to
exploiting this - because we're in the

render and we only have one bug, we need
to turn this into a memory disclosure. And

so you know fortunately this bug can be
triggered repeatedly. And so the idea here

is triggering it once gives you this
decrement-by-n primitive. And so when

you're releasing - you know if you ever
hit zero - you'll trigger the destructor

again. And so essentially what you want to
do for the leak is to not trigger the

destructor because it will blow up, but
rather find a string somewhere in memory

where it is a string pointing to the heap
and then decrement the string pointer so

then it starts like sliding somewhere else
into the heap. So that when you read that

string back you're actually leaking heap
data. And so, we did that. So there's some

object that had a standard C++ string at
the beginning. On Windows the first..

keyword[?] is like the, er, pointer
to the string data. So you decrement this.

It is actually a cookie object, so we just
read the cookie back from the browser and

then in the cookie value we see the leaked
bytes. And then from there there was a

vtable access that we can control in the
destructor. So we make another fake object

that looks like it has one reference left.
Make it hit zero so the destructor is

triggered and then this AppCache thing
gets confused and essentially calls a

control vtable pointer. And then from
there, those are the primitives you need

to write an exploit and then it was just a
matter of putting it together. And if

you're curious about that again you should
look forward to Niklas' talk. And so just

a summary: essentially starting all the
way from the beginning you want to be

practicing deliberately. Keep working
constantly, and keep identifying gaps and

actively working to improve. It sounds
weird but you want to keep that in mind.

And use this new technique with LibFuzzer
and protobuf-mutator - I can promise you

it's not going to be the last time you see
someone using this. And I mentioned I've

started on XNU and we'll see some initial
results pretty soon on that. It's working.

So yeah, and lastly never give up. It may
take months but it's fine. So with that I

guess I'll open to questions. Yeah, thank
you.

Applause

Herald: Okay thank you for that talk. If
you have a question please line up at the

microphones in the room and try to limit
your question to one single sentence. If

you would like to leave at this point
please do that as quietly as possible so

that everyone else can still stay for the
questions. And also if you're listening on

a stream you can ask a question online.
Seems there is... no question? Ah there is

one. Microphone number 2, your question
please.

Mic 2: Hello. I just want to ask why have
you chosen Chrome for bug hunting? Was it

just like you picked one random browser
and you started?

Ned: Yeah no. I mean it's basically just
kind of the hardest thing I could think of

that I could plausibly do? You know just
for the purpose of getting better. And

there's more to it. I think Chrome the way
it's written is very amenable to research

and like I actually didn't know C++ before
I worked on Chrome!

So like learning looking at a great
example of the C++ code base and learning

from that was really helpful to me. And
you know I glossed over kind of my path

but I was actually finding random obscure
library bugs that weren't even reachable

at first. So, just the quality of Chrome
makes it so that what your training is the

real talent not just being able to
decipher bad code. So, I highly recommend

it. I can say that I definitely feel that
the two years I invested on that one

project completely helped me get better.
Herald: Thank you. Signal angel, question

from the internet.
Signal Angel: Hello, there is one question

from the internet.... inaudible
Ned: So the question is: "is it possible

to attack using Meltdown or SPECTRE?" I
don't know. I guess it's possible. I was

essentially focusing only on application
level bugs. So things that I could

trigger deterministically using only bugs
in the Chrome code itself. And so... I

mean also those things came along way
after I was doing my research. So you know

I can't comment on that but I'm sure
someone knows.

Herald: Thanks. Ned: Yeah.
Herald: I see no more people on the

microphones or questions on the internet.
Yeah. OK. Thanks for your talk and thanks

for Q&A.
Ned: Thank you.

Applause

postroll music

subtitles created by c3subtitles.de
in the year 2020. Join, and help us!