WEBVTT
00:00:18.209 --> 00:00:23.874
Herald-Angel: The Layman's Guide to Zero-
Day Engineering is our next talk by, and
00:00:23.874 --> 00:00:31.050
my colleagues out in Austin who run the
Pown2Own contest, assure me that our next
00:00:31.050 --> 00:00:35.700
speakers are really very much top of their
class, and I'm really looking forward to
NOTE Paragraph
00:00:35.700 --> 00:00:40.430
this talk for that. A capture the flag
contest like that requires having done a
00:00:40.430 --> 00:00:46.309
lot of your homework upfront so that you
have the tools at your disposal, at the
00:00:46.309 --> 00:00:50.920
time, so that you can win. And Marcus and
Amy are here to tell us something way more
NOTE Paragraph
00:00:50.920 --> 00:00:56.309
valuable about the actual tools they found,
but how they actually arrived at those
00:00:56.309 --> 00:01:00.659
tools, and you know, the process of going
to that. And I think that is going to be a
00:01:00.659 --> 00:01:08.250
very valuable recipe or lesson for us. So,
please help me welcome Marcus and Amy to a
00:01:08.250 --> 00:01:16.768
very much anticipated talk.
applause
00:01:16.768 --> 00:01:22.600
Marcus: All right. Hi everyone. Thank you
for making out to our talk this evening.
00:01:22.600 --> 00:01:26.409
So, I'd like to start by thanking the CCC
organizers for inviting us out here to
00:01:26.409 --> 00:01:30.329
give this talk. This was a unique
opportunity for us to share some of our
00:01:30.329 --> 00:01:35.810
experience with the community, and we're
really happy to be here. So yeah, I hope
00:01:35.810 --> 00:01:42.729
you guys enjoy. OK, so who are we? Well,
my name is Marcus Gaasedelen. I sometimes
00:01:42.729 --> 00:01:48.070
go by the handle @gaasedelen which is my
last name. And I'm joined here by my co-
00:01:48.070 --> 00:01:52.950
worker Amy who was also a good friend and
longtime collaborator. We worked for a
00:01:52.950 --> 00:01:56.391
company called Ret2 Systems. Ret2 is best
known publicly for its security research
00:01:56.391 --> 00:01:59.740
and development. Behind the scenes we do
consulting and have been pushing to
00:01:59.740 --> 00:02:04.040
improve the availability of security
education, and specialized security
00:02:04.040 --> 00:02:08.100
training, as well as, raising awareness
and sharing information like you're going
00:02:08.100 --> 00:02:14.069
to see today. So this talk has been
structured roughly to show our approach in
00:02:14.069 --> 00:02:18.370
breaking some of the world's most hardened
consumer software. In particular, we're
00:02:18.370 --> 00:02:23.349
going to talk about one of the Zero-Days
that we produce at Ret2 in 2018. And over
00:02:23.349 --> 00:02:28.019
the course of the talk, we hope to break
some common misconceptions about the
00:02:28.019 --> 00:02:31.260
process of Zero-Day Engineering. We're
going to highlight some of the
00:02:31.260 --> 00:02:36.799
observations that we've gathered and built
up about this industry and this trade over
00:02:36.799 --> 00:02:41.269
the course of many years now. And we're
going to try to offer some advice on how
00:02:41.269 --> 00:02:46.220
to get started doing this kind of work as
an individual. So, we're calling this talk
00:02:46.220 --> 00:02:51.580
a non-technical commentary about the
process of Zero-Day Engineering. At times,
00:02:51.580 --> 00:02:55.740
it may seem like we're stating the
obvious. But the point is to show that
00:02:55.740 --> 00:03:00.879
there's less magic behind the curtain than
most of the spectators probably realize.
00:03:00.879 --> 00:03:05.350
So let's talk about PWN2OWN 2018. For
those that don't know, PWN2OWN is an
00:03:05.350 --> 00:03:09.099
industry level security competition,
organized annually by Trend Micro's Zero-
00:03:09.099 --> 00:03:15.000
Day Initiative. PWN2OWN invites the top
security researchers from around the world
00:03:15.000 --> 00:03:19.810
to showcase Zero-Day exploits against high
value software targets such as premier web
00:03:19.810 --> 00:03:23.310
browsers, operating systems and
virtualization solutions, such as Hyper-V,
00:03:23.310 --> 00:03:28.540
VMware, VirtualBox, Xen, whatever.
So at Ret2, we thought it would be fun to
00:03:28.540 --> 00:03:32.560
play on PWN2OWN this year. Specifically we
wanted to target the competition's browser
00:03:32.560 --> 00:03:38.480
category. We chose to attack Apple's
Safari web browser on MacOS because it was
00:03:38.480 --> 00:03:44.700
new, it was mysterious. But also to avoid
any prior conflicts of interest. And so
00:03:44.700 --> 00:03:48.159
for this competition, we ended up
developing a type of Zero-Day, known
00:03:48.159 --> 00:03:56.019
typically as a single click RCE or Safari
remote, kind of as some industry language.
00:03:56.019 --> 00:04:00.219
So what this means is that we could gain
remote, root level access to your Macbook,
00:04:00.219 --> 00:04:05.059
should you click a single malicious link
of ours. That's kind of terrifying. You
00:04:05.059 --> 00:04:10.390
know, a lot of you might feel like you're
very prone to not clicking malicious
00:04:10.390 --> 00:04:15.091
links, or not getting spearfished. But it's
so easy. Maybe you're in a coffee shop, maybe
00:04:15.091 --> 00:04:19.970
I just man-in-the-middle your connection.
It's pretty, yeah, it's a pretty scary
00:04:19.970 --> 00:04:23.360
world. So this is actually a picture that
you took on stage at PWN2OWN 2018,
00:04:23.360 --> 00:04:27.940
directly following our exploit attempt.
This is actually Joshua Smith from ZDI
00:04:27.940 --> 00:04:33.020
holding the competition machine after our
exploit had landed, unfortunately, a
00:04:33.020 --> 00:04:37.020
little bit too late. But the payload at
the end of our exploit would pop Apple's
00:04:37.020 --> 00:04:41.370
calculator app and a root shell on the
victim machine. This is usually used to
00:04:41.370 --> 00:04:45.930
demonstrate code execution. So, for fun we
also made the payload change the desktop's
00:04:45.930 --> 00:04:50.666
background to the Ret2 logo. So that's
what you're seeing there. So, what makes a
00:04:50.666 --> 00:04:54.620
Zero-Day a fun case study, is that we had
virtually no prior experience with Safari
00:04:54.620 --> 00:04:58.750
or MacOS going into this event. We
literally didn't even have a single
00:04:58.750 --> 00:05:03.220
Macbook in the office. We actually had to
go out and buy one. And, so as a result
00:05:03.220 --> 00:05:07.050
you get to see, how we as expert
researchers approach new and unknown
00:05:07.050 --> 00:05:12.060
software targets. So I promised that this
was a non-technical talk which is mostly
00:05:12.060 --> 00:05:16.710
true. That's because we actually publish
all the nitty gritty details for the
00:05:16.710 --> 00:05:21.530
entire exploit chain as a verbose six part
blog series on our blog this past summer.
00:05:21.530 --> 00:05:26.534
It's hard to make highly tactical talks
fun and accessible to all audiences. So
00:05:26.534 --> 00:05:31.210
we've reserved much of the truly technical
stuff for you to read at your own leisure.
00:05:31.210 --> 00:05:34.550
It's not a prerequisite for this talk, so
don't feel bad if you haven't read those.
00:05:34.550 --> 00:05:39.310
So with that in mind, we're ready to
introduce you to the very 1st step of what
00:05:39.310 --> 00:05:44.870
we're calling, The Layman's Guide to Zero-
Day Engineering. So, at the start of this
00:05:44.870 --> 00:05:48.730
talk, I said we'd be attacking some of the
most high value and well protected
00:05:48.730 --> 00:05:54.543
consumer software. This is no joke, right?
This is a high stakes game. So before any
00:05:54.543 --> 00:05:58.759
of you even think about looking at code,
or searching for vulnerabilities in these
00:05:58.759 --> 00:06:02.590
products, you need to set some
expectations about what you're going to be
00:06:02.590 --> 00:06:08.190
up against. So this is a picture of you.
You might be a security expert, a software
00:06:08.190 --> 00:06:11.757
engineer, or even just an enthusiast. But
there are some odd twists of self-
00:06:11.757 --> 00:06:16.009
loathing, you find yourself interested in
Zero-Days, and the desire to break some
00:06:16.009 --> 00:06:22.380
high impact software, like a web browser.
But it's important to recognize that
00:06:22.380 --> 00:06:25.759
you're looking to devise some of the
largest, most successful organizations of
00:06:25.759 --> 00:06:29.720
our generation. These types of companies have
every interest in securing their products
00:06:29.720 --> 00:06:33.410
and building trust with consumers. These
vendors have steadily been growing their
00:06:33.410 --> 00:06:36.449
investments in software and device
security, and that trend will only
00:06:36.449 --> 00:06:41.300
continue. You see cyber security in
headlines every day, hacking, you know,
00:06:41.300 --> 00:06:45.220
these systems compromised. It's only
getting more popular. You know, there's
00:06:45.220 --> 00:06:49.319
more money than ever in this space. This
is a beautiful mountain peak that
00:06:49.319 --> 00:06:53.330
represents your mission of, I want to
craft a Zero-Day. But you're cent up this
00:06:53.330 --> 00:06:58.410
mountain is not going to be an easy task.
As an individual, the odds are not really
00:06:58.410 --> 00:07:02.940
in your favor. This game is sort of a free
for all, and everyone is at each other's
00:07:02.940 --> 00:07:06.770
throats. So, in one corner is the vendor,
might as well have infinite money and
00:07:06.770 --> 00:07:10.729
infinite experience. In another corner, is
the rest of the security research
00:07:10.729 --> 00:07:16.199
community, fellow enthusiasts, other
threat actors. So, all of you are going to
00:07:16.199 --> 00:07:21.120
be fighting over the same terrain, which
is the code. This is unforgiving terrain
00:07:21.120 --> 00:07:25.669
in and of itself. But the vendor has home
field advantage. So these obstacles are
00:07:25.669 --> 00:07:30.419
not fun, but it's only going to get worse
for you. Newcomers often don't prepare
00:07:30.419 --> 00:07:34.810
themselves for understanding what kind of
time scale they should expect when working
00:07:34.810 --> 00:07:39.840
on these types of projects. So, for those
of you who are familiar with the Capture
00:07:39.840 --> 00:07:44.550
The Flag circuit. These competitions
usually are time boxed from 36 to 48
00:07:44.550 --> 00:07:49.191
hours. Normally, they're over a weekend. We
came out of that circuit. We love the
00:07:49.191 --> 00:07:54.880
sport. We still play. But how long does it
take to develop a Zero-Day? Well, it can
00:07:54.880 --> 00:07:58.780
vary a lot. Sometimes, you get really
lucky. I've seen someone produce a
00:07:58.780 --> 00:08:05.340
Chrome-/V8-bug in 2 days. Other times,
it's taken two weeks. Sometimes, it takes
00:08:05.340 --> 00:08:10.360
a month. But sometimes, it can actually
take a lot longer to study and exploit new
00:08:10.360 --> 00:08:13.960
targets. You need to be thinking, you
know, you need to be looking at time in
00:08:13.960 --> 00:08:19.509
these kind of scales. And so it could take
3.5 months. It could take maybe even 6
00:08:19.509 --> 00:08:23.021
months for some targets. The fact of the
matter is that it's almost impossible to
00:08:23.021 --> 00:08:28.370
tell how long the process is going take
you. And so unlike a CTF challenge,
00:08:28.370 --> 00:08:33.140
there's no upper bound to this process of
Zero-Day Engineering. There's no guarantee
00:08:33.140 --> 00:08:37.270
that the exploitable bugs you need to make
a Zero-Day, even exist in the software
00:08:37.270 --> 00:08:43.400
you're targeting. You also don't always
know what you're looking for, and you're
00:08:43.400 --> 00:08:47.540
working on projects that are many order
magnitudes larger than any sort of
00:08:47.540 --> 00:08:51.150
educational resource. We're talking
millions of lines of code where your
00:08:51.150 --> 00:08:56.780
average CTF challenge might be a couple
hundred lines to see at most. So I can
00:08:56.780 --> 00:09:01.673
already see the tear and self-doubt in
some of your eyes. But I really want to
00:09:01.673 --> 00:09:06.640
stress that you shouldn't be too hard on
yourself about this stuff. As a novice,
00:09:06.640 --> 00:09:11.470
you need to keep these caveats in mind and
accept that failure is not unlikely in the
00:09:11.470 --> 00:09:15.746
journey. All right? So please check this
box before watching the rest of the talk.
00:09:17.406 --> 00:09:21.010
So having built some psychological
foundation for the task at hand, the next
00:09:21.010 --> 00:09:28.130
step in the Layman's Guide is what we call
reconnaissance. So this is kind of a goofy
00:09:28.130 --> 00:09:33.566
slide, but yes, even Metasploit reminds
you to start out doing recon. So with
00:09:33.566 --> 00:09:36.530
regard to Zero-Day Engineering,
discovering vulnerabilities against large
00:09:36.530 --> 00:09:40.606
scale software can be an absolutely
overwhelming experience. Like that
00:09:40.606 --> 00:09:44.790
mountain, it's like, where do I start?
What hill do I go up? Like, where do I
00:09:44.790 --> 00:09:49.330
even go from there? So to overcome this,
it's vital to build foundational knowledge
00:09:49.330 --> 00:09:53.470
about the target. It's also one of the
least glamorous parts of the Zero-Day
00:09:53.470 --> 00:09:57.540
development process. And it's often
skipped by many. You don't see any of the
00:09:57.540 --> 00:10:01.000
other speakers really talking about this
so much. You don't see blog posts where
00:10:01.000 --> 00:10:05.480
people are like, I googled for eight hours
about Apple Safari before writing a Zero-
00:10:05.480 --> 00:10:11.160
Day for it. So you want to aggregate and
review all existing research related to
00:10:11.160 --> 00:10:17.266
your target. This is super, super
important. So how do we do our recon? Well
00:10:17.266 --> 00:10:21.790
the simple answer is Google everything.
This is literally us Googling something,
00:10:21.790 --> 00:10:25.210
and what we do is we go through and we
click, and we download, and we bookmark
00:10:25.210 --> 00:10:29.580
every single thing for about five pages.
And you see all those buttons that you
00:10:29.580 --> 00:10:33.542
never click at the bottom of Google. All
the things are related searches that you
00:10:33.542 --> 00:10:37.370
might want to look at. You should
definitely click all of those. You should
00:10:37.370 --> 00:10:40.960
also go through at least four or five
pages and keep downloading and saving
00:10:40.960 --> 00:10:48.040
everything that looks remotely relevant.
So you just keep doing this over, and
00:10:48.040 --> 00:10:53.621
over, and over again. And you just Google,
and Google, and Google everything that you
00:10:53.621 --> 00:10:58.730
think could possibly be related. And the
idea is, you know, you just want to grab
00:10:58.730 --> 00:11:02.260
all this information, you want to
understand everything you can about this
00:11:02.260 --> 00:11:07.766
target. Even if it's not Apple Safari
specific. I mean, look into V8, look into
00:11:07.766 --> 00:11:14.200
Chrome, look into Opera, look into Chakra,
look into whatever you want. So the goal
00:11:14.200 --> 00:11:19.370
is to build up a library of security
literature related to your target and its
00:11:19.370 --> 00:11:26.010
ecosystem. And then, I want you to read
all of it. But I don't want you, don't,
00:11:26.010 --> 00:11:29.120
don't force yourself to understand
everything in your sack in your
00:11:29.120 --> 00:11:32.720
literature. The point of this exercise is
to build additional concepts about the
00:11:32.720 --> 00:11:36.996
software, its architecture and its
security track record. By the end of the
00:11:36.996 --> 00:11:40.120
reconnaissance phase, you should aim to be
able to answer these kinds of questions
00:11:40.120 --> 00:11:45.727
about your target. What is the purpose of
the software? How is it architected? Can
00:11:45.727 --> 00:11:50.640
anyone describe what WebKit's architecture
is to me? What are its major components?
00:11:50.640 --> 00:11:55.880
Is there a sandbox around it? How do you
debug it? How did the developers debug it?
00:11:55.880 --> 00:12:00.550
Are there any tips and tricks, are there
special flags? What does its security
00:12:00.550 --> 00:12:04.265
track record look like? Does it have
historically vulnerable components? Is
00:12:04.265 --> 00:12:10.630
there existing writeups, exploits, or
research in it? etc. All right, we're
00:12:10.630 --> 00:12:16.450
through reconnaissance. Step 2 is going to
be target selection. So, there's actually
00:12:16.450 --> 00:12:20.190
a few different names that you could maybe
call this. Technically, we're targeting
00:12:20.190 --> 00:12:24.684
Apple's Safari, but you want to try and
narrow your scope. So what we're looking
00:12:24.684 --> 00:12:32.520
at here is a TreeMap visualization of the
the Web Kit source. So Apple's Safari web
00:12:32.520 --> 00:12:36.030
browser is actually built on top of the
Web Kit framework which is essentially a
00:12:36.030 --> 00:12:42.000
browser engine. This is Open Source. So
yeah, this is a TreeMap visualization of
00:12:42.000 --> 00:12:47.000
the source directory where files are
sorted in by size. So each of those boxes
00:12:47.000 --> 00:12:53.020
is essentially a file. While all the grey
boxes, the big gray boxes are directories.
00:12:53.020 --> 00:13:02.240
All the sub squares are files. In each
file is sized, based on its lines of code.
00:13:02.240 --> 00:13:07.490
Hue, the blue hues represent approximate
maximum cyclomatic complexity detected in
00:13:07.490 --> 00:13:10.810
each source file. And you might be
getting, anyway, you might be getting
00:13:10.810 --> 00:13:14.191
flashbacks back to that picture of that
mountain peak. How do you even start to
00:13:14.191 --> 00:13:17.546
hunt for security vulnerabilities in a
product or codebase of this size?
00:13:17.546 --> 00:13:22.070
3 million lines of code. You know, I've
maybe written like, I don't know, like a
00:13:22.070 --> 00:13:28.687
100,000 lines of C or C++ in my life, let
alone read or reviewed 3 million. So the
00:13:28.687 --> 00:13:33.963
short answer to breaking this problem down
is that you need to reduce your scope of
00:13:33.963 --> 00:13:39.950
valuation, and focus on depth over
breadth. And this is most critical when
00:13:39.950 --> 00:13:44.980
attacking extremely well picked over
targets. Maybe you're probing an IoT
00:13:44.980 --> 00:13:47.600
device? You can probably just sneeze at
that thing and you are going to find
00:13:47.600 --> 00:13:52.424
vulnerabilities. But you know, you're
fighting on a very different landscape here.
00:13:52.424 --> 00:13:59.820
And so you need to be very detailed
with your review. So reduce your scope.
00:13:59.820 --> 00:14:03.950
Our reconnaissance and past experience
with exploiting browsers had lead us to
00:14:03.950 --> 00:14:09.090
focus on WebKit's JavaScript engine,
highlighted up here in orange. So, bugs in
00:14:09.090 --> 00:14:14.120
JS engines, when it comes to browsers, are
generally regarded as extremely powerful
00:14:14.120 --> 00:14:18.460
bugs. But they're also few and far
between, and they're kind of becoming more
00:14:18.460 --> 00:14:23.550
rare, as more of you are looking for bugs.
More people are colliding, they're dying
00:14:23.550 --> 00:14:28.920
quicker. And so, anyway, let's try to
reduce our scope. So we reduce our scope
00:14:28.920 --> 00:14:33.820
from 3 million down in 350,000 lines of
code. Here, we'll zoom into that orange.
00:14:33.820 --> 00:14:37.200
So now we're looking at the JavaScript
directory, specifically the JavaScript
00:14:37.200 --> 00:14:42.490
core directory. So this is a JavaScript
engine within WebKit, as used by Safari,
00:14:42.490 --> 00:14:47.820
on MacOS. And specifically, to further
reduce our scope, we chose to focus on the
00:14:47.820 --> 00:14:52.821
highest level interface of the JavaScript
core which is the runtime folder. So this
00:14:52.821 --> 00:14:57.800
contains code that's almost 1 for 1
mappings to JavaScript objects and methods
00:14:57.800 --> 00:15:05.791
in the interpreter. So, for example,
Array.reverse, or concat, or whatever.
00:15:05.791 --> 00:15:10.901
It's very close to what you JavaScript
authors are familiar with. And so this is
00:15:10.901 --> 00:15:16.836
what the runtime folder looks like, at
approximately 70,000 lines of code. When
00:15:16.836 --> 00:15:21.630
we were spinning up for PWN2OWN, we said,
okay, we are going to find a bug in this
00:15:21.630 --> 00:15:25.680
directory in one of these files, and we're
not going to leave it until we have, you
00:15:25.680 --> 00:15:30.784
know, walked away with something. So if we
take a step back now. This is what we
00:15:30.784 --> 00:15:34.592
started with, and this is what we've done.
We've reduced our scope. So it helps
00:15:34.592 --> 00:15:39.490
illustrate this, you know, whittling
process. It was almost a little bit
00:15:39.490 --> 00:15:44.310
arbitrary. There's a lot, previously,
there's been a lot of bugs in the runtime
00:15:44.310 --> 00:15:51.380
directory. But it's really been cleaned up
the past few years. So anyway, this is
00:15:51.380 --> 00:15:56.870
what we chose for our RCE. So having spent
a number of years going back and forth
00:15:56.870 --> 00:16:00.930
between attacking and defending, I've come
to recognize that bad components do not
00:16:00.930 --> 00:16:05.200
get good fast. Usually researchers are
able to hammer away at these components
00:16:05.200 --> 00:16:10.520
for years before they reach some level of
acceptable security. So to escape Safari's
00:16:10.520 --> 00:16:15.084
sandbox, we simply looked at the security
trends covered during the reconnaissance phase.
00:16:15.084 --> 00:16:18.790
So, this observation, historically
bad components will often take years to
00:16:18.790 --> 00:16:23.542
improve, means that we chose to look at
WindowServer. And for those that don't
00:16:23.542 --> 00:16:29.390
know, WindowServer is a root level system
service that runs on MacOS. Our research
00:16:29.390 --> 00:16:35.570
turned up a trail of ugly bugs from a
MacOS, from essentially the WindowServer
00:16:35.570 --> 00:16:42.910
which is accessible to the Safari sandbox.
And in particular, when we're doing our
00:16:42.910 --> 00:16:47.110
research, we're looking at ZDI's website,
and you can just search all their
00:16:47.110 --> 00:16:52.910
advisories that they've disclosed. And in
particular in 2016, there is over 10 plus
00:16:52.910 --> 00:16:57.220
vulnerabilities report to ZDI that were
used as sandbox escapes or privilege
00:16:57.220 --> 00:17:03.406
escalation style issues. And so, these are
only vulnerabilities that are reported to
00:17:03.406 --> 00:17:09.500
ZDI. If you look in 2017, there is 4 all,
again, used for the same purpose. I think,
00:17:09.500 --> 00:17:15.600
all of these were actually, probably used
at PWN2OWN both years. And then in 2018,
00:17:15.600 --> 00:17:19.720
there is just one. And so, this is 3
years. Over the span of 3 years where
00:17:19.720 --> 00:17:25.010
people were hitting the same exact
component, and Apple or researchers around
00:17:25.010 --> 00:17:28.810
the world could have been watching, or
listening and finding bugs, and fighting
00:17:28.810 --> 00:17:36.230
over this land right here. And so, it's
pretty interesting. I mean, they give some
00:17:36.230 --> 00:17:42.200
perspective. The fact of the matter is
that it's hard to write, it's really hard
00:17:42.200 --> 00:17:46.250
for bad components to improve quickly.
Nobody wants to try and sit down and
00:17:46.250 --> 00:17:50.497
rewrite bad code. And vendors are
terrified, absolutely terrified of
00:17:50.497 --> 00:17:55.400
shipping regressions. Most vendors will
only patch or modify old bad code only
00:17:55.400 --> 00:18:02.370
when they absolutely must. For example,
when a vulnerability is reported to them.
00:18:02.370 --> 00:18:07.930
And so, as listed on this slide, there's a
number of reasons why a certain module or
00:18:07.930 --> 00:18:12.780
component has a terrible security track
record. Just try to keep in mind, that's
00:18:12.780 --> 00:18:17.740
usually a good place to look for more
bugs. So if you see a waterfall of bugs
00:18:17.740 --> 00:18:22.660
this year in some component, like, wasm or
JIT, maybe you should be looking there,
00:18:22.660 --> 00:18:27.401
right? Because that might be good for a
few more years. All right.
00:18:28.060 --> 00:18:32.240
Step three. So after all this talk, we are finally
getting to a point where we can start
00:18:32.240 --> 00:18:38.470
probing and exploring the codebase in
greater depth. This step is all about bug hunting.
00:18:38.470 --> 00:18:45.280
So as an individual researcher or
small organization, the hardest part of
00:18:45.280 --> 00:18:48.957
the Zero-Day engineering process is
usually discovering and exploiting a
00:18:48.957 --> 00:18:53.400
vulnerability. That's just kind of from
our perspective. This can maybe vary from
00:18:53.400 --> 00:18:58.040
person to person. But you know, we don't
have a hundred million dollars to spend on
00:18:58.040 --> 00:19:06.360
fuzzers, for example. And so we literally
have one Macbook, right? So it's kind of
00:19:06.360 --> 00:19:10.920
like looking for a needle in a haystack.
We're also well versed in the exploitation
00:19:10.920 --> 00:19:14.938
process itself. And so those end up being
a little bit more formulaic for ourselves.
00:19:14.938 --> 00:19:18.500
So there are two core strategies for
finding exploitable vulnerabilities.
00:19:18.500 --> 00:19:22.133
There's a lot of pros and cons to both of
these approaches. But I don't want to
00:19:22.133 --> 00:19:25.448
spend too much time talking about their
strengths or weaknesses. So they're all
00:19:25.448 --> 00:19:30.610
listed here, the short summary is that
fuzzing is the main go-to strategy for
00:19:30.610 --> 00:19:36.970
many security enthusiasts. Some of the key
perks is that it's scalable and almost
00:19:36.970 --> 00:19:41.841
always yields result. And so, spoiler
alert, but later in the software industry,
00:19:41.841 --> 00:19:48.640
we fuzzed both of our bugs. Both the bugs
that we use for a full chain. And we know
00:19:48.640 --> 00:19:53.806
it's 2018. These things are still falling
out with some very trivial means. OK. So,
00:19:53.806 --> 00:19:58.860
source review is the other main strategy.
Source review is often much harder for
00:19:58.860 --> 00:20:02.746
novices, but it can produce some high
quality bugs when performed diligently.
00:20:02.746 --> 00:20:06.200
You know if you're looking to just get
into this stuff, I would say, start real
00:20:06.200 --> 00:20:12.930
simple, start with fuzzing and see how
fare you get. So, yeah, for the purpose of
00:20:12.930 --> 00:20:16.490
this talk, we are mostly going to focus on
fuzzing. This is a picture from the
00:20:16.490 --> 00:20:21.080
dashboard of a simple, scalable fuzzing
harness we built for JavaScript core. This
00:20:21.080 --> 00:20:25.457
is when we were ramping up for PWN2OWN and
trying to build our chain. It was a
00:20:25.457 --> 00:20:30.310
grammar based JavaScript fuzzer, based on
Mozilla's Darma. There is nothing fancy
00:20:30.310 --> 00:20:34.723
about it. This is a snippet of some of
what some of its output looked like. We
00:20:34.723 --> 00:20:37.860
had only started building it out when we
actually found the exploitable
00:20:37.860 --> 00:20:42.520
vulnerability that we ended up using. So
we haven't really played with this much
00:20:42.520 --> 00:20:48.030
since then, but it's, I mean, it shows
kind of how easy it was to get where we
00:20:48.030 --> 00:20:55.221
needed to go. So, something like we'd like
to stress heavily to the folks who fuzz,
00:20:55.221 --> 00:20:59.720
is that it really must be treated as a
science for these competitive targets.
00:20:59.720 --> 00:21:04.180
Guys, I know code coverage isn't the best
metric, but you absolutely must use some
00:21:04.180 --> 00:21:08.552
form of introspection to quantify the
progress and reach of your fuzzing. Please
00:21:08.552 --> 00:21:13.730
don't just fuzz blindly. So our fuzzer
would generate web based code coverage
00:21:13.730 --> 00:21:18.160
reports of our grammars every 15 minutes,
or so. This allowed us to quickly iterate
00:21:18.160 --> 00:21:22.594
upon our fuzzer, helping generate more
interesting complex test cases. A good
00:21:22.594 --> 00:21:26.050
target is 60 percent code coverage. So you
can see that in the upper right hand
00:21:26.050 --> 00:21:29.050
corner. That's kind of what we were
shooting for. Again, it really varies from
00:21:29.050 --> 00:21:33.849
target to target. This was also just us
focusing on the runtime folder. If you see
00:21:33.849 --> 00:21:39.400
in the upper left hand corner. And so,
something that we have observed, again
00:21:39.400 --> 00:21:45.910
over many targets and exotic, exotic
targets, is that bugs almost always fall
00:21:45.910 --> 00:21:51.624
out of what we call the hard fought final
coverage percentages. And so, what this
00:21:51.624 --> 00:21:55.960
means is, you might work for a while,
trying to build up your coverage, trying
00:21:55.960 --> 00:22:02.030
to, you know, build a good set of test
cases, or Grammar's for fuzzing, and then
00:22:02.030 --> 00:22:06.030
you'll hit that 60 percent, and you'll be,
okay, what am I missing now? Like everyone
00:22:06.030 --> 00:22:09.550
gets that 60 percent, let's say. But then,
once you start inching a little bit
00:22:09.550 --> 00:22:15.150
further is when you start fining a lot of
bugs. So, for example, we will pull up
00:22:15.150 --> 00:22:19.190
code, and we'll be like, why did we not
hit those blocks up there? Why are those
00:22:19.190 --> 00:22:22.630
grey box? Why did we never hit those in
our millions of test cases? And we'll go
00:22:22.630 --> 00:22:26.020
find that that's some weird edge case, or
some unoptimized condition, or something
00:22:26.020 --> 00:22:32.160
like that, and we will modify our test
cases to hit that code. Other times we'll
00:22:32.160 --> 00:22:36.230
actually sit down pull it up on our
projector and talk through some of that
00:22:36.230 --> 00:22:39.420
and we'll be like: What the hell is going
on there? This is actually, it's funny,
00:22:39.420 --> 00:22:44.341
this is actually a live photo that I took
during our pawn2own hunt. You know as
00:22:44.341 --> 00:22:48.430
cliche as this picture is of hackers
standing in front of like a dark screen in
00:22:48.430 --> 00:22:52.090
a dark room, this was absolutely real. You
know we were we were just reading some
00:22:52.090 --> 00:23:02.190
code. And so it's good to rubber ducky
among co-workers and to hash out ideas to
00:23:02.190 --> 00:23:10.520
help confirm theories or discard them. And
so yeah this kinda leads us to the next
00:23:10.520 --> 00:23:15.220
piece of advice is when you're doing
source reviews so this applies to both
00:23:15.220 --> 00:23:21.060
debugging or assessing those corner cases
and whatnot. If you're ever unsure about
00:23:21.060 --> 00:23:24.800
the code that you're reading you
absolutely should be using debuggers and
00:23:24.800 --> 00:23:30.250
dynamic analysis. So as painful as it can
maybe be to set up a JavaScript core or
00:23:30.250 --> 00:23:35.500
debug this massive C++ application that's
dumping these massive call stacks that are
00:23:35.500 --> 00:23:39.900
100 [steps] deep you need to learn those
tools or you are never going to be able to
00:23:39.900 --> 00:23:46.700
understand the amount of context necessary
for some of these bugs and complex code.
00:23:46.700 --> 00:23:55.010
So for example one of our blog posts makes
extensive use of rr to reverse or to root
00:23:55.010 --> 00:23:58.831
cause the vulnerability that we end up
exploiting. It was a race condition in the
00:23:58.831 --> 00:24:03.270
garbage collector - totally wild bug.
There's probably, I said there's probably
00:24:03.270 --> 00:24:07.830
3 people on earth that could have spotted
this bug through source review. It
00:24:07.830 --> 00:24:12.690
required immense knowledge of code base in
my opinion to be able to recognize this as
00:24:12.690 --> 00:24:16.250
a vulnerability. We found it through
fuzzing, we had a root cause it using time
00:24:16.250 --> 00:24:23.630
travel debugging. Mozilla's rr which is an
amazing project. And so, yeah. Absolutely
00:24:23.630 --> 00:24:28.100
use debuggers. This is an example of a
call stack again, just using a debugger to
00:24:28.100 --> 00:24:32.043
dump the callstack from a function that
you are auditing can give you an insane
00:24:32.043 --> 00:24:36.500
amount of context as to how that function
is used, what kind of data it's operating
00:24:36.500 --> 00:24:42.294
on. Maybe, you know, what kind of areas of
the codebase it's called from. You're not
00:24:42.294 --> 00:24:46.490
actually supposed to be able to read the
size or read the slide but it's a
00:24:46.490 --> 00:24:56.370
backtrace from GDB that is 40 or 50 call
stacks deep. All right. So there is this
00:24:56.370 --> 00:25:01.420
huge misconception by novices that new
code is inherently more secure and that
00:25:01.420 --> 00:25:07.054
vulnerabilities are only being removed
from code bases, not added. This is almost
00:25:07.054 --> 00:25:11.980
patently false and this is something that
I've observed over the course of several
00:25:11.980 --> 00:25:18.377
years. Countless targets you know, code
from all sorts of vendors. And there's
00:25:18.377 --> 00:25:24.340
this really great blog post put out by
Ivan from GPZ this past fall and in his
00:25:24.340 --> 00:25:29.469
blog post he basically ... so one year ago
he fuzzed WebKit using his fuzzer
00:25:29.469 --> 00:25:33.250
called Domato. He found a bunch of
vulnerabilities, he reported them and then
00:25:33.250 --> 00:25:39.680
he open sourced the fuzzer. But then this
year, this fall, he downloads his fuzzer,
00:25:39.680 --> 00:25:43.553
ran it again with little to no changes,
just to get things up and running. And
00:25:43.553 --> 00:25:47.680
then he found another eight plus
exploitable use after free vulnerabilities.
00:25:47.680 --> 00:25:51.400
So what's really amazing
about this, is when you look at these last
00:25:51.400 --> 00:25:55.590
two columns that I have highlighted in
red, virtually all the bugs he found had
00:25:55.590 --> 00:26:03.950
been introduced or regressed in the past
12 months. So yes, new vulnerabilities get
00:26:03.950 --> 00:26:11.110
introduced every single day. The biggest
reason new code is considered harmful, is
00:26:11.110 --> 00:26:16.270
simply that it's not had years to sit in
market. This means it hasn't had time to
00:26:16.270 --> 00:26:20.843
mature, it hasn't been tested exhaustively
like the rest of the code base. As soon as
00:26:20.843 --> 00:26:24.786
that developer pushes it, whenever it hits
release, whenever it hits stable that's
00:26:24.786 --> 00:26:29.000
when you have a billion users pounding at
- it let's say on Chrome. I don't know how
00:26:29.000 --> 00:26:33.020
big that user base is but it's massive and
that's a thousand users around the world
00:26:33.020 --> 00:26:37.537
just using the browser who are effectively
fuzzing it just by browsing the web. And
00:26:37.537 --> 00:26:41.120
so of course you're going to manifest
interesting conditions that will cover
00:26:41.120 --> 00:26:47.046
things that are not in your test cases and
unit testing. So yeah, it's not uncommon.
00:26:47.046 --> 00:26:50.490
The second point down here is that it's
not uncommon for new code to break
00:26:50.490 --> 00:26:55.360
assumptions made elsewhere in the code
base. And this is also actually extremely
00:26:55.360 --> 00:26:59.720
common. The complexity of these code bases
can be absolutely insane and it can be
00:26:59.720 --> 00:27:04.263
extremely hard to tell if let's say some
new code that Joe Schmoe, the new
00:27:04.263 --> 00:27:10.420
developer, adds breaks some paradigm held
by let's say the previous owner of the
00:27:10.420 --> 00:27:14.870
codebase. He maybe doesn't understand it
as well - you know, maybe it could be an
00:27:14.870 --> 00:27:22.996
expert developer who just made a mistake.
It's super common. Now a piece of advice.
00:27:22.996 --> 00:27:27.200
This should be a no brainer for bug
hunting but novices often grow impatient
00:27:27.200 --> 00:27:30.320
and start hopping around between code and
functions and getting lost or trying to
00:27:30.320 --> 00:27:35.920
chase use-after-frees or bug classes
without really truly understanding what
00:27:35.920 --> 00:27:41.780
they're looking for. So a great starting
point is always identify the sources of
00:27:41.780 --> 00:27:45.700
user input or the way that you can
interface with the program and then just
00:27:45.700 --> 00:27:50.890
follow the data, follow it down. You know
what functions parse it, what manipulates
00:27:50.890 --> 00:27:58.611
your data, what reads it, what writes to
it. You know just keep it simple. And so
00:27:58.611 --> 00:28:03.570
when we're looking for our sandbox escapes
when you're looking at window server and
00:28:03.570 --> 00:28:07.370
our research had showed that there's all
of these functions we don't know anything
00:28:07.370 --> 00:28:12.000
about Mac but we read this blog post from
Keine that was like "Oh there's all these
00:28:12.000 --> 00:28:15.700
functions that you can send data to in
window server" and apparently there's
00:28:15.700 --> 00:28:21.090
about six hundred and there are all these
functions prefixed with underscore
00:28:21.090 --> 00:28:28.250
underscore x. And so the 600 end points
will parse and operate upon data that we
00:28:28.250 --> 00:28:33.430
send to them. And so to draw a rough
diagram, there is essentially this big red
00:28:33.430 --> 00:28:37.760
data tube from the safari sandbox to the
windows server system service. This tube
00:28:37.760 --> 00:28:43.651
can deliver arbitrary data that we control
to all those six hundred end points. We
00:28:43.651 --> 00:28:48.001
immediately thought let's just try to man-
in-the-middle this data pipe, so that we
00:28:48.001 --> 00:28:52.260
can see what's going on. And so that's
exactly what we did. We just hooked up
00:28:52.260 --> 00:28:58.898
FRIDA to it, another open source DBI. It's
on GitHub. It's pretty cool. And we're
00:28:58.898 --> 00:29:04.590
able to stream all of the messages flowing
over this pipe so we can see all this data
00:29:04.590 --> 00:29:08.870
just being sent into the window server
from all sorts of applications - actually
00:29:08.870 --> 00:29:12.800
everything on Mac OS talks to this. The
window server is responsible for drawing
00:29:12.800 --> 00:29:17.490
all your windows on the desktop, your
mouse clicks, your whatever. It's kind of
00:29:17.490 --> 00:29:23.820
like explorer.exe on Windows. So you know
we see all this data coming through, we
00:29:23.820 --> 00:29:29.050
see all these crazy messages, all these
unique message formats, all these data
00:29:29.050 --> 00:29:33.948
buffers that it's sending in and this is
just begging to be fuzzed. And so we said
00:29:33.948 --> 00:29:38.080
"OK, let's fuzz it" and we're getting all
hyped and I distinctly remember thinking
00:29:38.080 --> 00:29:42.440
maybe we can jerry-rig AFL into the
window server or let's mutate these
00:29:42.440 --> 00:29:51.059
buffers with Radamsa or why don't we just
try flipping some bits. So that's what we
00:29:51.059 --> 00:29:57.030
did. So I actually had a very timely tweet
just a few weeks back that echoed this
00:29:57.030 --> 00:30:02.590
exact experience. He said that "Looking at
my Security / Vulnerability research
00:30:02.590 --> 00:30:06.753
career, my biggest mistakes were almost
always trying to be too clever. Success
00:30:06.753 --> 00:30:11.587
hides behind what is the dumbest thing
that could possibly work." The takeaway
00:30:11.587 --> 00:30:18.030
here is that you should always start
simple and iterate. So this is our Fuzz
00:30:18.030 --> 00:30:22.520
farm. It's a single 13 inch MacBook Pro. I
don't know if this is actually going to
00:30:22.520 --> 00:30:26.220
work, it's not a big deal if it doesn't.
I'm only gonna play a few seconds of it.
00:30:26.220 --> 00:30:31.953
This is me literally placing my wallet on
the enter key and you can see this box
00:30:31.953 --> 00:30:35.490
popping up and we're fuzzing - our fuzzer
is running now and flipping bits in the
00:30:35.490 --> 00:30:39.366
messages. And the screen is changing
colors. You're going to start seeing the
00:30:39.366 --> 00:30:42.905
boxes freaking out. It's going all over
the place. This is because the bits are
00:30:42.905 --> 00:30:46.660
being flipped, it's corrupting stuff, it's
changing the messages. Normally, this
00:30:46.660 --> 00:30:51.330
little box is supposed to show your
password hint. But the thing is by holding
00:30:51.330 --> 00:30:56.240
the enter key on the locked screen. All
this traffic was being generated to the
00:30:56.240 --> 00:31:00.490
window server, and every time the window
server crashed - you know where it brings
00:31:00.490 --> 00:31:03.809
you? It brings you right back to your lock
screen. So we had this awesome fuzzing
00:31:03.809 --> 00:31:16.003
setup by just holding the enter key.
Applause
00:31:16.003 --> 00:31:21.810
And we you know we lovingly titled that
picture "Advanced Persistent Threat" in
00:31:21.810 --> 00:31:29.630
our blog. So this is a crash that we got
out of the fuzzer. This occurred very
00:31:29.630 --> 00:31:34.299
quickly after ... this was probably within
the first 24 hours. So we found a ton of
00:31:34.299 --> 00:31:38.670
crashes, we didn't even explore all of
them. There is probably a few still
00:31:38.670 --> 00:31:44.380
sitting on our server. But there's lots
and all the rest ... lots of garbage. But
00:31:44.380 --> 00:31:48.780
then this one stands out in particular:
Anytime you see this thing up here that says
00:31:48.780 --> 00:31:53.679
"EXC_BAD_ACCESS" with a big number up
there with address equals blah blah blah.
00:31:53.679 --> 00:31:57.520
That's a really bad place to be. And so
this is a bug that we end up using at
00:31:57.520 --> 00:32:01.250
pwn2own to perform our sandbox escape if
you want to read about it again, it's on
00:32:01.250 --> 00:32:05.710
the blog, we're not going to go too deep
into it here. So maybe some of you have
00:32:05.710 --> 00:32:12.590
seen the infosec comic. You know it's all
about how people try to do these really
00:32:12.590 --> 00:32:17.320
cool clever things. They get ... People
can get too caught up trying to inject so
00:32:17.320 --> 00:32:21.930
much science and technology into these
problems that they often miss the forest
00:32:21.930 --> 00:32:26.750
for the trees. And so here we are in the
second panel. We just wrote this really
00:32:26.750 --> 00:32:31.832
crappy little fuzzer and we found our bug
pretty quickly. And this guy's really
00:32:31.832 --> 00:32:38.670
upset. Which brings us to the
misconception that only expert researchers
00:32:38.670 --> 00:32:42.950
with blank tools can find bugs. And so you
can fill in the blank with whatever you
00:32:42.950 --> 00:32:51.000
want. It can be cutting edge tools, state
of the art, state sponsored, magic bullet.
00:32:51.000 --> 00:32:58.720
This is not true. There are very few
secrets. So the next observation: you
00:32:58.720 --> 00:33:03.350
should be very wary of any bugs that you
find quickly. A good mantra is that an
00:33:03.350 --> 00:33:08.840
easy to find bug is just as easily found
by others. And so what this means is that
00:33:08.840 --> 00:33:13.260
soon after our blog post went out ...
actually at pwn2own 2018 we actually knew
00:33:13.260 --> 00:33:18.640
we had collided with fluorescence one of
the other competitors. We both struggled
00:33:18.640 --> 00:33:24.580
with exploiting this issue ... is a
difficult bug to exploit. And we were ...
00:33:24.580 --> 00:33:29.884
we had some very creative exploit, it was
very strange. But there was some
00:33:29.884 --> 00:33:33.490
discussion after the fact on Twitter by
nadge - started by Neddy -he's probably
00:33:33.490 --> 00:33:36.299
out here, actually speaking tomorrow. You
guys should go see this talk about the
00:33:36.299 --> 00:33:42.610
Chrome IPC. That should be really good.
But there is some discussion on Twitter,
00:33:42.610 --> 00:33:47.170
that Ned had started, and Nicholas, who is
also here, said "well, at least three
00:33:47.170 --> 00:33:52.130
teams found it separately". So at least
us, fluorescence and Nicholas had found
00:33:52.130 --> 00:33:57.320
this bug. And we were all at pwn2own, so
you can think how many people out there
00:33:57.320 --> 00:34:01.200
might have also found this. There's
probably at least a few. How many people
00:34:01.200 --> 00:34:07.160
actually tried to weaponize this thing?
Maybe not many. It is kind of a difficult
00:34:07.160 --> 00:34:14.790
bug. And so there are probably at least a
few other researchers who are aware of
00:34:14.790 --> 00:34:21.070
this bug. So yeah, that kinda closes the,
you know, if you found a bug very quickly
00:34:21.070 --> 00:34:25.360
especially with fuzzing, you can almost
guarantee that someone else has found it.
00:34:25.360 --> 00:34:31.040
So I want to pass over the next section to
Amy to continue.
00:34:31.040 --> 00:34:38.360
Amy: So we just talked a bunch about, you
know, techniques and expectations when
00:34:38.360 --> 00:34:42.070
you're actually looking for the bug. Let
me take over here and talk a little bit
00:34:42.070 --> 00:34:48.179
about what to expect when trying to
exploit whatever bug you end up finding.
00:34:48.179 --> 00:34:53.600
Yeah and so we have the exploit
development as the next step. So OK, you
00:34:53.600 --> 00:34:57.221
found a bug right, you've done the hard
part. You were looking at whatever your
00:34:57.221 --> 00:35:01.090
target is, maybe it's a browser maybe it's
the window server or the kernel or
00:35:01.090 --> 00:35:05.784
whatever you're trying to target. But the
question is how do you actually do the rest?
00:35:05.784 --> 00:35:10.500
How do you go from the bug to
actually popping a calculator onto the
00:35:10.500 --> 00:35:15.650
screen? The systems that you're working
with have such a high level of complexity
00:35:15.650 --> 00:35:19.880
that even just like understanding you know
enough to know how your bug works it might
00:35:19.880 --> 00:35:23.650
not be enough to actually know how to
exploit it. So we try to like brute force
00:35:23.650 --> 00:35:28.560
our way to an exploit, is that a good
idea? Well all right before we try to
00:35:28.560 --> 00:35:34.390
tackle your bug let's take a step back and
ask a slightly different question. How do
00:35:34.390 --> 00:35:39.410
we actually write an exploit like this in
general? Now I feel like a lot of people
00:35:39.410 --> 00:35:44.461
consider these kind of exploits maybe be
in their own league at least when you
00:35:44.461 --> 00:35:49.750
compare them to something like maybe what
you'd do at a CTF competition or something
00:35:49.750 --> 00:35:55.859
simpler like that. And if you were for
example to be given a browser exploit
00:35:55.859 --> 00:36:00.340
challenge at a CTF competition it may seem
like an impossibly daunting task has just
00:36:00.340 --> 00:36:04.620
been laid in front of you if you've never
done this stuff before. So how can we work
00:36:04.620 --> 00:36:09.350
to sort of change that view? And you know
it might be kind of cliche but I actually
00:36:09.350 --> 00:36:14.090
think the best way to do it is through
practice. And I know everyone says "oh how
00:36:14.090 --> 00:36:19.540
do you get good", "oh, practice". But I
think that this is actually very valuable
00:36:19.540 --> 00:36:24.930
for this and the way that practicing
actually comes out is that, well, before we
00:36:24.930 --> 00:36:29.010
talked a lot about consuming everything
you could about your targets, like
00:36:29.010 --> 00:36:33.740
searching for everything you could that's
public, downloading it, trying to read it
00:36:33.740 --> 00:36:36.960
even if you don't understand it, because
you'll hopefully gleam something from it
00:36:36.960 --> 00:36:41.550
it doesn't hurt but maybe your goal now
could be actually trying to understand it
00:36:41.550 --> 00:36:46.810
as at least as much as you can. You know
it's going to be... it's not going to be
00:36:46.810 --> 00:36:53.130
easy. These are very intricate systems
that we're attacking here. And so it will
00:36:53.130 --> 00:36:57.150
be a lot of work to understand this stuff.
But for every old exploit you can work
00:36:57.150 --> 00:37:02.240
your way through, the path will become
clearer for actually exploiting these
00:37:02.240 --> 00:37:10.600
targets. So as because I focused mostly on
browser work and I did that browser part
00:37:10.600 --> 00:37:16.540
of our chain, at least the exploitation
part. I have done a lot of exploits and
00:37:16.540 --> 00:37:21.080
read a ton of browser exploits and one
thing that I have found is that a lot of
00:37:21.080 --> 00:37:26.170
them have very very similar structure. And
they'll have similar techniques in them
00:37:26.170 --> 00:37:30.923
they'll have similar sort of primitives
that are being used to build up the
00:37:30.923 --> 00:37:37.660
exploit. And so that's one observation.
And to actually illustrate that I have an
00:37:37.660 --> 00:37:42.770
example. So alongside us at this at
PWN2OWN this spring we had Samuel Groffs
00:37:42.770 --> 00:37:48.920
of Phoenix. He's probably here right now.
So he was targeting Safari just like we
00:37:48.920 --> 00:37:53.550
were. But his bug was in the just in time
compiler, the JIT, which converts
00:37:53.550 --> 00:38:00.061
JavaScript to the machine code. Our bug
was nowhere near that. It was over in the
00:38:00.061 --> 00:38:06.250
garbage collector so a completely
different kind of bug. But the bug here is
00:38:06.250 --> 00:38:11.014
super reliable. It was very very clean. I
recommend you go look at it online. It's a
00:38:11.014 --> 00:38:18.850
very good resource. And then, a few months
later, at PWN2OWN Mobile, so another pwning
00:38:18.850 --> 00:38:24.430
event, we have Fluoroacetate, which was an
amazing team who managed to pretty much
00:38:24.430 --> 00:38:28.071
pwn everything they could get their hands
on at that competition, including an
00:38:28.071 --> 00:38:33.103
iPhone which of course iPhone uses Safari
so they needed a Safari bug. The safari
00:38:33.103 --> 00:38:37.690
bug that they had was very similar in
structure to the previous bug earlier that
00:38:37.690 --> 00:38:43.320
year, at least in terms of how the bug
worked and what you could do with it. So
00:38:43.320 --> 00:38:47.760
now you could exploit both of these bugs
with very similar exploit code almost in
00:38:47.760 --> 00:38:52.950
the same way. There were a few tweaks you
had to do because Apple added a few things
00:38:52.950 --> 00:39:01.100
since then. But the path between bug and
code execution was very similar. Then,
00:39:01.100 --> 00:39:07.070
even a few months after that, there is a
CTF called "Realworld CTF" which took
00:39:07.070 --> 00:39:11.050
place in China and as the title suggests
they had a lot of realistic challenges
00:39:11.050 --> 00:39:18.280
including Safari. So of course my team
RPISEC was there and they woke me up in
00:39:18.280 --> 00:39:23.000
the middle of the night and tasked me with
solving it. And so I was like "Okay, okay
00:39:23.000 --> 00:39:27.760
I'll look at this". And I looked at it and
it was a JIT bug and I've never actually
00:39:27.760 --> 00:39:34.720
before that looked at the Safari JIT. And
so, you know, I didn't have much previous
00:39:34.720 --> 00:39:40.490
experience doing that, but because I had
taken the time to read all the public
00:39:40.490 --> 00:39:45.300
exploits. So I read all the other PWN2OWN
competitors exploits and I read all the
00:39:45.300 --> 00:39:49.470
other things that people were releasing
for different CVEs. I had seen a bug like
00:39:49.470 --> 00:39:54.790
this before very similar and I knew how to
exploit it, so I could... I was able to
00:39:54.790 --> 00:39:59.460
quickly build the path from bug to code
exec and we actually managed to get first
00:39:59.460 --> 00:40:03.350
blood on the challenge which was really
really cool.
00:40:03.350 --> 00:40:12.020
Applaus
So... So what does this actually mean?
00:40:12.020 --> 00:40:19.160
Well I think not every bug is going to be
that easily to swap into an exploit but I
00:40:19.160 --> 00:40:23.350
do think that understanding old exploits
is extremely valuable if you're trying to
00:40:23.350 --> 00:40:28.840
exploit new bugs. A good place to start if
you're interested in looking at old bugs
00:40:28.840 --> 00:40:34.170
is on places like this with the js-vuln-db,
which is a basically a repository of a
00:40:34.170 --> 00:40:39.200
whole bunch of JavaScript bugs and proof
of concepts and sometimes even exploits
00:40:39.200 --> 00:40:43.690
for them. And so if you were to go through
all of those, I guarantee by the end you'd
00:40:43.690 --> 00:40:49.560
have a great understanding of the types of
bugs that are showing up these days and
00:40:49.560 --> 00:40:55.430
probably how to exploit most of them.
And... But there aren't that many bugs
00:40:55.430 --> 00:41:02.430
that get published that are full exploits.
There's only a couple a year maybe. So
00:41:02.430 --> 00:41:05.260
what do you do from there once you've read
all those in and you need to learn more?
00:41:05.260 --> 00:41:12.540
Well maybe start trying to exploit other
bugs yourself so you can go... For
00:41:12.540 --> 00:41:16.260
example, I like Chrome because they have a
very nice list of all their
00:41:16.260 --> 00:41:19.510
vulnerabilities that they post every time
they have an update and they even link you
00:41:19.510 --> 00:41:24.980
to the issue, so you can go and see
exactly what was wrong and so take some of
00:41:24.980 --> 00:41:30.500
these for example, at the very top you
have out-of-bounds write in V8. So we
00:41:30.500 --> 00:41:34.000
could click on that and go and see what
the bug was and then could try to write an
00:41:34.000 --> 00:41:38.130
exploit for it. And then by the end we all
have had a much better idea of how to
00:41:38.130 --> 00:41:43.970
exploit an out-of-bounds write in V8 and
we've now done it ourselves too. So this
00:41:43.970 --> 00:41:47.650
is a chance to sort of apply what you've
learned. But you say OK that's a lot of
00:41:47.650 --> 00:41:53.030
work. You know I have to do all kinds of
other stuff, I'm still in school or I have
00:41:53.030 --> 00:41:59.690
a full time job. Can't I just play CTFs?
Well it's a good question. The question is
00:41:59.690 --> 00:42:03.310
how much do CTFs actually help you with
these kind of exploits. I do think that
00:42:03.310 --> 00:42:06.310
you can build a very good mindset for this
because you need a very adversarial
00:42:06.310 --> 00:42:13.410
mindset to do this sort of work. But a lot
of the times the challenges don't really
00:42:13.410 --> 00:42:17.270
represent the real world exploitation.
There was a good tweet just the other day,
00:42:17.270 --> 00:42:23.890
like a few days ago, where we were saying
that yeah libc is... like random libc
00:42:23.890 --> 00:42:28.660
challenges - Actually I don't think
it's... Yes. It's libc here. Yeah. - are
00:42:28.660 --> 00:42:33.330
often very artificial and don't carry much
value to real world because they're very
00:42:33.330 --> 00:42:38.570
specific. Some people love these sort of
very specific CTF challenges, but I don't
00:42:38.570 --> 00:42:43.410
think that there's as much value as there
could be. However a lot of... There's been
00:42:43.410 --> 00:42:48.040
a couple CTFs recently and historically as
well that have had pretty realistic
00:42:48.040 --> 00:42:56.580
challenges in them. In fact right now is a
CTF, 35c3 CTF is running and they have 3
00:42:56.580 --> 00:43:00.250
browser exploit challenges, they have a
full chain safari challenge, they have a
00:43:00.250 --> 00:43:05.950
virtual box challenge, It's like it's
pretty crazy and it's crazy to see people
00:43:05.950 --> 00:43:10.810
solve those challenges in such a short
time span too. But I think it's definitely
00:43:10.810 --> 00:43:15.020
something that you can look at afterwards
even if you don't manage to get through
00:43:15.020 --> 00:43:19.940
one of those challenges today. But
something to try to work on. And so these
00:43:19.940 --> 00:43:24.990
are the sort of new CTFs are actually
pretty good for people who want to jump
00:43:24.990 --> 00:43:31.690
off to this kind of real estate or a real
exploit development work. However it can
00:43:31.690 --> 00:43:36.310
be kind of scary for newer newcomers to
the CTF scene because suddenly you know
00:43:36.310 --> 00:43:39.681
it's your first CTF and they're asking you
to exploit Chrome and you're like what...
00:43:39.681 --> 00:43:45.730
what is going on here. So there, it is a
double edged sword sometimes. All right so
00:43:45.730 --> 00:43:50.590
now we found the bug and we have
experience, so what do we actually do?
00:43:50.590 --> 00:43:54.610
Well you have to kind of have to get lucky
though because even if you've had a ton of
00:43:54.610 --> 00:43:58.820
experience that doesn't necessarily mean
that you can instantly write an exploit
00:43:58.820 --> 00:44:03.390
for a bug. Our javascript exploit was kind
of like that, it was kind of nice, we knew
00:44:03.390 --> 00:44:09.170
what to do very right away but the
brows... or our sandbox exploit did not
00:44:09.170 --> 00:44:14.110
fit into a nice box of a previous exploit
that we had seen. So it took a lot of
00:44:14.110 --> 00:44:19.020
effort. Quickly I'll show... So this was
the actual bug that we exploited for the
00:44:19.020 --> 00:44:25.340
Sandbox. It's a pretty simple bug. It's a
integer issue where index is signed which
00:44:25.340 --> 00:44:30.160
means it can be negative. So normally it
expects a value like 4 but we could give
00:44:30.160 --> 00:44:34.920
it a value like negative 3 and that would
make it go out of bounds and we could
00:44:34.920 --> 00:44:39.830
corrupt memory. So very simple bug not
like a crazy complex one like some of the
00:44:39.830 --> 00:44:44.340
other ones we've seen. But does that mean
that this exploit is going to be really
00:44:44.340 --> 00:44:52.440
simple? Well let's see... That's a lot of
code. So our exploit for this bug ended up
00:44:52.440 --> 00:44:59.400
being about 1300 lines. And so that's
pretty crazy. And you're probably
00:44:59.400 --> 00:45:04.750
wondering how it gets there but I do want
to say just be aware that when you do find
00:45:04.750 --> 00:45:09.410
it simple looking bug, it might not be
that easy to solve or to exploit. And it
00:45:09.410 --> 00:45:14.654
might take a lot of effort but don't get
discouraged if it happens to you. It just
00:45:14.654 --> 00:45:19.420
means it's time to ride the exploit
development rollercoaster. And basically
00:45:19.420 --> 00:45:24.150
what that means is there's a lot of ups
and downs to an exploit and we have to
00:45:24.150 --> 00:45:28.010
basically ride the rollercoaster until
hopefully we have it, the exploit,
00:45:28.010 --> 00:45:36.020
finished and we had to do that for our
sandbox escape. And so to say we found the
00:45:36.020 --> 00:45:41.880
bug and we had a bunch of great ideas we'd
previously seen a bug exploited like this
00:45:41.880 --> 00:45:47.250
by Kiehne and we read their papers and we
had a great idea but then we were like OK
00:45:47.250 --> 00:45:51.422
OK it's going to work we just have to make
sure this one bit is not set and it was
00:45:51.422 --> 00:45:55.750
like in a random looking value, so we
assumed it would be fine. But turns out
00:45:55.750 --> 00:46:00.690
that bit is always set and we have no idea
why and no one else knows why, so thank
00:46:00.690 --> 00:46:06.450
you Apple for that. And so OK maybe we can
work around it, maybe we can figure out a
00:46:06.450 --> 00:46:11.190
way to unset it and we're like oh yes we
can delete it. It's going to work again!
00:46:11.190 --> 00:46:14.670
Everything will be great! Until we realize
that actually breaks the rest of the
00:46:14.670 --> 00:46:20.900
exploit. So it's this back and forth, it's
an up and down. And you know sometimes
00:46:20.900 --> 00:46:26.920
when you solve one issue you think you've
got what you need and then another issue
00:46:26.920 --> 00:46:30.930
shows up.
So it's all about making incremental
00:46:30.930 --> 00:46:35.870
progress towards removing all the issues
that are in your way, getting at least
00:46:35.870 --> 00:46:38.770
something that works.
Marcus: And so just as a quick aside, this
00:46:38.770 --> 00:46:41.450
all happened within like 60 minutes one
night.
00:46:41.450 --> 00:46:44.920
Amy: Yeah.
Marcus: Amy saw me just like I was
00:46:44.920 --> 00:46:49.609
walking, out of breath, I was like Are you
kidding me? There's two bugs that tripped
00:46:49.609 --> 00:46:54.110
us up that made this bug much more
difficult to exploit. And there is no good
00:46:54.110 --> 00:46:58.320
reason for why those issues were there and
that it was a horrible experience.
00:46:58.320 --> 00:47:04.260
Amy: But it's still one I'd recommend. And
so this rollercoaster it actually applies
00:47:04.260 --> 00:47:10.950
to the entire process not just for, you
know, the exploit development because
00:47:10.950 --> 00:47:15.940
you'll have it when you find crashes that
don't actually lead to vulnerabilities or
00:47:15.940 --> 00:47:20.340
unexplainable crashes or super unreliable
exploits. You just have to keep pushing
00:47:20.340 --> 00:47:24.450
your way through until eventually you
hopefully you get to the end of the ride
00:47:24.450 --> 00:47:32.740
and you've got yourself a nice exploit.
And so now. OK. So we assume, OK, we've
00:47:32.740 --> 00:47:36.400
written an exploit at this point. Maybe
it's not the most reliable thing but it
00:47:36.400 --> 00:47:42.060
works like I can get to my code exec every
now and then. So we have to start talking
00:47:42.060 --> 00:47:46.721
about the payload. So what is the payload
exactly? A payload is whatever your
00:47:46.721 --> 00:47:51.430
exploits trying to actually do. It could
be trying to open up a calculator on the
00:47:51.430 --> 00:47:56.870
screen, it could be trying to launch your
sandbox escape exploit, it could be trying
00:47:56.870 --> 00:48:01.859
to clean up your system after your
exploit. And by that, I mean fix the
00:48:01.859 --> 00:48:05.870
program that you're actually exploiting.
So in CTF we don't get a lot of practice
00:48:05.870 --> 00:48:11.240
with this because we're so used to doing
'system', you know, 'cat flag', and then
00:48:11.240 --> 00:48:14.910
it doesn't matter if the entire program is
crashing down in flames around us because
00:48:14.910 --> 00:48:19.670
we got the flag. So in this case yeah you
cat the flag and then it crashes right
00:48:19.670 --> 00:48:24.410
away because you didn't have anything
after your action. But in the real world
00:48:24.410 --> 00:48:27.760
it kind of matters all the more, so here's
an example like what would happen if your
00:48:27.760 --> 00:48:32.640
exploit didn't clean up after itself, and
just crashes and goes back to the login
00:48:32.640 --> 00:48:37.590
screen. This doesn't look very good. If
you're at a conference like Pwn2Own this
00:48:37.590 --> 00:48:44.390
won't work, I don't think that they would
let you win if this happened. And so, it's
00:48:44.390 --> 00:48:48.589
very important to try to go back and fix
up any damage that you've done to the
00:48:48.589 --> 00:48:55.339
system, before it crashes, right after you
finished. Right. And so actually running
00:48:55.339 --> 00:49:00.981
your payload, so, a lot of times we see or
in the exploits we see that you'll get to
00:49:00.981 --> 00:49:05.560
the code exec here which is just CCs
which means INT3 which just tells the
00:49:05.560 --> 00:49:10.700
program to stop or trap to break point and
all the exploits you see most of the time
00:49:10.700 --> 00:49:13.580
they just stop here. They don't tell you
anymore and to be fair you know they've
00:49:13.580 --> 00:49:16.820
gotten a new code exec they're just
talking about the exploit but you, you
00:49:16.820 --> 00:49:20.349
still have to figure out how to do your
payload because unless you want to write
00:49:20.349 --> 00:49:26.349
those 1300 lines of code in handwritten
assembly and then make it into shellcode,
00:49:26.349 --> 00:49:32.230
you're not going to have a good time. And
so, we had to figure out a way to actually
00:49:32.230 --> 00:49:37.570
take our payload, write it to the file
system in the only place that the sandbox
00:49:37.570 --> 00:49:42.560
would let us, and then we could run it
again, as a library, and then it would go
00:49:42.560 --> 00:49:50.460
and actually do our exploit. And so yeah.
And so now that you've like assembled
00:49:50.460 --> 00:49:56.430
everything you're almost done here. You
have your exploit working. You get a
00:49:56.430 --> 00:50:00.020
calculator pops up. This was actually our
sandbox escape of running and popping
00:50:00.020 --> 00:50:04.320
calculator and proving that we had brute
code exec, but we're not completely done
00:50:04.320 --> 00:50:09.630
yet because we need to do a little bit
more, which is exploit reliability. We
00:50:09.630 --> 00:50:12.740
need to make sure that our exploit is
actually as reliable as we want it to
00:50:12.740 --> 00:50:17.270
because if it only works 1 in 100 times
that's not going to be very good. For
00:50:17.270 --> 00:50:22.310
Pwn2own, we ended up building a harness
for our Mac which would let us run the
00:50:22.310 --> 00:50:26.160
exploit multiple times, and then collect
information about it so we could look
00:50:26.160 --> 00:50:30.500
here, and we could see, very easily how
often it would fail and how often it would
00:50:30.500 --> 00:50:36.130
succeed, and then we could go and get more
information, maybe a log, and other stuff
00:50:36.130 --> 00:50:41.760
like how long it ran. And this made it
very easy to iterate over our exploit, and
00:50:41.760 --> 00:50:48.290
try to correct issues and, make it better
and more reliable. We found that most of
00:50:48.290 --> 00:50:52.589
our failures were coming from our heap
groom, which is where we try to line all
00:50:52.589 --> 00:50:57.100
your memory in certain ways, but there's
not much that you can do there in our
00:50:57.100 --> 00:51:00.570
situation, so we tried to make it as best
as we could and then accepted the
00:51:00.570 --> 00:51:06.680
reliability that we got. You, something
else might want to test on is a bunch of
00:51:06.680 --> 00:51:11.410
multiple devices. For example our
javascript exploit was a race condition,
00:51:11.410 --> 00:51:15.300
so that means the number of cpus on the
device and the speed of the cpu actually
00:51:15.300 --> 00:51:20.349
might matter when you're running your
exploit. You might want to try different
00:51:20.349 --> 00:51:23.990
operating systems or different operating
system versions, because even if they're
00:51:23.990 --> 00:51:28.180
all vulnerable, they may have different
quirks, or tweaks, that you have to do to
00:51:28.180 --> 00:51:32.981
actually make your exploit work reliably
on all of them. We had to, we wanted to
00:51:32.981 --> 00:51:37.340
test on the MacOS beta as well as the
normal MacOS at least, so that we could
00:51:37.340 --> 00:51:42.089
make sure it worked in case Apple pushed
an update right before the competition. So
00:51:42.089 --> 00:51:44.359
we did make sure that some parts of our
code, and our exploit, could be
00:51:44.359 --> 00:51:49.090
interchanged. So for example, we have
addresses here that are specific to the
00:51:49.090 --> 00:51:52.500
operating system version, and we could
swap those out very easily by changing
00:51:52.500 --> 00:51:58.619
what part of the code is done here. Yeah.
And then also if you're targeting some
00:51:58.619 --> 00:52:01.610
browsers you might be interested in
testing them on mobile too even if you're
00:52:01.610 --> 00:52:06.400
not targeting the mobile device. Because a
lot of times the bugs might also work on a
00:52:06.400 --> 00:52:10.250
phone, or at least the initial bug will.
And so that's another interesting target
00:52:10.250 --> 00:52:17.030
you might be interested in if you weren't
thinking about it originally. So in
00:52:17.030 --> 00:52:20.170
general what I'm trying to say is try
throwing your exploit at everything you
00:52:20.170 --> 00:52:25.690
can, and hopefully you will be able to
recover some reliability percentages or
00:52:25.690 --> 00:52:30.640
figure out things that you overlooked in
your initial testing. Alright. I'm gonna
00:52:30.640 --> 00:52:35.250
throw it back over, for the final section.
Marcus: You're in the final section here.
00:52:35.250 --> 00:52:39.210
So I didn't get to spend as much time as I
would have liked on this section, but I
00:52:39.210 --> 00:52:43.250
think it's an important discussion to have
on here. And so the very last step of our
00:52:43.250 --> 00:52:50.849
layman's guide is about responsibilities.
And so this is critical. And so you've
00:52:50.849 --> 00:52:54.940
listened to our talk. You've seen us
develop the skeleton keys to computers and
00:52:54.940 --> 00:53:01.710
systems and devices. We can create doors
into computers and servers and people's
00:53:01.710 --> 00:53:06.490
machines, you can invade privacy, you can
deal damage to people's lives and
00:53:06.490 --> 00:53:12.610
companies and systems and countries and so
there is a lot of you have to be very
00:53:12.610 --> 00:53:17.870
careful with these. And so everyone in
this room, you know if you take any of our
00:53:17.870 --> 00:53:22.869
advice going into this stuff, you know,
please acknowledge what you're getting
00:53:22.869 --> 00:53:27.869
into and what can be done to people. And
so, there's at least one example that's
00:53:27.869 --> 00:53:31.090
kind of related, that, you know I pulled
out quickly that, you know, quickly came
00:53:31.090 --> 00:53:35.860
to mind. It was in 2016. I have to say i
remember this day actually sitting at
00:53:35.860 --> 00:53:42.820
work. And there is this massive ddos, that
plagued the Internet, at least in the
00:53:42.820 --> 00:53:48.110
U.S., and it took down all your favorite
sites:Twitter, Amazon, Netflix, Etsy,
00:53:48.110 --> 00:53:54.000
Github, Spotify, Reddit. I remember the
whole Internet came to a crawl in the U.S.
00:53:54.000 --> 00:54:01.310
This is the L3 outage map. This was
absolutely insane. And I remember people
00:54:01.310 --> 00:54:05.240
were bouncing off the walls like crazy,
you know. After the fact and they were all
00:54:05.240 --> 00:54:09.490
referencing Bruce Schneier's blog, and
there on Twitter there's all this
00:54:09.490 --> 00:54:13.800
discussion popping up that "this was
likely a state actor". This was a newly
00:54:13.800 --> 00:54:19.370
sophisticated ddos attack. Bruce suggested
it was China or Russia, or you know, some
00:54:19.370 --> 00:54:23.160
nation state and the blog post was
specifically titled someone is learning
00:54:23.160 --> 00:54:28.430
how to take down the Internet. But then a
few months later we figured out that this
00:54:28.430 --> 00:54:32.260
was called the mirai botnet and it was
actually just a bunch of kids trying to
00:54:32.260 --> 00:54:38.500
ddos each other minecraft servers. And so
it's a... it's scary because you know I
00:54:38.500 --> 00:54:45.630
have a lot of respect for the young people
and how talented they are. And it's a... But
00:54:45.630 --> 00:54:51.690
people need to be very conscious about the
damage that can be caused by these things.
00:54:51.690 --> 00:54:57.780
Mirai, they weren't using 0days per se.
Later. Now nowadays they are using 0days
00:54:57.780 --> 00:55:00.747
but back then they weren't, they were just
using an IoT based botnet. One of the
00:55:00.747 --> 00:55:04.977
biggest in the world, our highest
throughput. But it was incredibly
00:55:04.977 --> 00:55:11.710
damaging. And you know, so when you you,
it's hard to recognize the power of an
00:55:11.710 --> 00:55:17.660
0day until you are wielding it. And so
that's why it's not the first step of the
00:55:17.660 --> 00:55:20.930
layman's guide. Once you finish this
process you will come to realize the
00:55:20.930 --> 00:55:25.510
danger that you can cause, but also the
danger that you might be putting yourself
00:55:25.510 --> 00:55:33.070
in. And so I kind of want to close on that
please be very careful. All right. And so,
00:55:33.070 --> 00:55:36.940
that's all we have. This is the
conclusion. The layman's guide, that is
00:55:36.940 --> 00:55:41.536
the summary. And if you have any questions
we'll take them now. Otherwise if we run
00:55:41.536 --> 00:55:44.890
out of time you can catch us after the talk,
and we'll have some cool stickers too,
00:55:44.890 --> 00:55:51.329
so...
Applause
00:55:51.329 --> 00:55:59.385
Herald-Angel: Wow, great talk. Thank you.
We have very very little time for
00:55:59.385 --> 00:56:03.480
questions. If somebody is very quick they
can come up to one of the microphones in
00:56:03.480 --> 00:56:08.070
the front, we will handle one but
otherwise, will you guys be available
00:56:08.070 --> 00:56:10.460
after the talk?
Marcus: We will be available after the
00:56:10.460 --> 00:56:14.600
talk, if ou want to come up and chat. We
might get swarmed but we also have some
00:56:14.600 --> 00:56:17.680
cool Rekto stickers, so, come grab them if
you want them.
00:56:17.680 --> 00:56:21.339
Herald-angel: And where can we find you.
Marcus: We'll be over here. We'll try to
00:56:21.339 --> 00:56:22.720
head out to the back
Herald-angel: Yeah yeah because we have
00:56:22.720 --> 00:56:25.390
another talk coming up in a moment or so.
Marcus: OK.
00:56:25.390 --> 00:56:29.859
Herald-angel: I don't see any questions.
So I'm going to wrap it up at this point,
00:56:29.859 --> 00:56:34.210
but as I said the speakers will be
available. Let's give this great speech
00:56:34.210 --> 00:56:35.360
another round of applause.
00:56:35.360 --> 00:56:40.266
Applause
00:56:40.266 --> 00:56:42.251
Outro
00:56:42.251 --> 00:57:04.000
subtitles created by c3subtitles.de
in the year 2020. Join, and help us!