34C3 - May contain DTraces of FreeBSD

music

applause

Raichoo: Yeah sorry about that - beamers
or projectors, I don't like them. They

don't like me either. So this is a little
heads up - this is going to be the only

slide I'm going to show you today so,
"slide", because I think doing stuff like

that in a terminal might be a little bit
more interesting for you. But sadly

something is getting cut off so I we have
to improvise a little bit. But anyway, so

today I will be able to talk about two of
my favorite things right now which are

FreeBSD and DTrace. But this talk has been
capped down to 30 minutes so we'll be

focusing a little more on the DTrace part.
So there will be a little bit less BSD

than I anticipated. And also adjusted
everything a little bit to fit better into

the resilience track so hopefully you'll
enjoy that. So before we begin, who here

is actually using DTrace? Okay more than
I expected but still not as many as I

would like to see. So hopefully after this
talk you will think, "oh, this is a really

awesome tool, I gotta learn it." Because I
totally love it - it changed the way I do

a lot of stuff. So for those of you who do
not know what DTrace is, first, let me

fill you in on this stuff. So it's open
source, it originated on Solaris, and been

developed currently on illumos which is a
fork from OpenSolaris. It has been ported

to FreeBSD, NetBSD, OS X, there's also a
port for Linux called next called DTrace

for Linux. I think it's done by a person
called Paul Fox. It's been ported to QNX

and the OpenBSD folks are currently doing
some work to get the technology like

DTrace on their system. And I think
there's a port for Windows? I don't know

if this is actually true, but it is it's
kind of cool because then that means it's

basically everywhere. So, most of you
would probably know static tools like

strace. We have a very similar tool on
FreeBSD that is called truss, and what

truss and strace are doing is - you can
attach them to a process and look at the

system calls that this process is
emitting. So in case something is going

wrong you can well look inside of the
program, which can be kind of useful when

you're trying to find a problem. It's
kind of handy but it's also pretty

limited. Because first of all it really
really slows down the process that you're

currently looking at. So if you want to
debug a performance issue, you're pretty

much out of luck there. And also it's kind
of like, narrow down - you can just look

at one process. Which is also like bad
thing because the system that we currently

have - all these systems are very
complex: we have a lot of layers. You have

virtual file systems, you have virtual
memory, you have network, you have

databases, processes communicating with
each other. And in case you are using a

high-level programming language, you might
also have a runtime system. So it's a

little operating system on top of your
operating system. So when something goes

wrong in a system that has such large
complexity, something happens that we call

the blame game. And the blame game - it's
never your fault, it's always someone

else's. So what we want to be able to do
is we want to look at the system as a

whole, so we can correlate all the data
and come up with some meaningful answers

when something is really going wrong in
there. And also, we don't want to

switch out all the processes for
debug processes to make that happen,

because as these things are all -- every
problem happens in production. It never

happens on the development box. So like,
switching out all the processes - that's

totally out of the picture. So to do that
in an arbitrary way, to like, instrument

the system in an arbitrary way, we sort of
need like a programming language. So, we

need to describe - when that happens,
please submit data so I can see what's

going on. So this kind of implies a
programming language. And DTrace comes

with such a programming language - it's a
little bit reminiscent of awk cross with

C? It's pretty simple to learn - you can
pick it up 20 up to pick it up in 20

minutes and you can start churning out
your first DTrace scripts. So like awk, if

you know awk, awk can be used to analyze
large bodies of text. Dtrace is pretty

much the same, but for system behavior -
so a little bit mind boggling, but

probably I can show you what I mean by
that. And also, as a bonus we don't want

to slow down the system, so we want to be
able to do things like performance

debugging, performance tests like that. So
I've prepared this little demo here, and.

So since we had some issues here probably
this is not -- I have to play around a

little bit. So what I'm going to do is
I'm going to look at a very very naive way

to -- excuse me for a second -- very naive
way to -- give me a second -- so very

naive way to authenticate a user. And
there's a lot of stuff wrong with this

code, but like what we're going to do is
we're going to take a user string as

input, and then we're going to just
compare it to another, to a secret. So I

know, the the secret in here is like in
plain text I know this is a problem, but

this is a little bit artificial. But I
just want to get my point across. So from

an algorithmic perspective, this check
function is correct: so we take a string

we take another string and we compare
them. So everything's fine and easy. So if

you look at the way string compare works
and what it does, it's essentially

taking these two strings and it's
comparing every character bit by bit. So

when it finds the first pair of characters
that do not match up, it's going to stop.

So we can we can conclude something about
from that - so if it takes very short if

if this function this check function takes
a very short amount of time, then, what

will happen is it will terminate earlier.
And if our password guess is better, it

will take well, it will take longer. And
if we can measure that we can basically

extract information from that running
algorithm. So I wrote a little driver

program in Haskell that basically just
iterates over an alphabet and just feeds

this one letter into that program,
And I'm going to use DTrace to get some

timing information. So let me start the
driver. So this is now just running in the

background. And you cannot see what I'm
typing there, but don't worry - these

scripts will all be; I will push them on
my github. So DTrace now produces this

nice little distribution. So if you if you
were if you were able to see the entire

alphabet, you would see that everything
except "D" behaves differently. So if you

squint a little, what you see there is
DTrace the D letter takes a couple of

nanoseconds longer. This is the precision
that I'm measuring here - ten to minus

nine seconds. Like really precise. And D
takes longer than everything else, so it's

a little bit cut off there, but trust me.
I know it sound like Donald Trump I'm

saying that. So yeah, and from that let's
just enter a letter. And now the password

and now the script clears everything and
it's going to guess the next letter. So

sadly this is cut off, because you would
see that this distribution radically

changed. It looks completely different,
and so we can play that game a little bit.

So let's just roll with that.
And like every three seconds the script is

going to recompute looking at the new
distribution. And you can probably see

where this is going. So here you can see,
okay, and now it just - it just takes

about like three seconds for me to guess
the next letter. So, and this is not a

problem that is only of
something that happens when you do string

compares. This can happen with
basically everything - so it's especially

in things like cryptographic stuff where
you don't want to have some information

leaked out. So this is what we call a
timing side channel attack. So I could

essentially use DTrace to analyze
the real binary. So I didn't change the

binary - I didn't have some some debug
code there. This is like the actual binary

that I would put into production. So
what's important about out that, is to

take the actual binary, is some of these
these timing side channels might be

introduced by a compiler optimization. And
when you insert debug code into that code,

then it might actually go away. So, you
want to look at the real code that you're

putting into production. Let me show you
the script that I came up with to write

that. So there are three interesting
things in this script. So and and don't

worry - this is the more
complicated example, I just want to like

inspire your ideas. Because the things
that you can do with DTrace that's pretty

much - the sky's the limit. You can
come up with the weirdest ideas, and so

this is more complicated example. I'm
going to show you simpler ones. So to

demonstrate how we got here. So there are
three interesting things in this code. The

first one is something that we call a
probe. So a probe is a point of

instrumentation in the system. So whenever
a certain event happens in the system this

probe is going to fire. And in this case,
the begin probe like marks the state

the moment when the script starts. So the
second interesting thing is this clause.

So this clause is basically what this
probe is going to execute - what's going

to be executed once that probe fires. So
it's a little block of code.

And this probe is a little bit more
interesting, because it tells us

something about the structure of how such
a probe looks like. Because every

probe is uniquely identified by a four
tuple. So it's like four components that

uniquely identify a probe. And the first
one is called the first part of this

tuple is called the provider, and I'm
going to talk about providers in a couple

of seconds and what they are. The second
one is called the module. Third one is

called the function. And the last one is
called the name. So these four pieces of

data, like, they identify a probe
uniquely. So the third thing that is

interesting here is, sadly something that
I don't have time to talk about today,

this is called an aggregation. And this
single line that you see here is

essentially responsible for accumulating
all this data to print out this

distribution stuff - to generate this
distribution. So this is built

into DTrace. You don't have to do that
yourself. As it, when you look at this

script, it's like 42 lines of code.
And I came up with the first prototype

after five minutes. So it's not a lot
of stuff to do to get something out of

that. So it's very useful to have things -
if you use DTrace you

will use this a lot for performance
debugging so it's kind of neat that we

have that. So yeah, let's talk a little
bit about providers, and this will

probably also will be cut off. So I'm
going to cheat a little bit here - I'm

just going to double that. So let's talk
about providers -- oh that's handy --

so I got 27 providers here and the number
of providers vary from operating system to

operating system. But these are the
ones that I can see right now. There are

other providers that can be come into
existence when you demand them. So I have

these 27 providers, and we're going to
look at the syscall provider and the FBT

provider first. So, every provider knows
how to instrument a specific part of the

system. So the syscall provider knows how
to instrument the syscall table. That's not

very surprising. So if you can look at the
syscall provider and here you can see

essentially every system call entry and
return that FreeBSD offers. So

here you can see this four tuple, like,
the provider syscall, FreeBSD is the

module, and so on. So these are all the
system calls that I have in my system. And

the other provider that I want to look at
is the so called FBT provider, and that is

pretty astonishing. The FBT provider, FBT
stands for "function boundary tracer" and

what it allows us to do, it allows us to
trace every single function in the kernel.

So I can look at the entire kernel at
functions, as they are being called. So to

illustrate that I wrote a little, very
simple DTrace script and this is probably,

look at the upper half please, so this is
probably one of the first DTrace scripts

that you will come up with, it's a
fairly simple example, so let's break it

down. So I'm going to instrument the mmap
system call. For those of you who do not

know what the mmap system call is, what
you can do with it is you can so you can

take a file and map that into the address
space of your process, so very dumbed down

version. So whenever we enter the mmap
system call we are going to set the

variable "follow" to one, and what this
"self at" means: this is essentially a

thread local variable and we're going to
associate that variable with the thread

that we're currently inspecting. Then I'm
going to do something pretty, that sounds

scary but I'm going to instrument the
entire kernel. Every function entry and

every function return, I'm going to
instrument that and say "please emit data

when you do that". And this is what we
call a predicate, so this is where the

awkiness of the DTrace programming
language comes in. So this is a predicate

and whenever that evaluates to true
then the probe is going to fire, so in

this case when we are in the thread that
we're currently tracing we're going to

emit data. And this is just an empty
clause, we just want to know "hey we got

here". So when we exit the mmap
system call and the predicate is set we're

going to set the variable "follow" to
zero, because every uninitialized variable

in DTrace is set to zero, so this pretty
much amounts to deallocating that variable

and then we're going to exit cleanly. So
let me run that. So it takes a couple of

seconds and boom. So you saw a little
pause here, that was when the DTrace guard

reverted the driver, the kernel. So now
you can see every function call that

happens inside the mmap system call. And
this is a little bit hard on the eyes, so

let me pass this flag here and now you can
have nice to read indentation. So

now you might say "I don't like that. You
are injecting code into the kernel. That

is, that sounds dangerous" and yeah, but
let me show you something that I find

really interesting. So I'm not
going too much into depth here, but this

is a byte code, so every DTrace script
gets compiled to bytecode and this

bytecode gets sent to the kernel and in
the kernel you have a virtual machine that

interprets that bytecode. So in case you
write a script that for some reason might

go rogue on your kernel, it like allocates
too much memory, takes too much time, this

virtual machine can just say "okay, stop
it" and just going to revert all the

changes that happened to your kernel, and
that's kinda handy. And it's not a new

idea, so if you're using TCP dump it's
basically the same approach. They also

have this kind of bytecode, so that's just
a little excursion here. This is called

BPF, Berkeley Packet Filter, so it's not
an entirely new idea. So everything I

showed you until now was "hey, I can look
when function calls happen". that's not

very much information, so we're going to
increase the amount of information that we

get out of the system with every example.
So let me look at the actual kernel. So I

had to restart my machine, so my setup is
basically gone now. So let's look at this

VM fault function. So this is, this is the
source code of the operating system that

I'm running right now. This is FreeBSD
current 12 and the VM fault function;

remember the mmap system call that I told
you? So the mmap system call

I told you can bring, like map a file
into your address space. And it doesn't

necessarily have to load the entire file,
so whenever we are touching a page in the

system, like a memory page, this machine
is four kilobytes and it's no super pages

here, so whenever it touches a piece of
memory that you didn't bring into memory

yet, we're generating something that's
called a page fault, and then this

function gets called. So here let's look
at the arguments, and I'm going to skip

the zeroeth argument, to look at the first
argument. So this is the address that

provoked that page fault, this is the
type and these are the flags and I'm going

to show you something to make that a
little bit more readable. So what about

this one? So you see it's a pointer and
this is a big structure, so we want

to be able to look at that structure. And
just probably should do this here, so

let's look at this VM fault script here.
So this is, make this a little bit more,

so this is, don't pay too much attention
to this code, this this is basically just

boilerplate to make make stuff readable
and this is where the actual action is

happening. So this is, so what I'm doing
there is I'm instrumenting the VM

fault function and whenever we enter it
then we're going to use some information

that DTrace gives us for free. So this is
execname, this is the name of the

currently running executable that provoked
the page fault, this is the process ID and

here we have a bunch of argument
variables. So these arg1, arg2, arg3,

that are essentially just integers, so
nothing too fancy there. But we wanna

look, wanna be able to look at that
struct. And here I'm going to use this

args array, and this args array
is kind of special, because it has typing

information about the arguments. So when
you run that, so you're referencing that

pointer there with the star, excuse me,
and let's just run that and maybe, that's

a start yeah. So this is an in-kernel
data structure that we can now look

at. So DTrace enabled us to look at in-
memory data structures as the system runs.

And this is really really powerful.
In in the DTrace script I could use all

these fields like I can manipulate this
args array, this value in there, just like

just like every other variable; I
can pretty much work like I was in C. So

how is it doing that? There is something
that's called CTF, that's not capture the

flag, it's, this is the, the Compact C
Tracing Format, so you can see that but

there is a man page in FreeBSD, and
there's a little segment in the kernel

binary, where all this typing information
is stored. I don't know how that compares

to modern DWARF but yeah this is what
DTrace is working with. So now you might

ask yourself "Why on earth would I do
that? Why on earth would I look at virtual

memory, because, yeah, um, this stuff is
safe isn't it? I mean there's no bugs in

there." Except when they are. Anyone
remembers remembers "Dirty COW"? So this

was a very nasty vulnerability in the
Linux kernel and that that was a problem

in the virtual memory management. So it
allowed you to write to a file that you

didn't own as a regular user. So you could
essentially just write to a binary that

had "set UID" set. Very unpleasant, but
I'm not going to bash the Linux folks

here, this is just, I just want to show
you these things are hard. And the first

fix for this problem was in 2005 and then
it came back in 2016. So now that's fixed

and then it came back with "Huge Dirty
COW" in 2017, so this is, I mean this

was there for way over a decade.
These things are hard to debug. And this

is what I like about these systems, so not
having, not having tools like DTrace to

figure out what's going on inside of the
system somehow, to me, amounts to security

by obscurity. And I've heard that some
people who are developing exploits for

systems that have DTrace they say "Oh, I
really like developing exploits on these

systems, because the tooling is so great!"
Yeah, but, to be honest this is cool,

because an exploit is a proof of concept
and coming up with these exploits quickly

is very usable, because you know what's
going on you can show "Hey, this is going

wrong". I had situations, where
people were telling me "Oh, this is this

is not a problem with our program, this is
this weird operating system that you're

using. Like Solaris, weird operating
system." And, yeah, and then I churned out

some DTrace scripts and "No, it's
actually your problem". "Oh, now I can see

that on my Linux box!" Magic. So,
everything I showed you until now was

very, very much related to function calls
and we want to have a little bit more

semantics here, because you might want to
write a script that inspects protocols,

stuff like TCP, UDP stuff like that. So,
you don't want to know which function

inside of the kernel is responsible for
handling your TCP/IP stuff, so DTrace

comes with something that's called static
providers and I'm just going to show the

apropos here. So these are, so every
static provider has a main page which is

kind of handy - documentation whoo - and
you can see there is an I/O provider if

you are interested in looking at this guy:
Oh, IP for looking at IPv4 and IPv6,

TCP... This one is pretty cool, it's about
scheduling behavior. So, "what does my

scheduler do?" And if you look at that, you
can see some interesting stuff like length

priority if you ever saw things like
priority inversion, stuff like that, now

you can see that happen. I'm a nerd, I
find this interesting for some reason, I

don't know. And it's also pretty
interesting to figure out what's going on,

"why is this getting de-scheduled all the
time?" So, some interesting things going

on there. So, I'm running a little bit
short on time here, but I just quickly

want to show you something - this is all
kernel stuff right now - can we do that

with userspace? Of course. So, there was
one provider that didn't show up when I

had my provider listing, but was in the
DTrace script where I did this timing

attack stuff. And that's called the PID
provider. And the PID provider generates

probes on demand, because a process might
have a lot of probes and you will shortly

see why and this is why I'm going to use a
very small program which is called "true",

and true just exits with exit code zero.
So, nothing too exciting going on here,

and this dollar target gets substituted
in, we get the process ID there. And this

is everything that happens when I'm
executing this program you see this is a

little bit more fine-grained than the FBT
provider, because now we can trace every

single instruction inside of that
function, which is kind of a handy. It's a

scriptable debugger. So, these numbers are
the instructional offsets inside of that

function. We can also look at - so this is
everything in the true segment - we can

also look at libraries that got linked in
and there's a lot of stuff happening in

libc for example when you run true.
So, one last thing that I wanted to show

you because it consumed a week of my life:
I'm using a lot of Haskell and the Mac OS

people, they also have DTrace and they
have GHC Haskell DTrace support - so the

Glasgow Haskell compiler - and glorious...
they have probes to analyze what's going

on inside of the runtime system. So, I
thought "I want to have that, I have

DTrace, why doesn't it work on FreeBSD?"
So, after a week of fighting with make

files and linkers, that works: If you
check out the recent GHC repository and

build it on FreeBSD, you get all the nice
stuff that I'm going to show you now. So,

this is a very boring program - it just
starts 32 green threads and schedules them

all over the place - and now I can do
something like this: phone rings I can

ring a telephone.
laughter

No, that would be
interesting... So, you can also use

wildcards - and not as name of the probe -
and this is what's going on inside, like

GC garbage collection and all this stuff.
Now you can look at this and write useful

DTrace scripts that also take my runtime
system into account. So, stuff like that

exists for I think Python - I'm not
entirely sure because I don't use it -

nodejs same, Postgres - I used it but not
with DTrace right now - and what a find

interesting: Firefox. When you run
JavaScript in your Firefox, it actually

has a provider, so you can trace
JavaScript running in your browser with

DTrace, so after everything I just showed
you, there might be some stuff going on

there. So yeah, this is basically
everything I wanted to show you and I

think I'm going to wrap out, because
otherwise we're not going to have a lot of

time for questions and maybe you have
some. So yeah, thanks.

applause
Herald: Thank you very much Raichoo. We

are actually over time already, but we
have two more minutes because we started

three minutes late, so if there are any
really quick questions, possibly from the

internet... There is one, the signal angel
says, let's hear it.

Question: Yeah, hi, okay. So, the question
is, "which changes are actually necessary

to do in the kernel of an operating system
to support DTrace?"

Answer: That's a lot of work. So, it's not
something like you do in a weekend. This

is... So, the person who started the work
on FreeBSD has sadly passed away now, but

I think they took a couple of years to
have everything in place, so you have to

have stuff like the CTF thing that I
showed you, which is what OpenBSD is

currently working on. And then you need
all those those magic gizmos, like kernel

modules and stuff like that. So, it takes
a lot of time, but it's been ported to

most operating systems that are available
and in use right now. So yeah, hope this

answers the question.
Herald: Excellent and there are no more

questions here in the room. I will thank
Raichoo and you can find him outside of

the room and also on Twitter at "raichoo"
if you have any more further question.

postroll music

subtitles created by c3subtitles.de
in the year 2020. Join, and help us!