reproducible_builds_status_update.webm

Next talk

Chris and Holger are going to talk to us again

about reproducible builds and tell us
where they're up to.

Thanks very much

The outline of this talk is from last year
we realised there were a lot of questions.

The rough plan is to quickly go over
what reproducible builds are

I guess everyone is up to speed

but getting everyone on the same page
would be a good idea.

Then Holger's going to jump in
and give the status update

and then we're going to talk about
future work, questions etc

What is the actual problem we're
solving here?

You can always inspect the source code of
free software for malicious flaws

or just flaws as well.

Unfortunately distributions provide
precompiled binaries to end users.

So can you actually trust this
compilation process has not

introduced flaws of its own?

The problem is it seems very effective if
you want to go after end users

you can go after developers.
Because if you go infect a developers

machine you will then infect all the
users of the software they generate.

Financial incentives. There always were
but they are even more so these days

with mobile phone etc.

You can also have very subtle flaws.
This one in particular there was a

root exploit in OpenSSH just by changing
a compare equal.

That sort of assembler jump thing and it
gives you root

but with only a single bit difference in
the binary.

Which is not to shabby.

Then you have all sorts of cute demos
where you load up the source code in VIM

and it just looks like 'Hello world' but
when you compile it with GCC

your kernel rootkit just goes 'oh I'm
going to give you a different file'

and self replicates of them like that.

Difficult to trust the process.

And there's some recent history as well
around Xcodeghost and iOS

and adverts and things like that.

You can Google those things.
Really scary stuff.

The last example is actually coming from
a CIA design paper from 2012.

Which was then found in the wild in 2014.
So these exploits are actually happening.

People are targeting developers to get
users.

Xcodeghost had 20 milllion user
installations.

It was probably not the CIA or NSA but
we don't know who it was.

There are many people who do these
exploits in the wild.

Yeah it's not just 'Here's this cute
thing we can talk about'.

It's actually happening.

The motivation is to ensure no flaws are
introduced during the build process.

We do this by ensuring the build always
produces identical results.

Then multiple parties do the same thing.

I build it, you build it, your friends
build it etc

An an attacker would need to infect
everyone simultaneously

otherwise they'd be detected.
For example if my machine was compromised

I would suddenly come up with a
different result.

I would come up with different binaries.

And you'd be 'what's going on here' and
eventually we would discover

that my machine was rootkitted etc.

You probably know it but identically
means bit by bit identical.

As that is really the same.

Yeah, bit, SHA, MD5 whatever you want.

There are a bunch of challenges here.
The biggest one is timestamps.

A lot of software just loves to include
timestamps everywhere.

Documentation, underscore underscore date
and underscore underscore time macros

Just all over the place, in file names etc
Things like that.

Builds often vary by locale and timezone.

Different new lines, different sorting
orders for example collations.

Different versions of libraries. I'm not
sure what this refers to exactly.

Moving on.

Non-deterministic file ordering for
example Shell Globs are not really defined

to be, I say not really defined they
aren't defined to come out in normal order.

Also read syscall, it doesn't actually
promise any particular ordering.

Dictionary/hash key ordering. So this is
in things like Perl and python

you use a key or a hash. If you iterate over the keys with that it's a non-determinative order.

If your build system loops over such a
hash or a dictionary

then the results from this build could be
non-reproducible and non-deterministic.

And also things like files in the part of
the build process will just adsorb

stuff from the surrounding environment
like umask and all that kind of

stuff that lives outside there.

Build paths is a very interesting one which
we cover in greater detail on another slide.

Also specifying the environment, we'll also
cover this one in the build info slides.

So not only are there privacy and security
advantages of using,

moving towards reproducible builds there
are also technical advantages.

It's faster to build if you basically
keep hitting cache.

I'm pretty certain this is why Google are
interested in it.

Because of the amount of
compilation they do

they're just going to save a whole bucket
load of money just by

'Oh we don't need to rebuild this because
it's the same SHA' etc

It's very nice to test revisions and
changes I use all out tools

when doing QA uploads or NMUs you
rebuild a package

and then you compare to the previous one.
And as the only things that have changed

should be the things that you've changed,
there haven't been all sorts

of random other nonsense being
reorderd with timestamps added.

You can get rid of all that noise and
just be 'oh yeah brilliant I can see that

the patch I've applied here has actually
changed the behaviour of the program'

and only that. It hasn't done all sorts
of wierd wierd stuff.

So you have safer uploads in that sense.

Speaking of safety a reproducible build
won't go talking to the Internet

like a lot of modern package managers
like to do. Mathen style ones.

Also a reproducible build will typically
not have any

non-deterministic failure modes.

So there's a lot of tests and test suites
in Debian that will

try and test things like 'Oh is this
algorithm N squared or bigger than N'.

And it will try doing that by running some sort of bench mark

and fail if it doesn't meet some sort of
arbitrary time difference and

that's obviously that's not reliable. So we
get rid of all those nonsense things.

It also finds bugs in really weird
locales. We build in French,

Swiss-French, and it just comes up with
all sorts of nonsense.

Or timezones, if you build in UTC-12 then
this date library doesn't work anymore

and it's like 'you had one job
to be a date library'.

[audience laughter]

It's pretty scary and some pretty
cute bugs.

It also detects if your machine is, you
just have a broken ??? [8:28].

We build a year and a month in the
future. You find things like the

maintainer has added a pre-generated SSL
certificate to their tests and

it expires in the year. And so it breaks.

We're preemptively detecting that fail to
??? [9:01] source.