Stretching out for trustworthy reproducible builds creating bit by bit identical binaries

Welcome and good morning

This is the reproducible builds team,
talking about

"Stretching out towards trustworthy
computing"

[Applause]

We're 4 on stage, but actually this is a
team effort.

All these people listed here have
contributed to the project at one point.

The 4 of us, that's

Lunar − me

there's Dhole,

Chris Lamb − lamby

and Holger.

But actually, this is DebConf and so a lot
more of us have been or are

currently here and so, if you want to
thank anybody that is working on this

you need to actually thank all of
these folks

'cause, yay.

[Applause]

[Holger] The people in blue are here.

[Lunar] Let's get started.

Quick recap on what we're talking
about.

We have software, it's made from source.

Source is readable by humans or at least
a good amount of humans.

In this room it's good.

Binary, readable by computer and some
tiny fraction of humanity.

Going from source to binary is called
build, or like building or compiling

and we're doing free software and
free software is awesome because

we can actually run these binaries like
we want

We can actually study the software, how
it's been made by studying the source

and by studying the source we can assess
that it does what it's supposed to do

and not something else that does not

have malware, or trojans or security bugs

So we have the binary that can be used,
fine.

We have the source that can be verified.

Problem is that right now, the only way we
know that a binary that we get…

We have to trust a website or a Debian
repository that says

"Well, this binary has been made with this
source"

But there's no way we can actually prove
that.

This is actually a problem that has been
well explained by

Mike Perry and Seth Schoen at the 31c3
in Hamburg last december.

For example, Seth Schoen made a proof of
concept exploit for the Linux kernel

that when GCC was called, the kernel would
without modifying anything on the disk

when the kernel detects that GCC is going
to read a C file, it will insert some

extra lines of code, and these lines of
code can be a very bad thing

in the case of 31c3 talk I was just
recalling.

Actually, you can even have developers
who are in very good faith, who have

totally secure dev machines, or they
thought they have,

who have reviewed all their source code
for any bugs

and we would still get totally owned as
soon as their computer gets compromised

or one of the build demons from Debian
gets compromised for example.

This is not, like, hypothetical threats
here we're discussing

A couple of months after Seth an Mike's
talk at 31c3,

the Intercept revealed from the Snowden
leaks

that at a CIA conference in 2012, one
of the talks that happened

was about a project called Strawhorse.

Strawhorse is about modifying Apple XCode,
which is the development environment

for MacOS 10 and iOS applications

and well, they were modifying XCode so
it would produce,

without the developer knowing,

binaries with trojans, malware,
??? binaries, lots of bad things.

So, solution:

enable anyone to reproduce identical
binary packages from a given source.

Because if using a source, using the same
environment,

multiple people on different computers, on
different networks, at different times,

can all get the same thing
from the same source

all the same binary, byte for byte,

then there's a good chance that…

Well, everybody could be owned,
but let's be more joyful and say that

probably, if everybody gets the same
result, there was actually no problem

and everybody is safe.

We call that solution
"reproducible builds"

Yay.

[Applause]

Actually, it's not only about security.

For Debian, we have, if you're doing
"Multi-arch: same" packages,

well they only have the same bytes if
they are built for different architectures,

the files in the package.

Debug packages, you can create at a later
time, if you forgot to have debug packages

in the first place,

you can pass the no-strip option later and
because the package is reproducible,

you will get the debug symbols that work
for software that has been shipped already

We do early detection of FTBFS that way

because if we try pretty quickly
to reproduce a build,

then it has to work.

It's useful for build profiles.

We can get smaller .deb deltas,

because from one version to the next we
might have the same content.

We can do validation of cross-builds,

Helmut Grohne can talk to you about that.

And also, Niels Thykier told me that

he was very interested in reproducible
builds because it would enable him to

test debhelper better, because

if the package builds reproducibly,
then he makes a change to debhelper

he can rebuild the package ???

the same version of a package with a newer
debhelper and see what has changed

and this change can be isolated to only
what he has worked on debhelper

for example.

And, oh my.

The whole world is watching us.

Since two years or a year and a half ago,
everybody I meet in security conference,

in hacker conference, in free software
conference is like

"Oh you're working on that,
that's awesome."

And, I mean, I've been the one doing quite
a lot of talks, and everybody comes to me

and I'm like "Wow wow, this is way bigger",

but we're actually leading the field here.

Yay Debian.

[Applause]

[Holger] So, we are not the only ones
leading the field,

Bitcoin and Tor made their software
reproducible before us,

Coreboot also succeeded, if you build
Coreboot without any payload,

that's 100% reproducible.

FreeBSD has a page on their wiki since
2013

saying there are 5 reproducibility issues
in their base system.

We're at the moment trying to
confirm this.

On jenkins.debian.net, I've also set up
now tests for FreeBSD, NetBSD,

Coreboot and OpenWrt.

So if you go to
reproducible.debian.net/

you get that tested.

And there's more in the pipeline.

There are other projects interested
as well.

NetBSD also has a variable ???
which you can set

and that builds reproducibly.

Though they think "I'm keeping some
timestamps ??? and then

filtering them out later".

We disagree.

So this is how Debian looks like,
Debian Sid,

but this is a lie.

This is not the truth.

This is just our test setup.

Sid is not like this.

For Sid, it's all orange, there's zero
reprodicibility in Sid today.

But we'll talk now and in the following
round table,

it's to actually make Sid reproducible.

The current status is

we're working on this in Debian since
two years ago.

We have weekly reports about our project
now since May

and we've given several talks, especially
in the last year

and all these talks, presentation, also
other stuff is linked in the wiki.

There's a page with information about
Debian, these BSDs,

other Linuxes, upstream ???
all on this wiki.

Since DebConf14, which is merely
a year ago,

we've made quite some changes.

We have introduced
strip-nondeterminism

which is called by dh at the end
of the build of the package

and will normalize some things
which Chris will explain later

We have decided on a fixed build path

because the build path is leaked
in the binaries and several things

We didn't find a way yet to make
the build path arbitrary.

We designed a way to record the build
environment

because to rebuild, you need to recreate
the build environment.

We set up this Jenkins setup.

We wrote diffoscope which used to be
called debbindiff

which shows differences between two
packages or two directories or

two filesystems by now.

There's SOURCEDATEEPOCH, which is a way
that the tools expose

the last modification of the source.

Because the build date, people want to
include the build date

because they think this is a
meaningful indication:

when a build was done,
which software used.

But if the build always recreates
the same results

the build date becomes meaningless

and the really interesting thing is
the latest modification of the source.

We have written patches for the tools

[Lunar] strip-nondeterminism:
is Andrew Ayer in the audience?

Yay! He did it!

It's written in Perl because we didn't
want to have a new build dependency

in all Debian packages.

Basically it takes anything and tries
to normalize it as much as it can

replacing timestamps or file permissions
or removing some issues.

It's working very well on many formats,
it's meant to be extensible

so we can actually add more things and
it's run by dh at the end of the process, as Holger said.

The .buildinfo is currently a proposal
we have not yet totally agreed

but we are generating them as part
of the test we have

and basically it's a new control file that
will tie the sources, the generated binary

the packages that were used to build this
binary and their version.

The idea is that we can use this file to
reinstall all the specific versions from snapshot

So we recreate the same build environment
then we can just start the build from that source

that was mentioned and see if the binary
that has been generated matches.

What it looks like for now, you see there is
a source binary, the build path

because currently we don't have any good
post-processing tool for buildpaths

in elf and dwarf binaries, we just decided
to specify the build path so when we do

a later rebuild we use that path and be safe.

The source is ???, the binary is .deb and
a list of packages with the versions.

We currently use the base files version
to know which Debian release is to be used

as the basis.

[Holger] The general procedure for testing is:
we build the source, we save the results,

we modify the environment and we build
it again and compare the results.

That was a shell script last year which I
put on jenkins and then it exploded a bit

and now we have 67 jenkins jobs running on
7 hosts.

Since last week we have 4 armhf small boards
where we will be able to test armhf,

but very slowly.

We have two new amd64 build nodes.

The code is now split into Python and bash
scripts.

For all the other distro testing there's a
lot of bash code now which is mostly

boilerplate and it's 5 lines or something
to build FreeBSD and 5 lines to build NetBSD

but there's 100 lines boilercode around so it's
really not that much code.

We do test Testing, Unstable and Experimental.

For arm we only start with Unstable.

We do like hardware so if you have hardware
to donate to us, that would be great,

we need ssh and then root basically.

We are testing Coreboot, OpenWrt and the
BSD's, soon I will also set up a Fedora test

I don't want to test all the 20,000 Fedora
packages but just 200 or something:

the base system of Fedora to examine how
rpm works

to get really the whole Free Software world
reproducible.

This is all run on ProfitBricks hardware
since 2002, so thanks to ProfitBricks.

This is the variations we do for Debian.

It's the hostname, username, timezone,
locale.

Chris will explain what modifications
this causes, variances...

We are not testing at the moment differences
in date so the date is always the same

the time is a bit different.

[Lunar] Well almost! Because we cheat with
the timezone, we use one timezone that is

GMT-14 and then GMT+12 so it's more than
24 hours appart.

[Holger] On the first of the month we
sometimes find new bugs where there's

packages which record the month.

We don't have variations of the CPU type
at the moment.

Both time and CPU type variations, we'll
have them about one or two weeks

the nodes are being prepared at the moment.

Then we will test all the meaningful
variations we could think of.

There will be probably some packages which
build different according to the number of

number of CD drives attached or whatever
things, but those will be find by you.

[Lunar] We are doing all these tests because
we want when you rebuild a package on

your machine that if any this is different from
the build deamons in Debian you get

the same results.

We use this to detect this problems early
before you actually a ??? that we have

to investigate when someone rebuilds a
package on their machine.