WEBVTT
00:00:00.762 --> 00:00:02.081
Welcome and good morning
00:00:03.719 --> 00:00:07.221
This is the reproducible builds team,
talking about
00:00:07.221 --> 00:00:09.941
"Stretching out towards trustworthy
computing"
00:00:12.280 --> 00:00:19.921
[Applause]
00:00:22.281 --> 00:00:25.881
We're 4 on stage, but actually this is a
team effort.
00:00:25.881 --> 00:00:30.521
All these people listed here have
contributed to the project at one point.
00:00:30.521 --> 00:00:33.191
The 4 of us, that's
00:00:33.191 --> 00:00:34.161
Lunar − me
00:00:34.161 --> 00:00:35.282
there's Dhole,
00:00:35.282 --> 00:00:36.381
Chris Lamb − lamby
00:00:36.381 --> 00:00:37.601
and Holger.
00:00:38.540 --> 00:00:42.571
But actually, this is DebConf and so a lot
more of us have been or are
00:00:42.571 --> 00:00:46.961
currently here and so, if you want to
thank anybody that is working on this
00:00:46.961 --> 00:00:49.182
you need to actually thank all of
these folks
00:00:49.182 --> 00:00:50.981
'cause, yay.
00:00:51.421 --> 00:00:56.201
[Applause]
00:00:57.331 --> 00:00:59.601
[Holger] The people in blue are here.
00:01:03.902 --> 00:01:05.570
[Lunar] Let's get started.
00:01:05.570 --> 00:01:07.884
Quick recap on what we're talking
about.
00:01:07.884 --> 00:01:10.902
We have software, it's made from source.
00:01:10.902 --> 00:01:15.032
Source is readable by humans or at least
a good amount of humans.
00:01:15.032 --> 00:01:17.443
In this room it's good.
00:01:17.443 --> 00:01:24.180
Binary, readable by computer and some
tiny fraction of humanity.
00:01:24.180 --> 00:01:30.040
Going from source to binary is called
build, or like building or compiling
00:01:30.040 --> 00:01:33.181
and we're doing free software and
free software is awesome because
00:01:33.181 --> 00:01:37.541
we can actually run these binaries like
we want
00:01:37.541 --> 00:01:44.001
We can actually study the software, how
it's been made by studying the source
00:01:44.001 --> 00:01:48.541
and by studying the source we can assess
that it does what it's supposed to do
00:01:48.541 --> 00:01:50.741
and not something else that does not
00:01:50.741 --> 00:01:56.461
have malware, or trojans or security bugs
00:01:56.461 --> 00:02:00.822
So we have the binary that can be used,
fine.
00:02:00.822 --> 00:02:04.021
We have the source that can be verified.
00:02:04.021 --> 00:02:10.461
Problem is that right now, the only way we
know that a binary that we get…
00:02:10.461 --> 00:02:15.681
We have to trust a website or a Debian
repository that says
00:02:15.681 --> 00:02:18.261
"Well, this binary has been made with this
source"
00:02:18.261 --> 00:02:23.001
But there's no way we can actually prove
that.
00:02:23.001 --> 00:02:27.400
This is actually a problem that has been
well explained by
00:02:27.400 --> 00:02:33.541
Mike Perry and Seth Schoen at the 31c3
in Hamburg last december.
00:02:33.541 --> 00:02:41.481
For example, Seth Schoen made a proof of
concept exploit for the Linux kernel
00:02:41.481 --> 00:02:52.061
that when GCC was called, the kernel would
without modifying anything on the disk
00:02:52.061 --> 00:02:58.962
when the kernel detects that GCC is going
to read a C file, it will insert some
00:02:58.962 --> 00:03:06.260
extra lines of code, and these lines of
code can be a very bad thing
00:03:06.260 --> 00:03:09.100
in the case of 31c3 talk I was just
recalling.
00:03:09.100 --> 00:03:17.901
Actually, you can even have developers
who are in very good faith, who have
00:03:17.901 --> 00:03:21.302
totally secure dev machines, or they
thought they have,
00:03:21.302 --> 00:03:24.361
who have reviewed all their source code
for any bugs
00:03:24.361 --> 00:03:31.022
and we would still get totally owned as
soon as their computer gets compromised
00:03:31.022 --> 00:03:34.122
or one of the build demons from Debian
gets compromised for example.
00:03:34.122 --> 00:03:41.241
This is not, like, hypothetical threats
here we're discussing
00:03:41.241 --> 00:03:45.801
A couple of months after Seth an Mike's
talk at 31c3,
00:03:45.801 --> 00:03:48.940
the Intercept revealed from the Snowden
leaks
00:03:48.940 --> 00:03:56.281
that at a CIA conference in 2012, one
of the talks that happened
00:03:56.281 --> 00:03:58.921
was about a project called Strawhorse.
00:03:58.921 --> 00:04:04.942
Strawhorse is about modifying Apple XCode,
which is the development environment
00:04:04.942 --> 00:04:08.861
for MacOS 10 and iOS applications
00:04:08.861 --> 00:04:11.140
and well, they were modifying XCode so
it would produce,
00:04:11.140 --> 00:04:13.482
without the developer knowing,
00:04:13.482 --> 00:04:23.021
binaries with trojans, malware,
watermarked binaries, lots of bad things.
00:04:23.181 --> 00:04:25.401
So, solution:
00:04:25.401 --> 00:04:29.421
enable anyone to reproduce identical
binary packages from a given source.
00:04:29.421 --> 00:04:34.901
Because if using a source, using the same
environment,
00:04:34.901 --> 00:04:39.981
multiple people on different computers, on
different networks, at different times,
00:04:39.981 --> 00:04:42.881
can all get the same thing
from the same source
00:04:42.881 --> 00:04:44.621
all the same binary, byte for byte,
00:04:44.621 --> 00:04:47.081
then there's a good chance that…
00:04:47.081 --> 00:04:54.881
Well, everybody could be owned,
but let's be more joyful and say that
00:04:54.881 --> 00:04:58.841
probably, if everybody gets the same
result, there was actually no problem
00:04:58.841 --> 00:05:01.381
and everybody is safe.
00:05:01.841 --> 00:05:04.461
We call that solution
"reproducible builds"
00:05:06.661 --> 00:05:07.684
Yay.
00:05:08.481 --> 00:05:10.562
[Applause]
00:05:12.721 --> 00:05:14.681
Actually, it's not only about security.
00:05:14.681 --> 00:05:19.061
For Debian, we have, if you're doing
"Multi-arch: same" packages,
00:05:19.061 --> 00:05:24.880
well they only have the same bytes if
they are built for different architectures,
00:05:24.880 --> 00:05:27.800
the files in the package.
00:05:27.800 --> 00:05:34.462
Debug packages, you can create at a later
time, if you forgot to have debug packages
00:05:34.462 --> 00:05:35.741
in the first place,
00:05:35.741 --> 00:05:42.480
you can pass the no-strip option later and
because the package is reproducible,
00:05:42.480 --> 00:05:46.582
you will get the debug symbols that work
for software that has been shipped already
00:05:46.582 --> 00:05:50.360
We do early detection of FTBFS that way
00:05:50.360 --> 00:05:53.581
because if we try pretty quickly
to reproduce a build,
00:05:53.581 --> 00:05:55.141
then it has to work.
00:05:55.141 --> 00:05:58.321
It's useful for build profiles.
00:05:58.321 --> 00:06:01.682
We can get smaller .deb deltas,
00:06:01.682 --> 00:06:05.381
because from one version to the next we
might have the same content.
00:06:05.381 --> 00:06:08.861
We can do validation of cross-builds,
00:06:08.861 --> 00:06:11.721
Helmut Grohne can talk to you about that.
00:06:11.721 --> 00:06:16.702
And also, Niels Thykier told me that
00:06:16.702 --> 00:06:21.081
he was very interested in reproducible
builds because it would enable him to
00:06:21.081 --> 00:06:23.621
test debhelper better, because
00:06:23.621 --> 00:06:28.510
if the package builds reproducibly,
then he makes a change to debhelper,
00:06:28.510 --> 00:06:32.111
then he can rebuild
00:06:32.111 --> 00:06:36.131
the same version of a package with a newer
debhelper and see what has changed
00:06:36.131 --> 00:06:39.921
and this change can be isolated to only
what he has worked on debhelper
00:06:39.921 --> 00:06:41.601
for example.
00:06:43.211 --> 00:06:45.042
And, oh my.
00:06:45.042 --> 00:06:47.820
The whole world is watching us.
00:06:47.820 --> 00:06:56.482
Since two years or a year and a half ago,
everybody I meet in security conference,
00:06:56.482 --> 00:06:59.101
in hacker conference, in free software
conference is like
00:06:59.101 --> 00:07:01.160
"Oh you're working on that,
that's awesome."
00:07:01.160 --> 00:07:08.621
And, I mean, I've been the one doing quite
a lot of talks, and everybody comes to me
00:07:08.621 --> 00:07:11.102
and I'm like "Wow wow, this is way bigger",
00:07:11.102 --> 00:07:15.682
but we're actually leading the field here.
00:07:15.682 --> 00:07:18.580
Yay Debian.
00:07:18.580 --> 00:07:25.560
[Applause]
00:07:25.952 --> 00:07:29.320
[Holger] So, we are not the only ones
leading the field,
00:07:29.320 --> 00:07:33.241
Bitcoin and Tor made their software
reproducible before us,
00:07:33.241 --> 00:07:36.901
Coreboot also succeeded, if you build
Coreboot without any payload,
00:07:36.901 --> 00:07:38.561
that's 100% reproducible.
00:07:38.561 --> 00:07:43.702
FreeBSD has a page on their wiki since
2013
00:07:43.702 --> 00:07:48.840
saying there are 5 reproducibility issues
in their base system.
00:07:48.840 --> 00:07:51.861
We're at the moment trying to
confirm this.
00:07:51.861 --> 00:07:57.080
On jenkins.debian.net, I've also set up
now tests for FreeBSD, NetBSD,
00:07:57.080 --> 00:07:58.921
Coreboot and OpenWrt.
00:07:58.921 --> 00:08:02.881
So if you go to
reproducible.debian.net/
00:08:02.881 --> 00:08:04.901
you get that tested.
00:08:04.901 --> 00:08:07.941
And there's more in the pipeline.
00:08:07.941 --> 00:08:10.981
There are other projects interested
as well.
00:08:10.981 --> 00:08:15.121
NetBSD also has a variable MKREPRO
which you can set
00:08:15.121 --> 00:08:17.162
and that builds reproducibly.
00:08:17.162 --> 00:08:20.281
Though they think "I'm keeping some
timestamps it's fine" and then
00:08:20.281 --> 00:08:21.901
filtering them out later".
00:08:21.901 --> 00:08:23.201
We disagree.
00:08:23.201 --> 00:08:28.400
So this is how Debian looks like,
Debian Sid,
00:08:28.400 --> 00:08:30.020
but this is a lie.
00:08:30.020 --> 00:08:31.601
This is not the truth.
00:08:31.601 --> 00:08:33.921
This is just our test setup.
00:08:33.921 --> 00:08:35.659
Sid is not like this.
00:08:35.659 --> 00:08:40.100
For Sid, it's all orange, there's zero
reprodicibility in Sid today.
00:08:40.100 --> 00:08:43.962
But we'll talk now and in the following
round table,
00:08:43.962 --> 00:08:46.781
it's to actually make Sid reproducible.
00:08:46.781 --> 00:08:52.461
The current status is
00:08:52.461 --> 00:08:57.981
we're working on this in Debian since
two years ago.
00:08:57.981 --> 00:09:01.941
We have weekly reports about our project
now since May
00:09:01.941 --> 00:09:07.201
and we've given several talks, especially
in the last year
00:09:07.201 --> 00:09:11.422
and all these talks, presentation, also
other stuff is linked in the wiki.
00:09:11.422 --> 00:09:14.921
There's a page with information about
Debian, these BSDs,
00:09:14.921 --> 00:09:18.960
other Linuxes, upstream softwares
all on this wiki.
00:09:22.850 --> 00:09:26.561
Since DebConf14, which is merely
a year ago,
00:09:26.861 --> 00:09:28.720
we've made quite some changes.
00:09:28.720 --> 00:09:32.781
We have introduced
strip-nondeterminism
00:09:32.781 --> 00:09:38.880
which is called by dh at the end
of the build of the package
00:09:38.880 --> 00:09:45.463
and will normalize some things
which Chris will explain later
00:09:45.463 --> 00:09:50.041
We have decided on a fixed build path
00:09:50.041 --> 00:09:53.761
because the build path is leaked
in the binaries and several things
00:09:53.761 --> 00:09:57.083
We didn't find a way yet to make
the build path arbitrary.
00:09:57.083 --> 00:10:03.461
We designed a way to record the build
environment
00:10:03.461 --> 00:10:08.061
because to rebuild, you need to recreate
the build environment.
00:10:08.061 --> 00:10:11.621
We set up this Jenkins setup.
00:10:11.621 --> 00:10:17.381
We wrote diffoscope which used to be
called debbindiff
00:10:17.381 --> 00:10:21.401
which shows differences between two
packages or two directories or
00:10:21.401 --> 00:10:23.601
two filesystems by now.
00:10:23.601 --> 00:10:30.931
There's SOURCEDATEEPOCH, which is a way
that the tools expose
00:10:30.931 --> 00:10:33.691
the last modification of the source.
00:10:33.691 --> 00:10:37.280
Because the build date, people want to
include the build date
00:10:37.280 --> 00:10:39.461
because they think this is a
meaningful indication:
00:10:39.461 --> 00:10:42.261
when a build was done,
which software used.
00:10:42.462 --> 00:10:45.565
But if the build always recreates
the same results
00:10:45.565 --> 00:10:47.340
the build date becomes meaningless
00:10:47.340 --> 00:10:50.681
and the really interesting thing is
the latest modification of the source.
00:10:52.391 --> 00:10:55.841
We have written patches for the tools
00:10:58.081 --> 00:11:03.591
[Lunar] strip-nondeterminism:
is Andrew Ayer in the audience?
00:11:03.591 --> 00:11:05.921
Yay! He did it!
00:11:05.921 --> 00:11:12.361
It's written in Perl because we didn't
want to have a new build dependency
00:11:12.361 --> 00:11:13.564
in all Debian packages.
00:11:13.564 --> 00:11:18.443
Basically it takes anything and tries
to normalize it as much as it can
00:11:18.443 --> 00:11:27.101
replacing timestamps or file permissions
or removing some issues.
00:11:27.101 --> 00:11:30.901
It's working very well on many formats,
it's meant to be extensible
00:11:30.901 --> 00:11:38.000
so we can actually add more things and
it's run by dh at the end of the process, as Holger said.
00:11:38.000 --> 00:11:45.422
The .buildinfo is currently a proposal
we have not yet totally agreed
00:11:45.422 --> 00:11:48.581
but we are generating them as part
of the test we have
00:11:48.581 --> 00:11:56.882
and basically it's a new control file that
will tie the sources, the generated binary
00:11:56.882 --> 00:12:00.560
the packages that were used to build this
binary and their version.
00:12:00.560 --> 00:12:08.503
The idea is that we can use this file to
reinstall all the specific versions from snapshot
00:12:08.503 --> 00:12:16.701
So we recreate the same build environment
then we can just start the build from that source
00:12:16.701 --> 00:12:20.981
that was mentioned and see if the binary
that has been generated matches.
00:12:23.051 --> 00:12:28.403
What it looks like for now, you see there is
a source binary, the build path
00:12:28.403 --> 00:12:33.881
because currently we don't have any good
post-processing tool for buildpaths
00:12:33.881 --> 00:12:41.061
in elf and dwarf binaries, we just decided
to specify the build path so when we do
00:12:41.061 --> 00:12:44.941
a later rebuild we use that path and be safe.
00:12:44.941 --> 00:12:51.522
The source is dsc, the binary is .deb and
a list of packages with the versions.
00:12:53.322 --> 00:13:01.931
We currently use the base files version
to know which Debian release is to be used
00:13:01.931 --> 00:13:04.044
as the basis.
00:13:11.331 --> 00:13:17.851
[Holger] The general procedure for testing is:
we build the source, we save the results,
00:13:17.851 --> 00:13:22.663
we modify the environment and we build
it again and compare the results.
00:13:22.663 --> 00:13:31.741
That started as a shell script last year which I
put on jenkins and then it exploded a bit
00:13:32.002 --> 00:13:36.141
and now we have 67 jenkins jobs running on
7 hosts.
00:13:36.141 --> 00:13:44.821
Since last week we have 4 armhf small boards
where we will be able to test armhf,
00:13:44.821 --> 00:13:46.001
but very slowly.
00:13:46.001 --> 00:13:48.901
We have two new amd64 build nodes.
00:13:48.901 --> 00:13:53.041
The code is now split into Python and bash
scripts.
00:13:53.041 --> 00:13:58.541
For all the other distro testing there's a
lot of bash code now which is mostly
00:13:58.541 --> 00:14:04.741
boilerplate and it's 5 lines or something
to build FreeBSD and 5 lines to build NetBSD
00:14:04.741 --> 00:14:08.841
but there's 100 lines boilercode around so it's
really not that much code.
00:14:08.841 --> 00:14:12.521
We do test Testing, Unstable and Experimental.
00:14:12.521 --> 00:14:15.561
For arm we only start with Unstable.
00:14:15.561 --> 00:14:22.063
We do like hardware so if you have hardware
to donate to us, that would be great,
00:14:22.063 --> 00:14:24.961
we need ssh and then root basically.
00:14:26.921 --> 00:14:34.150
We are testing Coreboot, OpenWrt and the
BSD's, soon I will also set up a Fedora test
00:14:34.150 --> 00:14:39.681
I don't want to test all the 20,000 Fedora
packages but just 200 or something:
00:14:39.681 --> 00:14:44.121
the base system of Fedora to examine how
rpm works
00:14:44.121 --> 00:14:47.741
to get really the whole Free Software world
reproducible.
00:14:47.741 --> 00:14:53.460
This is all run on ProfitBricks hardware
since 2002, so thanks to ProfitBricks.
00:14:56.861 --> 00:15:00.081
This is the variations we do for Debian.
00:15:01.741 --> 00:15:07.261
It's the hostname, username, timezone,
locale.
00:15:07.261 --> 00:15:14.100
Chris will explain what modifications
this causes, variances...
00:15:14.100 --> 00:15:19.061
We are not testing at the moment differences
in date so the date is always the same
00:15:19.061 --> 00:15:20.381
the time is a bit different.
00:15:20.381 --> 00:15:25.521
[Lunar] Well almost! Because we cheat with
the timezone, we use one timezone that is
00:15:25.521 --> 00:15:32.241
GMT-14 and then GMT+12 so it's more than
24 hours appart.
00:15:32.650 --> 00:15:35.941
[Holger] On the first of the month we
sometimes find new bugs where there's
00:15:35.941 --> 00:15:38.002
packages which record the month.
00:15:41.002 --> 00:15:43.821
We don't have variations of the CPU type
at the moment.
00:15:45.782 --> 00:15:50.901
Both time and CPU type variations, we'll
have them about one or two weeks
00:15:50.901 --> 00:15:53.663
the nodes are being prepared at the moment.
00:15:53.663 --> 00:16:00.801
Then we will test all the meaningful
variations we could think of.
00:16:01.234 --> 00:16:05.021
There will be probably some packages which
build different according to the number of
00:16:05.021 --> 00:16:11.141
number of CD drives attached or whatever
things, but those will be find by you.
00:16:12.491 --> 00:16:16.891
[Lunar] We are doing all these tests because
we want when you rebuild a package on
00:16:16.891 --> 00:16:22.022
your machine that if any this is different from
the build deamons in Debian you get
00:16:22.022 --> 00:16:23.291
the same results.
00:16:23.291 --> 00:16:30.011
We use this to detect this problems early
before you actually a false positive that we have
00:16:30.011 --> 00:16:34.151
to investigate when someone rebuilds a
package on their machine.
00:16:37.321 --> 00:16:42.541
To understand the difference that we found
from one build to the other.
00:16:42.541 --> 00:16:50.541
It started also as a 10 lines shellscript
and then it felt okeyish
00:16:50.541 --> 00:16:51.941
and so Python!
00:16:51.941 --> 00:16:57.601
And now it's a lot of code and it actually
grew way beyond a Debian package.
00:16:57.601 --> 00:17:03.041
We changed the name, it was called debbindiff
but it's absolutely not tied to Debian anymore.
00:17:03.041 --> 00:17:07.421
It's called diffoscope, thanks to Jocelyn
for the name.
00:17:07.421 --> 00:17:12.020
Basically what it does: it tries to get to
the bottom of what is different between
00:17:12.020 --> 00:17:13.801
two archives or directories.
00:17:13.801 --> 00:17:22.260
Because it's not useful to compare bytes that
are compressed by gzip or xz, that will not
00:17:22.260 --> 00:17:27.021
lead you to understand what is different
you need to uncompress and look at
00:17:27.021 --> 00:17:32.801
uncompressed data, and if the thing actually
compressed is a tarball, you might actually
00:17:32.801 --> 00:17:35.060
want to compare the files inside the tarball.
00:17:35.060 --> 00:17:42.261
If there is a PDF inside this archive, you
don't want to compare the bytes of the PDF
00:17:42.261 --> 00:17:43.981
you want to compare the text of the PDF.
00:17:43.981 --> 00:17:49.681
So this is basically what diffoscope does,
it tries to transform anything that is
00:17:49.681 --> 00:17:56.601
a container and compare things in this
container and if they can be transformed into
00:17:56.601 --> 00:18:01.181
a human readable form it will try to do
that, and compare these human readable form.
00:18:01.181 --> 00:18:05.151
And if it doesn't find any difference but
there are still differences from the bin
00:18:05.151 --> 00:18:07.322
it will fall back to binary comparison.
00:18:07.521 --> 00:18:12.681
Try it, extend it; it's Python, it's modular,
it's great.
00:18:12.681 --> 00:18:23.041
It already supports squashfs, ISO, rpm,
gettext, mo files files and so many different things.
00:18:23.041 --> 00:18:29.981
You can have HTML output like that,
so this is what is displayed on many
00:18:29.981 --> 00:18:34.021
examples we've shown so far, and also
to make it easier for copy paste
00:18:34.021 --> 00:18:38.141
and post processing we have the text output.
00:18:38.141 --> 00:18:43.060
You can also use it to review packages before
uploading them to Debian.
00:18:43.060 --> 00:18:49.161
It does fuzzy matching, so even if the
directory is different in the archive it will
00:18:49.161 --> 00:18:52.441
find it like git does.
00:18:52.441 --> 00:18:58.561
It has grown way more beyond just build
reproducibly. A useful tool.
00:19:01.451 --> 00:19:07.371
[Dhole] In order to solve timestamp issues, we are
proposing the SOURCEDATEEPOCH variable.
00:19:07.381 --> 00:19:11.851
This is because most of the times having
the build date embedded in a package
00:19:11.851 --> 00:19:16.391
is not useful for the user, because you could
take a really old package and build it today
00:19:16.391 --> 00:19:19.001
and that day would not be useful.
00:19:19.001 --> 00:19:25.740
We are standardizing a replacement for build
dates so that tools can use it.
00:19:25.740 --> 00:19:31.961
When this value is set, the tool instead of
embedding the current date, it will embed
00:19:31.961 --> 00:19:37.501
the date taken from SOURCEDATEEPOCH which
will contain a Unix epoch timestamp.
00:19:37.501 --> 00:19:42.881
This is a general solution we are trying to
standardize so that not only Debian uses it,
00:19:42.881 --> 00:19:48.020
but other Free Software projects and
distributions and in the case of Debian,
00:19:48.020 --> 00:19:52.461
we set this variable to the latest Debian
changelog entry timestamp.
00:19:55.101 --> 00:20:00.721
We have already been sending patches to
different packages, mostly it's documentation
00:20:00.721 --> 00:20:06.061
generation. So here's a list of bugs that
we have opened which have been closed
00:20:06.061 --> 00:20:12.291
and merged; so it's help2man, epydoc,
ghostscript, texi2html and sphinx.
00:20:12.291 --> 00:20:18.991
We are both sending these patches to Debian
and upstream so all the distributions can
00:20:18.991 --> 00:20:28.161
use them, and we have also been sending
patches to other packages which are still
00:20:28.161 --> 00:20:32.160
open, so we encourage you to take a look
at these packages if you are the maintainer
00:20:32.160 --> 00:20:34.622
and merge the patch.
00:20:36.161 --> 00:20:41.181
[Lunar] Thanks to Daniel Kahn Gillmor and
Ximin Luo for pushing this proposal forward.
00:20:41.181 --> 00:20:45.541
And also lots of these patches have been
written by Akira and Dhole as part of their
00:20:45.541 --> 00:20:48.681
Google Summer of Code, and you work really
great.
00:20:52.021 --> 00:20:56.741
[Applause]
00:21:02.722 --> 00:21:07.721
[Dhole] The gcc patch is: gcc uses two
macros which are _DATE and TIME_
00:21:07.721 --> 00:21:14.300
which embed the timestamp and I wrote a
patch so that if SOURCEDATEEPOCH is set
00:21:14.300 --> 00:21:18.740
instead of adding the current time, it takes
the time from that variable.
00:21:18.740 --> 00:21:25.560
I sent this patch to gcc, it's still there
forgotten with many other patches
00:21:25.560 --> 00:21:29.401
but hopefully at some point they will
realize that this is interesting and they
00:21:29.401 --> 00:21:30.421
will merge it.
00:21:38.881 --> 00:21:46.331
[Lamby] Hey. Let's very quickly run you
through some really simple ways
00:21:46.331 --> 00:21:50.441
to fixing packages. The details don't
necessarily matter, it's just to give you
00:21:50.450 --> 00:21:55.641
of what needs to be changed and basically
to point out that it's not rocket science.
00:21:55.641 --> 00:21:57.661
So you can just come in and jump in.
00:21:57.661 --> 00:22:07.861
For example gzip, it's a very old tool
and they decided to add timestamps when
00:22:07.861 --> 00:22:11.942
you generate it, but it's an easy fix, you
just add -n flag.
00:22:11.942 --> 00:22:19.981
Some other things easy to change: some
Python stuff had tag_date=True, which
00:22:19.981 --> 00:22:25.081
I don't know if you can see it but adds a
timestamp to eggs. You just change it to
00:22:25.081 --> 00:22:26.401
False to get rid of it.
00:22:26.401 --> 00:22:34.342
Static libraries, they are just ar archives
so the same format as .deb, and you
00:22:34.342 --> 00:22:37.941
can just use binutils or strip-nondeterminism
tool.
00:22:37.941 --> 00:22:44.361
PNG has timestamps for some reason, you can
get rid of them, that's ImageMagick and it's
00:22:44.361 --> 00:22:49.301
a bit ugly, but also strip-nondeterminism
gets rid of it.
00:22:49.301 --> 00:22:54.641
Tarballs are quite interesting, they will
by default capture user and group
00:22:54.641 --> 00:22:58.341
you just pass --owner=root bla bla bla...
00:22:58.341 --> 00:23:04.502
Ordering, this is interesting as well, it
will usually use file system ordering
00:23:04.502 --> 00:23:10.641
which is completely non-deterministic. So
you need to sort with LC_ALL=C.
00:23:14.741 --> 00:23:18.781
[Lunar] Think about the locale! Because
sorting order varies from local to the next.
00:23:22.681 --> 00:23:28.081
[Lamby] They also take timestamps, again
you can set --mtime or you can mock around
00:23:28.081 --> 00:23:31.041
with find/xargs/touch bla bla...
00:23:31.041 --> 00:23:37.401
Lots of other files have timestamps: Erlang
files for no reason, even upstream don't
00:23:37.401 --> 00:23:39.601
know why they added a timestamp.
00:23:42.481 --> 00:23:48.601
We have now a patch for SOURCEDATEEPOCH,
which I think landed a couple days ago.
00:23:49.881 --> 00:23:57.141
Here's an interesting one, not necessarily
the current build timestamp, so this is a
00:23:57.141 --> 00:24:04.642
timezone dependent date which Ruby loads
and then saves incorrectly as your local time.
00:24:04.642 --> 00:24:07.420
This gets mangled, so that's patching.
00:24:07.420 --> 00:24:14.501
I'm going from changing individual packages
to more toolchain things as you can see.
00:24:14.501 --> 00:24:21.001
Upstream configure scripts, you can maybe
see the top that it just uses hostname
00:24:21.001 --> 00:24:26.241
for no reason. Sometimes you can override
it in debian/rules just by exporting something
00:24:26.241 --> 00:24:31.921
or passing a variable to dh_autobuild or
whatever. That's just a little bit more
00:24:31.921 --> 00:24:33.880
involved, you have to look at it more
carefully.
00:24:33.880 --> 00:24:40.441
Perl hash order, lot of Perl uses data
Data::Dumper to just output a bunch of stuff which
00:24:40.441 --> 00:24:46.761
is just not deterministic. So often just
setting Sortkeys, but sometimes it's
00:24:46.761 --> 00:24:48.461
a completely different solution.
00:24:48.461 --> 00:24:53.321
Header files, so you can maybe see that
they are using the timestamp essentially
00:24:53.321 --> 00:24:59.481
as a unique identifier, you probably have
to start re-writing these something saner
00:24:59.481 --> 00:25:03.901
because this is a wrong use of timestamp
anyway.
00:25:03.901 --> 00:25:12.801
More Makefiles, the deeper they timestamp
in the upstream package the more you have
00:25:12.801 --> 00:25:15.381
to start patching, so these kind of start
sucking a little.
00:25:15.381 --> 00:25:21.021
We've made a lot of toolchain changes, some
already mentioned, some of them already
00:25:21.021 --> 00:25:25.201
merged, see more in this link. Again,
details don't matter, just check it out
00:25:25.201 --> 00:25:29.781
it isn't crazy, it's just working out
what's different.
00:25:29.781 --> 00:25:35.061
In terms of the work done we've sent these
many patches: two patches a day,
00:25:35.061 --> 00:25:37.900
which is not too bad, on average.
00:25:40.170 --> 00:25:46.442
[Applause]
00:25:48.111 --> 00:25:50.762
[Holger] I can't clap because I sent three
or something like that
00:25:52.510 --> 00:25:53.742
[Lamby] Holger does three per day.
00:25:54.841 --> 00:25:59.642
And this doesn't count other bugs we found
in the process of building packages, like
00:25:59.642 --> 00:26:00.801
fail to build.
00:26:00.801 --> 00:26:08.361
This is blue the ones that are open and
orange are done.
00:26:08.361 --> 00:26:14.001
You can see that someone went a bit crazy
in February filing bugs and eventually they
00:26:14.001 --> 00:26:17.201
were being fixed; slowly.
00:26:18.421 --> 00:26:23.591
[Holger] And actually we filed more bugs
because the fail to build from source bugs
00:26:23.591 --> 00:26:28.702
are excluded, I think we filed 300 FTBFS
in the last two or three months.
00:26:30.851 --> 00:26:34.400
[Lamby] And those include fail to build
because of reproducibility things as well
00:26:34.400 --> 00:26:36.142
but we haven't split them up.
00:26:40.022 --> 00:26:46.641
[Lunar] What's left to be done because
Holger said "the graph is a lie".
00:26:46.641 --> 00:26:58.241
The main thing that is blocking a lot of
work is dpkg. Right now the output of dpkg
00:26:58.241 --> 00:27:09.261
will be not deterministic 100% of the time,
because of timestamps and at least the
00:27:09.261 --> 00:27:15.381
file ordering. We also have a patch that
creates these .buildinfo files that we've
00:27:15.381 --> 00:27:22.422
shown that works. It's not submitted yet
to dpkg because we need to agree on the
00:27:22.422 --> 00:27:27.021
format. At least we have ftpmaster or
maybe dpkg, well we have a lot of people
00:27:27.021 --> 00:27:29.500
and that's what we are going to do the
next hour.
00:27:29.500 --> 00:27:38.560
Debhelper also has a few changes; the make
mtimes, debhelper might also not be
00:27:38.560 --> 00:27:42.781
best place, maybe we want that in dpkg.
00:27:42.781 --> 00:27:47.501
I've been trying to put patches in tar so
we can make it easier. It's complicated to
00:27:47.501 --> 00:27:54.240
see where's the best place but so far we've
been doing our tests with this frame and it works.
00:27:54.240 --> 00:27:59.961
[Holger] In our repository we have these
packages with these bugs fixed so when
00:27:59.961 --> 00:28:03.623
you want to test reproducibility issues on
your own machine you need to use the
00:28:03.623 --> 00:28:07.192
repository which has these patches applied
at the moment.
00:28:07.192 --> 00:28:10.201
In pure sid you cannot create reproducible
packages.
00:28:10.461 --> 00:28:18.201
[Lunar] I heard that the SOURCEDATEEPOCH
patch is in git already, so it's going to happen.
00:28:18.201 --> 00:28:26.640
cdbs also needed to export SOURCEDATEEPOCH
and we are starting to do more infrastructure
00:28:26.640 --> 00:28:34.381
work: Josch mainly and Akira on sbuild,
because we wanted to have this
00:28:34.381 --> 00:28:40.421
srebuild script, where you give it a
buildinfo and it will do the rebuild and
00:28:40.421 --> 00:28:47.101
it needs changes in build daemon for the
build path and also a couple of changes in
00:28:47.101 --> 00:28:48.940
sbuild itself.
00:28:48.940 --> 00:28:53.301
[Holger] And the script is not ready yet,
this "Finish" means it uses our repository
00:28:53.301 --> 00:28:56.962
at the moment, we need to change it to only
use Sid and snapshot.
00:28:56.962 --> 00:29:01.700
[Lunar] So there is the buildd issue that
we need to discuss
00:29:01.700 --> 00:29:08.742
and we also need to see how we could include
or not, or somewhere give this buildinfo
00:29:08.742 --> 00:29:12.501
control file to the world so they can
rebuild the packages, so it's not yet
00:29:12.501 --> 00:29:14.421
clear where's the best place to store
them.
00:29:14.421 --> 00:29:20.821
Because adding 22,000 files, some
people get cranky of this idea.
00:29:20.821 --> 00:29:25.841
[Holger] It's more than 22,000 files, it's
22,000 source packages multiplied by
00:29:25.841 --> 00:29:30.400
10 architectures; but there's a lot of
arch builds so that's probably 100,000
00:29:30.400 --> 00:29:37.561
buildinfo files, multiplied by Stretch and
Sid, so it's 200,000 files or more on
00:29:37.561 --> 00:29:40.001
the file servers and on the mirrors we
would like to have it.
00:29:40.001 --> 00:29:43.881
That's the same amount of files which are
currently there. The mirror operators are
00:29:43.881 --> 00:29:49.001
currently not happy, they will not take it,
so our current idea is just concatenate
00:29:49.001 --> 00:29:54.681
all these files into one file that's 140 MB
uncompressed, 40 MB compressed.
00:29:54.681 --> 00:29:56.441
That's easier to handle.
00:29:56.441 --> 00:29:59.880
And then probably have a service
buildinfo.debian.org where you can
00:29:59.880 --> 00:30:02.780
download individual buildinfo files if you
need them.
00:30:03.660 --> 00:30:09.901
[Lunar] And so when we will be done with
all that we can maybe add a final patch
00:30:09.984 --> 00:30:16.301
it would be to Debian policy, mandating
Debian packages be reproducible.
00:30:19.511 --> 00:30:22.731
[Applause]
00:30:24.232 --> 00:30:31.151
I can say again that the dream of mine is
that we would stop uploading .deb when
00:30:31.151 --> 00:30:37.881
we upload a package, but instead just upload
the hash of the binary, have the buildd
00:30:37.881 --> 00:30:43.051
build again this package and only if these
two match they can enter the archive.
00:30:43.051 --> 00:30:47.864
So we are sure that at least the two
machines, the developer machine and the
00:30:47.864 --> 00:30:50.521
build deamon agree that they've built the
same thing.
00:30:51.021 --> 00:30:55.401
[Applause]
00:30:58.162 --> 00:31:02.781
[Holger] I share this dream but I think
having this in policy is a mass requirement
00:31:02.781 --> 00:31:15.920
sadly something only for Stretch + 1, but
I'm curious if we had fixed dpkg and
00:31:15.920 --> 00:31:21.661
debhelper now, would you think we should
upgrade all these wishlist bugs to important now?
00:31:22.631 --> 00:31:26.341
[Audience] Yes!
00:31:31.062 --> 00:31:33.586
[Holger] We'll talk about this later soon.
00:31:34.322 --> 00:31:36.642
[Lunar] But before that we actually have
work to do.
00:31:40.141 --> 00:31:43.641
[Dhole] In order to fix your package, the
first thing you can do is go to
00:31:43.641 --> 00:31:51.342
reproducible.debian.net/package, and you
can the web interface where you can see
00:31:51.342 --> 00:31:56.190
notes on the package, we have tags to
identify different issues that make packages
00:31:56.190 --> 00:31:59.180
not reproducible, with links to the wiki
about how to solve them.
00:32:04.811 --> 00:32:08.742
[Holger] When you see this, you want to
click on this debbindiff link.
00:32:08.742 --> 00:32:12.441
It's still called debbindiff not diffoscope,
this will show all the differences,
00:32:12.441 --> 00:32:17.081
if there is a note. If the package is
unreproducible and there's no note
00:32:17.081 --> 00:32:21.041
it will automatically display the
debbindiff, and if your package is fine
00:32:21.041 --> 00:32:23.241
there's here a sun.
00:32:29.500 --> 00:32:34.141
[Dhole] You can also see an entry in the
tracker, stating if your package is
00:32:34.141 --> 00:32:35.481
reproducible or not.
00:32:38.751 --> 00:32:45.601
You can also find information in DDPO and
DMD. You can find tips on the wiki it's
00:32:45.601 --> 00:32:53.940
ReproducibleBuilds wiki, we are working on
a Howto to have detailed steps on different
00:32:53.940 --> 00:33:01.141
issues and how to solve them. Lunar gave
a talk at CCCamp where there's many issues
00:33:01.141 --> 00:33:05.462
really well explained and the solutions for
them.
00:33:05.462 --> 00:33:11.261
You can also come to our irc channel which
is #debian-reproducible and ask for help
00:33:11.261 --> 00:33:12.881
or go to the mailing-list.
00:33:14.241 --> 00:33:21.231
In order to test locally if your package is
reproducible right now we are using a
00:33:21.231 --> 00:33:29.401
script that uses pbuilder in a custom
configuration, you need to set up our
00:33:29.401 --> 00:33:35.240
reproducible repository. In the Howto in
the wiki there's the steps on how to set up
00:33:35.240 --> 00:33:38.861
the chroot and everything, it's documented
in the wiki.
00:33:38.861 --> 00:33:44.282
Diffoscope is in unstable and today it's
going in Stretch.
00:33:44.282 --> 00:33:53.961
We plan to add these scripts to rebuild
packages in different settings in debscripts
00:33:53.961 --> 00:34:04.181
once dpkg is good, and we welcome you
tomorrow to the hacking session from
00:34:04.181 --> 00:34:06.981
2 to 7 in Stockholm room.
00:34:10.461 --> 00:34:15.201
[Lunar] That's for fixing your packages,
please do that. If you want to have even
00:34:15.201 --> 00:34:18.742
more fun, then test your own package, join
us!
00:34:18.742 --> 00:34:25.002
This is the past year of my life, it has
been awesome because the team has been
00:34:25.002 --> 00:34:32.341
so great, it's been friendly atmosphere, lots of
new understanding so many things you didn't
00:34:32.341 --> 00:34:39.241
want to learn about that you had to learn
about, and basically it feels very good to
00:34:39.241 --> 00:34:46.462
be part of this actual changing the world
thing. It's just software but it has some
00:34:46.462 --> 00:34:51.121
profound effect. I've been told that the
work we've been doing is being tossed
00:34:51.121 --> 00:34:58.161
around in Cisco and Google and Facebook;
all these big dot com companies bla bla,
00:34:58.161 --> 00:35:01.561
they actually want to do that as well even
though they are not doing Free Software,
00:35:01.561 --> 00:35:03.401
which I find wired, but whatever.
00:35:03.401 --> 00:35:09.701
So what do we do? We review packages, we
have these notes when we actually try to
00:35:09.701 --> 00:35:13.031
identify, so when the maintainer comes
they don't have to think to much about
00:35:13.031 --> 00:35:19.131
the problem and just fix it. We try to
identify common trends so when many
00:35:19.131 --> 00:35:23.741
packages have the same problem we make an
entry and explain and maybe think about fixes
00:35:23.741 --> 00:35:26.821
that could apply to the whole archive.
00:35:26.821 --> 00:35:33.501
We work on this reproducible.debian.net
jenkins setup, the scripts.
00:35:33.501 --> 00:35:41.441
We hack on the diffoscope tool, we make
strip-nondeterminism better, we propose
00:35:41.441 --> 00:35:45.442
changes for the toolchains when there are
needs, some need a lot of patches,
00:35:45.442 --> 00:35:59.322
most of the bugs we have reported on
individual packages have patches.
00:36:01.352 --> 00:36:03.791
[Holger] Bugs have patches
[Lunar] Yes!
00:36:04.261 --> 00:36:08.801
And also we are actually writing some more
general documentation from the
00:36:08.801 --> 00:36:15.182
understanding of these things we have been
having, we are preparing a reproducible
00:36:15.182 --> 00:36:22.461
builds Howto to explain to the Free Software
world how they can do it so it's about some
00:36:22.461 --> 00:36:26.541
of what Chris explained but also more
general consideration on what if you're
00:36:26.541 --> 00:36:29.441
not Debian and you want your thing
reproducible when you distribute as an
00:36:29.441 --> 00:36:35.720
independent vendor. So we want to work on
reference documentation so the whole world
00:36:35.720 --> 00:36:37.421
can actually do that.
00:36:38.901 --> 00:36:43.142
We do a lot of talks as you've seen and
it's been fun, and with all these
00:36:43.142 --> 00:36:48.960
presentations we've made so far it's all
in git. And everybody is free to take one
00:36:48.960 --> 00:36:52.941
of these slide decks and run with it
somewhere, translate it...
00:36:56.891 --> 00:36:58.600
Questions?
00:37:01.240 --> 00:37:04.161
We have to run with the microphone, because
there's no mic anymore.
00:37:14.221 --> 00:37:17.201
[Question] I just wanted to make two quick
comments: so first of all diffoscope is
00:37:17.201 --> 00:37:22.141
really awesome, not only for reproducibility
but also for example if you change your
00:37:22.141 --> 00:37:27.400
debian/rules in some way and want to see if
the package is the same afterwards because
00:37:27.400 --> 00:37:31.561
you just cleaned up a bit, that's really
awesome for that, so thank you.
00:37:31.561 --> 00:37:37.481
And also I think the work you're doing now
is something that in 20 years time we're
00:37:37.481 --> 00:37:41.340
going to look back towards it and think,
well, of course builds should be
00:37:41.340 --> 00:37:44.481
reproducible, so thank you very much for
your work!
00:37:44.820 --> 00:37:49.021
[Applause]
00:37:52.400 --> 00:38:02.981
[Question] When reproducibility becomes
part of the Debian policy, will there be a
00:38:02.981 --> 00:38:05.900
lintian --reproducible?
00:38:09.180 --> 00:38:12.281
[Holger] I don't think lintian can detect
that because lintian works on the source
00:38:12.281 --> 00:38:15.021
package and you need to build the package
for this.
00:38:15.681 --> 00:38:20.580
[Lamby] Things that could be detected by
lintian from a static analysis point of view,
00:38:20.580 --> 00:38:26.241
yeah I'm sure, like looking for gzip
without -n for example, but that wouldn't
00:38:26.241 --> 00:38:28.941
be conclusive from lintian point of view.
00:38:29.361 --> 00:38:33.061
[Lunar] One thing that I really wanted to
diffoscope at some point - the code is made
00:38:33.061 --> 00:38:37.540
the way that it's possible - it's to have
hints so when it actually looks up
00:38:37.540 --> 00:38:44.500
differences between two packages then you
can have an idea, suggest you: hey you need
00:38:44.500 --> 00:38:49.800
to remove that timestamps, or you should
sort these keys. It's not done yet, but if
00:38:49.800 --> 00:38:52.580
anybody wants to do patches it's totally
doable.
00:38:58.481 --> 00:39:04.021
[Question] Thank you for the work, have
you thought about reproducible images?
00:39:04.620 --> 00:39:06.001
[Holger] It's on the todo list.
00:39:07.701 --> 00:39:15.061
Before images we need reproducible package
installation, and then we need reproducible
00:39:15.061 --> 00:39:19.561
images like squashfs has some things which
are not reproducible, but the package
00:39:19.561 --> 00:39:23.281
installation is not reproducible at the
moment because apt installs packages in
00:39:23.281 --> 00:39:27.861
arbitrary order and then the post-inst
create for example users which get
00:39:27.861 --> 00:39:32.682
user-ids in the order the packages are
installed, so for that to fix either apt
00:39:32.682 --> 00:39:39.421
needs a way to install in a deterministic
order, but it's on the todo list file.
00:39:39.842 --> 00:39:43.881
[Lunar] Pabs started a wiki page a couple
of months ago that is called reproducible
00:39:43.881 --> 00:39:50.260
install. This is very important if we want
tools like Tails to actually be reproducible
00:39:50.260 --> 00:39:54.740
so some people will work on that, we do
want to work on that.
00:39:54.740 --> 00:39:58.541
[Lamby] It's quite a deep problem for
example d-i will install different stuff
00:39:58.541 --> 00:40:02.261
depending on your hardware, so that's
immediately not reproducible.
00:40:02.261 --> 00:40:04.681
It'd be great.
00:40:06.541 --> 00:40:09.661
[Question] I've been working on a couple
of my packages to get them reproducible
00:40:09.672 --> 00:40:16.001
build, but I was often wondering if I
should fix it in my package or actually
00:40:16.001 --> 00:40:23.221
that it should be fixed in higher up and I
guess I've been adding some fixes to my
00:40:23.221 --> 00:40:28.441
packages which may in the future even not
be needed anymore and then it's just
00:40:28.441 --> 00:40:30.802
unnecessary code as well.
00:40:30.802 --> 00:40:35.560
So how do you see where things should be
fixed and how should we as package
00:40:35.560 --> 00:40:37.580
maintainers go about with this?
00:40:38.441 --> 00:40:44.021
[Holger] There's many things which there's
the easy fix to whatever: set the timezone in
00:40:44.021 --> 00:40:50.661
debhelper or better in dpkg to UTC, but
that will not fix the upstream bugs, so
00:40:50.661 --> 00:40:56.560
actually it's better not to fix, set the
timezone or other things deterministically
00:40:56.560 --> 00:41:00.721
in these tools but rather have them fixed
upstream, that's what we want.
00:41:00.721 --> 00:41:06.941
Some things we will need to fix them in
dpkg to get a meaningful result but
00:41:06.941 --> 00:41:11.662
basically we want rather these distributions
with just build from source which don't have
00:41:11.662 --> 00:41:15.061
debian/rules and they just build with
upstream Makefiles, we want the fixes
00:41:15.061 --> 00:41:17.321
to land there.
00:41:18.101 --> 00:41:21.500
[Lunar] We've been experimenting for two
and this is a lot of trials and errors,
00:41:21.500 --> 00:41:26.191
trying something, see how it fails, or
maybe we can do better than that and
00:41:26.191 --> 00:41:30.021
changing. And I know this can be frustrating
at some point because you do changes
00:41:30.021 --> 00:41:35.601
and they all become unneeded, but in the
end this is how we make stuff that matters.
00:41:35.601 --> 00:41:40.981
And we move forward, it's not because we're
trying to make the big picture at once,
00:41:40.981 --> 00:41:46.261
and I know in Debian we sometimes try to do
that, so we experiment and learn from it.
00:41:46.582 --> 00:41:51.741
[Question] An example that I'm now looking
into is actually the documentation is built
00:41:51.741 --> 00:41:57.761
for this package by looking in all the files
and generating but, for instances the
00:41:57.761 --> 00:42:05.501
index file is sorted, but I guess upstream
would say: well, if you set some ordering
00:42:05.501 --> 00:42:11.360
in your LC parameters you want this page
to be order as you want, instead of forcing
00:42:11.360 --> 00:42:16.181
it in the sort, so I'm really wondering:
should I now upstream this or should
00:42:16.181 --> 00:42:18.741
I just fix it in my rules because that's
the logical place?
00:42:20.571 --> 00:42:29.361
[Lunar] Both. No, there's no good answer,
I'm quite a strong proponent on the idea
00:42:29.361 --> 00:42:34.582
that if you use a computer you should be
able to talk and have the computer talk to
00:42:34.582 --> 00:42:41.041
you in the language that you choose, so if
people want to have gcc error messages
00:42:41.041 --> 00:42:45.321
in German, they should have it.
00:42:45.321 --> 00:42:50.920
But local sorting, this is the kind of
LC_ALL that can be very local and that
00:42:50.920 --> 00:42:54.181
you can do for just one tool, it's fine to
do that.
00:42:55.821 --> 00:42:59.281
[Question] Do you have ideas on making
sources reproducible? Like upstreams
00:42:59.281 --> 00:43:03.941
calling make dist, or this infamous
autogen.sh files?
00:43:06.742 --> 00:43:12.461
[Lunar] I don't think that anybody in the
team has looked into that yet, source
00:43:12.461 --> 00:43:22.661
files are easy to analyze way more than
binary packages so, it would still be great
00:43:22.661 --> 00:43:29.781
to have easier ways; you have source
tarballs be byte for byte identical,
00:43:29.781 --> 00:43:37.331
but it's not as an issue as it is for
binaries. If people want to look in that
00:43:37.331 --> 00:43:39.041
they should.
00:43:43.561 --> 00:43:48.701
[Question] Do you know a way to make git
archive build something reproducible?
00:43:50.441 --> 00:43:51.832
[Lunar] Well pristine-tar
00:43:51.881 --> 00:43:53.241
[Question] Yes, but without it.
00:43:54.461 --> 00:43:58.301
[Holger] There's one tool. You want to use
a new one? Then write it.
00:44:01.681 --> 00:44:04.902
Why not use that tool which does the job?
00:44:05.541 --> 00:44:07.721
pristine-tar does it.
00:44:10.641 --> 00:44:16.541
[Lunar] This is for source and so that's
another issue that what we are actually
00:44:16.541 --> 00:44:17.801
currently working on.
00:44:21.751 --> 00:44:25.911
[Holger] You're welcome to join the team and
extend our scope to sources.
00:44:28.582 --> 00:44:30.641
[Lunar] How many questions, two?
00:44:33.052 --> 00:44:35.540
Two more questions, two or three.
00:44:44.361 --> 00:44:52.421
[Question] So if there is a couple of other
environment variables that could be set
00:44:52.421 --> 00:44:59.241
in the environment to increase
reproducibility, where to put them?
00:44:59.241 --> 00:45:07.743
In the rules file? Or in the generic build
environment of all packages, or where
00:45:07.743 --> 00:45:09.680
should these things be placed?
00:45:13.442 --> 00:45:20.000
[Lamby] It'd be nice if upstream fixed it,
so if we just change it in debian/rules
00:45:20.000 --> 00:45:28.621
that's just only helping us, so often take
it upstream, would be the ideal solution.
00:45:28.621 --> 00:45:30.631
Are you referring to something else?
00:45:31.901 --> 00:45:40.131
[Question] For example many hashmaps have
randomized data in the hash function, so if
00:45:40.131 --> 00:45:46.961
you have some code that relies on hash
order, at least some implementations of
00:45:46.961 --> 00:45:56.991
hash functions are leaving them be seeded
rather than using something random for
00:45:56.991 --> 00:46:02.681
a build thing, but you want the randomness
in your hash functions for normal users
00:46:02.681 --> 00:46:10.941
because else your hashmaps get open
to attacks.
00:46:12.301 --> 00:46:13.821
[Lamby] Correct, yes.
00:46:16.261 --> 00:46:21.571
[Lunar] In these cases we send patches
adding sort everywhere for the keys and
00:46:21.571 --> 00:46:27.741
it's solved. For very few cases, for Perl for
example you can set and environment
00:46:27.741 --> 00:46:33.261
variable and some maintainers prefer to do
that. But usually we try to push these
00:46:33.261 --> 00:46:35.900
changes upstream, because they are simple
enough and they like it.
00:46:35.900 --> 00:46:38.941
Actually it makes testing easier to them.
00:46:42.261 --> 00:46:45.141
There was one in the back, there.
00:46:53.281 --> 00:46:56.111
[Lunar] That's the last question
00:46:56.341 --> 00:46:59.801
[Question] Follow up question to what we
had here before.
00:46:59.801 --> 00:47:10.041
You showed an open bug report against gcc
to support SOURCEDATEEPOCH to cover
00:47:10.041 --> 00:47:20.401
the mdate and mtime timestamps, so I have
patches to patch them out in my packages.
00:47:20.401 --> 00:47:24.362
Should I remove those patches and if so,
when?
00:47:26.361 --> 00:47:30.162
[Lunar] Have you seen any more emails
from the gcc maintainers?
00:47:33.902 --> 00:47:39.900
[Dhole] The mail is forgotten, I guess we
should ping it again, and see if they
00:47:39.900 --> 00:47:49.501
reply, because what I read from the gcc
website is that only the replies from
00:47:49.501 --> 00:47:54.861
maintainers are the ones that matter, and
I think no maintainer replied to the
00:47:54.861 --> 00:47:58.162
message, so we should ping again.
00:47:59.482 --> 00:48:02.560
[Question] That was just an example, my
question was more general.
00:48:02.560 --> 00:48:08.821
At which time should I remove my patches
to fix things which were fixed higher up
00:48:08.821 --> 00:48:12.420
in the toolchain? Or should I just leave
them in there?
00:48:13.082 --> 00:48:14.221
[Holger] Once they are in Sid.
00:48:15.131 --> 00:48:16.661
[Question] Ok thanks!
00:48:18.256 --> 00:48:20.042
[Lunar] Ok, I guess we're out of time.
00:48:20.261 --> 00:48:22.241
Thank you for listening.
00:48:22.660 --> 00:48:25.901
[Applause]
00:48:25.901 --> 00:48:28.861
[Lunar] Fix your packages!