0:00:00.000,0:00:13.009
33c3 preroll music
0:00:13.009,0:00:16.870
Herald: So, the last speaker for this[br]morning is Trammell. He is doing
0:00:16.870,0:00:21.510
awesome research on "Bootstrapping[br]more secure laptops or servers" and
0:00:21.510,0:00:27.480
he is doing that basically by moving the[br]root of trust into the write-protected room.
0:00:27.480,0:00:32.829
He is building an open-source custom[br]firmware, so big ups for that, and also
0:00:32.829,0:00:36.759
encouraging the research on this field[br]which I believe it's super interesting.
0:00:36.759,0:00:42.199
Thanks![br]applause
0:00:42.199,0:00:46.979
applause
0:00:46.979,0:00:49.790
Trammel Hudson: So I'm Trammell Hudson[br]with Two Sigma Investments, and for the
0:00:49.790,0:00:54.310
past several years I've been researching[br]firmware security vulnerabilities
0:00:54.310,0:00:59.420
and looked at how they affect systems.[br]Two years ago, I presented my work
0:00:59.420,0:01:05.030
on Thunderstrike here at CCC. And this was[br]the first firmware attack against MacBooks
0:01:05.030,0:01:09.249
that allowed an attacker to override[br]the motherboard bootrom.
0:01:09.249,0:01:14.310
The year after that, I collaborated with[br]Xeno Kovah and Corey Kallenberg
0:01:14.310,0:01:19.200
from LegbaCore - both of whom[br]are now at Apple, doing firmware work.
0:01:19.200,0:01:24.990
And we ported a bunch of Windows UEFI[br]vulnerabilities over to the Mac, and
0:01:24.990,0:01:29.640
showed that the software platform - the[br]UEFI software platform, you know - allowed
0:01:29.640,0:01:35.659
very portable attacks to be done. This[br]also allowed a remote attacker with code
0:01:35.659,0:01:41.689
execution on your machine to override the[br]motherboard bootrom. But, more than just
0:01:41.689,0:01:46.530
breaking things, what we like to do is[br]take the things apart and understand how
0:01:46.530,0:01:51.360
they work and document them, so that other[br]people can build systems on top of them.
0:01:51.360,0:01:55.370
And that's why I'm really excited to be[br]talking to you all about my project
0:01:55.370,0:02:00.630
"Heads", which is a open source[br]firmware and boot loader
0:02:00.630,0:02:06.429
for laptops and servers.[br]The name is kind of a play
0:02:06.429,0:02:11.239
on the popular 'Tails' distribution[br]which is a stateless Linux
0:02:11.239,0:02:15.070
for when you don't want to leave any traces[br]of what you're doing on your machine.
0:02:15.070,0:02:18.680
Heads is for the opposite case: it's where[br]you want to be able to trust the machine,
0:02:18.680,0:02:23.570
and you want to be able to trust that the[br]data you store on the machine is safe
0:02:23.570,0:02:28.640
and unmodified. And, let's back up[br]for a quick minute and just talk about
0:02:28.640,0:02:34.380
why firmware security is so important,[br]that this is the code that is
0:02:34.380,0:02:39.490
executed by the CPU when it comes out of[br]reset. This is the first instruction that
0:02:39.490,0:02:43.840
the CPU executes. And so it's in a really[br]privileged position to be able to
0:02:43.840,0:02:50.680
circumvent any sort of OS or other[br]policies. And there's no shortage of talks
0:02:50.680,0:02:55.820
that you can watch on interesting[br]attack vectors using firmware-based
0:02:55.820,0:03:01.300
malware. One that I really liked[br]was last year at DEFCON. The
0:03:01.300,0:03:06.050
Intel Advanced Threat Research[br]presented an attack that showed
0:03:06.050,0:03:11.830
how firmware could - malicious firmware -[br]could circumvent the hypervisors. They
0:03:11.830,0:03:17.920
then went further and showed how buggy[br]firmware allowed a unprivileged guest
0:03:17.920,0:03:25.200
inside a virtual machine to escalate into[br]privileges inside the hypervisor. And for
0:03:25.200,0:03:29.750
that reason, it's really important that[br]firmware vulnerabilities and firmware bugs
0:03:29.750,0:03:36.370
have a way to get patched. These aren't[br]just theoretical research vulnerabilities,
0:03:36.370,0:03:41.420
either. We know that there are malicious[br]organizations and hacking groups that are
0:03:41.420,0:03:48.269
selling firmware rootkits to whoever will[br]pay - including nation-state adversaries,
0:03:48.269,0:03:55.590
that are using them for their persistent[br]threats. And they are very persistent,
0:03:55.590,0:04:00.560
because they are in the motherboard[br]bootrom So you've reinstalled the OS -
0:04:00.560,0:04:06.959
they're still there, you swap out the hard[br]drive - they're still there. And some
0:04:06.959,0:04:14.890
vendors are even bundling these rootkits[br]into their official ROMs. That they are
0:04:14.890,0:04:22.019
using them to install the bloatware, or[br]whatever adware they want to put into the
0:04:22.019,0:04:28.189
OS. So even after you reinstall a clean[br]version of the OS, this particular vendors
0:04:28.189,0:04:34.840
system would install its own additions.[br]Some of those had vulnerabilities that
0:04:34.840,0:04:41.139
could then be exploited by attackers. This[br]particular case, the vendor received
0:04:41.139,0:04:46.450
enough bad press, that they released a[br]firmware update that patched this
0:04:46.450,0:04:50.919
vulnerability. And they had to do that.[br]This wasn't something that the users could
0:04:50.919,0:04:55.689
do on their own - they couldn't update the[br]the firmware in their machine the
0:04:55.689,0:05:02.389
way they do with their operating system or[br]an application. And in fact, most firmware
0:05:02.389,0:05:08.780
vulnerabilities never see patches get[br]deployed out to the end-user.
0:05:08.780,0:05:15.499
Part of the reason for that is that the[br]the firmware is usually four or five
0:05:15.499,0:05:21.699
companies, removed from the end-user. That[br]there's the open source reference
0:05:21.699,0:05:28.100
implementation from Intel, called eh[br]TianoCore or EDK II. When vulnerabilities
0:05:28.100,0:05:33.689
are patched in there, they have to get[br]pulled by the independent BIOS vendor, and
0:05:33.689,0:05:39.499
merged into the IBV tree, and then the[br]BIOS vendor sells that to the device
0:05:39.499,0:05:44.710
manufacturers. So they have to package up[br]a release the thing gets pulled by the
0:05:44.710,0:05:49.389
device manufacturer. It has to get QA'd it[br]against however many motherboards they
0:05:49.389,0:05:55.550
want to test it on. And then it has to get[br]again pulled by the original equipment
0:05:55.550,0:06:00.830
manufacturer to get rebranded and whatever[br]value they want to add. And then sometimes
0:06:00.830,0:06:05.479
it has to go through the operating system[br]vendor to even make it out to the end-
0:06:05.479,0:06:10.400
user. And as a result of this, most of the[br]time, products do not receive any updates
0:06:10.400,0:06:15.409
after they've been sold. There's one[br]exception: in this chart you can see that
0:06:15.409,0:06:21.249
Apple builds their own firmware, and in my[br]work with them, I've been really pleased
0:06:21.249,0:06:26.179
that they've rolled out patches for eight[br]years of hardware. Which is above and
0:06:26.179,0:06:34.939
beyond what any other firmware vendor is[br]doing right now. When EFI was introduced,
0:06:34.939,0:06:43.599
it brought a lot of complexity, and the[br]Linux community was very skeptical as to
0:06:43.599,0:06:48.300
what the value was going to be provided by[br]all this complexity. It's basically an
0:06:48.300,0:06:54.199
entire operating systems worth of code.[br]And it's not, that the 16-bit real-mode
0:06:54.199,0:07:01.289
BIOS was all that much better. In fact, it[br]had its own set of issues. But it was
0:07:01.289,0:07:08.590
small, it was simple, it did one thing, it[br]did it okay. And it took a long time for
0:07:08.590,0:07:15.259
UEFI to even become widely supported. But[br]even now most systems ship with both the
0:07:15.259,0:07:19.999
UEFI and a BIOS compatibility module. So[br]they've basically doubled their attack
0:07:19.999,0:07:25.369
surface for potential bugs and[br]vulnerabilities. So the state of the
0:07:25.369,0:07:33.110
firmware world today is that updates are[br]rare, patches, if they ever come out, take
0:07:33.110,0:07:35.599
a long time to make it through the[br]process.
0:07:35.599,0:07:41.449
Users can't fix things on their own, and[br]we can't see what's inside, since most of
0:07:41.449,0:07:45.339
them are built with closed source[br]components - and that's not a great state
0:07:45.339,0:07:52.649
for something that is as privileged as a[br]firmware. So it's my belief, that firmware
0:07:52.649,0:07:57.050
needs to be built with open source. It[br]must be flexible, so we can adapt it to
0:07:57.050,0:08:02.580
our needs for our systems. It needs to be[br]built with software that we understand and
0:08:02.580,0:08:08.770
we use for, in, other applications, so[br]that it can get widely tested and well
0:08:08.770,0:08:12.869
tested. It needs to be built in a[br]reproducible manner, so that we can be
0:08:12.869,0:08:18.330
secure against that build chain attacks.[br]And it needs to be cryptographically
0:08:18.330,0:08:23.189
measured, so that we can be sure that[br]this, what we flash on the system, is what
0:08:23.189,0:08:30.159
is actually running on the system. And[br]that's the philosophy behind Heads - it's
0:08:30.159,0:08:36.429
built on the free software coreboot[br]firmware, plus a Linux kernel in ROM, that
0:08:36.429,0:08:44.959
acts as a bootloader, and then a lot of[br]security research and tools, that help us
0:08:44.959,0:08:52.420
try to build slightly more secure systems.[br]Using Linux as a bootloader is not a
0:08:52.420,0:08:58.370
particularly new idea. Back in the 1990s[br]when we started building large scale Linux
0:08:58.370,0:09:07.840
clusters, we were very frustrated with the[br]inflexibility of DHCP and PXE booting
0:09:07.840,0:09:13.230
large machines, that even with those[br]frustrations, we built one, that was about
0:09:13.230,0:09:21.060
the 30th fastest in the world on the top[br]500. Meanwhile, my colleague Ron Minich at
0:09:21.060,0:09:25.960
Los Alamos was also building large[br]clusters, and had the observation that the
0:09:25.960,0:09:33.489
BIOS enumerates all the buses, initializes[br]a bunch of devices, finds the Linux kernel
0:09:33.489,0:09:38.680
and in the Linux kernel, enumerates all[br]the buses, initializes all the devices,
0:09:38.680,0:09:43.640
and he thought: "This is this is silly,[br]why are we doing this twice?".
0:09:43.640,0:09:49.730
So he had the idea to build a version of[br]Linux, that ran in the ROM. He called this
0:09:49.730,0:09:56.459
project "Linux BIOS" and it went on to[br]power the MRC cluster, which was the third
0:09:56.459,0:10:04.850
fastest machine in the world in 2003. In[br]2008 Linux BIOS underwent a major
0:10:04.850,0:10:10.100
refactoring and it was renamed to[br]"Coreboot". And Google chose to use
0:10:10.100,0:10:14.690
Coreboot as the firmware on their[br]Chromebooks - which are at this point the
0:10:14.690,0:10:24.190
only non-UEFI x86-based laptops that you[br]can buy. And they've done some really some
0:10:24.190,0:10:28.280
great work in trying to lock down the[br]configuration and the firmware on the
0:10:28.280,0:10:35.779
Chromebooks. Ron, who has only moved to[br]Google in 2011 and is continuing to work
0:10:35.779,0:10:47.350
on the Coreboot project there. So Coreboot[br]has three stages that it goes through as
0:10:47.350,0:10:53.820
it starts up the machine: the first one is[br]a very small amount of real-mode assembly,
0:10:53.820,0:11:00.699
because your modern 64-bit laptop still[br]boots up in real mode with 16-bit just
0:11:00.699,0:11:08.389
like its 1970's. So that's a very small[br]amount of code - about 1.5K. That in turn
0:11:08.389,0:11:14.170
sets up a C runtime environment with[br]what's called "Cache-as-RAM mode" and it
0:11:14.170,0:11:20.330
calls into the ROM stage which is about[br]70K. "Heads" has moved the TPM
0:11:20.330,0:11:26.800
initialization early in the ROM stage[br]before DRAM is set up to help measure the
0:11:26.800,0:11:33.069
boot block and provide a static route of[br]trust that's hopefully a little bit more
0:11:33.069,0:11:38.279
secure. And because that measurement is[br]done early on, our trusted computing base
0:11:38.279,0:11:44.660
is actually quite small. This is a[br]fraction of 1% of the size of a UEFI
0:11:44.660,0:11:50.339
firmware, which is usually about 16 MB[br]once it's uncompressed.
0:11:50.339,0:11:55.730
The ROM stage then measures and executes[br]the RAM stage, which does the more
0:11:55.730,0:12:01.120
traditional "walk the bus, find / figure[br]out what devices are on there", but it
0:12:01.120,0:12:04.269
doesn't initialize them: it just[br]enumerates them and generates the
0:12:04.269,0:12:09.320
descriptors for Linux. It also installs[br]the SMM handler for system management
0:12:09.320,0:12:16.710
mode. It then measures and jumps into the[br]payload - and that whole process takes
0:12:16.710,0:12:23.669
less than a second. So it is able to get[br]into the payload and actually get to the
0:12:23.669,0:12:30.730
user code very, very quickly. On something[br]like the X230 it's able to go from power
0:12:30.730,0:12:36.720
on to a interactive recovery shell in less[br]than 2 seconds. And that includes bring up
0:12:36.720,0:12:42.560
the TPM, doing cryptographic measurements[br]and assessing the state of the system.
0:12:42.560,0:12:46.790
Because we now have Linux at this point,[br]we have all the flexibility that comes
0:12:46.790,0:12:53.550
with that. We can implement boot scripts[br]with the full power of the shell or the C
0:12:53.550,0:12:57.600
language runtime:[br]we're not stuck with the limited functions
0:12:57.600,0:13:02.889
of UEFI. Linux supports lots of different[br]file systems. It supports lots of
0:13:02.889,0:13:07.139
different devices. It supports lots of[br]different encryption methods. And this
0:13:07.139,0:13:12.830
gives us the ability to use any of them[br]for your specific application. In
0:13:12.830,0:13:19.850
contrast, UEFI, which supports unencrypted[br]FAT file systems on the first drive that
0:13:19.850,0:13:23.589
have to be in the first 1 GiB or[br]something, it's really, really limited as
0:13:23.589,0:13:31.810
to how it can find its boot device.[br]There's a saying in the in the Open Source
0:13:31.810,0:13:35.889
community that with enough eyes all bugs[br]are shallow.
0:13:35.889,0:13:42.610
And Linux has a lot more eyes looking at[br]it. That the device drivers and the file
0:13:42.610,0:13:50.249
systems and the encryption have been[br]reviewed by both white hat and black hat
0:13:50.249,0:13:56.709
people around the world. The UEFI versions[br]of these do not have that same level of
0:13:56.709,0:14:03.160
scrutiny, so we're using both the UEFI[br]drivers and then having to run whatever on
0:14:03.160,0:14:08.079
top of it, increases the attack surface.[br]But by putting Linux in the ROM and
0:14:08.079,0:14:15.480
depending on its drivers we've reduced our[br]attack surface very dramatically. More
0:14:15.480,0:14:21.240
importantly though, Coreboot and Linux are[br]Open Source, so it is possible to build
0:14:21.240,0:14:25.480
custom versions for the device drivers[br]that you need, for the file systems that
0:14:25.480,0:14:30.839
you need. It's possible to fix bugs when[br]they come out, and sign and install your
0:14:30.839,0:14:37.089
own kernels. You don't have to wait for[br]the vendor to get around to doing it.
0:14:37.089,0:14:44.889
And the third major component of heads is[br]a tool called "kexec", which is a system
0:14:44.889,0:14:50.269
call that was added for the Linux BIOS[br]project back in 2003 by Eric Peterman,
0:14:50.269,0:14:56.230
that allows a running kernel to do a[br]graceful shutdown and start a new kernel
0:14:56.230,0:15:01.060
without having to go through the reboot[br]process. So this allowed, on their
0:15:01.060,0:15:06.510
application, it allowed them to do very[br]fast reboots of their cluster nodes. And
0:15:06.510,0:15:11.939
in the "Heads" case, it allows us to act[br]as a bootloader where we can find the real
0:15:11.939,0:15:16.149
kernel, that you want to run, and exec it.[br]Because "Heads" is quite small. It has to
0:15:16.149,0:15:22.689
fit in 4 MB of ROM. So it's not something[br]that you learn to run as a day-to-day sort
0:15:22.689,0:15:31.229
of OS. Hopefully this won't explode on me[br]again.
0:15:36.269,0:15:40.799
Because we have the Bourne Shell, most of[br]the policies and the startup scripts in
0:15:40.799,0:15:47.779
"Heads" are implemented as as shell[br]scripts. In this case we were able to pass
0:15:47.779,0:15:52.939
in a new set of command line parameters, a[br]new initial RAM disk, and in this case we
0:15:52.939,0:15:58.620
can even start a hypervisor. And all of[br]that can happen very very quickly, as well
0:15:58.620,0:16:07.269
as with with a good degree of security.[br]So, those are the building blocks that
0:16:07.269,0:16:12.980
"Heads" is built on: Coreboot, Linux and[br]tools like "kexec". But it now gives us a
0:16:12.980,0:16:16.710
really nice platform to begin[br]experimenting with additional security
0:16:16.710,0:16:23.240
features. And before we go too deep down[br]the rabbit hole of, you know, security and
0:16:23.240,0:16:27.509
threat models, I want to quote my friend[br]Steph, who said that, "Your threat model
0:16:27.509,0:16:31.019
is not my threat model."[br]But you know your threat model is okay as
0:16:31.019,0:16:35.730
well, that we all have different things we[br]want to protect, from different attackers,
0:16:35.730,0:16:40.389
who are willing to spend different amounts[br]of effort to go after them. And the nice
0:16:40.389,0:16:46.129
thing about having an open source is: We[br]can build systems tailored to your
0:16:46.129,0:16:50.930
individual threat model. So a lot of these[br]things may not actually apply to your
0:16:50.930,0:16:59.060
specific threats, but the fact that we can[br]build them is a great capability. Last
0:16:59.060,0:17:05.990
year Joanna Rutkowska reminded us that[br]firmware is not just in our CPU. Firmware
0:17:05.990,0:17:12.809
is in our Wi-Fi card. It is in our GPU. It[br]is in our SSD. It is in our keyboards. And
0:17:12.809,0:17:19.189
all of these devices might be trying to[br]subvert the boot process.
0:17:19.189,0:17:25.500
One way to handle that is to take Peter[br]Stuge's advice of disassembling the
0:17:25.500,0:17:30.401
machine and ripping out anything we can't[br]control. If this is your threat model, his
0:17:30.401,0:17:35.090
instructions are really worth following.[br]They're really thorough about what pieces
0:17:35.090,0:17:41.070
are potentially of concern. And you're[br]going, right now, you'll have to open up
0:17:41.070,0:17:46.560
your laptop to install "Heads." It's not[br]quite as easy as it is to install most
0:17:46.560,0:17:52.660
Linux distributions, because we have to[br]flash it into the motherboard boot ROM.
0:17:52.660,0:17:56.659
While we're in there, now, we take[br]advantage of some features that, to the
0:17:56.659,0:18:02.279
best of my knowledge, no UEFI system is[br]using. These flash chips have a hardware
0:18:02.279,0:18:09.610
write-protect mode, where you can specify[br]part of the chip is write-only. Excuse me.
0:18:09.610,0:18:15.480
Is read-only, write-once. And this gives[br]us our immutable boot block in which to
0:18:15.480,0:18:20.340
store the trusted computing base, the TCB,[br]so that we can measure the rest of the
0:18:20.340,0:18:26.270
system. We also then suggest disconnecting[br]the write-protect PIN from the
0:18:26.270,0:18:32.940
motherboard, which protects against[br]certain classes of attacks, like the Intel
0:18:32.940,0:18:38.899
close chassis adapter, that allows[br]external JTAG of the CPU.
0:18:38.899,0:18:42.440
Depending on your threat model you might[br]want to cover that chip in epoxy as well,
0:18:42.440,0:18:48.499
to frustrate Evil Maid attacks that want[br]to do a physical programming on it.
0:18:48.499,0:18:53.490
Disconnecting the write-protect PIN also[br]serves to protect from other devices on
0:18:53.490,0:18:58.890
the machine that have access to those[br]PINs. Devices like the Management Engine,
0:18:58.890,0:19:08.630
which is a really scary CPU inside the[br]CPU. Rudolf Marek two years ago at CCC
0:19:08.630,0:19:17.610
called it the "Matroyshka CPU".[br]And Igor Skochinsky detailed what are the
0:19:17.610,0:19:23.549
capabilities of the management engine. And[br]they're really worrisome, that it runs a
0:19:23.549,0:19:31.060
opaque obfuscated blob of code about 5 MiB[br]that the CPU can't see. The management
0:19:31.060,0:19:35.980
engine can read and write all of main[br]memory. It can read from the keyboard and
0:19:35.980,0:19:42.059
video. It can receive Java bytecodes over[br]the network and execute them on behalf of
0:19:42.059,0:19:46.510
someone outside the machine. And it's[br]listening on the network even when the
0:19:46.510,0:19:51.530
system is powered off.[br]So this is basically a rootkit inside the
0:19:51.530,0:19:58.480
chipset, as some folks have called it. So[br]that concerned me a lot, and I spent some
0:19:58.480,0:20:03.500
time looking at how its firmware images[br]are built, and realized that we can build
0:20:03.500,0:20:10.059
modified, reduced functionality firmware[br]for it that removes all of the rootkit
0:20:10.059,0:20:16.150
functions, and just leaves the "CPU bring[br]up" module. This takes that 5 MiB and
0:20:16.150,0:20:23.720
shrinks it down to about about 40 KiB of[br]space. So we don't know exactly what it's
0:20:23.720,0:20:28.460
doing in that 40 KiB, but we at least know[br]it doesn't have a device driver, or a Java
0:20:28.460,0:20:31.719
Virtual Machine, or a lot of the other[br]functions.
0:20:31.719,0:20:37.710
And we've successfully done this on both[br]Sandy Bridge and Ivy Bridge like the X230
0:20:37.710,0:20:43.380
ThinkPads as well as modern Skylake CPUs[br]like the Chell Chromebook. And that's
0:20:43.380,0:20:47.809
really encouraging, that if we can apply[br]this to more modern hardware, that allows
0:20:47.809,0:20:53.570
us to, you know, move away from our five-[br]year-old ThinkPads to something a little
0:20:53.570,0:21:01.580
shinier. The management engine isn't the[br]only device that might be trying subvert
0:21:01.580,0:21:05.799
the boot process. Again Johanna showed us[br]there are lots of things to be worried
0:21:05.799,0:21:12.159
about. Intel's UEFI architects, Yao and[br]Zimmer, recommend that firmware turn on
0:21:12.159,0:21:19.630
the IOMMU, now called VTD, to protect[br]against rogue devices. To the best of my
0:21:19.630,0:21:24.409
knowledge, since they've written this[br]guide, no UEFI firmware is taking
0:21:24.409,0:21:29.200
advantage of the IOMMU. So, it's a great[br]piece of hardware to have, but it doesn't
0:21:29.200,0:21:35.720
help if you don't turn it on. Linux,[br]meanwhile has no problem taking advantage
0:21:35.720,0:21:40.220
of it, so we use it, what we get[br]essentially - we get that DMA protection
0:21:40.220,0:21:48.120
for free by using Linux as our bootloader[br]in the ROM. Another way devices, rogue
0:21:48.120,0:21:52.360
devices, can try to interfere with the[br]boot process is by providing option ROMs,
0:21:52.360,0:21:59.169
which are executable code to be run by the[br]BIOS, that have a sort of a device driver.
0:21:59.169,0:22:03.630
And this code can do things like about log[br]keystrokes and then try to exfiltrate
0:22:03.630,0:22:11.909
passwords as we see here. That problem was[br]initially reported in 2007 by John Heasman
0:22:11.909,0:22:18.919
at BlackHat, again by Snare in 2012, and[br]again by my work on Thunderstrike. And as
0:22:18.919,0:22:24.290
of last week a official fix has finally[br]rolled out for it to close that particular
0:22:24.290,0:22:29.950
vulnerability.[br]Folks who are using coreboot have this as
0:22:29.950,0:22:35.600
a option that they can say, this, I am[br]concerned about this threat, let me let me
0:22:35.600,0:22:40.470
fix this, let me disable this function.[br]And they point out that it might cause
0:22:40.470,0:22:45.990
degraded functionality, but that's[br]something you can QA on your own system.
0:22:45.990,0:22:51.240
And in practice with Linux as your[br]bootloader in the ROM, you don't use the
0:22:51.240,0:22:56.610
option ROMs for anything. Everything is[br]done with with Linux's own device drivers,
0:22:56.610,0:23:02.750
so you're not dependent on the whatever[br]limited functionality the option ROM
0:23:02.750,0:23:07.830
provided.[br]So, now that we have we've taken our
0:23:07.830,0:23:12.220
building blocks and we hopefully have[br]protected the boot process, and hopefully
0:23:12.220,0:23:16.659
the code that's running is what we think[br]it is, we need to turn to how do we secure
0:23:16.659,0:23:22.780
the secrets on the machine. And I'm a huge[br]fan of the TPM, the Trusted Platform
0:23:22.780,0:23:28.590
Module, now I know in the free software[br]community it's been largely unwelcome. It
0:23:28.590,0:23:34.840
has not received a very welcome reception,[br]because of the way it's been used for DRM
0:23:34.840,0:23:41.530
and other other user-hostile things. Since[br]we control the TPM from the first
0:23:41.530,0:23:47.260
instruction in the in the boot block,[br]we're able to use it in ways that we want
0:23:47.260,0:23:55.140
to, so we don't have to enable DRM, but we[br]can use it to protect our secrets. And the
0:23:55.140,0:23:59.289
the way that it does that, is it keeps[br]track of what code is executed as the
0:23:59.289,0:24:06.049
system boots, and it hashes that code into[br]special registers called PCRs. And the
0:24:06.049,0:24:11.230
idea is that, you can extend the PCR by[br]hashing the next module of code, and then
0:24:11.230,0:24:17.460
hashing that with the previous hash, and[br]this creates a chain of trust that allows
0:24:17.460,0:24:24.690
to say: If these hashes match the expected[br]values, only the code, that we want to
0:24:24.690,0:24:34.040
have run, has run. And then the TPM will[br]only decrypt the the disk encryption key
0:24:34.040,0:24:40.260
if those PCRs match, which means that the[br]code that we want to have run, is what has
0:24:40.260,0:24:47.049
been executed. This means if someone[br]manages to overwrite the, the non-write
0:24:47.049,0:24:51.500
protected part of the ROM, that would[br]change those measurements and the TPM
0:24:51.500,0:24:59.019
won't reveal the key to them. It also[br]takes a user password and that password is
0:24:59.019,0:25:04.929
validated by the TPM hardware itself,[br]which gives us hardware rate-limiting on
0:25:04.929,0:25:10.850
how often an attacker can try. It also[br]gives us the ability to do hardware-based
0:25:10.850,0:25:15.820
retry limits, so that the TPM will flush[br]the key, if they if an attacker in
0:25:15.820,0:25:21.980
possession of your machine tries too long.[br]That does mean there's now another way to
0:25:21.980,0:25:27.299
lose your disk encryption key.[br]And there's the the old joke about there
0:25:27.299,0:25:31.149
are two types of people with encrypted[br]drives - those who have lost data due to
0:25:31.149,0:25:36.750
forgetting their key, and and those who[br]will. So what "Heads" does, when you
0:25:36.750,0:25:41.810
generate your key, is, it takes that key[br]and splits it into multiple pieces, that
0:25:41.810,0:25:46.789
you can then share either to friends or to[br]backup services, where each piece doesn't
0:25:46.789,0:25:53.030
let you decrypt it. But you can combine[br]them with Shamir secret sharing to
0:25:53.030,0:25:58.169
regenerate the cryptographic disk[br]encryption key.
0:25:58.169,0:26:05.019
We're also able to take advantage of best[br]practices like using the, including the
0:26:05.019,0:26:11.889
disk encryption key headers in the PCRs[br]that we use to seal the disks. This avoids
0:26:11.889,0:26:15.769
a certain class of Evil Maid attack, where[br]someone swaps out your drive; may not be
0:26:15.769,0:26:19.100
in your threat model, but it is easy to[br]deal with just a few lines of shell
0:26:19.100,0:26:28.309
script. So we hopefully we now trust that[br]the system is running the code we think it
0:26:28.309,0:26:33.690
is, but how does it prove to us that it is[br]actually our machine, that someone hasn't
0:26:33.690,0:26:38.889
snuck into our hotel room and swapped out[br]everything and left, left up, carefully
0:26:38.889,0:26:43.539
replaced our stickers to make us believe[br]we're typing our password into to our own
0:26:43.539,0:26:51.929
computer. Some anti-Evil Maid toolkits[br]will encrypt a secret, secret phrase, and
0:26:51.929,0:26:56.419
then display it to you if and only if the[br]PCR is match, but that's subject to a
0:26:56.419,0:27:03.970
replay attack. What Matthew Garrett[br]demonstrated last year at 32C3, was, using
0:27:03.970,0:27:10.840
the time, a time-based one-time password[br]used by Google Authenticator to protect
0:27:10.840,0:27:16.760
the firmware and have it attest to the[br]user, that it is unmodified. So when the
0:27:16.760,0:27:21.710
system boots, and it goes through[br]measuring all of the various components
0:27:21.710,0:27:29.960
the, the TPM will only release the secret[br]if those PCRs match. The firmware then
0:27:29.960,0:27:33.550
hashes that along with the current time[br]and generates a six digit value that it
0:27:33.550,0:27:38.519
prints on the screen. You compare that to[br]what's on your phone and that tells you
0:27:38.519,0:27:46.791
whether or not you can trust the machine.[br]It's a, it's a great idea, and it's
0:27:46.791,0:27:51.780
implemented again as a very small shell[br]script to read, read the value from the
0:27:51.780,0:27:58.630
TPM, unseal it and then compute the[br]the hash of it.
0:27:58.630,0:28:04.390
This also allows us to start making a[br]transition from the TPM-rooted, a TPM
0:28:04.390,0:28:10.031
static route of trust to a PGP-based[br]trust, where most importantly, this is a
0:28:10.031,0:28:14.540
TPM key - excuse me, this is a PGP key,[br]that you, the owner of the computer
0:28:14.540,0:28:19.820
control, not some random vendor or some[br]random hardware device manufacturer that's
0:28:19.820,0:28:25.659
going to lose the key and allow malware[br]like Stuxnet to use it to circumvent
0:28:25.659,0:28:30.940
security. The boot script[br]in "Heads", again, it's a
0:28:30.940,0:28:37.200
small shell script, is able to use a GPG[br]to verify the next stages of the, the
0:28:37.200,0:28:45.010
hypervisor, the initial RAM disk and the[br]kernel. And it also then uses the the
0:28:45.010,0:28:50.820
TPM's counters to help prevent against[br]rollback, where someone takes your drive
0:28:50.820,0:28:54.660
and rolls it back to a previous version,[br]perhaps with a vulnerability that they can
0:28:54.660,0:28:59.380
exploit. So this, this[br]allows the, allows us to be
0:28:59.380,0:29:03.679
sure, that not only are we running the[br]firmware, the OS that we think we should
0:29:03.679,0:29:10.799
be running, it ensures us that someone[br]hasn't been able to substitute one that, a
0:29:10.799,0:29:16.700
vulnerable version. Having the[br]PGP key also allows us to take
0:29:16.700,0:29:24.720
advantage of an idea from Android and[br]Chrome OS which is the the dm-verity read-
0:29:24.720,0:29:31.010
only root filesystem - this hashes all of[br]the blocks and that hashes all of the the
0:29:31.010,0:29:38.320
hashes and so on up until it gets to a[br]root hash in the tree that is then signed.
0:29:38.320,0:29:45.029
This allows the kernel, on every read[br]access, in logarithmic time to verify the
0:29:45.029,0:29:51.580
essentially a signature on that data. This[br]does require a read-only root filesystem,
0:29:51.580,0:29:59.719
but it gives us even more confidence that[br]the system has been untampered with. Once
0:29:59.719,0:30:05.100
you're running your OS, it's good to have[br]you know some security conscious thoughts
0:30:05.100,0:30:10.990
as well, you know. Heads is mostly focused[br]on, "how do we securely transition to an
0:30:10.990,0:30:17.529
OS" and that OS you run is up to you. I[br]like Qubes - it's reasonably secure, it's
0:30:17.529,0:30:23.630
highly recommended by people who know[br]about endpoint security. And the Qubes
0:30:23.630,0:30:30.480
team recognizes that firmware security is[br]a vital piece of system security. For
0:30:30.480,0:30:35.149
their next release, Qubes are for -[br]they're going to require the machines have
0:30:35.149,0:30:39.101
open source firmware, such as coreboot.[br]And I hope that Heads is going to be a
0:30:39.101,0:30:46.330
piece of that. I've also been working with[br]with the Qubes software and have modified
0:30:46.330,0:30:53.000
it to work with things like the dm-verity[br]read-only root filesystem. This now allows
0:30:53.000,0:30:58.809
the user to lock down the configuration,[br]so that someone can't tamper with their
0:30:58.809,0:31:05.850
their setup. It also gives you a recovery[br]mode that allows you to fix things up and
0:31:05.850,0:31:12.139
re-sign the OS. Reproducible builds are[br]really important so that everyone can
0:31:12.139,0:31:19.490
verify what that the builds match what[br]they should. In the case of of Heads we
0:31:19.490,0:31:22.429
have a lot of upstream dependencies that[br]aren't reproducible so we're working with
0:31:22.429,0:31:28.190
them to try to patch them. We've patched[br]Xen - they've accepted that commit. We've
0:31:28.190,0:31:33.669
also built some tools to let you build[br]initial RAM disks in a reproducible way.
0:31:33.669,0:31:37.389
This works with Qubes, with Heads, and[br]we're hoping to other Linux distributions
0:31:37.389,0:31:41.960
pick it up as well.[br]All of our tree is cryptographically
0:31:41.960,0:31:48.230
signed, so hopefully GitHub is not trying[br]to slip in any patches. And it is open
0:31:48.230,0:31:52.890
source so we encourage anyone to read[br]through it. No NDA is required, unlike most
0:31:52.890,0:31:59.570
of the UEFI sources. So that's sort of the[br]state of where things are - it's pretty
0:31:59.570,0:32:04.679
much in very beta, but it is usable. But[br]there are a lot of areas where we could
0:32:04.679,0:32:08.679
continue to do research - things like the[br]embedded controllers on Chromebooks are
0:32:08.679,0:32:14.359
open source. We can use those to help with[br]our root of trust as well. Porting
0:32:14.359,0:32:17.070
coreboot to more modern platforms would[br]let us take advantage of things like
0:32:17.070,0:32:23.440
tamper switches and Intel Boot Guard. I'm[br]also working on porting coreboot over to
0:32:23.440,0:32:30.029
server platforms so that we can use it for[br]more secure cloud hosting. Servers have a
0:32:30.029,0:32:36.480
very different threat model from from[br]laptops, and a lot of things have firmware
0:32:36.480,0:32:41.090
that we have to be concerned about. One[br]collaboration I'm looking at there is with
0:32:41.090,0:32:45.989
the open BMC project to be able to take[br]advantage of the open source in the
0:32:45.989,0:32:54.159
management controller for the servers. And[br]I'm also collaborating with the Mass Open
0:32:54.159,0:33:01.129
Cloud project that's trying to build[br]secure bare metal clouds. I'm cautiously
0:33:01.129,0:33:06.590
optimistic about enclaves and how they[br]will help us with the security, especially
0:33:06.590,0:33:10.269
in a environment where we control the[br]firmware and we can ensure that the
0:33:10.269,0:33:17.270
enclaves are set up in a safe way. There[br]are a lot of issues on GitHub, again,
0:33:17.270,0:33:25.150
please welcome contributions. I hope[br]everyone gets inspired to to work on the
0:33:25.150,0:33:31.080
installing this on their laptop. And if if[br]you are interested I'll be hanging out at
0:33:31.080,0:33:38.220
the coreboot assembly later today and[br]occasionally this week. The coreboot team
0:33:38.220,0:33:42.610
has a bunch of people here. They have[br]flash programmers and can help you install
0:33:42.610,0:33:50.990
coreboot on your laptop. The source code[br]for Heads is available on GitHub, and a
0:33:50.990,0:33:56.419
annotated version of this talk is up on my[br]website, and welcome comments and feedback
0:33:56.419,0:34:02.340
on it. So thank you all for coming to hear[br]about this project I hope that everyone is
0:34:02.340,0:34:06.899
and you know as excited about open source[br]firmware as I am and I'd love to take any
0:34:06.899,0:34:09.918
questions that you all have.
0:34:09.918,0:34:21.739
Applause
0:34:28.549,0:34:31.730
Question: Thanks for your great talk -[br]this is very interesting. Do you have any
0:34:31.730,0:34:39.009
advice for the 95% of us who are stuck on[br]non coreboot compatible laptops.
0:34:39.009,0:34:42.890
Trammell: Buy a Chromebook?[br]Laughter
0:34:42.890,0:34:52.951
Trammell: It's hard to trust the closed-[br]source firmware. It certainly; there are
0:34:52.951,0:34:55.920
people we have to trust - there are[br]institutions we have to trust. We have to
0:34:55.920,0:35:01.620
trust Intel to some extent, and Intel is[br]responsible for, you know, both our CPUs
0:35:01.620,0:35:09.370
and a lot of the firmware. Depending on[br]your threat model, firmware attacks may
0:35:09.370,0:35:15.280
not be a huge concern for your particular[br]machine. Or they might be, you know, of
0:35:15.280,0:35:20.470
grave concern, in which case even just[br]doing some of the things that Peter Stuge
0:35:20.470,0:35:30.020
suggested of clipping the write-protect[br]pin on the on the chip, removing things
0:35:30.020,0:35:35.200
that might be hostile and attacking your[br]system. His guides are really good one to
0:35:35.200,0:35:43.100
follow for the hardware security.[br]Question: I was wondering if you also
0:35:43.100,0:35:49.610
support ARM - I just saw Intel laptops so[br]I was just wondering.
0:35:49.610,0:35:55.231
Trammell: So ARM it has a lot of[br]advantages as a CPU. It only has 20 years
0:35:55.231,0:36:00.750
of legacy baggage rather than 40. And the[br]boot process on it is much much simpler
0:36:00.750,0:36:06.820
since it doesn't have to go through real[br]mode to long mode to paging and all the
0:36:06.820,0:36:15.140
other steps. The downside to a lot of ARMs[br]is that they their boot code is a few is
0:36:15.140,0:36:22.190
on die and outside of the control of the[br]user. Luckily, most of that boot code is
0:36:22.190,0:36:29.280
fairly simple, and can establish - and[br]some of them will establish a hardware
0:36:29.280,0:36:37.490
root of trust. But in general, yeah that[br]the ARM to U-Boot to whatever seems to
0:36:37.490,0:36:44.090
work out pretty well. I know there's been[br]some interest in, "can U-Boot be replaced
0:36:44.090,0:36:49.920
with a Linux BIOS or coreboot like thing",[br]and I suspect the folks at the booth would
0:36:49.920,0:36:54.700
be able to talk more about that.[br]Question: And then just a followup - so if
0:36:54.700,0:37:00.510
coreboot or Libreboot supports the[br]platform - Heads will work too right?
0:37:00.510,0:37:05.590
Trammell: Essentially yes, yeah.[br]Heads is a payload for coreboot.
0:37:05.590,0:37:13.030
Question: OK.[br]Herald: It's there on the left.
0:37:13.030,0:37:17.960
Signal Angel: Thank you - there's a[br]question from the Internet about coreboot.
0:37:17.960,0:37:23.430
Coreboot has blobs included and, for[br]example, binary blobs from Intel with all
0:37:23.430,0:37:28.750
the firmware support package and all that[br]stuff. How can we call coreboot secure,
0:37:28.750,0:37:33.060
then, in the light of this - let alone[br]open source?
0:37:33.060,0:37:37.890
Trammell: So the intel FSP is a[br]significant concern. This is the firmware
0:37:37.890,0:37:42.010
support package that is required to[br]initialize the memory controllers on on
0:37:42.010,0:37:52.200
modern Intel cpus. On older CPUs, such as[br]the the Sandy Bridge and Ivy Bridge, the
0:37:52.200,0:37:57.040
coreboot and Libreboot are able to[br]initialize the memory natively without
0:37:57.040,0:38:05.870
having to go into the FSB. However, if you[br]look at what coreboot is doing in the MRC
0:38:05.870,0:38:10.930
on those platforms, it tends to just be[br]poking a bunch of registers with values
0:38:10.930,0:38:20.170
that seem to work. And it's modern memory[br]controllers are so complex that, and Intel
0:38:20.170,0:38:26.700
is unwilling to document them, that[br]without extensive NDA's that it's very
0:38:26.700,0:38:32.330
hard to build any sort of memory[br]initialization. So what we can't say it's
0:38:32.330,0:38:38.610
a hundred percent free software, we can at[br]least we can ensure that the FSP is
0:38:38.610,0:38:47.200
measured and it's unchanging. We can also[br]look at the state of things that it sets
0:38:47.200,0:38:52.210
up and include those in our measurements.[br]So even if it doesn't get us one hundred
0:38:52.210,0:38:57.850
percent open source - and as far as I know[br]the only system that does that right now
0:38:57.850,0:39:05.280
is bunnies' Novena laptop. At least we can[br]measure it and we can know that it hasn't
0:39:05.280,0:39:10.810
been tampered with from[br]what we initially installed.
0:39:10.810,0:39:17.260
Question: And before so this is a great[br]project and I'd like to ask why you did
0:39:17.260,0:39:23.650
certain architectural decisions. The[br]specific combination of Linux and shell.
0:39:23.650,0:39:29.840
So why didn't you choose a BSD kernel which[br]are usually perceived to be more secure
0:39:29.840,0:39:34.720
and of a higher quality, and why did you[br]choose a shell over let's say, Python
0:39:34.720,0:39:41.230
or Haskell, which are also often[br]perceived of higher quality?
0:39:41.230,0:39:45.400
Trammell: There is a lot of desire to[br]support Python in Heads. The downside is
0:39:45.400,0:39:51.910
that there's a very limited space: the[br]X230 bootrom, for instance, has 4
0:39:51.910,0:40:02.070
megabytes of available space. The Python[br]interpreter is a couple megs already. In
0:40:02.070,0:40:09.080
terms of why Linux over BSD - the kexec[br]system call is a core component of this:
0:40:09.080,0:40:14.940
to be able to do a graceful shut down and[br]transfer from the Linux kernel to another
0:40:14.940,0:40:22.370
kernel or - to any multi-boot compliant[br]kernels, which includes BSD. It is a
0:40:22.370,0:40:31.010
necessary feature if it, if BSD had such[br]functionality, that it would be a fine
0:40:31.010,0:40:37.225
just a choice for the the internal boot[br]ROM/boot loader.
0:40:45.700,0:40:53.601
Question: Thanks for great work. How to[br]perform updates of coreboot and its
0:40:53.601,0:41:00.391
payload when it's binaries used in[br]measurement for releasing encryption key
0:41:00.391,0:41:07.150
then when you update coreboot this[br]measurement will change and you will know
0:41:07.150,0:41:12.250
no longer will be able to boot the system[br]- how to solve that problem?
0:41:12.250,0:41:19.180
Trammell: So migrating encryption keys[br]with TPM requires a explicit step of
0:41:19.180,0:41:24.060
retrieving the key from the TPM with the[br]current configuration and then resealing
0:41:24.060,0:41:30.210
it with the new configuration. One[br]advantage of a reproducible build is the
0:41:30.210,0:41:37.240
hashes of the of all the firmware stages[br]can be published - it can be pre-computed,
0:41:37.240,0:41:41.460
and then the PCR values can be pre-[br]computed so you can read you can seal the
0:41:41.460,0:41:52.170
keys for the new values. In terms of the[br]update process for the Heads payload - one
0:41:52.170,0:41:56.550
of the things that we're working on is[br]being able to have an even more minimal
0:41:56.550,0:42:03.500
heads that has just a USB device driver[br]that you can boot into, copy your new
0:42:03.500,0:42:08.830
payload, and then install that elsewhere[br]on the chip. And part of that process
0:42:08.830,0:42:15.200
would involve resealing any of the keys[br]that you need to transfer.
0:42:15.200,0:42:17.270
Herald: Another question from the[br]Internet.
0:42:17.270,0:42:21.570
Signal Angel: Thank you. On your webpage[br]you implemented Heads on ThinkPads only.
0:42:21.570,0:42:28.250
How much work is still needed to translate[br]this to, let's say, non-ThinkPads?
0:42:28.250,0:42:32.890
Trammell: ThinkPads are really popular[br]with the security community. It's quite
0:42:32.890,0:42:36.280
interesting to look out at you know the[br]hall here and see how many ThinkPads there
0:42:36.280,0:42:41.520
are. And as a result the coreboot[br]community has been very supportive of
0:42:41.520,0:42:48.820
ThinkPads. There's, other than the[br]ThinkPads and the Chromebooks, there
0:42:48.820,0:42:53.270
aren't a lot of devices that support[br]coreboot out of the box. And that's
0:42:53.270,0:42:58.050
something that I hope would change - I[br]hope that some OEMs would realize there's
0:42:58.050,0:43:04.560
value in provide an open source firmware[br]and move to move to using it. Both as a
0:43:04.560,0:43:10.590
cost-saving measure as well as a freedom[br]measure. In terms of the difficulty
0:43:10.590,0:43:15.780
important coreboot to a platform - I[br]haven't successfully done that yet, but I
0:43:15.780,0:43:23.350
suspect the people at the assembly would[br]be happy to discuss that further.
0:43:23.350,0:43:29.400
Question: Would you plan to rework[br]embedded controller firmware on ThinkPads
0:43:29.400,0:43:38.420
because it's remain close at birth which[br]still has an access to PC bus and probably
0:43:38.420,0:43:43.440
couldn't be trusted.[br]Trammell: So your question is how do we
0:43:43.440,0:43:48.520
how do we replace the EC?[br]Question: Yes do plan to replace EC with
0:43:48.520,0:43:52.150
open source firmware as in the[br]Chromebooks?
0:43:52.150,0:43:57.530
Trammell: So the Chromebook has open[br]source EC. The part of building coreboot
0:43:57.530,0:44:03.190
for Chromebook involves installing the ARM[br]cross compiler to build the EC firmware.
0:44:03.190,0:44:08.790
And the Chromebooks actually have a really[br]elegant protocol for the EC to attest to
0:44:08.790,0:44:14.430
the CPU that is running the firmware that[br]you think it is running. On other
0:44:14.430,0:44:23.700
platforms, this would require a lot more[br]research. The doc; many of the EC chipsets
0:44:23.700,0:44:27.840
have data sheets available, so it's[br]possible to read through and see how they
0:44:27.840,0:44:33.070
work. And most of them have updatable[br]firmware. In case of the ThinkPads,
0:44:33.070,0:44:37.410
there's a module in the ThinkPad BIOS that[br]will do that update. There's, we would
0:44:37.410,0:44:41.480
need to figure out what that protocol[br]looks like.
0:44:41.480,0:44:47.810
Question: Sorry yes I mean if you have[br]working prototype on ThinkPads probably
0:44:47.810,0:44:56.870
want to add remaining business open[br]sources EC on ThinkPads as well in the
0:44:56.870,0:45:00.550
first place.[br]Trammell: I'm sorry, I don't think I
0:45:00.550,0:45:09.540
understood your follow-up.[br]Question: Okay. So if you if you have a
0:45:09.540,0:45:18.240
working prototype on ThinkPads and only on[br]ThinkPads, will you finish somewhat soon
0:45:18.240,0:45:28.711
current existing prototype of open source[br]EC exists in college shade by Lincoln's or
0:45:28.711,0:45:36.830
you are playing to extend your work on[br]other platforms and finish this bits
0:45:36.830,0:45:39.700
later.[br]Trammell: Yeah right now I have not
0:45:39.700,0:45:46.910
personally made any progress on the[br]ThinkPad EC. I was looking into it because
0:45:46.910,0:45:51.700
I have a modified keyboard on my ThinkPad[br]that that needs to updated EC firmware,
0:45:51.700,0:46:00.740
but I haven't actually gotten into that.[br]This is a area of open research
0:46:00.740,0:46:03.900
Signal Angel: Thank you, two quick[br]questions from the IRC - are you planning
0:46:03.900,0:46:07.690
to use systemd in the boot process[br]is the first one
0:46:07.690,0:46:11.030
Laughter[br]Signal Angel: And the second one let's say
0:46:11.030,0:46:16.110
you flash your firmware at the Congress[br]right here with the help of a hardware
0:46:16.110,0:46:22.900
programmer. Can you update when there's a[br]new version or do you have to currently
0:46:22.900,0:46:29.620
need the hardware access to update?[br]Trammell: Right now you can update
0:46:29.620,0:46:40.030
afterwards at great risk, because you can[br]leave the flash writeable, which would
0:46:40.030,0:46:45.710
allow you to flash after the fact. We are[br]still working on a good procedure for
0:46:45.710,0:46:53.540
doing software-only firmware updates once[br]the immutable boot block is installed. And
0:46:53.540,0:46:58.950
to the other question - did I mention that[br]we are really short on space and we don't
0:46:58.950,0:47:07.712
want to put any large applications like[br]systemd on there.
0:47:07.712,0:47:11.292
Questioner: It was like good one, thanks.
0:47:11.292,0:47:19.242
applause
0:47:19.242,0:47:28.159
postroll music
0:47:28.159,0:47:42.811
subtitles created by c3subtitles.de[br]in the year 2018