WEBVTT 00:00:00.000 --> 00:00:13.009 33c3 preroll music 00:00:13.009 --> 00:00:16.870 Herald: So, the last speaker for this morning is Trammell. He is doing 00:00:16.870 --> 00:00:21.510 awesome research on "Bootstrapping more secure laptops or servers" and 00:00:21.510 --> 00:00:27.480 he is doing that basically by moving the root of trust into the write-protected room. 00:00:27.480 --> 00:00:32.829 He is building an open-source custom firmware, so big ups for that, and also 00:00:32.829 --> 00:00:36.759 encouraging the research on this field which I believe it's super interesting. 00:00:36.759 --> 00:00:42.199 Thanks! applause 00:00:42.199 --> 00:00:46.979 applause 00:00:46.979 --> 00:00:49.790 Trammel Hudson: So I'm Trammell Hudson with Two Sigma Investments, and for the 00:00:49.790 --> 00:00:54.310 past several years I've been researching firmware security vulnerabilities 00:00:54.310 --> 00:00:59.420 and looked at how they affect systems. Two years ago, I presented my work 00:00:59.420 --> 00:01:05.030 on Thunderstrike here at CCC. And this was the first firmware attack against MacBooks 00:01:05.030 --> 00:01:09.249 that allowed an attacker to override the motherboard bootrom. 00:01:09.249 --> 00:01:14.310 The year after that, I collaborated with Xeno Kovah and Corey Kallenberg 00:01:14.310 --> 00:01:19.200 from LegbaCore - both of whom are now at Apple, doing firmware work. 00:01:19.200 --> 00:01:24.990 And we ported a bunch of Windows UEFI vulnerabilities over to the Mac, and 00:01:24.990 --> 00:01:29.640 showed that the software platform - the UEFI software platform, you know - allowed 00:01:29.640 --> 00:01:35.659 very portable attacks to be done. This also allowed a remote attacker with code 00:01:35.659 --> 00:01:41.689 execution on your machine to override the motherboard bootrom. But, more than just 00:01:41.689 --> 00:01:46.530 breaking things, what we like to do is take the things apart and understand how 00:01:46.530 --> 00:01:51.360 they work and document them, so that other people can build systems on top of them. 00:01:51.360 --> 00:01:55.370 And that's why I'm really excited to be talking to you all about my project 00:01:55.370 --> 00:02:00.630 "Heads", which is a open source firmware and boot loader 00:02:00.630 --> 00:02:06.429 for laptops and servers. The name is kind of a play 00:02:06.429 --> 00:02:11.239 on the popular 'Tails' distribution which is a stateless Linux 00:02:11.239 --> 00:02:15.070 for when you don't want to leave any traces of what you're doing on your machine. 00:02:15.070 --> 00:02:18.680 Heads is for the opposite case: it's where you want to be able to trust the machine, 00:02:18.680 --> 00:02:23.570 and you want to be able to trust that the data you store on the machine is safe 00:02:23.570 --> 00:02:28.640 and unmodified. And, let's back up for a quick minute and just talk about 00:02:28.640 --> 00:02:34.380 why firmware security is so important, that this is the code that is 00:02:34.380 --> 00:02:39.490 executed by the CPU when it comes out of reset. This is the first instruction that 00:02:39.490 --> 00:02:43.840 the CPU executes. And so it's in a really privileged position to be able to 00:02:43.840 --> 00:02:50.680 circumvent any sort of OS or other policies. And there's no shortage of talks 00:02:50.680 --> 00:02:55.820 that you can watch on interesting attack vectors using firmware-based 00:02:55.820 --> 00:03:01.300 malware. One that I really liked was last year at DEFCON. The 00:03:01.300 --> 00:03:06.050 Intel Advanced Threat Research presented an attack that showed 00:03:06.050 --> 00:03:11.830 how firmware could - malicious firmware - could circumvent the hypervisors. They 00:03:11.830 --> 00:03:17.920 then went further and showed how buggy firmware allowed a unprivileged guest 00:03:17.920 --> 00:03:25.200 inside a virtual machine to escalate into privileges inside the hypervisor. And for 00:03:25.200 --> 00:03:29.750 that reason, it's really important that firmware vulnerabilities and firmware bugs 00:03:29.750 --> 00:03:36.370 have a way to get patched. These aren't just theoretical research vulnerabilities, 00:03:36.370 --> 00:03:41.420 either. We know that there are malicious organizations and hacking groups that are 00:03:41.420 --> 00:03:48.269 selling firmware rootkits to whoever will pay - including nation-state adversaries, 00:03:48.269 --> 00:03:55.590 that are using them for their persistent threats. And they are very persistent, 00:03:55.590 --> 00:04:00.560 because they are in the motherboard bootrom So you've reinstalled the OS - 00:04:00.560 --> 00:04:06.959 they're still there, you swap out the hard drive - they're still there. And some 00:04:06.959 --> 00:04:14.890 vendors are even bundling these rootkits into their official ROMs. That they are 00:04:14.890 --> 00:04:22.019 using them to install the bloatware, or whatever adware they want to put into the 00:04:22.019 --> 00:04:28.189 OS. So even after you reinstall a clean version of the OS, this particular vendors 00:04:28.189 --> 00:04:34.840 system would install its own additions. Some of those had vulnerabilities that 00:04:34.840 --> 00:04:41.139 could then be exploited by attackers. This particular case, the vendor received 00:04:41.139 --> 00:04:46.450 enough bad press, that they released a firmware update that patched this 00:04:46.450 --> 00:04:50.919 vulnerability. And they had to do that. This wasn't something that the users could 00:04:50.919 --> 00:04:55.689 do on their own - they couldn't update the the firmware in their machine the 00:04:55.689 --> 00:05:02.389 way they do with their operating system or an application. And in fact, most firmware 00:05:02.389 --> 00:05:08.780 vulnerabilities never see patches get deployed out to the end-user. 00:05:08.780 --> 00:05:15.499 Part of the reason for that is that the the firmware is usually four or five 00:05:15.499 --> 00:05:21.699 companies, removed from the end-user. That there's the open source reference 00:05:21.699 --> 00:05:28.100 implementation from Intel, called eh TianoCore or EDK II. When vulnerabilities 00:05:28.100 --> 00:05:33.689 are patched in there, they have to get pulled by the independent BIOS vendor, and 00:05:33.689 --> 00:05:39.499 merged into the IBV tree, and then the BIOS vendor sells that to the device 00:05:39.499 --> 00:05:44.710 manufacturers. So they have to package up a release the thing gets pulled by the 00:05:44.710 --> 00:05:49.389 device manufacturer. It has to get QA'd it against however many motherboards they 00:05:49.389 --> 00:05:55.550 want to test it on. And then it has to get again pulled by the original equipment 00:05:55.550 --> 00:06:00.830 manufacturer to get rebranded and whatever value they want to add. And then sometimes 00:06:00.830 --> 00:06:05.479 it has to go through the operating system vendor to even make it out to the end- 00:06:05.479 --> 00:06:10.400 user. And as a result of this, most of the time, products do not receive any updates 00:06:10.400 --> 00:06:15.409 after they've been sold. There's one exception: in this chart you can see that 00:06:15.409 --> 00:06:21.249 Apple builds their own firmware, and in my work with them, I've been really pleased 00:06:21.249 --> 00:06:26.179 that they've rolled out patches for eight years of hardware. Which is above and 00:06:26.179 --> 00:06:34.939 beyond what any other firmware vendor is doing right now. When EFI was introduced, 00:06:34.939 --> 00:06:43.599 it brought a lot of complexity, and the Linux community was very skeptical as to 00:06:43.599 --> 00:06:48.300 what the value was going to be provided by all this complexity. It's basically an 00:06:48.300 --> 00:06:54.199 entire operating systems worth of code. And it's not, that the 16-bit real-mode 00:06:54.199 --> 00:07:01.289 BIOS was all that much better. In fact, it had its own set of issues. But it was 00:07:01.289 --> 00:07:08.590 small, it was simple, it did one thing, it did it okay. And it took a long time for 00:07:08.590 --> 00:07:15.259 UEFI to even become widely supported. But even now most systems ship with both the 00:07:15.259 --> 00:07:19.999 UEFI and a BIOS compatibility module. So they've basically doubled their attack 00:07:19.999 --> 00:07:25.369 surface for potential bugs and vulnerabilities. So the state of the 00:07:25.369 --> 00:07:33.110 firmware world today is that updates are rare, patches, if they ever come out, take 00:07:33.110 --> 00:07:35.599 a long time to make it through the process. 00:07:35.599 --> 00:07:41.449 Users can't fix things on their own, and we can't see what's inside, since most of 00:07:41.449 --> 00:07:45.339 them are built with closed source components - and that's not a great state 00:07:45.339 --> 00:07:52.649 for something that is as privileged as a firmware. So it's my belief, that firmware 00:07:52.649 --> 00:07:57.050 needs to be built with open source. It must be flexible, so we can adapt it to 00:07:57.050 --> 00:08:02.580 our needs for our systems. It needs to be built with software that we understand and 00:08:02.580 --> 00:08:08.770 we use for, in, other applications, so that it can get widely tested and well 00:08:08.770 --> 00:08:12.869 tested. It needs to be built in a reproducible manner, so that we can be 00:08:12.869 --> 00:08:18.330 secure against that build chain attacks. And it needs to be cryptographically 00:08:18.330 --> 00:08:23.189 measured, so that we can be sure that this, what we flash on the system, is what 00:08:23.189 --> 00:08:30.159 is actually running on the system. And that's the philosophy behind Heads - it's 00:08:30.159 --> 00:08:36.429 built on the free software coreboot firmware, plus a Linux kernel in ROM, that 00:08:36.429 --> 00:08:44.959 acts as a bootloader, and then a lot of security research and tools, that help us 00:08:44.959 --> 00:08:52.420 try to build slightly more secure systems. Using Linux as a bootloader is not a 00:08:52.420 --> 00:08:58.370 particularly new idea. Back in the 1990s when we started building large scale Linux 00:08:58.370 --> 00:09:07.840 clusters, we were very frustrated with the inflexibility of DHCP and PXE booting 00:09:07.840 --> 00:09:13.230 large machines, that even with those frustrations, we built one, that was about 00:09:13.230 --> 00:09:21.060 the 30th fastest in the world on the top 500. Meanwhile, my colleague Ron Minich at 00:09:21.060 --> 00:09:25.960 Los Alamos was also building large clusters, and had the observation that the 00:09:25.960 --> 00:09:33.489 BIOS enumerates all the buses, initializes a bunch of devices, finds the Linux kernel 00:09:33.489 --> 00:09:38.680 and in the Linux kernel, enumerates all the buses, initializes all the devices, 00:09:38.680 --> 00:09:43.640 and he thought: "This is this is silly, why are we doing this twice?". 00:09:43.640 --> 00:09:49.730 So he had the idea to build a version of Linux, that ran in the ROM. He called this 00:09:49.730 --> 00:09:56.459 project "Linux BIOS" and it went on to power the MRC cluster, which was the third 00:09:56.459 --> 00:10:04.850 fastest machine in the world in 2003. In 2008 Linux BIOS underwent a major 00:10:04.850 --> 00:10:10.100 refactoring and it was renamed to "Coreboot". And Google chose to use 00:10:10.100 --> 00:10:14.690 Coreboot as the firmware on their Chromebooks - which are at this point the 00:10:14.690 --> 00:10:24.190 only non-UEFI x86-based laptops that you can buy. And they've done some really some 00:10:24.190 --> 00:10:28.280 great work in trying to lock down the configuration and the firmware on the 00:10:28.280 --> 00:10:35.779 Chromebooks. Ron, who has only moved to Google in 2011 and is continuing to work 00:10:35.779 --> 00:10:47.350 on the Coreboot project there. So Coreboot has three stages that it goes through as 00:10:47.350 --> 00:10:53.820 it starts up the machine: the first one is a very small amount of real-mode assembly, 00:10:53.820 --> 00:11:00.699 because your modern 64-bit laptop still boots up in real mode with 16-bit just 00:11:00.699 --> 00:11:08.389 like its 1970's. So that's a very small amount of code - about 1.5K. That in turn 00:11:08.389 --> 00:11:14.170 sets up a C runtime environment with what's called "Cache-as-RAM mode" and it 00:11:14.170 --> 00:11:20.330 calls into the ROM stage which is about 70K. "Heads" has moved the TPM 00:11:20.330 --> 00:11:26.800 initialization early in the ROM stage before DRAM is set up to help measure the 00:11:26.800 --> 00:11:33.069 boot block and provide a static route of trust that's hopefully a little bit more 00:11:33.069 --> 00:11:38.279 secure. And because that measurement is done early on, our trusted computing base 00:11:38.279 --> 00:11:44.660 is actually quite small. This is a fraction of 1% of the size of a UEFI 00:11:44.660 --> 00:11:50.339 firmware, which is usually about 16 MB once it's uncompressed. 00:11:50.339 --> 00:11:55.730 The ROM stage then measures and executes the RAM stage, which does the more 00:11:55.730 --> 00:12:01.120 traditional "walk the bus, find / figure out what devices are on there", but it 00:12:01.120 --> 00:12:04.269 doesn't initialize them: it just enumerates them and generates the 00:12:04.269 --> 00:12:09.320 descriptors for Linux. It also installs the SMM handler for system management 00:12:09.320 --> 00:12:16.710 mode. It then measures and jumps into the payload - and that whole process takes 00:12:16.710 --> 00:12:23.669 less than a second. So it is able to get into the payload and actually get to the 00:12:23.669 --> 00:12:30.730 user code very, very quickly. On something like the X230 it's able to go from power 00:12:30.730 --> 00:12:36.720 on to a interactive recovery shell in less than 2 seconds. And that includes bring up 00:12:36.720 --> 00:12:42.560 the TPM, doing cryptographic measurements and assessing the state of the system. 00:12:42.560 --> 00:12:46.790 Because we now have Linux at this point, we have all the flexibility that comes 00:12:46.790 --> 00:12:53.550 with that. We can implement boot scripts with the full power of the shell or the C 00:12:53.550 --> 00:12:57.600 language runtime: we're not stuck with the limited functions 00:12:57.600 --> 00:13:02.889 of UEFI. Linux supports lots of different file systems. It supports lots of 00:13:02.889 --> 00:13:07.139 different devices. It supports lots of different encryption methods. And this 00:13:07.139 --> 00:13:12.830 gives us the ability to use any of them for your specific application. In 00:13:12.830 --> 00:13:19.850 contrast, UEFI, which supports unencrypted FAT file systems on the first drive that 00:13:19.850 --> 00:13:23.589 have to be in the first 1 GiB or something, it's really, really limited as 00:13:23.589 --> 00:13:31.810 to how it can find its boot device. There's a saying in the in the Open Source 00:13:31.810 --> 00:13:35.889 community that with enough eyes all bugs are shallow. 00:13:35.889 --> 00:13:42.610 And Linux has a lot more eyes looking at it. That the device drivers and the file 00:13:42.610 --> 00:13:50.249 systems and the encryption have been reviewed by both white hat and black hat 00:13:50.249 --> 00:13:56.709 people around the world. The UEFI versions of these do not have that same level of 00:13:56.709 --> 00:14:03.160 scrutiny, so we're using both the UEFI drivers and then having to run whatever on 00:14:03.160 --> 00:14:08.079 top of it, increases the attack surface. But by putting Linux in the ROM and 00:14:08.079 --> 00:14:15.480 depending on its drivers we've reduced our attack surface very dramatically. More 00:14:15.480 --> 00:14:21.240 importantly though, Coreboot and Linux are Open Source, so it is possible to build 00:14:21.240 --> 00:14:25.480 custom versions for the device drivers that you need, for the file systems that 00:14:25.480 --> 00:14:30.839 you need. It's possible to fix bugs when they come out, and sign and install your 00:14:30.839 --> 00:14:37.089 own kernels. You don't have to wait for the vendor to get around to doing it. 00:14:37.089 --> 00:14:44.889 And the third major component of heads is a tool called "kexec", which is a system 00:14:44.889 --> 00:14:50.269 call that was added for the Linux BIOS project back in 2003 by Eric Peterman, 00:14:50.269 --> 00:14:56.230 that allows a running kernel to do a graceful shutdown and start a new kernel 00:14:56.230 --> 00:15:01.060 without having to go through the reboot process. So this allowed, on their 00:15:01.060 --> 00:15:06.510 application, it allowed them to do very fast reboots of their cluster nodes. And 00:15:06.510 --> 00:15:11.939 in the "Heads" case, it allows us to act as a bootloader where we can find the real 00:15:11.939 --> 00:15:16.149 kernel, that you want to run, and exec it. Because "Heads" is quite small. It has to 00:15:16.149 --> 00:15:22.689 fit in 4 MB of ROM. So it's not something that you learn to run as a day-to-day sort 00:15:22.689 --> 00:15:31.229 of OS. Hopefully this won't explode on me again. 00:15:36.269 --> 00:15:40.799 Because we have the Bourne Shell, most of the policies and the startup scripts in 00:15:40.799 --> 00:15:47.779 "Heads" are implemented as as shell scripts. In this case we were able to pass 00:15:47.779 --> 00:15:52.939 in a new set of command line parameters, a new initial RAM disk, and in this case we 00:15:52.939 --> 00:15:58.620 can even start a hypervisor. And all of that can happen very very quickly, as well 00:15:58.620 --> 00:16:07.269 as with with a good degree of security. So, those are the building blocks that 00:16:07.269 --> 00:16:12.980 "Heads" is built on: Coreboot, Linux and tools like "kexec". But it now gives us a 00:16:12.980 --> 00:16:16.710 really nice platform to begin experimenting with additional security 00:16:16.710 --> 00:16:23.240 features. And before we go too deep down the rabbit hole of, you know, security and 00:16:23.240 --> 00:16:27.509 threat models, I want to quote my friend Steph, who said that, "Your threat model 00:16:27.509 --> 00:16:31.019 is not my threat model." But you know your threat model is okay as 00:16:31.019 --> 00:16:35.730 well, that we all have different things we want to protect, from different attackers, 00:16:35.730 --> 00:16:40.389 who are willing to spend different amounts of effort to go after them. And the nice 00:16:40.389 --> 00:16:46.129 thing about having an open source is: We can build systems tailored to your 00:16:46.129 --> 00:16:50.930 individual threat model. So a lot of these things may not actually apply to your 00:16:50.930 --> 00:16:59.060 specific threats, but the fact that we can build them is a great capability. Last 00:16:59.060 --> 00:17:05.990 year Joanna Rutkowska reminded us that firmware is not just in our CPU. Firmware 00:17:05.990 --> 00:17:12.809 is in our Wi-Fi card. It is in our GPU. It is in our SSD. It is in our keyboards. And 00:17:12.809 --> 00:17:19.189 all of these devices might be trying to subvert the boot process. 00:17:19.189 --> 00:17:25.500 One way to handle that is to take Peter Stuge's advice of disassembling the 00:17:25.500 --> 00:17:30.401 machine and ripping out anything we can't control. If this is your threat model, his 00:17:30.401 --> 00:17:35.090 instructions are really worth following. They're really thorough about what pieces 00:17:35.090 --> 00:17:41.070 are potentially of concern. And you're going, right now, you'll have to open up 00:17:41.070 --> 00:17:46.560 your laptop to install "Heads." It's not quite as easy as it is to install most 00:17:46.560 --> 00:17:52.660 Linux distributions, because we have to flash it into the motherboard boot ROM. 00:17:52.660 --> 00:17:56.659 While we're in there, now, we take advantage of some features that, to the 00:17:56.659 --> 00:18:02.279 best of my knowledge, no UEFI system is using. These flash chips have a hardware 00:18:02.279 --> 00:18:09.610 write-protect mode, where you can specify part of the chip is write-only. Excuse me. 00:18:09.610 --> 00:18:15.480 Is read-only, write-once. And this gives us our immutable boot block in which to 00:18:15.480 --> 00:18:20.340 store the trusted computing base, the TCB, so that we can measure the rest of the 00:18:20.340 --> 00:18:26.270 system. We also then suggest disconnecting the write-protect PIN from the 00:18:26.270 --> 00:18:32.940 motherboard, which protects against certain classes of attacks, like the Intel 00:18:32.940 --> 00:18:38.899 close chassis adapter, that allows external JTAG of the CPU. 00:18:38.899 --> 00:18:42.440 Depending on your threat model you might want to cover that chip in epoxy as well, 00:18:42.440 --> 00:18:48.499 to frustrate Evil Maid attacks that want to do a physical programming on it. 00:18:48.499 --> 00:18:53.490 Disconnecting the write-protect PIN also serves to protect from other devices on 00:18:53.490 --> 00:18:58.890 the machine that have access to those PINs. Devices like the Management Engine, 00:18:58.890 --> 00:19:08.630 which is a really scary CPU inside the CPU. Rudolf Marek two years ago at CCC 00:19:08.630 --> 00:19:17.610 called it the "Matroyshka CPU". And Igor Skochinsky detailed what are the 00:19:17.610 --> 00:19:23.549 capabilities of the management engine. And they're really worrisome, that it runs a 00:19:23.549 --> 00:19:31.060 opaque obfuscated blob of code about 5 MiB that the CPU can't see. The management 00:19:31.060 --> 00:19:35.980 engine can read and write all of main memory. It can read from the keyboard and 00:19:35.980 --> 00:19:42.059 video. It can receive Java bytecodes over the network and execute them on behalf of 00:19:42.059 --> 00:19:46.510 someone outside the machine. And it's listening on the network even when the 00:19:46.510 --> 00:19:51.530 system is powered off. So this is basically a rootkit inside the 00:19:51.530 --> 00:19:58.480 chipset, as some folks have called it. So that concerned me a lot, and I spent some 00:19:58.480 --> 00:20:03.500 time looking at how its firmware images are built, and realized that we can build 00:20:03.500 --> 00:20:10.059 modified, reduced functionality firmware for it that removes all of the rootkit 00:20:10.059 --> 00:20:16.150 functions, and just leaves the "CPU bring up" module. This takes that 5 MiB and 00:20:16.150 --> 00:20:23.720 shrinks it down to about about 40 KiB of space. So we don't know exactly what it's 00:20:23.720 --> 00:20:28.460 doing in that 40 KiB, but we at least know it doesn't have a device driver, or a Java 00:20:28.460 --> 00:20:31.719 Virtual Machine, or a lot of the other functions. 00:20:31.719 --> 00:20:37.710 And we've successfully done this on both Sandy Bridge and Ivy Bridge like the X230 00:20:37.710 --> 00:20:43.380 ThinkPads as well as modern Skylake CPUs like the Chell Chromebook. And that's 00:20:43.380 --> 00:20:47.809 really encouraging, that if we can apply this to more modern hardware, that allows 00:20:47.809 --> 00:20:53.570 us to, you know, move away from our five- year-old ThinkPads to something a little 00:20:53.570 --> 00:21:01.580 shinier. The management engine isn't the only device that might be trying subvert 00:21:01.580 --> 00:21:05.799 the boot process. Again Johanna showed us there are lots of things to be worried 00:21:05.799 --> 00:21:12.159 about. Intel's UEFI architects, Yao and Zimmer, recommend that firmware turn on 00:21:12.159 --> 00:21:19.630 the IOMMU, now called VTD, to protect against rogue devices. To the best of my 00:21:19.630 --> 00:21:24.409 knowledge, since they've written this guide, no UEFI firmware is taking 00:21:24.409 --> 00:21:29.200 advantage of the IOMMU. So, it's a great piece of hardware to have, but it doesn't 00:21:29.200 --> 00:21:35.720 help if you don't turn it on. Linux, meanwhile has no problem taking advantage 00:21:35.720 --> 00:21:40.220 of it, so we use it, what we get essentially - we get that DMA protection 00:21:40.220 --> 00:21:48.120 for free by using Linux as our bootloader in the ROM. Another way devices, rogue 00:21:48.120 --> 00:21:52.360 devices, can try to interfere with the boot process is by providing option ROMs, 00:21:52.360 --> 00:21:59.169 which are executable code to be run by the BIOS, that have a sort of a device driver. 00:21:59.169 --> 00:22:03.630 And this code can do things like about log keystrokes and then try to exfiltrate 00:22:03.630 --> 00:22:11.909 passwords as we see here. That problem was initially reported in 2007 by John Heasman 00:22:11.909 --> 00:22:18.919 at BlackHat, again by Snare in 2012, and again by my work on Thunderstrike. And as 00:22:18.919 --> 00:22:24.290 of last week a official fix has finally rolled out for it to close that particular 00:22:24.290 --> 00:22:29.950 vulnerability. Folks who are using coreboot have this as 00:22:29.950 --> 00:22:35.600 a option that they can say, this, I am concerned about this threat, let me let me 00:22:35.600 --> 00:22:40.470 fix this, let me disable this function. And they point out that it might cause 00:22:40.470 --> 00:22:45.990 degraded functionality, but that's something you can QA on your own system. 00:22:45.990 --> 00:22:51.240 And in practice with Linux as your bootloader in the ROM, you don't use the 00:22:51.240 --> 00:22:56.610 option ROMs for anything. Everything is done with with Linux's own device drivers, 00:22:56.610 --> 00:23:02.750 so you're not dependent on the whatever limited functionality the option ROM 00:23:02.750 --> 00:23:07.830 provided. So, now that we have we've taken our 00:23:07.830 --> 00:23:12.220 building blocks and we hopefully have protected the boot process, and hopefully 00:23:12.220 --> 00:23:16.659 the code that's running is what we think it is, we need to turn to how do we secure 00:23:16.659 --> 00:23:22.780 the secrets on the machine. And I'm a huge fan of the TPM, the Trusted Platform 00:23:22.780 --> 00:23:28.590 Module, now I know in the free software community it's been largely unwelcome. It 00:23:28.590 --> 00:23:34.840 has not received a very welcome reception, because of the way it's been used for DRM 00:23:34.840 --> 00:23:41.530 and other other user-hostile things. Since we control the TPM from the first 00:23:41.530 --> 00:23:47.260 instruction in the in the boot block, we're able to use it in ways that we want 00:23:47.260 --> 00:23:55.140 to, so we don't have to enable DRM, but we can use it to protect our secrets. And the 00:23:55.140 --> 00:23:59.289 the way that it does that, is it keeps track of what code is executed as the 00:23:59.289 --> 00:24:06.049 system boots, and it hashes that code into special registers called PCRs. And the 00:24:06.049 --> 00:24:11.230 idea is that, you can extend the PCR by hashing the next module of code, and then 00:24:11.230 --> 00:24:17.460 hashing that with the previous hash, and this creates a chain of trust that allows 00:24:17.460 --> 00:24:24.690 to say: If these hashes match the expected values, only the code, that we want to 00:24:24.690 --> 00:24:34.040 have run, has run. And then the TPM will only decrypt the the disk encryption key 00:24:34.040 --> 00:24:40.260 if those PCRs match, which means that the code that we want to have run, is what has 00:24:40.260 --> 00:24:47.049 been executed. This means if someone manages to overwrite the, the non-write 00:24:47.049 --> 00:24:51.500 protected part of the ROM, that would change those measurements and the TPM 00:24:51.500 --> 00:24:59.019 won't reveal the key to them. It also takes a user password and that password is 00:24:59.019 --> 00:25:04.929 validated by the TPM hardware itself, which gives us hardware rate-limiting on 00:25:04.929 --> 00:25:10.850 how often an attacker can try. It also gives us the ability to do hardware-based 00:25:10.850 --> 00:25:15.820 retry limits, so that the TPM will flush the key, if they if an attacker in 00:25:15.820 --> 00:25:21.980 possession of your machine tries too long. That does mean there's now another way to 00:25:21.980 --> 00:25:27.299 lose your disk encryption key. And there's the the old joke about there 00:25:27.299 --> 00:25:31.149 are two types of people with encrypted drives - those who have lost data due to 00:25:31.149 --> 00:25:36.750 forgetting their key, and and those who will. So what "Heads" does, when you 00:25:36.750 --> 00:25:41.810 generate your key, is, it takes that key and splits it into multiple pieces, that 00:25:41.810 --> 00:25:46.789 you can then share either to friends or to backup services, where each piece doesn't 00:25:46.789 --> 00:25:53.030 let you decrypt it. But you can combine them with Shamir secret sharing to 00:25:53.030 --> 00:25:58.169 regenerate the cryptographic disk encryption key. 00:25:58.169 --> 00:26:05.019 We're also able to take advantage of best practices like using the, including the 00:26:05.019 --> 00:26:11.889 disk encryption key headers in the PCRs that we use to seal the disks. This avoids 00:26:11.889 --> 00:26:15.769 a certain class of Evil Maid attack, where someone swaps out your drive; may not be 00:26:15.769 --> 00:26:19.100 in your threat model, but it is easy to deal with just a few lines of shell 00:26:19.100 --> 00:26:28.309 script. So we hopefully we now trust that the system is running the code we think it 00:26:28.309 --> 00:26:33.690 is, but how does it prove to us that it is actually our machine, that someone hasn't 00:26:33.690 --> 00:26:38.889 snuck into our hotel room and swapped out everything and left, left up, carefully 00:26:38.889 --> 00:26:43.539 replaced our stickers to make us believe we're typing our password into to our own 00:26:43.539 --> 00:26:51.929 computer. Some anti-Evil Maid toolkits will encrypt a secret, secret phrase, and 00:26:51.929 --> 00:26:56.419 then display it to you if and only if the PCR is match, but that's subject to a 00:26:56.419 --> 00:27:03.970 replay attack. What Matthew Garrett demonstrated last year at 32C3, was, using 00:27:03.970 --> 00:27:10.840 the time, a time-based one-time password used by Google Authenticator to protect 00:27:10.840 --> 00:27:16.760 the firmware and have it attest to the user, that it is unmodified. So when the 00:27:16.760 --> 00:27:21.710 system boots, and it goes through measuring all of the various components 00:27:21.710 --> 00:27:29.960 the, the TPM will only release the secret if those PCRs match. The firmware then 00:27:29.960 --> 00:27:33.550 hashes that along with the current time and generates a six digit value that it 00:27:33.550 --> 00:27:38.519 prints on the screen. You compare that to what's on your phone and that tells you 00:27:38.519 --> 00:27:46.791 whether or not you can trust the machine. It's a, it's a great idea, and it's 00:27:46.791 --> 00:27:51.780 implemented again as a very small shell script to read, read the value from the 00:27:51.780 --> 00:27:58.630 TPM, unseal it and then compute the the hash of it. 00:27:58.630 --> 00:28:04.390 This also allows us to start making a transition from the TPM-rooted, a TPM 00:28:04.390 --> 00:28:10.031 static route of trust to a PGP-based trust, where most importantly, this is a 00:28:10.031 --> 00:28:14.540 TPM key - excuse me, this is a PGP key, that you, the owner of the computer 00:28:14.540 --> 00:28:19.820 control, not some random vendor or some random hardware device manufacturer that's 00:28:19.820 --> 00:28:25.659 going to lose the key and allow malware like Stuxnet to use it to circumvent 00:28:25.659 --> 00:28:30.940 security. The boot script in "Heads", again, it's a 00:28:30.940 --> 00:28:37.200 small shell script, is able to use a GPG to verify the next stages of the, the 00:28:37.200 --> 00:28:45.010 hypervisor, the initial RAM disk and the kernel. And it also then uses the the 00:28:45.010 --> 00:28:50.820 TPM's counters to help prevent against rollback, where someone takes your drive 00:28:50.820 --> 00:28:54.660 and rolls it back to a previous version, perhaps with a vulnerability that they can 00:28:54.660 --> 00:28:59.380 exploit. So this, this allows the, allows us to be 00:28:59.380 --> 00:29:03.679 sure, that not only are we running the firmware, the OS that we think we should 00:29:03.679 --> 00:29:10.799 be running, it ensures us that someone hasn't been able to substitute one that, a 00:29:10.799 --> 00:29:16.700 vulnerable version. Having the PGP key also allows us to take 00:29:16.700 --> 00:29:24.720 advantage of an idea from Android and Chrome OS which is the the dm-verity read- 00:29:24.720 --> 00:29:31.010 only root filesystem - this hashes all of the blocks and that hashes all of the the 00:29:31.010 --> 00:29:38.320 hashes and so on up until it gets to a root hash in the tree that is then signed. 00:29:38.320 --> 00:29:45.029 This allows the kernel, on every read access, in logarithmic time to verify the 00:29:45.029 --> 00:29:51.580 essentially a signature on that data. This does require a read-only root filesystem, 00:29:51.580 --> 00:29:59.719 but it gives us even more confidence that the system has been untampered with. Once 00:29:59.719 --> 00:30:05.100 you're running your OS, it's good to have you know some security conscious thoughts 00:30:05.100 --> 00:30:10.990 as well, you know. Heads is mostly focused on, "how do we securely transition to an 00:30:10.990 --> 00:30:17.529 OS" and that OS you run is up to you. I like Qubes - it's reasonably secure, it's 00:30:17.529 --> 00:30:23.630 highly recommended by people who know about endpoint security. And the Qubes 00:30:23.630 --> 00:30:30.480 team recognizes that firmware security is a vital piece of system security. For 00:30:30.480 --> 00:30:35.149 their next release, Qubes are for - they're going to require the machines have 00:30:35.149 --> 00:30:39.101 open source firmware, such as coreboot. And I hope that Heads is going to be a 00:30:39.101 --> 00:30:46.330 piece of that. I've also been working with with the Qubes software and have modified 00:30:46.330 --> 00:30:53.000 it to work with things like the dm-verity read-only root filesystem. This now allows 00:30:53.000 --> 00:30:58.809 the user to lock down the configuration, so that someone can't tamper with their 00:30:58.809 --> 00:31:05.850 their setup. It also gives you a recovery mode that allows you to fix things up and 00:31:05.850 --> 00:31:12.139 re-sign the OS. Reproducible builds are really important so that everyone can 00:31:12.139 --> 00:31:19.490 verify what that the builds match what they should. In the case of of Heads we 00:31:19.490 --> 00:31:22.429 have a lot of upstream dependencies that aren't reproducible so we're working with 00:31:22.429 --> 00:31:28.190 them to try to patch them. We've patched Xen - they've accepted that commit. We've 00:31:28.190 --> 00:31:33.669 also built some tools to let you build initial RAM disks in a reproducible way. 00:31:33.669 --> 00:31:37.389 This works with Qubes, with Heads, and we're hoping to other Linux distributions 00:31:37.389 --> 00:31:41.960 pick it up as well. All of our tree is cryptographically 00:31:41.960 --> 00:31:48.230 signed, so hopefully GitHub is not trying to slip in any patches. And it is open 00:31:48.230 --> 00:31:52.890 source so we encourage anyone to read through it. No NDA is required, unlike most 00:31:52.890 --> 00:31:59.570 of the UEFI sources. So that's sort of the state of where things are - it's pretty 00:31:59.570 --> 00:32:04.679 much in very beta, but it is usable. But there are a lot of areas where we could 00:32:04.679 --> 00:32:08.679 continue to do research - things like the embedded controllers on Chromebooks are 00:32:08.679 --> 00:32:14.359 open source. We can use those to help with our root of trust as well. Porting 00:32:14.359 --> 00:32:17.070 coreboot to more modern platforms would let us take advantage of things like 00:32:17.070 --> 00:32:23.440 tamper switches and Intel Boot Guard. I'm also working on porting coreboot over to 00:32:23.440 --> 00:32:30.029 server platforms so that we can use it for more secure cloud hosting. Servers have a 00:32:30.029 --> 00:32:36.480 very different threat model from from laptops, and a lot of things have firmware 00:32:36.480 --> 00:32:41.090 that we have to be concerned about. One collaboration I'm looking at there is with 00:32:41.090 --> 00:32:45.989 the open BMC project to be able to take advantage of the open source in the 00:32:45.989 --> 00:32:54.159 management controller for the servers. And I'm also collaborating with the Mass Open 00:32:54.159 --> 00:33:01.129 Cloud project that's trying to build secure bare metal clouds. I'm cautiously 00:33:01.129 --> 00:33:06.590 optimistic about enclaves and how they will help us with the security, especially 00:33:06.590 --> 00:33:10.269 in a environment where we control the firmware and we can ensure that the 00:33:10.269 --> 00:33:17.270 enclaves are set up in a safe way. There are a lot of issues on GitHub, again, 00:33:17.270 --> 00:33:25.150 please welcome contributions. I hope everyone gets inspired to to work on the 00:33:25.150 --> 00:33:31.080 installing this on their laptop. And if if you are interested I'll be hanging out at 00:33:31.080 --> 00:33:38.220 the coreboot assembly later today and occasionally this week. The coreboot team 00:33:38.220 --> 00:33:42.610 has a bunch of people here. They have flash programmers and can help you install 00:33:42.610 --> 00:33:50.990 coreboot on your laptop. The source code for Heads is available on GitHub, and a 00:33:50.990 --> 00:33:56.419 annotated version of this talk is up on my website, and welcome comments and feedback 00:33:56.419 --> 00:34:02.340 on it. So thank you all for coming to hear about this project I hope that everyone is 00:34:02.340 --> 00:34:06.899 and you know as excited about open source firmware as I am and I'd love to take any 00:34:06.899 --> 00:34:09.918 questions that you all have. 00:34:09.918 --> 00:34:21.739 Applause 00:34:28.549 --> 00:34:31.730 Question: Thanks for your great talk - this is very interesting. Do you have any 00:34:31.730 --> 00:34:39.009 advice for the 95% of us who are stuck on non coreboot compatible laptops. 00:34:39.009 --> 00:34:42.890 Trammell: Buy a Chromebook? Laughter 00:34:42.890 --> 00:34:52.951 Trammell: It's hard to trust the closed- source firmware. It certainly; there are 00:34:52.951 --> 00:34:55.920 people we have to trust - there are institutions we have to trust. We have to 00:34:55.920 --> 00:35:01.620 trust Intel to some extent, and Intel is responsible for, you know, both our CPUs 00:35:01.620 --> 00:35:09.370 and a lot of the firmware. Depending on your threat model, firmware attacks may 00:35:09.370 --> 00:35:15.280 not be a huge concern for your particular machine. Or they might be, you know, of 00:35:15.280 --> 00:35:20.470 grave concern, in which case even just doing some of the things that Peter Stuge 00:35:20.470 --> 00:35:30.020 suggested of clipping the write-protect pin on the on the chip, removing things 00:35:30.020 --> 00:35:35.200 that might be hostile and attacking your system. His guides are really good one to 00:35:35.200 --> 00:35:43.100 follow for the hardware security. Question: I was wondering if you also 00:35:43.100 --> 00:35:49.610 support ARM - I just saw Intel laptops so I was just wondering. 00:35:49.610 --> 00:35:55.231 Trammell: So ARM it has a lot of advantages as a CPU. It only has 20 years 00:35:55.231 --> 00:36:00.750 of legacy baggage rather than 40. And the boot process on it is much much simpler 00:36:00.750 --> 00:36:06.820 since it doesn't have to go through real mode to long mode to paging and all the 00:36:06.820 --> 00:36:15.140 other steps. The downside to a lot of ARMs is that they their boot code is a few is 00:36:15.140 --> 00:36:22.190 on die and outside of the control of the user. Luckily, most of that boot code is 00:36:22.190 --> 00:36:29.280 fairly simple, and can establish - and some of them will establish a hardware 00:36:29.280 --> 00:36:37.490 root of trust. But in general, yeah that the ARM to U-Boot to whatever seems to 00:36:37.490 --> 00:36:44.090 work out pretty well. I know there's been some interest in, "can U-Boot be replaced 00:36:44.090 --> 00:36:49.920 with a Linux BIOS or coreboot like thing", and I suspect the folks at the booth would 00:36:49.920 --> 00:36:54.700 be able to talk more about that. Question: And then just a followup - so if 00:36:54.700 --> 00:37:00.510 coreboot or Libreboot supports the platform - Heads will work too right? 00:37:00.510 --> 00:37:05.590 Trammell: Essentially yes, yeah. Heads is a payload for coreboot. 00:37:05.590 --> 00:37:13.030 Question: OK. Herald: It's there on the left. 00:37:13.030 --> 00:37:17.960 Signal Angel: Thank you - there's a question from the Internet about coreboot. 00:37:17.960 --> 00:37:23.430 Coreboot has blobs included and, for example, binary blobs from Intel with all 00:37:23.430 --> 00:37:28.750 the firmware support package and all that stuff. How can we call coreboot secure, 00:37:28.750 --> 00:37:33.060 then, in the light of this - let alone open source? 00:37:33.060 --> 00:37:37.890 Trammell: So the intel FSP is a significant concern. This is the firmware 00:37:37.890 --> 00:37:42.010 support package that is required to initialize the memory controllers on on 00:37:42.010 --> 00:37:52.200 modern Intel cpus. On older CPUs, such as the the Sandy Bridge and Ivy Bridge, the 00:37:52.200 --> 00:37:57.040 coreboot and Libreboot are able to initialize the memory natively without 00:37:57.040 --> 00:38:05.870 having to go into the FSB. However, if you look at what coreboot is doing in the MRC 00:38:05.870 --> 00:38:10.930 on those platforms, it tends to just be poking a bunch of registers with values 00:38:10.930 --> 00:38:20.170 that seem to work. And it's modern memory controllers are so complex that, and Intel 00:38:20.170 --> 00:38:26.700 is unwilling to document them, that without extensive NDA's that it's very 00:38:26.700 --> 00:38:32.330 hard to build any sort of memory initialization. So what we can't say it's 00:38:32.330 --> 00:38:38.610 a hundred percent free software, we can at least we can ensure that the FSP is 00:38:38.610 --> 00:38:47.200 measured and it's unchanging. We can also look at the state of things that it sets 00:38:47.200 --> 00:38:52.210 up and include those in our measurements. So even if it doesn't get us one hundred 00:38:52.210 --> 00:38:57.850 percent open source - and as far as I know the only system that does that right now 00:38:57.850 --> 00:39:05.280 is bunnies' Novena laptop. At least we can measure it and we can know that it hasn't 00:39:05.280 --> 00:39:10.810 been tampered with from what we initially installed. 00:39:10.810 --> 00:39:17.260 Question: And before so this is a great project and I'd like to ask why you did 00:39:17.260 --> 00:39:23.650 certain architectural decisions. The specific combination of Linux and shell. 00:39:23.650 --> 00:39:29.840 So why didn't you choose a BSD kernel which are usually perceived to be more secure 00:39:29.840 --> 00:39:34.720 and of a higher quality, and why did you choose a shell over let's say, Python 00:39:34.720 --> 00:39:41.230 or Haskell, which are also often perceived of higher quality? 00:39:41.230 --> 00:39:45.400 Trammell: There is a lot of desire to support Python in Heads. The downside is 00:39:45.400 --> 00:39:51.910 that there's a very limited space: the X230 bootrom, for instance, has 4 00:39:51.910 --> 00:40:02.070 megabytes of available space. The Python interpreter is a couple megs already. In 00:40:02.070 --> 00:40:09.080 terms of why Linux over BSD - the kexec system call is a core component of this: 00:40:09.080 --> 00:40:14.940 to be able to do a graceful shut down and transfer from the Linux kernel to another 00:40:14.940 --> 00:40:22.370 kernel or - to any multi-boot compliant kernels, which includes BSD. It is a 00:40:22.370 --> 00:40:31.010 necessary feature if it, if BSD had such functionality, that it would be a fine 00:40:31.010 --> 00:40:37.225 just a choice for the the internal boot ROM/boot loader. 00:40:45.700 --> 00:40:53.601 Question: Thanks for great work. How to perform updates of coreboot and its 00:40:53.601 --> 00:41:00.391 payload when it's binaries used in measurement for releasing encryption key 00:41:00.391 --> 00:41:07.150 then when you update coreboot this measurement will change and you will know 00:41:07.150 --> 00:41:12.250 no longer will be able to boot the system - how to solve that problem? 00:41:12.250 --> 00:41:19.180 Trammell: So migrating encryption keys with TPM requires a explicit step of 00:41:19.180 --> 00:41:24.060 retrieving the key from the TPM with the current configuration and then resealing 00:41:24.060 --> 00:41:30.210 it with the new configuration. One advantage of a reproducible build is the 00:41:30.210 --> 00:41:37.240 hashes of the of all the firmware stages can be published - it can be pre-computed, 00:41:37.240 --> 00:41:41.460 and then the PCR values can be pre- computed so you can read you can seal the 00:41:41.460 --> 00:41:52.170 keys for the new values. In terms of the update process for the Heads payload - one 00:41:52.170 --> 00:41:56.550 of the things that we're working on is being able to have an even more minimal 00:41:56.550 --> 00:42:03.500 heads that has just a USB device driver that you can boot into, copy your new 00:42:03.500 --> 00:42:08.830 payload, and then install that elsewhere on the chip. And part of that process 00:42:08.830 --> 00:42:15.200 would involve resealing any of the keys that you need to transfer. 00:42:15.200 --> 00:42:17.270 Herald: Another question from the Internet. 00:42:17.270 --> 00:42:21.570 Signal Angel: Thank you. On your webpage you implemented Heads on ThinkPads only. 00:42:21.570 --> 00:42:28.250 How much work is still needed to translate this to, let's say, non-ThinkPads? 00:42:28.250 --> 00:42:32.890 Trammell: ThinkPads are really popular with the security community. It's quite 00:42:32.890 --> 00:42:36.280 interesting to look out at you know the hall here and see how many ThinkPads there 00:42:36.280 --> 00:42:41.520 are. And as a result the coreboot community has been very supportive of 00:42:41.520 --> 00:42:48.820 ThinkPads. There's, other than the ThinkPads and the Chromebooks, there 00:42:48.820 --> 00:42:53.270 aren't a lot of devices that support coreboot out of the box. And that's 00:42:53.270 --> 00:42:58.050 something that I hope would change - I hope that some OEMs would realize there's 00:42:58.050 --> 00:43:04.560 value in provide an open source firmware and move to move to using it. Both as a 00:43:04.560 --> 00:43:10.590 cost-saving measure as well as a freedom measure. In terms of the difficulty 00:43:10.590 --> 00:43:15.780 important coreboot to a platform - I haven't successfully done that yet, but I 00:43:15.780 --> 00:43:23.350 suspect the people at the assembly would be happy to discuss that further. 00:43:23.350 --> 00:43:29.400 Question: Would you plan to rework embedded controller firmware on ThinkPads 00:43:29.400 --> 00:43:38.420 because it's remain close at birth which still has an access to PC bus and probably 00:43:38.420 --> 00:43:43.440 couldn't be trusted. Trammell: So your question is how do we 00:43:43.440 --> 00:43:48.520 how do we replace the EC? Question: Yes do plan to replace EC with 00:43:48.520 --> 00:43:52.150 open source firmware as in the Chromebooks? 00:43:52.150 --> 00:43:57.530 Trammell: So the Chromebook has open source EC. The part of building coreboot 00:43:57.530 --> 00:44:03.190 for Chromebook involves installing the ARM cross compiler to build the EC firmware. 00:44:03.190 --> 00:44:08.790 And the Chromebooks actually have a really elegant protocol for the EC to attest to 00:44:08.790 --> 00:44:14.430 the CPU that is running the firmware that you think it is running. On other 00:44:14.430 --> 00:44:23.700 platforms, this would require a lot more research. The doc; many of the EC chipsets 00:44:23.700 --> 00:44:27.840 have data sheets available, so it's possible to read through and see how they 00:44:27.840 --> 00:44:33.070 work. And most of them have updatable firmware. In case of the ThinkPads, 00:44:33.070 --> 00:44:37.410 there's a module in the ThinkPad BIOS that will do that update. There's, we would 00:44:37.410 --> 00:44:41.480 need to figure out what that protocol looks like. 00:44:41.480 --> 00:44:47.810 Question: Sorry yes I mean if you have working prototype on ThinkPads probably 00:44:47.810 --> 00:44:56.870 want to add remaining business open sources EC on ThinkPads as well in the 00:44:56.870 --> 00:45:00.550 first place. Trammell: I'm sorry, I don't think I 00:45:00.550 --> 00:45:09.540 understood your follow-up. Question: Okay. So if you if you have a 00:45:09.540 --> 00:45:18.240 working prototype on ThinkPads and only on ThinkPads, will you finish somewhat soon 00:45:18.240 --> 00:45:28.711 current existing prototype of open source EC exists in college shade by Lincoln's or 00:45:28.711 --> 00:45:36.830 you are playing to extend your work on other platforms and finish this bits 00:45:36.830 --> 00:45:39.700 later. Trammell: Yeah right now I have not 00:45:39.700 --> 00:45:46.910 personally made any progress on the ThinkPad EC. I was looking into it because 00:45:46.910 --> 00:45:51.700 I have a modified keyboard on my ThinkPad that that needs to updated EC firmware, 00:45:51.700 --> 00:46:00.740 but I haven't actually gotten into that. This is a area of open research 00:46:00.740 --> 00:46:03.900 Signal Angel: Thank you, two quick questions from the IRC - are you planning 00:46:03.900 --> 00:46:07.690 to use systemd in the boot process is the first one 00:46:07.690 --> 00:46:11.030 Laughter Signal Angel: And the second one let's say 00:46:11.030 --> 00:46:16.110 you flash your firmware at the Congress right here with the help of a hardware 00:46:16.110 --> 00:46:22.900 programmer. Can you update when there's a new version or do you have to currently 00:46:22.900 --> 00:46:29.620 need the hardware access to update? Trammell: Right now you can update 00:46:29.620 --> 00:46:40.030 afterwards at great risk, because you can leave the flash writeable, which would 00:46:40.030 --> 00:46:45.710 allow you to flash after the fact. We are still working on a good procedure for 00:46:45.710 --> 00:46:53.540 doing software-only firmware updates once the immutable boot block is installed. And 00:46:53.540 --> 00:46:58.950 to the other question - did I mention that we are really short on space and we don't 00:46:58.950 --> 00:47:07.712 want to put any large applications like systemd on there. 00:47:07.712 --> 00:47:11.292 Questioner: It was like good one, thanks. 00:47:11.292 --> 00:47:19.242 applause 00:47:19.242 --> 00:47:28.159 postroll music 00:47:28.159 --> 00:47:42.811 subtitles created by c3subtitles.de in the year 2018