1
00:00:00,000 --> 00:00:19,759
35c3 prerol music
2
00:00:19,759 --> 00:00:26,630
Herald: So Trammell Hudson, who is
standing here, he's taking things apart.
3
00:00:26,630 --> 00:00:34,370
Don't worry not life on stage, but he will
give us a proof of concept and some
4
00:00:34,370 --> 00:00:39,559
details and functionalities about hardware
implants. So the same things that we heard
5
00:00:39,559 --> 00:00:45,480
from Bloomberg article talking about Apple
and super microcomputers with implants
6
00:00:45,480 --> 00:00:52,879
that, yeah, were implanted into those,
into those computers. And I'm really
7
00:00:52,879 --> 00:00:57,680
excited to see this in action. Please give
a warm round of applause to Trammel
8
00:00:57,680 --> 00:01:02,590
Hudson!
9
00:01:02,590 --> 00:01:07,510
applause
10
00:01:07,510 --> 00:01:11,600
Trammell: Before we begin talking about
hardware implants just two quick
11
00:01:11,600 --> 00:01:16,310
disclaimers. The first from my employer
Two Sigma investments as it says are
12
00:01:16,310 --> 00:01:21,910
chocolate bars. This is not investment
advice. And secondly I don't actually know
13
00:01:21,910 --> 00:01:26,920
what the story is behind the super micro
story. No one outside of Bloomberg and
14
00:01:26,920 --> 00:01:32,450
their sources do. But I have spent a lot
of time thinking about hardware implants
15
00:01:32,450 --> 00:01:38,200
starting with the thunderstrike firmware
attack against mac books as well as the
16
00:01:38,200 --> 00:01:45,420
thunderstrike 2 where we were able to get
software to write into the firmware on the
17
00:01:45,420 --> 00:01:50,560
mac books. I've also been thinking a lot
about how to defend against hardware
18
00:01:50,560 --> 00:01:54,420
implants with things like the heads
firmware for slightly more secure laptops
19
00:01:54,420 --> 00:01:59,420
and also as part of my co-lead on the
Linux boot project. We're thinking about
20
00:01:59,420 --> 00:02:10,080
how to protect servers from physical and
software attacks. So with all of this
21
00:02:10,080 --> 00:02:14,910
concentrated thinking about firmware and
hardware attacks, I was really excited
22
00:02:14,910 --> 00:02:20,720
when I saw the Bloomberg story back in
October. But what really intrigued me was
23
00:02:20,720 --> 00:02:26,440
the animated image that they had at the
header that highlighted one small part of
24
00:02:26,440 --> 00:02:32,920
the board as where the implant was, but
what I found really interesting is that is
25
00:02:32,920 --> 00:02:40,250
exactly where I would install a hardware
implant as they described on the SPI bus.
26
00:02:40,250 --> 00:02:44,610
A lot of other people in the hardware and
from our security community thought it
27
00:02:44,610 --> 00:02:50,140
sounded plausible. Other people pointed
out that supply chain attacks come up
28
00:02:50,140 --> 00:02:56,070
periodically and they are definitely a
concern. Some people thought the attack as
29
00:02:56,070 --> 00:03:03,320
described was entirely implausible and in
general we sort of had a Whiskey Tango
30
00:03:03,320 --> 00:03:08,160
Foxtrot moment as everybody scrambled to
figure out what's going on inside their
31
00:03:08,160 --> 00:03:14,540
machines. So, let's step back very quickly
and review what the key claims that
32
00:03:14,540 --> 00:03:22,340
Bloomberg alleged happened. First they
said that Amazon's testers found a tiny
33
00:03:22,340 --> 00:03:27,250
microchip that wasn't part of the board's
original design that had been disguised to
34
00:03:27,250 --> 00:03:33,350
look like a signaling condition signal
condition coupler and that these illicit
35
00:03:33,350 --> 00:03:39,790
chips were connected to the baseboard
management controller or the BMC which
36
00:03:39,790 --> 00:03:44,210
gave them access to machines that were
turned off. That might sound kind of
37
00:03:44,210 --> 00:03:49,870
extreme, but that's actually what the role
of the BMC is, that in most servers the
38
00:03:49,870 --> 00:03:55,280
BMC is running any time the machine is
hooked up to power and it's connected to
39
00:03:55,280 --> 00:04:01,910
the power supplies so that it can turn the
machine on and turn it off. Frequently you
40
00:04:01,910 --> 00:04:06,780
want to be able to do this over a network
so it has its own dedicated LAN port but
41
00:04:06,780 --> 00:04:14,180
it can also share the LAN port with the
with the main system. Serial over LAN is a
42
00:04:14,180 --> 00:04:19,180
really useful way to debug the systems so
it provides that functionality. It can
43
00:04:19,180 --> 00:04:27,350
also provide fake USB volumes to allow to
to do unintended OS installation. A lot of
44
00:04:27,350 --> 00:04:33,430
sites also won't remote KVM so it has VGA
but that VGA support means that it's on
45
00:04:33,430 --> 00:04:40,370
the PCIe BUS and because some PCIe it can
do DMA into main memory. It also is
46
00:04:40,370 --> 00:04:47,000
typically muxed into the SPI flash for
the host firmware, which allows it to
47
00:04:47,000 --> 00:04:51,820
modify it and on some systems it's even
connected to the TPM which allows it to
48
00:04:51,820 --> 00:04:59,930
circumvent the corporate of trust. So with
all of this capability inside this chip
49
00:04:59,930 --> 00:05:06,919
it's really unfortunate that they are
really not well put together. The head of
50
00:05:06,919 --> 00:05:11,150
Azure security says they have no
protection against attacks. There's no
51
00:05:11,150 --> 00:05:15,530
ability to detect if an attack has
happened and there's no ability to recover
52
00:05:15,530 --> 00:05:22,449
from an attack. So having a hardware
implant on the BMC is a really big
53
00:05:22,449 --> 00:05:32,030
concern. The other claim in the article is
that it affected 30 different companies
54
00:05:32,030 --> 00:05:39,930
including Apple and Bloomberg alleges that
Apple found malicious chips independently
55
00:05:39,930 --> 00:05:44,980
on their super micro boards. Went to the
FBI about it and that they then severed
56
00:05:44,980 --> 00:05:52,100
ties with Super Micro. This particular
claim was interesting because it
57
00:05:52,100 --> 00:05:57,570
corroborated a story that had shown up
back in early 2017 that Apple had removed
58
00:05:57,570 --> 00:06:03,050
Super Micro from their data centers. Apple
denied that there was a firmware issue.
59
00:06:03,050 --> 00:06:10,190
But it's interesting that perhaps these
two were related. The third set of claims
60
00:06:10,190 --> 00:06:16,090
is that on some of these implants they
were actually put between the layers on
61
00:06:16,090 --> 00:06:23,210
the PCB and then the most explosive claim
is that this was done by operatives from
62
00:06:23,210 --> 00:06:33,580
China, the Chinese People's Liberation
Army. With a story with this you know this
63
00:06:33,580 --> 00:06:39,389
many claims and this significant of
allegations we'd hoped that it would be
64
00:06:39,389 --> 00:06:45,430
really well sourced and for a normal story
17 independent sources that Bloomberg
65
00:06:45,430 --> 00:06:52,490
editors agreed to grant anonymity to,
including six national security, two
66
00:06:52,490 --> 00:06:57,340
people inside of AWS and three senior
insiders at Apple seems like pretty solid
67
00:06:57,340 --> 00:07:03,110
sourcing, except as soon as this article
is published everyone denied it. The
68
00:07:03,110 --> 00:07:09,080
Director of National Intelligence said
they'd seen no evidence of this. Amazon
69
00:07:09,080 --> 00:07:13,990
said that they've never found any issues
of modified hardware nor have they been
70
00:07:13,990 --> 00:07:21,000
engaged with the government over it. Apple
was even more blunt. CEO Tim Cook said
71
00:07:21,000 --> 00:07:27,590
this did not happen. There is no truth to
this. And Super Micro wrote a fairly
72
00:07:27,590 --> 00:07:32,150
lengthy letter about what they do to
protect their supply chain and why they
73
00:07:32,150 --> 00:07:38,990
think this attack did not happen. And it
is worth going through to look at some of
74
00:07:38,990 --> 00:07:44,880
the things that they say that they do to
protect their supply chain. They point out
75
00:07:44,880 --> 00:07:50,700
that if there's any unauthorized physical
alterations during the manufacturing
76
00:07:50,700 --> 00:07:56,949
process other design elements would not
match and those things would be detected.
77
00:07:56,949 --> 00:08:03,300
To sort of understand how circuit boards
are made, I recently visited a PCB factory
78
00:08:03,300 --> 00:08:11,080
in Guangzhou. This is not a super micro
factory. This is just a holiday photos. So
79
00:08:11,080 --> 00:08:16,760
in order to add new vias they would have
to modify the drill files which would then
80
00:08:16,760 --> 00:08:22,050
get electroplated. If they had to add new
traces, they would have to be able to
81
00:08:22,050 --> 00:08:29,400
subvert the masking and etching process
and any changes to either the drills or
82
00:08:29,400 --> 00:08:34,909
the etching on individual layers would be
caught by the optical inspection that's
83
00:08:34,909 --> 00:08:41,479
done on these bare circuit boards.
Additionally the allegation that things
84
00:08:41,479 --> 00:08:47,110
were inserted between circuit boards would
require that the lamination process be
85
00:08:47,110 --> 00:08:55,329
subverted and that the implant somehow
aligned into the system. If that implant
86
00:08:55,329 --> 00:09:00,410
changes any of the connectivity the flying
protesters would pick it up or the bed of
87
00:09:00,410 --> 00:09:05,980
nails testers which checks all of the
connectivity of all the traces to make
88
00:09:05,980 --> 00:09:09,300
sure that there are no shorts and to make
sure that everything that is supposed to
89
00:09:09,300 --> 00:09:16,679
be connected is electrically conductive.
So it would be very difficult to
90
00:09:16,679 --> 00:09:22,110
circumvent the production process at this
stage. And it also would be very difficult
91
00:09:22,110 --> 00:09:27,709
to contain because the PCB factory doesn't
know which customers are going to receive
92
00:09:27,709 --> 00:09:34,470
those circuit boards. Super Micro also
points out that during the assembly
93
00:09:34,470 --> 00:09:40,480
process when the parts are installed they
have their employees on site the whole
94
00:09:40,480 --> 00:09:47,559
time. On my same holiday trip I also
visited some PCB assembly companies and
95
00:09:47,559 --> 00:09:53,589
spoke with companies that are using doing
contract manufacturing and they said that
96
00:09:53,589 --> 00:09:59,089
they also send their employees to the
production line to observe the pick and
97
00:09:59,089 --> 00:10:05,600
place machines and the reflow and the rest
of the surface mount assembly. Their big
98
00:10:05,600 --> 00:10:10,089
concern is that if they don't have someone
there the parts that are fed in the pick
99
00:10:10,089 --> 00:10:17,660
in place will be replaced with either
counterfeits or with salvaged parts. I
100
00:10:17,660 --> 00:10:23,459
visited the electronics market in ???????
bay where there are people desoldering
101
00:10:23,459 --> 00:10:29,190
e-waste and then sorting the parts into
bins and selling these salvaged components
102
00:10:29,190 --> 00:10:34,589
by the kilo and for a few extra renminbi
they'll put them on rails for you so that
103
00:10:34,589 --> 00:10:41,660
you can save a few pennies on your
production process. The other concern that
104
00:10:41,660 --> 00:10:46,489
these companies have, is not just salvaged
parts but straight up counterfeits.
105
00:10:46,489 --> 00:10:52,439
Especially for things that cost more than
a few dollars each. The Arduino community
106
00:10:52,439 --> 00:10:59,139
was hit a few years ago with a bunch of
counterfeit FTDI chips where the internal
107
00:10:59,139 --> 00:11:07,600
construction was entirely different. In
this case it caused reliability issues but
108
00:11:07,600 --> 00:11:11,550
you can imagine from a security
perspective this is really worrisome that
109
00:11:11,550 --> 00:11:15,709
parts that look identical might have
completely different functionality inside
110
00:11:15,709 --> 00:11:25,379
of them. Super Micro also mentions that
they X-ray their main boards to look for
111
00:11:25,379 --> 00:11:32,000
anomalies and I wasn't able to take any
photos inside the factory there was doing
112
00:11:32,000 --> 00:11:38,230
x-rays. But in this Wikipedia photo we can
clearly see active components like this
113
00:11:38,230 --> 00:11:45,670
SOIC chip are different from things like
the SMD resistors and capacitors. So if an
114
00:11:45,670 --> 00:11:51,220
attacker were trying to subvert the supply
chain by putting a disguise component it
115
00:11:51,220 --> 00:11:56,670
could be detected at this step. Another
interesting thing in this photo are these
116
00:11:56,670 --> 00:12:02,680
inductors that are encased in dip
packages. This is really common in a lot
117
00:12:02,680 --> 00:12:07,439
of Ethernet boards and occasionally people
have thought they had some sort of
118
00:12:07,439 --> 00:12:13,589
hardware implant when they found inductors
in their ethernet jacks but it's pretty
119
00:12:13,589 --> 00:12:19,799
it's fairly common and it shows it pretty
clearly on the x-ray. Some other security
120
00:12:19,799 --> 00:12:26,069
researchers like Sophia D'Antoine did an
extensive teardown of Super Micro boards
121
00:12:26,069 --> 00:12:33,439
including X-ray analysis and her group
found a few oddities but nothing.. they
122
00:12:33,439 --> 00:12:37,529
didn't find anything malicious. There were
no smoking guns. They just appeared to be
123
00:12:37,529 --> 00:12:43,190
sort of supply chain type things. You can
read her blog post for more details about
124
00:12:43,190 --> 00:12:49,319
where they found things that shouldn't
have been there. But turned out to be just
125
00:12:49,319 --> 00:13:00,879
actual signal condition components. So
super micro in their ???? letter, they
126
00:13:00,879 --> 00:13:07,239
keep reenforcing that the manufacturing
process that is the assembly process, it's
127
00:13:07,239 --> 00:13:11,179
during the manufacturing process and I
agree with them. It would be very
128
00:13:11,179 --> 00:13:17,939
difficult to circumvent security in a
reasonable way in that part of the
129
00:13:17,939 --> 00:13:23,189
process. But that's not the only place
this could happen. We know that national
130
00:13:23,189 --> 00:13:30,309
security agencies intercept shipments of
computer hardware and then have their
131
00:13:30,309 --> 00:13:37,249
tailored access operations open the
computers, install hardware implants,
132
00:13:37,249 --> 00:13:43,670
reseal them and then have them continue on
their way in shipment. The NSA even has a
133
00:13:43,670 --> 00:13:51,199
catalog of hardware implants like this
JTAG implant Ethernet jacks with embedded
134
00:13:51,199 --> 00:13:57,009
computers in them as well as firmware
specific ones that target servers SNM(?)
135
00:13:57,009 --> 00:14:05,490
and then some that can do data
exfiltration via RF. So that's sort of
136
00:14:05,490 --> 00:14:09,481
tailored access operations is really ideal
for this supply chain attack because it
137
00:14:09,481 --> 00:14:16,699
allows them to contain the exploit to a
single customer. It allows them fairly
138
00:14:16,699 --> 00:14:21,180
good concealment as well as good cover
that if it's discovered it's really hard
139
00:14:21,180 --> 00:14:25,769
to attribute where things went wrong. Now
unlike if you find something inside your
140
00:14:25,769 --> 00:14:34,230
motherboard between the layers you know
that had to have happened at the factory.
141
00:14:34,230 --> 00:14:47,040
So Super Micro also claim that this was
technically implausible, that it was
142
00:14:47,040 --> 00:14:52,559
highly unlikely that unauthorized hardware
would function properly because a third
143
00:14:52,559 --> 00:15:02,470
party with lack of complete knowledge of
the design. I think that's inaccurate,
144
00:15:02,470 --> 00:15:07,639
both because we know the NSA does it and
also because I have done it.
145
00:15:07,639 --> 00:15:10,319
laughter
146
00:15:10,319 --> 00:15:16,059
Really, all that you need to know is that
these are common components. These flash
147
00:15:16,059 --> 00:15:20,310
chips show up on all the boards. You can
search the internet for the data sheet and
148
00:15:20,310 --> 00:15:25,989
find exactly how it's wired into the rest
of the system. And the only thing that we
149
00:15:25,989 --> 00:15:33,499
need to know to communicate to the BMC is
the serial output pin from this component,
150
00:15:33,499 --> 00:15:43,429
so the BMC flash is connected over to the
BMC CPU via the serial output and it goes
151
00:15:43,429 --> 00:15:51,589
through a small series resistor and that
is where my implant goes in. Mine's a
152
00:15:51,589 --> 00:15:56,670
little bit larger than that resistor. It
clicks onto the board and it has a small
153
00:15:56,670 --> 00:16:03,009
FPGA that hangs offside but it's
completely plausible to fit it into
154
00:16:03,009 --> 00:16:12,139
something that small in fact a modern ARM
M0 fits in the space of two transistors
155
00:16:12,139 --> 00:16:18,350
from a 65 002 from a few years ago. The
Moore's Law means we can pack an amazing
156
00:16:18,350 --> 00:16:28,329
amount of CPU into a very very small
amount of space. So on that 0 6 0 3
157
00:16:28,329 --> 00:16:36,100
resistor could fit around 100 cortex M0 it
would be plenty powerful for this system.
158
00:16:36,100 --> 00:16:42,379
The problem is we only have those two pins
so ordinarily on the spy flashing you need
159
00:16:42,379 --> 00:16:47,720
at least six pens but we don't have power
and ground so we have to passively power
160
00:16:47,720 --> 00:16:53,059
this through the data signal that's
passing through it. We don't have the chip
161
00:16:53,059 --> 00:16:59,959
select pin so we have to guess when this
chip has been talked to. We don't have the
162
00:16:59,959 --> 00:17:04,980
data input pin so we don't know what
addresses are being read or what commands
163
00:17:04,980 --> 00:17:11,190
are being sent. We have to reconstruct it
from the data output pin and we also don't
164
00:17:11,190 --> 00:17:18,900
have a clock pin so we have to figure out
how to synchronize to that clock. Lastly
165
00:17:18,900 --> 00:17:22,890
we don't have the ability to make
arbitrary data changes. All we can do is
166
00:17:22,890 --> 00:17:29,060
disconnect the pin from the BMC so we can
only turn 1 bits into 0 bits. We can't go
167
00:17:29,060 --> 00:17:35,300
the other way around. So with these
limitations we can still do some pretty
168
00:17:35,300 --> 00:17:40,920
interesting things. Recovering the clock
is actually pretty easy. We can look at
169
00:17:40,920 --> 00:17:49,670
the data stream and find the shortest bit
transitions from 0 1 0 or 1 0 1 to
170
00:17:49,670 --> 00:17:55,060
estimate what the clock is which allows us
to then reconstruct that data stream being
171
00:17:55,060 --> 00:18:00,870
sent to the BMC and if we look at the
flash contents we can see that a lot of it
172
00:18:00,870 --> 00:18:07,570
is being fairly random noise but a lot of
it is all white which in this case would
173
00:18:07,570 --> 00:18:15,110
mean that it's all one bits. So if we look
at the way the flash is organized we can
174
00:18:15,110 --> 00:18:19,380
see there's the u-boot bootloader and
that's executable. That's kind of
175
00:18:19,380 --> 00:18:25,230
difficult to make useful changes in, the
kernel and the root file system are both
176
00:18:25,230 --> 00:18:33,040
compressed so that they look effectively
like random noise but the nvram region is
177
00:18:33,040 --> 00:18:41,660
a jffs2 file system and this file system
??? 3 Megs, it's mostly empty and all that
178
00:18:41,660 --> 00:18:50,040
empty space is F F which is all ones. So
this is plenty of ones for us to work on.
179
00:18:50,040 --> 00:18:55,380
Additionally it has fairly nice headers
that we can we can match on. So when we
180
00:18:55,380 --> 00:19:00,570
see these magic bit masks we know when
we've entered different parts of the file
181
00:19:00,570 --> 00:19:06,990
system. So given that we can now
reconstruct the clock we can figure out
182
00:19:06,990 --> 00:19:13,310
where we are in the file system. This
hardware implant can start to inject new
183
00:19:13,310 --> 00:19:20,320
data into what was the empty space. So
this short file that we put in here is a
184
00:19:20,320 --> 00:19:28,020
small shell script and it is one of the
network configuration scripts, so this is
185
00:19:28,020 --> 00:19:37,350
where I'm going to try a live demo and I
hope this works. We're running in qemu
186
00:19:37,350 --> 00:19:45,660
since I didn't bring a Super Micro board
and what we have on the left is the flash
187
00:19:45,660 --> 00:19:50,530
console excuse me the hardware implant
console. And then on the right we have the
188
00:19:50,530 --> 00:19:57,353
serial console from the BMC so we can see
it has loaded the kernel and in a second
189
00:19:57,353 --> 00:20:03,430
it's going to we should see a bunch of
traffic, okay, so the implant is active.
190
00:20:03,430 --> 00:20:10,450
It has replaced the data when that nvram
file system was mounted the BMC is now
191
00:20:10,450 --> 00:20:18,780
continuing on doing its set up. It's going
to load a bunch of device drivers for that
192
00:20:18,780 --> 00:20:24,250
video. It pauses here for some reason that
I haven't diagnosed because that's that's
193
00:20:24,250 --> 00:20:27,040
not my job.
194
00:20:27,040 --> 00:20:29,140
laughter
195
00:20:29,140 --> 00:20:33,020
And eventually it's going to configure the
networks and it does that by running that
196
00:20:33,020 --> 00:20:43,010
shell script off of the nvram partition
here it starts KVM stuff brings up some
197
00:20:43,010 --> 00:20:53,390
things. Allright.
applause
198
00:20:53,390 --> 00:21:01,920
OK. So luckily we got to that point
without having to fake the demo. In the
199
00:21:01,920 --> 00:21:07,820
hardware it's really flaky. My version
works about one in eight times. But it
200
00:21:07,820 --> 00:21:12,041
doesn't typically cause a crash. So that's
actually good for concealment because it
201
00:21:12,041 --> 00:21:17,850
becomes now much harder to determine which
machines are affected. In qemu because
202
00:21:17,850 --> 00:21:21,640
it's emulating, it's a little more
reliable but it's still it's only two out
203
00:21:21,640 --> 00:21:26,760
of three. If we let the BMC boot a little
bit further it actually prints out this
204
00:21:26,760 --> 00:21:32,120
message. And if you hit enter it drops you
to a shell with no password and you can
205
00:21:32,120 --> 00:21:38,170
then just run commands as root on the BMC
and that's a lot easier than all this
206
00:21:38,170 --> 00:21:43,440
stuff with the SPI bus if you wanted to
build a hardware implant against it. I
207
00:21:43,440 --> 00:21:48,540
don't know where the serial port is on the
on the Super Micro but on a different tier
208
00:21:48,540 --> 00:21:54,030
1 server mainboard I was able to probe
around the oscilloscope and locate the
209
00:21:54,030 --> 00:22:00,830
serial console for the BMC. Figure out
it's 115 kbaud and it has the same code
210
00:22:00,830 --> 00:22:06,050
that you hit enter and you can run
commands there. So that's a much easier
211
00:22:06,050 --> 00:22:11,990
way to do it. A big question a lot of
people have is how do we actually detect
212
00:22:11,990 --> 00:22:18,100
this sort of flash implant. A lot of high
assurance sites replace all of their roms
213
00:22:18,100 --> 00:22:22,760
with ones that they flash themselves but
that doesn't get rid of the implant
214
00:22:22,760 --> 00:22:28,960
because it's outside of the ROM chip.
Likewise reading the ROM chip doesn't show
215
00:22:28,960 --> 00:22:35,321
anything because it's not in the ROM
itself it's it's outside of it. Even
216
00:22:35,321 --> 00:22:40,650
hooking up a logic analyzer to the bus and
watching as the machine boots and seeing
217
00:22:40,650 --> 00:22:45,780
the data stream coming out of the flash
won't actually reveal the implant because
218
00:22:45,780 --> 00:22:51,770
you'd have to put the logic probes on the
PGA pads on the flat on the BMC itself.
219
00:22:51,770 --> 00:22:58,140
And that's a much harder task. Some people
think "oh well we can see the weird
220
00:22:58,140 --> 00:23:03,150
network traffic when the BMC tries to
exfiltrate the data" but that would be
221
00:23:03,150 --> 00:23:08,030
that's only one way for the BMC to affect
things. There is a great talk a few years
222
00:23:08,030 --> 00:23:13,410
ago at DefCon from Intel ATR where they
showed how something that can control the
223
00:23:13,410 --> 00:23:19,020
system firmware can backdoor hypervisors.
And then they gave a use case where a
224
00:23:19,020 --> 00:23:26,180
unprivileged guest on a cloud system could
read all of the rest of physical memory so
225
00:23:26,180 --> 00:23:34,760
it could see all of the other guests
memory. So what do we do? The big problems
226
00:23:34,760 --> 00:23:39,560
is the BMC has way too many privileges.
It's connected to pretty much everything
227
00:23:39,560 --> 00:23:46,650
in the system but the BMC is not our only
concern. As @whitequark said, our PCs are
228
00:23:46,650 --> 00:23:52,300
just a bunch of embedded devices in a
trench coat and they all have firmware. In
229
00:23:52,300 --> 00:23:56,680
fact pretty much everything on your system
more complex than a resistor probably has
230
00:23:56,680 --> 00:24:01,270
firmware and if you have one of those
Super Micro implants maybe even your
231
00:24:01,270 --> 00:24:08,500
resistors have firmware as well. I've
found that the firmware and things like
232
00:24:08,500 --> 00:24:15,150
the power supplies can be used to gain
code execution on the BMC. It's really
233
00:24:15,150 --> 00:24:20,750
interesting how tightly connected all of
our systems are. And as Joe Fit's pointed
234
00:24:20,750 --> 00:24:26,700
out in his blackhat ???? talk, these are
not multimillion dollar attacks these are
235
00:24:26,700 --> 00:24:33,500
five euro bits of hardware that we now
have to really be worried about. I really
236
00:24:33,500 --> 00:24:38,480
like the guidelines that NIST has
published that suggests that we think
237
00:24:38,480 --> 00:24:43,650
about our systems more in this holistic
manner. Although the interpreting pretty
238
00:24:43,650 --> 00:24:50,290
much everything into the TPM is the
trusted platform module for doing this
239
00:24:50,290 --> 00:24:55,580
attestation and I think we as a community
need to do more to use the TPM. There
240
00:24:55,580 --> 00:25:01,060
actually a really good tool for securing
our systems but they are also potentially
241
00:25:01,060 --> 00:25:08,030
subject to their own hardware implants.
The NCC Group TPM genie is able to subvert
242
00:25:08,030 --> 00:25:14,600
the core root of trust by interposing on
the TPM. So a lot of folks are proposing
243
00:25:14,600 --> 00:25:19,160
we should move to other trusted execution
environments like SGX or Trustzone. And I
244
00:25:19,160 --> 00:25:24,960
think these have a lot of promise
especially for trusted cloud computing.
245
00:25:24,960 --> 00:25:30,970
There also is a lot of innovation in the
hardware roots of trust going on right now
246
00:25:30,970 --> 00:25:34,860
between the Google Titan, which initially
was for their servers and is now showing
247
00:25:34,860 --> 00:25:39,740
up on all of their chrome books. The
Microsoft Cerberus chip which again is the
248
00:25:39,740 --> 00:25:46,710
Azure system. They're actually publishing
their firmware and the ASIC design so that
249
00:25:46,710 --> 00:25:49,880
people can have a little more faith in it
and they hope it will become an open
250
00:25:49,880 --> 00:25:56,780
standard. And companies like Apple have
also gone their own way. With the T2 and
251
00:25:56,780 --> 00:26:00,620
the T2's are really amazing chip for
securing systems. But it does so at the
252
00:26:00,620 --> 00:26:06,790
expense of user freedom and that gets in
the way of what I think the real way that
253
00:26:06,790 --> 00:26:11,130
we need to.. we need to solve this
problem. We need to get rid of a lot of
254
00:26:11,130 --> 00:26:18,830
these secrets. Counter to what the Super
Micro CEO said, having a secret
255
00:26:18,830 --> 00:26:22,770
motherboard design does not make you more
secure. Things like the Open Compute
256
00:26:22,770 --> 00:26:27,140
hardware I think is a good vision for how
we can move forward that when you buy an
257
00:26:27,140 --> 00:26:33,030
Open Compute server it comes with full
schematics and gerber files. So that
258
00:26:33,030 --> 00:26:37,910
motivated customers can verify that the
systems that they're buying are the ones
259
00:26:37,910 --> 00:26:42,140
that they think they that they're buying
that all of the components are what they
260
00:26:42,140 --> 00:26:49,250
think they should be. I think the firmware
also needs more openness. Ronald Minnich,
261
00:26:49,250 --> 00:26:56,150
Google is my co-lead on Linux boot project
and we think that Linux in the firmware is
262
00:26:56,150 --> 00:27:03,821
a way forward to get a more secure more
flexible and more resilient system. We're
263
00:27:03,821 --> 00:27:09,981
working with a spin off project called
micro BMC that is using the Linux boot
264
00:27:09,981 --> 00:27:16,580
tools to build BMC firmware and this is
opensource. It's reproducibly built it can
265
00:27:16,580 --> 00:27:22,740
work with roots of trust attestation. It's
written in a memory safe language since
266
00:27:22,740 --> 00:27:27,740
it's a Google collaboration and go. And
more importantly we've thrown away all of
267
00:27:27,740 --> 00:27:31,240
the legacy features that have been a
source of a lot of security
268
00:27:31,240 --> 00:27:40,960
vulnerabilities in these systems. So did
it happen? I don't know. Is it technically
269
00:27:40,960 --> 00:27:44,520
possible? I think so. I hope I've
convinced all of you that this is
270
00:27:44,520 --> 00:27:50,770
definitely a technical possibility that we
need to be concerned about and I hope that
271
00:27:50,770 --> 00:27:56,260
the way forward through hardware roots of
trust with attestation and more
272
00:27:56,260 --> 00:28:01,400
importantly with open hardware so that we
know that what the machines were running
273
00:28:01,400 --> 00:28:07,130
are running code that we know.. the code
that we've built that we understand and
274
00:28:07,130 --> 00:28:13,080
that we can actually have a good chance of
being able to take control back of them.
275
00:28:13,080 --> 00:28:18,300
If you're interested in more discussion on
this and also on open firmware, there's an
276
00:28:18,300 --> 00:28:23,850
assembly here in this hall that has a
bunch folks working on a core boot and
277
00:28:23,850 --> 00:28:29,110
Linux boot and a lot of these projects
where you can help contribute and you can
278
00:28:29,110 --> 00:28:37,510
help also pressure vendors to make these
this standard and a way forward for a more
279
00:28:37,510 --> 00:28:42,000
secure computing. So thank you all for
coming. And I really enjoyed the chance to
280
00:28:42,000 --> 00:28:50,380
show off my modship of the state.
281
00:28:50,380 --> 00:28:56,030
applause
282
00:28:56,030 --> 00:29:02,600
Herald: Geat talk, thank you very much
Trammel. We have 10 minutes for questions
283
00:29:02,600 --> 00:29:11,080
so please line up at the microphones if
you have questions. And we also have a
284
00:29:11,080 --> 00:29:25,390
signal angel probably with questions from
the internet. So any questions? Microphone
285
00:29:25,390 --> 00:29:29,870
number three?
Mic 3: Yes, I was going to ask, what's
286
00:29:29,870 --> 00:29:35,650
your opinion on the Talos systems? The
openPOWER based ones?
287
00:29:35,650 --> 00:29:41,830
Trammell: So the question is about the
Talos power 9 based systems power 9 is a
288
00:29:41,830 --> 00:29:48,490
really interesting architecture. The.. it
is using a open firmware very similar to
289
00:29:48,490 --> 00:29:55,250
Linux boot called Petitboot that
moves Linux into the bootloader. I'm a big
290
00:29:55,250 --> 00:29:58,860
fan. There's a lot of folks in the
opensource community who are very excited
291
00:29:58,860 --> 00:30:07,540
about it. I'm hoping that there would be
more power nine systems coming out. I'm
292
00:30:07,540 --> 00:30:13,130
also very excited about the RISC-V
systems. I think having open source CPUs
293
00:30:13,130 --> 00:30:19,310
use is a real way that we can have more
assurance that our systems are what we
294
00:30:19,310 --> 00:30:22,800
think they are.
Herald: Thank you, microphone number two
295
00:30:22,800 --> 00:30:27,100
please.
Mic 2: Yes, thanks for the talk. I was
296
00:30:27,100 --> 00:30:32,810
wondering if you have just a scope probe
over this serial, cause it's just a serial
297
00:30:32,810 --> 00:30:37,320
resistor which we're replacing. If you put
just two scope probes on there and measure
298
00:30:37,320 --> 00:30:41,270
the voltage over it, in your situation
would the voltage change there once in a
299
00:30:41,270 --> 00:30:42,400
while?
Trammell: Yes, yes, yes.
300
00:30:42,400 --> 00:30:46,540
Mic 2: Well okay, in the normal case would
it actually be quite consistent current.
301
00:30:46,540 --> 00:30:56,890
Or if you lowered the input impedance of
the BMC chip who might already have fixed
302
00:30:56,890 --> 00:31:01,760
a part of the attack because the output
sourcing current of your exploit is
303
00:31:01,760 --> 00:31:04,900
probably limited due to the limited supply
you only can..
304
00:31:04,900 --> 00:31:12,390
Herald: Your question please?
Mic 2: Yes.. but.. do you see a way to get
305
00:31:12,390 --> 00:31:17,710
more power into your setup? Maybe using,
well other power sources, other than the
306
00:31:17,710 --> 00:31:22,650
two pins, or maybe somewhere of..
Trammell: Well, so the question is about,
307
00:31:22,650 --> 00:31:28,420
would there be a way to do more arbitrary
changes through redesigning the implant.
308
00:31:28,420 --> 00:31:34,190
One of the goals was to fit with only
those two pins so that a single piece on
309
00:31:34,190 --> 00:31:38,900
the motherboard could be replaced. With a
dual probe soldering iron and you can pop
310
00:31:38,900 --> 00:31:45,500
it out and stick a new one down in a
matter of seconds. So, yes, if you have
311
00:31:45,500 --> 00:31:51,809
more pins where you can get more power
from you can do much more interesting
312
00:31:51,809 --> 00:31:57,460
things. But that's.. would require a
different set of changes to the
313
00:31:57,460 --> 00:32:02,480
motherboard.
Herald: Thank you. Microphone 1 please.
314
00:32:02,480 --> 00:32:09,350
Mic 1: So, a lot of the -like- arguments
that these implants were not feasible by a
315
00:32:09,350 --> 00:32:13,820
Super Micro where you also show the
picture from the fab that you had to
316
00:32:13,820 --> 00:32:19,390
change the etching and the optical
inspection and so on and so on. But how
317
00:32:19,390 --> 00:32:27,870
probable would you rate the fact that some
acto just intercepted the manufacturing
318
00:32:27,870 --> 00:32:33,570
files and added that component already in
the file because then all the optical
319
00:32:33,570 --> 00:32:38,810
inspection and that would all say well
that matches what was sent to us. But that
320
00:32:38,810 --> 00:32:41,650
was not necessarily what Super Micro sent
to the fab.
321
00:32:41,650 --> 00:32:44,900
Trammell: So the question is, could
someone have modified all of the
322
00:32:44,900 --> 00:32:48,620
manufacturing files that went to the
factory, and that's absolutely a
323
00:32:48,620 --> 00:32:54,520
possibility. But that's also very likely
that that would be detected by Super Micro
324
00:32:54,520 --> 00:33:01,170
itself that in a lot of cases you don't
necessarily want to trust the company that
325
00:33:01,170 --> 00:33:05,930
is making the product to also test it. And
you probably want to have a separate
326
00:33:05,930 --> 00:33:11,059
company that does random spot checks to
verify that the boards are actually being
327
00:33:11,059 --> 00:33:16,460
produced to the specification that you..
that you desire. So it's certainly
328
00:33:16,460 --> 00:33:24,050
possible and I really don't want to
speculate as to the accuracy of that part
329
00:33:24,050 --> 00:33:31,030
of the story but yeah it would require
quite a bit more changes. And also would
330
00:33:31,030 --> 00:33:34,679
be much more likely to be detected in the
spot check.
331
00:33:34,679 --> 00:33:38,230
Herald: Great. Microphone number two
please.
332
00:33:38,230 --> 00:33:44,510
Mic 2: Yes, for a lot of motherboards
there are also quite a few components not
333
00:33:44,510 --> 00:33:53,750
populated some of which are on which you
could consider sensitive myths. Wouldn't
334
00:33:53,750 --> 00:33:59,430
that make it. Yeah exactly. Wouldn't that
make it very easy to do just pop something
335
00:33:59,430 --> 00:34:04,540
on there in parallel with one of the
components and not have it be detected
336
00:34:04,540 --> 00:34:08,329
because it's like the board is modified.
There is a component or you have no way of
337
00:34:08,329 --> 00:34:11,490
telling whether it had to be populated or
not?
338
00:34:11,490 --> 00:34:18,599
Trammell: Super Micro puts a lot of extra
pads on the board in this one particular
339
00:34:18,599 --> 00:34:28,700
one they have both 8 pin and 16 pin flash
chip pads that are just in parallel
340
00:34:28,700 --> 00:34:32,989
together. So depending on which chip is
cheaper that day of the week or who knows
341
00:34:32,989 --> 00:34:38,419
what, they will populate one or the other.
So that's why in this particular photo
342
00:34:38,419 --> 00:34:47,950
having the position of that circle on the
data output pin is very very interesting.
343
00:34:47,950 --> 00:34:56,659
Herald: Question answered? Okay. So one
more question on microphone number two
344
00:34:56,659 --> 00:35:00,400
please?
Mic 2: How far can signing of firmware be
345
00:35:00,400 --> 00:35:06,470
a solution to this problem?
Trammell: Signing firmware solves a lot of
346
00:35:06,470 --> 00:35:13,401
the issues. It does however not all
typically not all of the firmware are
347
00:35:13,401 --> 00:35:21,020
signed specifically is probably to be
signed in in a modern BMC. The kernel and
348
00:35:21,020 --> 00:35:25,789
maybe the root file system might be
signed. But the envy of RAM file system in
349
00:35:25,789 --> 00:35:32,589
this BMC is designed to be user modifiable
so it can't be signed by the manufacturer,
350
00:35:32,589 --> 00:35:41,340
so this sort of attack would work against
a signed BMC just as well. Also the "Hit
351
00:35:41,340 --> 00:35:49,509
enter to get a serial console" attack
circumvents any signing. There are things
352
00:35:49,509 --> 00:35:56,140
on the host firmware on the x86 like boot
card that do a really good job of making
353
00:35:56,140 --> 00:36:01,520
it harder to get code execution during the
boot process. But there have been several
354
00:36:01,520 --> 00:36:07,720
CVEs where it has been implemented poorly.
So even though signature's the firmware is
355
00:36:07,720 --> 00:36:13,800
signed, people have still managed to get
code execution during that process.
356
00:36:13,800 --> 00:36:18,329
Herald: Great. Thank you Trammell Hudson
again, a warm round of applause, thank you
357
00:36:18,329 --> 00:36:21,009
very much!
358
00:36:21,009 --> 00:36:24,009
applause
359
00:36:24,009 --> 00:36:25,529
35c3 postrol music
360
00:36:25,529 --> 00:36:52,000
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!