35C3 - Modchips of the State

35c3 prerol music

Herald: So Trammell Hudson, who is
standing here, he's taking things apart.

Don't worry not life on stage, but he will
give us a proof of concept and some

details and functionalities about hardware
implants. So the same things that we heard

from Bloomberg article talking about Apple
and super microcomputers with implants

that, yeah, were implanted into those,
into those computers. And I'm really

excited to see this in action. Please give
a warm round of applause to Trammel

Hudson!

applause

Trammell: Before we begin talking about
hardware implants just two quick

disclaimers. The first from my employer
Two Sigma investments as it says are

chocolate bars. This is not investment
advice. And secondly I don't actually know

what the story is behind the super micro
story. No one outside of Bloomberg and

their sources do. But I have spent a lot
of time thinking about hardware implants

starting with the thunderstrike firmware
attack against mac books as well as the

thunderstrike 2 where we were able to get
software to write into the firmware on the

mac books. I've also been thinking a lot
about how to defend against hardware

implants with things like the heads
firmware for slightly more secure laptops

and also as part of my co-leas on the
Linux boot project. We're thinking about

how to protect servers from physical and
software attacks. So with all of this

concentrated thinking about firmware and
hardware attacks, I was really excited

when I saw the Bloomberg story back in
October. But what really intrigued me was

the animated image that they had at the
header that highlighted one small part of

the board as where the implant was, but
what I found really interesting is that is

exactly where I would install a hardware
implant as they described on the SPI bus.

A lot of other people in the hardware and
from our security community thought it

sounded plausible. Other people pointed
out that supply chain attacks come up

periodically and they are definitely a
concern. Some people thought the attack as

described was entirely implausible and in
general we sort of had a Whiskey Tango

Foxtrot moment as everybody scrambled to
figure out what's going on inside their

machines. So, let's step back very quickly
and review what the key claims that

Bloomberg alleged happened. First they
said that Amazon's testers found a tiny

microchip that wasn't part of the board's
original design that had been disguised to

look like a signaling condition signal
condition coupler and that these illicit

chips were connected to the baseboard
management controller or the BMC which

gives them access to machines that were
turned off. That might sound kind of

extreme, but that's actually what the role
of the BMC is, that in most servers the

BMC is running any time the machine is
hooked up to power and it's connected to

the power supplies so that it can turn the
machine on and turn it off. Frequently you

want to be able to do this over a network
so it has its own dedicated LAN port but

it can also share the LAN port with the
with the main system. Serial over LAN is a

really useful way to debug the systems so
it provides that functionality. It can

also provide fake USB volumes to allow to
to do unintended OS installation. A lot of

sites also won't remote KVM so it has VGA
but that VGA support means that it's on

the PCIe BUS and because some PCIe it can
do DMA into main memory. It also is

typically mixed(?) into the SPI flash for
the host firmware, which allows it to

modify it and on some systems it's even
connected to the TPM which allows it to

circumvent the corporate of trust. So with
all of this capability inside this chip

it's really unfortunate that they are
really not well put together. The head of

Azure security says they have no
protection against attacks. There's no

ability to detect if an attack has
happened and there's no ability to recover

from an attack. So having a hardware
implant on the BMC is a really big

concern. The other claim in the article is
that it affected 30 different companies

including Apple and Bloomberg alleges that
Apple found malicious chips independently

on their super micro boards. Went to the
FBI about it and that they then severed

ties with Super Micro. This particular
claim was interesting because it

corroborated a story that had shown up
back in early 2017 that Apple had removed

Super Micro from their data centers. Apple
denied that there was a firmware issue.

But it's interesting that perhaps these
two were related. The third set of claims

is that on some of these implants they
were actually put between the layers on

the PCB and then the most explosive claim
is that this was done by operatives from

China, the Chinese People's Liberation
Army. With a story with this you know this

many claims and this significant of
allegations we'd hoped that it would be

really well sourced and for a normal story
17 independent sources that Bloomberg

editors agreed to grant anonymity to,
including six national security, two

people inside of AWS and three senior
insiders at Apple seems like pretty solid

sourcing, except as soon as this article
is published everyone denied it. The

Director of National Intelligence said
they'd seen no evidence of this. Amazon

said that they've never found any issues
of modified hardware nor have they been

engaged with the government over it. Apple
was even more blunt. CEO Tim Cook said

this did not happen. There is no truth to
this. And Super Micro wrote a fairly

lengthy letter about what they do to
protect their supply chain and why they

think this attack did not happen. And it
is worth going through to look at some of

the things that they say that they do to
protect their supply chain. They point out

that if there's any unauthorized physical
alterations during the manufacturing

process other design elements would not
match and those things would be detected.

To sort of understand how circuit boards
are made, I recently visited a PCB factory

in Guangzhou. This is not a super micro
factory. This is just a holiday photos. So

in order to add new vias they would have
to modify the drill files which would then

get electroplated. If they had to add new
traces, they would have to be able to

subvert the masking and etching process
and any changes to either the drills or

the etching on individual layers would be
caught by the optical inspection that's

done on these bare circuit boards.
Additionally the allegation that things

were inserted between circuit boards would
require that the lamination process be

subverted and that the implant somehow
aligned into the system. If that implant

changes any of the connectivity the flying
protesters would pick it up or the bed of

nails testers which checks all of the
connectivity of all the traces to make

sure that there are no shorts and to make
sure that everything that is supposed to

be connected is electrically conductive.
So it would be very difficult to

circumvent the production process at this
stage. And it also would be very difficult

to contain because the PCB factory doesn't
know which customers are going to receive

those circuit boards. Super Micro also
points out that during the assembly

process when the parts are installed they
have their employees on site the whole

time. On my same holiday trip I also
visited some PCB assembly companies and

spoke with companies that are using doing
contract manufacturing and they said that

they also send their employees to the
production line to observe the pick and

place machines and the reflow and the rest
of the surface mount assembly. Their big

concern is that if they don't have someone
there the parts that are fed in the pick

in place will be replaced with either
counterfeits or with salvaged parts. I

visited the electronics market in ???????
bay where there are people desoldering

e-waste and then sorting the parts into
bins and selling these salvaged components

by the kilo and for a few extra renminbi
they'll put them on rails for you so that

you can save a few pennies on your
production process. The other concern that

these companies have, is not just salvaged
parts but straight up counterfeits.

Especially for things that cost more than
a few dollars each. The Arduino community

was hit a few years ago with a bunch of
counterfeit FTDI chips where the internal

construction was entirely different. In
this case it caused reliability issues but

you can imagine from a security
perspective this is really worrisome that

parts that look identical might have
completely different functionality inside

of them. Super Micro also mentions that
they X-ray their main boards to look for

anomalies and I wasn't able to take any
photos inside the factory there was doing

x-rays. But in this Wikipedia photo we can
clearly see active components like this

SOIC chip are different from things like
the SMD resistors and capacitors. So if an

attacker were trying to subvert the supply
chain by putting a disguise component it

could be detected at this step. Another
interesting thing in this photo are these

inductors that are encased in dip
packages. This is really common in a lot

of Ethernet boards and occasionally people
have thought they had some sort of

hardware implant when they found inductors
in their ethernet jacks but it's pretty

it's fairly common and it shows it pretty
clearly on the x-ray. Some other security

researchers like Sophia D'Antoine did an
extensive teardown of Super Micro boards

including X-ray analysis and her group
found a few oddities but nothing.. they

didn't find anything malicious. There were
no smoking guns. They just appeared to be

sort of supply chain type things. You can
read her blog post for more details about

where they found things that shouldn't
have been there. But turned out to be just

actual signal condition components. So
super micro in their ???? letter, they

keep reenforcing that the manufacturing
process that is the assembly process, it's

during the manufacturing process and I
agree with them. It would be very

difficult to circumvent security in a
reasonable way in that part of the

process. But that's not the only place
this could happen. We know that national

security agencies intercept shipments of
computer hardware and then have their

tailored access operations open the
computers, install hardware implants,

reseal them and then have them continue on
their way in shipment. The NSA even has a

catalog of hardware implants like this
JTAG implant Ethernet jacks with embedded

computers in them as well as firmware
specific ones that target servers SNM(?)

and then some that can do data
exfiltration via RF. So that's sort of

tailored access operations is really ideal
for this supply chain attack because it

allows them to contain the exploit to a
single customer. It allows them fairly

good concealment as well as good cover
that if it's discovered it's really hard

to attribute where things went wrong. Now
unlike if you find something inside your

motherboard between the layers you know
that had to have happened at the factory.

So Super Micro also claim that this was
technically implausible, that it was

highly unlikely that unauthorized hardware
would function properly because a third

party with lack of complete knowledge of
the design. I think that's inaccurate,

both because we know the NSA does it and
also because I have done it.

laughter

Really, all that you need to know is that
these are common components. These flash

chips show up on all the boards. You can
search the internet for the data sheet and

find exactly how it's wired into the rest
of the system. And the only thing that we

need to know to communicate to the BMC is
the serial output pin from this component,

so the BMC flash is connected over to the
BMC CPU via the serial output and it goes

through a small series resistor and that
is where my implant goes in. Mine's a

little bit larger than that resistor. It
clicks onto the board and it has a small

FPGA that hangs offside but it's
completely plausible to fit it into

something that small in fact a modern ARM
M0 fits in the space of two transistors

from a 65 002 from a few years ago. The
Moore's Law means we can pack an amazing

amount of CPU into a very very small
amount of space. So on that 0 6 0 3

resistor could fit around 100 cortex M0 it
would be plenty powerful for this system.

The problem is we only have those two pins
so ordinarily on the spy flashing you need

at least six pens but we don't have power
and ground so we have to passively power

this through the data signal that's
passing through it. We don't have the chip

select pin so we have to guess when this
chip has been talked to. We don't have the

data input pin so we don't know what
addresses are being read or what commands

are being sent. We have to reconstruct it
from the data output pin and we also don't

have a clock pin so we have to figure out
how to synchronize to that clock. Lastly

we don't have the ability to make
arbitrary data changes. All we can do is

disconnect the pin from the BMC so we can
only turn 1 bits into 0 bits. We can't go

the other way around. So with these
limitations we can still do some pretty

interesting things. Recovering the clock
is actually pretty easy. We can look at

the data stream and find the shortest bit
transitions from 0 1 0 or 1 0 1 to

estimate what the clock is which allows us
to then reconstruct that data stream being

sent to the BMC and if we look at the
flash contents we can see that a lot of it

is being fairly random noise but a lot of
it is all white which in this case would

mean that it's all one bits. So if we look
at the way the flash is organized we can

see there's the u-boot bootloader and
that's executable. That's kind of

difficult to make useful changes in, the
kernel and the root file system are both

compressed so that they look effectively
like random noise but the nvram region is

a jffs2 file system and this file system
??? 3 Megs, it's mostly empty and all that

empty space is F F which is all ones. So
this is plenty of ones for us to work on.

Additionally it has fairly nice headers
that we can we can match on. So when we

see these magic bit masks we know when
we've entered different parts of the file

system. So given that we can now
reconstruct the clock we can figure out

where we are in the file system. This
hardware implant can start to inject new

data into what was the empty space. So
this short file that we put in here is a

small shell script and it is one of the
network configuration scripts, so this is

where I'm going to try a live demo and I
hope this works. We're running in qemu

since I didn't bring a Super Micro board
and what we have on the left is the flash

console excuse me the hardware implant
console. And then on the right we have the

serial console from the BMC so we can see
it has loaded the kernel and in a second

it's going to we should see a bunch of
traffic, okay, so the implant is active.

It has replaced the data when that nvram
file system was mounted the BMC is now

continuing on doing its set up. It's going
to load a bunch of device drivers for that

video. It pauses here for some reason that
I haven't diagnosed because that's that's

not my job.

laughter

And eventually it's going to configure the
networks and it does that by running that

shell script off of the nvram partition
here it starts KVM stuff brings up some

things. Allright.
applause

OK. So luckily we got to that point
without having to fake the demo. In the

hardware it's really flaky. My version
works about one in eight times. But it

doesn't typically cause a crash. So that's
actually good for concealment because it

becomes now much harder to determine which
machines are affected. In qemu because

it's emulating, it's a little more
reliable but it's still it's only two out

of three. If we let the BMC boot a little
bit further it actually prints out this

message. And if you hit enter it drops you
to a shell with no password and you can

then just run commands as root on the BMC
and that's a lot easier than all this

stuff with the SPI bus if you wanted to
build a hardware implant against it. I

don't know where the serial port is on the
on the Super Micro but on a different tier

1 server mainboard I was able to probe
around the oscilloscope and locate the

serial console for the BMC. Figure out
it's 115 kbaud and it has the same code

that you hit enter and you can run
commands there. So that's a much easier

way to do it. A big question a lot of
people have is how do we actually detect

this sort of flash implant. A lot of high
assurance sites replace all of their roms

with ones that they flash themselves but
that doesn't get rid of the implant

because it's outside of the ROM chip.
Likewise reading the ROM chip doesn't show

anything because it's not in the ROM
itself it's it's outside of it. Even

hooking up a logic analyzer to the bus and
watching as the machine boots and seeing

the data stream coming out of the flash
won't actually reveal the implant because

you'd have to put the logic probes on the
PGA pads on the flat on the BMC itself.

And that's a much harder task. Some people
think "oh well we can see the weird

network traffic when the BMC tries to
exfiltrate the data" but that would be

that's only one way for the BMC to affect
things. There is a great talk a few years

ago at DefCon from Intel ATR where they
showed how something that can control the

system firmware can backdoor hypervisors.
And then they gave a use case where a

unprivileged guest on a cloud system could
read all of the rest of physical memory so

it could see all of the other guests
memory. So what do we do? The big problems

is the BMC has way too many privileges.
It's connected to pretty much everything

in the system but the BMC is not our only
concern. As @whitequark said, our PCs are

just a bunch of embedded devices in a
trench coat and they all have firmware. In

fact pretty much everything on your system
more complex than a resistor probably has

firmware and if you have one of those
Super Micro implants maybe even your

resistors have firmware as well. I've
found that the firmware and things like

the power supplies can be used to gain
code execution on the BMC. It's really

interesting how tightly connected all of
our systems are. And as Joe Fit's pointed

out in his blackhat ???? talk, these are
not multimillion dollar attacks these are

five euro bits of hardware that we now
have to really be worried about. I really

like the guidelines that NIST has
published that suggests that we think

about our systems more in this holistic
manner. Although the interpreting pretty

much everything into the TPM is the
trusted platform module for doing this

attestation and I think we as a community
need to do more to use the TPM. There

actually a really good tool for securing
our systems but they are also potentially

subject to their own hardware implants.
The NCC Group TPM genie is able to subvert

the core root of trust by interposing on
the TPM. So a lot of folks are proposing

we should move to other trusted execution
environments like SGX or Trustzone. And I

think these have a lot of promise
especially for trusted cloud computing.

There also is a lot of innovation in the
hardware roots of trust going on right now

between the Google Titan, which initially
was for their servers and is now showing

up on all of their chrome books. The
Microsoft Cerberus chip which again is the

Azure system. They're actually publishing
their firmware and the ASIC design so that

people can have a little more faith in it
and they hope it will become an open

standard. And companies like Apple have
also gone their own way. With the T2 and

the T2's are really amazing chip for
securing systems. But it does so at the

expense of user freedom and that gets in
the way of what I think the real way that

we need to.. we need to solve this
problem. We need to get rid of a lot of

these secrets. Counter to what the Super
Micro CEO said, having a secret

motherboard design does not make you more
secure. Things like the Open Compute

hardware I think is a good vision for how
we can move forward that when you buy an

Open Compute server it comes with full
schematics and gerber files. So that

motivated customers can verify that the
systems that they're buying are the ones

that they think they that they're buying
that all of the components are what they

think they should be. I think the firmware
also needs more openness. Ronald Minnich,

Google is my co-lead on Linux boot project
and we think that Linux in the firmware is

a way forward to get a more secure more
flexible and more resilient system. We're

working with a spin off project called
micro BMC that is using the Linux boot

tools to build BMC firmware and this is
opensource. It's reproducibly built it can

work with roots of trust attestation. It's
written in a memory safe language since

it's a Google collaboration and go. And
more importantly we've thrown away all of

the legacy features that have been a
source of a lot of security

vulnerabilities in these systems. So did
it happen? I don't know. Is it technically

possible? I think so. I hope I've
convinced all of you that this is

definitely a technical possibility that we
need to be concerned about and I hope that

the way forward through hardware roots of
trust with attestation and more

importantly with open hardware so that we
know that what the machines were running

are running code that we know.. the code
that we've built that we understand and

that we can actually have a good chance of
being able to take control back of them.

If you're interested in more discussion on
this and also on open firmware, there's an

assembly here in this hall that has a
bunch folks working on a core boot and

Linux boot and a lot of these projects
where you can help contribute and you can

help also pressure vendors to make these
this standard and a way forward for a more

secure computing. So thank you all for
coming. And I really enjoyed the chance to

show off my modship of the state.

applause

Herald: Geat talk, thank you very much
Trammel. We have 10 minutes for questions

so please line up at the microphones if
you have questions. And we also have a

signal angel probably with questions from
the internet. So any questions? Microphone

number three?
Mic 3: Yes, I was going to ask, what's

your opinion on the Talos systems? The
openPOWER based ones?

Trammell: So the question is about the
Talos power 9 based systems power 9 is a

really interesting architecture. The.. it
is using a open firmware very similar to

Linux boot called Petty(??) boot that
moves Linux into the bootloader. I'm a big

fan. There's a lot of folks in the
opensource community who are very excited

about it. I'm hoping that there would be
more power nine systems coming out. I'm

also very excited about the brisque five
systems. I think having open source CPUs

use is a real way that we can have more
assurance that our systems are what we

think they are.
Herald: Thank you, microphone number two

please.
Mic 2: Yes, thanks for the talk. I was

wondering if you have just a scope probe
over this serial, cause it's just a serial

resistor which we're replacing. If you put
just two scope probes on there and measure

the voltage over it, in your situation
would the voltage change there once in a

while?
Trammell: Yes, yes, yes.

Mic 2: Well okay, in the normal case would
it actually be quite consistent current.

Or if you lowered the input impedance of
the BMC chip who might already have fixed

a part of the attack because the output
sourcing current of your exploit is

probably limited due to the limited supply
you only can..

Herald: Your question please?
Mic 2: Yes.. but.. do you see a way to get

more power into your setup? Maybe using,
well other power sources, other than the

two pins, or maybe somewhere of..
Trammell: Well, so the question is about,

would there be a way to do more arbitrary
changes through redesigning the implant.

One of the goals was to fit with only
those two pins so that a single piece on

the motherboard could be replaced. With a
dual probe soldering iron and you can pop

it out and stick a new one down in a
matter of seconds. So, yes, if you have

more pins where you can get more power
from you can do much more interesting

things. But that's.. would require a
different set of changes to the

motherboard.
Herald: Thank you. Microphone 1 please.

Mic 1: So, a lot of the -like- arguments
that these implants were not feasible by a

Super Micro where you also show the
picture from the fab that you had to

change the etching and the optical
inspection and so on and so on. But how

probable would you rate the fact that some
acto just intercepted the manufacturing

files and added that component already in
the file because then all the optical

inspection and that would all say well
that matches what was sent to us. But that

was not necessarily what Super Micro sent
to the fab.

Trammell: So the question is, could
someone have modified all of the

manufacturing files that went to the
factory, and that's absolutely a

possibility. But that's also very likely
that that would be detected by Super Micro

itself that in a lot of cases you don't
necessarily want to trust the company that

is making the product to also test it. And
you probably want to have a separate

company that does random spot checks to
verify that the boards are actually being

produced to the specification that you..
that you desire. So it's certainly

possible and I really don't want to
speculate as to the accuracy of that part

of the story but yeah it would require
quite a bit more changes. And also would

be much more likely to be detected in the
spot check.

Herald: Great. Microphone number two
please.

Mic 2: Yes, for a lot of motherboards
there are also quite a few components not

populated some of which are on which you
could consider sensitive myths. Wouldn't

that make it. Yeah exactly. Wouldn't that
make it very easy to do just pop something

on there in parallel with one of the
components and not have it be detected

because it's like the board is modified.
There is a component or you have no way of

telling whether it had to be populated or
not?

Trammell: Super Micro puts a lot of extra
pads on the board in this one particular

one they have both 8 pin and 16 pin flash
chip pads that are just in parallel

together. So depending on which chip is
cheaper that day of the week or who knows

what, they will populate one or the other.
So that's why in this particular photo

having the position of that circle on the
data output pin is very very interesting.

Herald: Question answered? Okay. So one
more question on microphone number two

please?
Mic 2: How far can signing of firmware be

a solution to this problem?
Trammell: Signing firmware solves a lot of

the issues. It does however not all
typically not all of the firmware are

signed specifically is probably to be
signed in in a modern BMC. The kernel and

maybe the root file system might be
signed. But the envy of RAM file system in

this BMC is designed to be user modifiable
so it can't be signed by the manufacturer,

so this sort of attack would work against
a signed BMC just as well. Also the "Hit

enter to get a serial console" attack
circumvents any signing. There are things

on the host firmware on the x86 like boot
card that do a really good job of making

it harder to get code execution during the
boot process. But there have been several

CVEs where it has been implemented poorly.
So even though signature's the firmware is

signed, people have still managed to get
code execution during that process.

Herald: Great. Thank you Trammell Hudson
again, a warm round of applause, thank you

very much!

applause

35c3 postrol music

Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!