Console Hacking 2016 (33c3)

33C3 preroll music

Herald: You have been
here on stage before.

You successfully tampered with the Wii,

You successfully tampered
with the PS3 and got

some legal challenges over there?

marcan: Some unfounded
legal challenges, yes.

Herald: And then you fucked,
and excuse my French over here

– by the way, that is number 8021 to get

the translation on your DECT phone.

So you fucked with the Wii U as well.

“Console Hacking 2016”,
here we go!

marcan: I’m a lazy guy, so I haven’t
turned on my computer yet for the slides.

So let me do that,
hopefully this will work.

My computer is a little bit special.
It runs a lot of Open Source software.

It runs FreeBSD.

applause

It even has things like OpenSSL
in there, and nginx.

And Cairo I think, and WebKit. It runs a
lot of interesting Open Source software.

But we all know that BSD is dying, so
we can make it run something a little bit

more interesting. And hopefully
give a presentation about it.

Let’s see if this works.

It’s a good start, black screen, you know.

It’s syncing to disk
and file system shutting down.

There we go!
applause

continued applause

And yes, I run Gentoo Linux.

applause

This is the “Does Wi-Fi work?” moment.
Hopefully.

NTP, yeah, no… “NTP failed”. Well,
that’s a bit annoying, but it still works.

Hello? Yeah, it takes a bit to boot.
It doesn’t run systemd, you know.

It’s sane, it’s a tiny bit slower,
but it’s sane.

There we go.
applause

This is the “Does my controller
work?” moment.

Bluetooth in Saal 1.
Okay, it does.

Alright, so let’s get started.

So this is “Console Hacking 2016 –
PS4: PC Master Race”.

I apologize for the horrible Nazi joke in
the subtitle, but it’s a Reddit thing.

“PC Master Race”, why? Well.
PS4, is it a PC? Is it not a PC?

But before we get started,
I would like to dedicate this talk

to my good friend Ben Byer
who we all know as “bushing”.

Unfortunately, he passed away
in February of this year and he was

a great hacker, he came to multiple
congresses, one of the nicest people

I’ve ever met. I’m sure that some of you
who have met him would agree with that.

If it weren’t for him, I wouldn’t be here.
So, thank you.

applause

Alright. So, the PS4.
Is it a PC? Is it not a PC?

Well, it’s a little bit different
from previous consoles.

It has x86, it’s an x86 CPU.
It runs FreeBSD, it runs WebKit.

It doesn’t have a hypervisor,
unfortunately.

Then again, the PS3 had a hypervisor
and it was useless, so there you go.

So this is different from the PS3,
but it’s not completely different.

It does have a security processor
that you can just ignore because

it doesn’t secure anything.
So that’s good.

So how to own a PS4? Well, you write
a WebKit exploit and you write

a FreeBSD exploit, duh. Right?
Everything runs WebKit,

and FreeBSD is not exactly the
most secure OS in the world,

especially not with Sony customizations.
So this is completely boring stuff.

Like, what’s the point of talking about
WebKit and FreeBSD exploits?

Instead, this talk is going to be about
something a little bit different.

First of all, after you run an exploit,
well, you know, step 3 “something”,

step 4 “PROFIT”. What is this about?
And not only that, though.

Before you write an exploit, you usually
want to have the code you’re trying

to exploit. And with WebKit and FreeBSD
you kinda do, but not the build they use,

and it’s customized. And it’s annoying to
write an exploit if you don’t have access

to the binary. So how do you get
the binary in the first place?

Well, you dump the code,
that’s an interesting step.

So let’s get started with step zero:
black-box code extraction, the fun way.

A long time ago
in a hackerspace far, far away

fail0verflow got together
after 31c3.

And we looked at the PS4 motherboard
and this is what we saw. So there’s

an Aeolia southbridge, that’s a codename,
by the way. Then there’s the Liverpool APU

which is the main processor.
It’s a GPU and a CPU

which is done by AMD, and
it has some RAM. And then

the southbridge connects to a bunch
of random crap like the USB ports,

a hard disk, which is USB. For some
inexplicable reason the internal disk

on the PS4 is USB. Like it’s SATA to USB,
and then to USB on the southbridge.

Even though it has SATA,
like, what? laughs

The Blu-ray drive is SATA. The Wi-Fi,
Bluetooth, SDIO and Ethernet is GMII.

Okay, how do we attack this?
Well, GDDR5…

What just…?
Oh. I have a screensaver, apparently!

That’s great.
laughter

I thought I killed that,
let me kill that screensaver real quick.

applause
Something had to fail, it always does.

I mean, of course I can
SSH into my PS4, right?

So there we go, okay.
Could have sworn I’d fix that. Anyway…

Which one of these interfaces
do you attack? Well, you know,

USB, SATA, SDIO, GMII – that’s
the raw ethernet interface, by the way –

all these are CPU-controlled. The CPU
issues commands and the devices reply.

The devices can’t really do anything. They
can’t write to memory or anything like that.

You can exploit USB if you
hide a bug in the USB driver,

but we’re back to the no-code issue.

DDR5, that would be great,
we could just write to our memory

and basically own the entire thing.
But it’s a very high-speed bus.

It’s definitely exploitable.
If you were making a secure system

don’t assume we can’t own DDR5,
because we will.

But it’s not the path of least resistance,
so we’re not gonna do that.

However, there’s a thing called
PCI Express in the middle there.

Hmm, that’s interesting!

PCIe is very fun for hacking –
even though it might seem intimidating –

because it’s bus mastering,
that means you can DMA to memory.

It’s complicated, and complicated things
are hard to implement properly.

It’s robust. People think that PCIe is this
voodoo-highspeed… No it’s not!

It’s high-speed, but you don’t need
matched traces to make it work.

It will run over wet string. You can hotwire
PCIe with pieces of wire and it will work.

At least at short distances anyway.
Believe me, it’s not as bad as you think.

It’s delay-tolerant, so you
can take your time to reply.

And the drivers are full of fail because
nobody writes a PCIe driver assuming

the device is evil even though of course
everybody should because devices can

and will be evil.
But nobody does that.

So, what can we do?
Well, we have a PCIe link,

let’s cut the lines and plug in the
southbridge to a PC motherboard

that we stick on the side. Now
the southbridge is a PCIe card for us.

And we connect the APU to an FPGA
board which then can pretend to be

a PCIe device. So we can man-in-the-middle
this PCIe bus and it’s now x1 width

instead of x4 because it’s easier that
way, but it will negotiate, that’s fine.

So how do we connect that
motherboard and the FPGA?

There’s of course many ways of doing this.
How many of you have done

any hardware hacking, even Arduino or
anything like that? Raise your hand!

I think that’s about a third to a half
or something like that, at least.

When you hack some hardware,
you meld some hardware,

after you blink an LED, what is the first
interface you use to talk to your hardware?

Serial port! So we run
PCIe over RS232 at 115 kBaud

which makes this PCIe…
laughter and applause

I said it was delay-tolerant!
So it makes this PCIe 0.00002x.

And eventually there was a
Gigabit ethernet port on the FPGA

so I upgraded to that, but I only got
around to doing it in one direction.

So now it’s PCIe 0.0002x in one direction
and 0.5x in the other direction

which has to make this one of the most
asymmetric buses in the world.

But it works, believe me.
This his hilarious.

We can run PCIe over serial out. Also, we
were ASCII encoding, so half the bandwidth.

It works fine. It’s fine.

So, PCIe 101.
It’s a reliable packet-switched network.

It uses a thing called
“Transaction Layer Packets”

which are basically just packets you send.
It can be… Memory Read, Memory Write,

IO Read, IO Write,
Configuration Read, Configuration Write.

There can be a message-signaled interrupt
which is a way of saying: “Hey,

listen to me!” by writing
to an address in memory.

Because we can write the thing,
so why not write for interrupts?

It has legacy interrupts
which are basically emulating the old

wire-low-for-interrupt-and-
high-for-no-interrupt thing,

you can tunnel that over PCIe.

And it has completions, which are
basically the replies. So if you read

a value from memory the completion
is what you get back with the value

you tried to read. So that’s PCIe,
we can just go wild with DMA.

We can just read all memory, dump
the kernel. Hey, it’s awesome, right?

Except there’s an IOMMU in the APU.

But... first, the IOMMU will protect
the devices. It will only let you access

what memory is mapped to your device.
So the host has to allow you

to read and write to memory.
But just because there’s an IOMMU

doesn’t mean that Sony uses it properly.
Here’s some pseudo-code,

it has a buffer on the stack, it says:
“please read from flash to this buffer”

with the correct length. Can anyone
see the problem with this code?

Well, it maps the buffer and it
reads and it unmaps the buffer.

But IOMMUs don’t just map
byte “foo” to byte “bar”,

they map pages, and
pages are 64k on the PS4.

So Sony has just mapped 64k
of its stack to the device so

it can just DMA straight into the stack,
basically the whole stack, and take over.

Now we got code execution, FreeBSD
kernel dump, and WebKit and OS libs dump,

just from mapping the flash.

Okay, that’s step zero.
We have the code.

But that’s not the PS4 that we did this
on, it was a giant mess of wires.

Someone here knows about that,
you know, flying over on Facebook.

We don’t make a ‘nice’ exploit.
We’ve done that because, as I said,

WebKit, FreeBSD, whatever.
What comes after that?

We want to do something.
Of course we want to run Linux, duh!

How do you go from FreeBSD to Linux?
It’s not a trivial process.

But you use something
that we call “ps4-kexec”.

So how does this work? It’s simple,
right? You just want to run Linux?

Just ‘jmp’ to Linux, right?
Well… kind of.

You need to load Linux into contiguous
physical RAM, set up boot parameters,

shut down FreeBSD cleanly, halt secondary
CPUs, make new pagetables etc.

A lot of random things. I’m not going to
bore you with this crap because you

can read the code. But there’s a lot
of iteration in getting this to work.

Let’s assume that you do all this magical
cleanup and you get Linux into

a nice state and you can ‘jmp’ Linux.
Now we jmp Linux, right? It’s cool.

Yeah, you can technically jmp to Linux,
and it will technically run

…for a little bit. And it will stop.

And you will not get any serial or any
video or anything. What’s going on here?

Let’s talk about hardware.
What is x86?

x86 is a mediocre instruction set
architecture by Intel.

It’s okay, I guess.
It’s not great.

PS4 is definitely x86, it’s x86-64.

What is a PC? Aah!
PC is a horrible, horrible thing

built upon piles and piles of legacy crap
dating back to 1981.

The PS4 is definitely -not- a PC.

That’s practically Sony-level hardware fail,
so it could be, but it’s not.

So what’s going on? A legacy PC

basically has an 8259 Programmable
Interrupt Controller,

a 8253 Programmable Interval Timer,
a UART at I/O 3f8h,

which is the standard address
for a serial port.

It has a PS/2 keyboard controller, 8042.
It has an RTC, a real-time clock

with a CMOS, and everyone
knows the CMOS, right?

MC146818 is the chip number for that. An
ISA bus – even if you think you don’t have

an ISA bus your computer has an ISA bus
inside the southbridge somewhere.

And it has VGA.
The PS4 doesn’t have -any- of these things.

So what do we do?
Let’s look a little bit how a PC works

and how a PS4 works. This is a general
simple PC system. There’s an APU

or an Intel Core CPU with a southbridge,
Intel calls it PCH, AMD FCH.

There’s an interface that is basically
PCIe although Intel calls it DMI and AMD

calls it UMI. DDR3 RAM and a bunch
of peripherals and SATA, whatever.

The PS4 kind of looks like that, right?
So you think this can’t be that dif…

What’s so hard about this?
Because all the crap I mentioned earlier

is in the southbridge on a PC, right?
The PS4 has a southbridge, right?

Right? Right? Umm… so
the southbridge, the AMD standard FCH

implements Intel legacy from 1981.
The Marvell Aeolia

– Marvell is the maker of the PS4
southbridge – implements Intel legacy

from 2002. What does that mean?
Ah! That’s no southbridge,

that’s a Marvell Armada SoC!
So it’s not actually a southbridge,

it was never a southbridge.
It’s an ARM system-on-a-chip CPU

with everything. It’s a descendant
from Intel StrongARM or XScale.

It has a bunch of peripherals.
And what they did is, they stuck

a PCIe bridge on the side and said: “Hey
x86, you can now use all my ARM shit.”

So it exposes all of its ARM peripherals
to the x86. They added some stuff

they really needed for PCs
and it has its own RAM.

Why do they do this? Well, it also runs
FreeBSD on the ARM in standby mode.

And that’s how they do the whole
“download updates in the background,

get content, update, whatever”.
All that crap is because they have

a separate OS on a separate chip running
in standby mode. Okay, that’s great, but

it’s also batshit insane.
laughter

Quick recap: This is what a
PCIe bus number looks like,

sorry, a device number.
It has a bus number, which is 8 bits,

a device number, which is 5 bits,
and a function number, which is 3 bits.

You’ve probably seen this in lspci
if you ever done that.

This is what a regular southbridge
looks like. It has a USB controller,

a PCI, ISA bridge, SATA, whatever.
And it has a bunch of devices.

So one southbridge pretends
to be multiple devices.

Because you only have three bits
for a function number so you can only have

up to eight functions in one device.

Intel southbridge just says:
“I’m device 14, 16, 1a, 1…,

I’m just a bunch of devices,
and you can talk to all of them.”

If you lspci on a roughly unpatched
Linux kernel on the PS4

you get something like this.
So the Aeolia first of all

clones itself into every PCIe device
because they were too lazy to do

“if device equals my number then
reply, otherwise don’t reply”. No,

they just said: “Oh, just reply to every
single PCIe device that might query”.

Linux sees the southbridge 31 different
times, which is kind of annoying

because it gets really confused when it
sees 31 clones of the same southbridge.

And then it has eight functions:
ACPI, ethernet, SATA, SDMC, PCIe,…

Eight functions, so all three bits.

Turns out, eight functions
are not enough for everybody.

Function no. 4, “PCI Express Glue”, has a
bridge config, MSI interrupt controller,

ICC – we’ll talk about that later –,
HPET timers, Flash controller,

RTC, timers, 2 serial ports, I2C… All
this smashed into one single PCIe device.

Linux has a minimum system requirement
to run on anything.

You need a timer, you need interrupts,
and you need some kind of console.

The PS4 has no PIT, no PIC and no standard
serial so none of the standard PC stuff

is going to work here. The board has
test points for an 8250 standard serial

in a different place. So we run
DMESG over that, okay, fine.

Linux has earlycon which we can
point to a serial port and say:

“Please send all your DMESG here
very early because I really want to see

what’s going on”. Doesn’t need IRQs,
you set console=uart8250,

the type, the address, the speed.
And you’ll see it says 3200 instead of

115 kBaud. That’s because their clock
is different. So you set 3200 but

it really means 115k.
And that gets you DMESG.

That actually gets you “Linux booting,
uncompressing”, whatever.

That’s pretty good.

Okay, we need a timer.
Because otherwise everything explodes.

Linux supports the TSC, a built-in CPU
timer which is super nice and super fun.

And PS4 has that. But Linux tries to
calibrate it against the legacy timer

which on the PS4 doesn’t exist
so that’s fail.

So again, the PS4 -really- is not a PC.

What we need to do here is
defining a new subarchitecture

because Linux supports this concept.
Says: “this is not a PC, this is a PS4”.

The bootloader tells Linux:
“Hey! This is a PS4!”

And then Linux says: “Okay, I’m not gonna
do the old timestamp calibration,

I’m gonna do it for the PS4” which has
a special code that we wrote

that calibrates against the PS4 timer.
And it disables the legacy crap.

So now this is officially
not a PC anymore.

Now we can talk about ACPI.

You might know ACPI for all its
horribleness and all its evilness

and all its Microsoft-y-ness.
ACPI - most people associate it with

“Suspend” and “Suspend to Hibernate”.
It’s not just power,

it does other stuff, too.
So we need ACPI for PCI config,

for the IOMMU, for the CPU frequency.

The PS4 of course has broken ACPI tables
because, of course it would be.

So we fixed them in ps4-kexec.

Now interrupts. We have timers,
we have serial, we fixed some stuff.

The PS4 does message-signaled interrupts
which is, what I said, the non-legacy,

the nice new thing where you just write
a value, and what you do is you tell

the device when you want to interrupt
“please write this value to this address”.

The device does that, and the CPU
interrupt controller sees that write

and says: “Oh, this is an interrupt”
and then just fires off that interrupt

into the CPU. That’s great.
It’s super fast and very efficient.

And the value directly tells the CPU:
“That’s the interrupt vector you have

to go to”. Okay, that’s the standard MSI
way there. Your computer does MSI that way.

This is how the PS4 does MSI: The Aeolia
ignores the MSI config registers

in the standard location. Instead of
has its own MSI controller,

all stuff that’s in Function 4,
which is that “glue” device.

Each function gets a shared address in
memory to write to and the top 27 bits

of data. And every sub function, because
you can’t do a lot of things into one place,

only gets the different 5 bits.
And all MSIs originate from Function 4,

so this device has to fire an interrupt,
then it goes to here, and then

that device fires an interrupt. Like… what…
this is all… what the hell is going on?

Seriously, this is really fucked up. And
– the i’s are missing in the front there.

But yeah. So, driver hell. Now the devices
are interdependent. Then the IRQ vector

location is not sequential, so that’s not
gonna work. And you need to modify

all the drivers. This is really painful to
develop for. So what we ended up doing

is there is a core driver that implements
an interrupt controller for this thing.

And then we have to make sure that loads
first, before the device driver. So Linux

has a mechanism for that. And we had to
patch the drivers. Some drivers we patched,

so to use these interrupts. And others
we wrapped around to use these interrupts.

Unfortunately, because of the top bit
thing, everything has to share one interrupt

within a function. Thankfully, we can fix
that with a IOMMU because it can read

direct interrupt. So we can say:
“Oh, interrupt no. 0 goes to here,

1 goes to here, 2 goes to here…”.
That’s great 'cause it's consecutive, right?

0 1 2 3 4 5… it’s obviously gonna have
the same top bits. But we have to fix

the ACPI table for that because it’s
broken. But this does work. So this

gets us interrupts that function and
they’re individual. So let’s look at

the check list: we have interrupts, timers,
early serial, late serial with interrupts.

We can get some user space, we can stash
some user space and binaries into the kernel.

And it will boot and you can get a console,
but you get a console and you try

writing commands and sometimes it hangs.
Okay. What’s going on there?

So it turns out that FreeBSD masks
interrupts with an AMD proprietary

register set. We had to clean that up,
too. And that fixes serial,

and all the other interrupts.
This took ages to find. It’s like: “why…

interrupts on CPU serial
sometimes don’t…, yeah”.

I ended up dumping register sets,
and I saw this #FFFFF here, not #FFFFF,

what’s that? But tracking through this
stack to find this was really annoying.

Alright. So we have the basics. We have
like a core platform we can run Linux on,

even though it won’t do anything
interesting. Add drivers!

So we have USB xHCI which has three
controllers in one device. Again, because

“Let’s make it insane!”. We have SDHCI,
that’s SDIO for the Wi-Fi and the Bluetooth.

Needs a non-standard config, it needs
quirks. Ethernet needs more hacks.

It’s still partially broken, it only runs at
Gigabit speed. If you plug in a 100Mbit/s

switch it just doesn’t send any data.
Not sure why.

And then all of this worked fine in
Linux 4.4, and then just three days ago

I think I tried to rebase on 4.9, and so
we have the latest and the greatest.

And everything failed. And DMA didn’t
work. And all the drivers were just

throwing their hands up in the air,
“what’s going on here?”.

exhales
Aeolia strikes back. So.

That’s what… the Aeolia looks like,
normally. So you have… again,

it’s an ARM SoC, it’s really not a device.
It’s like its own little system. But

it maps, it’s low 2 GB of the address base
to memory on the PC. And then the PC

has a window into its registers that it
can use to control those devices.

So the PC can kind of play with the
devices, and the DMA is to the same address

and that works great. Because it’s mapped
in the same place. And then has its own RAM,

in its own address space. This works fine.
But now we had an IOMMU. Because

we needed it for the interrupts. And the
IOMMU inserts its own address space

in between and says: “Okay, you can map
anything to anything you want, that’s great.“

It’s a page table, you can say “this
address goes to that address.”

Linux 4.4 did this: it would find some
addresses at the bottom of the IOMMU

address space, say: “page 1 goes to this,
page 2 goes to that, page 3 goes to that”.

And say: “device, you can now write to these
pages”. And they go to this place in the x86.

That worked fine. It turns out Linux 4.9,
or somewhere between 4.4 and 4.9

it started doing this: it would map pages
from the top of the IOMMU address space

and that’s fine for the IOMMU but it’s
not in the window in the Aeolia, so

you say “ethernet DMA to address
FExxx”, and instead of DMA-ing

to the RAM on the PC it DMA-s to the RAM
on the Aeolia which is not gonna work.

Effectively the Aeolia implements 31 bit
DMA, not 32 bit DMA because only

the bottom half is usable. It’s like why…
this is all really fucked up, guys!

Seriously. And this is littered all over
the code in Linux, so they seeded

more patches, and it works, but, yeah.

Painful. Okay. Devices, laying out (?)
devices’ work.

Now for something completely different.
Who can tell me who this character is?

That’s Starsha from Space Battleship Yamato.
And apparently that’s the code name

for the PS4 graphics chip. Or at least that’s
one of the code names. Because

they don’t seem to be able to agree
on like what the code names are.

It’s got “Liverpool” in some places, and
“Starsha” in other places. Then “ThebeJ”

in other places. And we think Sony calls
it “Starsha” and AMD calls it “Liverpool”

but we’re not sure. We are calling it
“Liverpool” everywhere just to avoid

confusion. Okay.
What’s this GPU about?

Well, it’s an AMD Sea
Islands generation GPU,

which is spelled CI instead of SI because
“S” was taken. It’s similar to other chips

in the generation. So at least that’s
not a bat shit crazy new thing.

But it does have quirks and customizations
and oddities and things that don’t work.

What we did is we took Bonaire which is
another GPU that is already supported

by Linux in that generation, and just kind
of added a new chip and said, okay,

do all the Bonaire stuff, and then change
things. And hopefully adapt it to the PS4.

So hacking AMD drivers, okay, well,
they’re open-source but AMD does not

publish register docs. They publish 3D
shader and command queue documentations,

so we get all the user space 3D rendering
commands, that’s documented. But they

don’t publish all the kernel hardware
register documentation. That’s what

we really want for hacking on drivers. So
that’s annoying. And you’re thinking

“the code is the documentation”,
right? “Just read the Linux drivers”.

That’s great. Yeah, but they’re incomplete,
then they have magic numbers, and

it’s, you know, you don’t know if you need
to write a new register that’s not there,

and it really sucks to try to write a GPU
driver by reading other GPU drivers

with no docs. So what do we do? We’re
hackers, right? We google. Everytime

we need information, hopefully Google will
find it because Google knows everything.

And any tip that you could find in any
forum or code dumped somewhere is

great. One of the things we found is we
googled this little string, “R8XXGPU”.

And we get nine results. And the second
result is this place, it’s “Siliconkit”,

token, was that okay? It’s an XML file.
And if we look at that it looks like

it’s an XML file that contains a dump of
the Bonaire GPU register documentation.

But it’s like broken XML, and it’s
incomplete, it stops at one point.

But like: “what’s this doing here?”
And where did this come from, right?

So let’s dig a little deeper. Okay Google,
what do you know about this website?

Well, there’s some random things like
whatthehellno.txt and whatthehellyes.txt

and some Excel files. Those are
really Excel like XML cell sheets.

And then there’s a thing in the (?) there
called RAI.GRAMMAR.4.TXT.

I wonder what that is. And it looks like
it’s a grammar, being a notation description

for a syntax, of some kind of register
documentation file. This looks like

an AMD internal format but it’s on this
website. Okay. So we have these two URLs,

/pragmatic/bonaire.xml
and /RAI/rai.grammar4.txt.

Let’s try something. How about maybe
/pragmatic/bonaire.rai – nah, it’s a 404.

Okay, /pragmatic/RAI/bonaire.rai – aah!
Bingo!

laughter and applause

So this is a full – almost full Bonaire
register documentation with like

full register field descriptions, breakdowns,
all the addresses. It’s not 100% but

like of the vast majority. This seems to
be AMD-internal stuff. And I looked

this guy up, and apparently he worked
at AMD at some point. So…

But yeah… This is really, really helpful
because now you know what everything

means, and debug registers, and… yeah.
So I wrote a working parser for this format.

This was effectively writing an XML parser,
something like convert this thing to XML

but it was all broken. Oh – he was writing
it in PHP, by the way, so there you go …

So I wrote a working one in Python and
you can dump it and then you can see

what each register means, and it’ll tell
you all the options. You can take

a register dump and map it to the (?)(?)
documented. You can diff dumps,

you can generic defines, it’s very useful
for AMD GPUs. And this, grossly speaking

applies to a lot of AMD GPUs, like they
share a lot of registers. So this is useful

for anyone hacking on AMD GPU stuff. Over
4.000 registers are documented in the …

just in the main GPU address space alone.
That’s great. Okay. So we have some docs.

How do we get to a frame buffer? So if you…
Israel (?) is HDMI it’s easy, right? The GPU

has HDMI, and if you query the GPU
information you actually get that it has

an HDMI port and a DisplayPort port. Okay,
maybe it’s unconnected, that’s fine, right?

But if you actually ask the GPU it tells
you: “HDMI is not connected, DP is connected”.

Okay. Yeah, they have an external HDMI
encoder from DisplayPort to HDMI because

just putting a wire from A to B is too
difficult, because this is Sony, so:

“let’s put a chip that converts some
protocol A to protocol B…” sighs

Yeah, yeah.
applause

It’s a Panasonic DisplayPort to HDMI
bridge, not documented by the way.

We parsed config to work, that’s why it
doesn’t just work. Even though some bridges do.

And you’d think, okay, it’s hooked up to the
GPU I2C bus, because GPUs have in the past

used these bridges, and, not this one
particularly but other AMD cards have had

various chips that they stuck in front. And
the code has support for talking to them

through the GPU I2C interface, right?
That’s easy. Yay, you wish – it’s a Sony.

sighs
Enter ICC! So, remember the ICC thing

in the Aeolia – it’s an RPC protocol you
use to send commands to an MCU that is

somewhere else on the motherboard. It’s
a message box system, so you write some

message to a memory place, and then you
tell: “Hey, read this message!” and then

it writes some message back, and it tells
you “Hey, it’s the reply!”.

The Aeolia – not the otherGPU – uses it for things like

Power Button, the LEDs, turning the power
on and off, and also the HDMI encoder I2C.

So now we have the dependency from the
GPU driver to the Aeolia driver, two different

PCI devices and two different… sighs
Yeah. And okay, again, ICC, but it’s I2C,

you know, I2C is a simple protocol.
You read a register, you write a register,

that’s all you need. It super simple.
Right? Now let’s make a byte code

fucking scripting engine to which you I2C
commands and delays and bit masking

and everything. And why, Sony, why, like
why would you do this? Well, because

ICC is so slow? That if you actually tried
to do one read and one write at a time

it takes 2 seconds to bring up HDMI.
exhales

Yeah…

I don’t even know at this point…
applause

I have no idea.
continued applause

And by the way this thing has commands
where you can send scripts in a script

to be run when certain events happen. So
“Yo dawg, I heard you like scripts, I put

scripts in your scripts so you can I2C
while you I2C”. Like: “let’s just go

even deeper at this point”, right? Yeah.
exhales

Okay. We wrote some code for this,
you need more hacks, it needs all

DisplayPort lanes up, Linux tries to downscale,
doesn’t work. Memory bandwidth calculation

is broken. Mouse cursor size is from the
previous GPU generation for some reason,

I guess they forgot to update that. So
wait! All this crap – we get a frame buffer.

But X won’t start. Ah. Well, it turns out
that PS4 uses a unified memory architecture

so it has a single memory pool that is
shared between the x86 and the GPU.

And games just put a texture in memory
and say: “Hey, GPU, render this!” and

that works great. And this makes a lot of
sense, and their driver uses this to the

fullest extents. So there’s a VRAM,
you know, the legacy… GPUs had

a separate VRAM and all these integrated
chip sets can emulate VRAM using a chunk

of the system memory. And you can usually
configure that in the BIOS if you have

a PC that does this. And PS4 sets it to
16 MB which is actually the lowest possible

setting. And 16 Megs is not enough to have
more than one Full HD frame buffer. So,

obviously, that’s going to explode in
Linux pretty badly. So what we do is

we actually reconfigure the memory
controller in the system to give 1 GB

of RAM to the VRAM, and we did it on the
psd-kexec. So it’s basically doing like

BIOSy things. We were reconfiguring the
Northbridge at this point to make this work.

But it works. And with this we can get X
to start because it can allocate its frame buffer.

But okay, it’s 3D time, right? – Neeaah,
GPU acceleration doesn’t quite work yet.

So we got at least, you know, X but let’s
talk a bit about the Radeon GPU

for a second. So when you want to draw
something on the GPU you send it a command

and you do this by putting it into ‘ring’
which is really just a structure in memory,

that’s a (?)(?)(?)(?). And it wraps around.
So that way you can queue things to be done

in the GPU, and then it does it on its own
and you can go and do other things.

There’s a Graphics Ring for drawing,
a Compute Ring for GPGPU, and a DMA Ring

for copying things around. The commands
are processed by the GPU Command Processor

which is really a bunch of different CPUs
inside the GPU. They are called F32.

And they run a proprietary AMD microcode.
So this is a custom architecture.

Also the rings can call out to IBs which
are indirect buffers. So you can say

basically “Call this piece of memory, do
this stuff there, return back to the ring”.

And that’s actually how the user space
thing does things. So this says:

“Draw this stuff” and it tells the kernel:
“Hey, draw this stuff”. And the kernel

tells the GPU: “Jump to that stuff,
read it come back, keep doing stuff”.

This is basically how most GPUs work but
Radeon specifically works like, you know…

with this F32 stuff. Okay. The driver
complains: “Ring 0 test failed”.

Technically (?), you test them, so at least
you know it has nice diagnostic,

and how does the test work? It’s really
easy. It writes a register with a value,

and then it tells the GPU with a command
“Please write this other value

to the register”, runs it and the checks
to see if the register was actually written

with the new value. So the write doesn’t
happen. Thankfully, thanks to that RAI file

earlier we found some debug registers that
tell you exactly what’s going on inside

the GPU. And it shows the Command
Processor is stuck, waiting for data

in the ring, so it needs more data.
After a NOP command?! Yeah…

NOP is hard, let’s go stalling. So packet
headers in this GPU thing have a size

that is SIZE-2. Whoever thought that was
a good idea. So a 2 word packet

has a size of zero. Then AMD implemented
a 1 word packet with a size of -1.

And old firmware doesn’t support that and
thinks: “Oh it’s 3FFF so I’m just gonna wait

for a shitload of code in the buffer”,
right? It turns out that Hawaii,

which is another GPU in the same gen
has the same problem with old firmware.

So they use a different NOP packet, so
there was an exception in the driver

for this. And we had to add ours to that.
But again – getting to this point, many,

many, many hours of headbanging.

Okay. We fixed that. Now it says:
“Ring 3 test failed”.

That’s the SDMA ring. That’s for copying
things in memory and it works

in the same way. It puts a value in RAM.
It tells the SDMA engine: “hey, write

a different value”. And checks. This time
we see the write happens but it writes “0”

instead if the 0xDEADBEEF or whatever.
Okay. So I tried this.

I put two Write commands in the ring
saying: “Write to one place, write to

a different place”. And this time,
if I saw, what it did is it wrote “1”

to the first destination and “0” to the
second destination. I’m thinking:

“Okay, it’s supposed to write 0xDEADBEEF…”
which is what you see there, it’s…

0xDEADBEEF is that word
with the value. It writes “1”.

Well, there’s a “1” there that
wasn’t there before, it was a “0”,

because of this padding, right? So it
turns out they have it off by four,

in the SDMA command parser
and it reads from four words later

than it should.
exhales

Again, this took many hours of
headbanging. It was like:

“Randomly try two commands, oh, one, one?”
– “One”.

So it reads four words too late but only
in ring buffers. Indirect buffers work fine.

That’s good because those come from user
space. So we don’t have to mock with those.

We can work around this, because it’s
only used in two places in the kernel,

by using a Fill command instead of a Write
command. That works fine. Again,…

how do they even make these mistakes?!
Okay. But still the GPU doesn’t work.

The ring tests pass but if you tried
to draw you get a bunch of page faults.

And it turns out that what happens is that
on the PS4 you can’t write the page table

registers from actual commands in the GPU
itself. You can write to them from the CPU

directly. You can say just: “Write memory
– memory register write”, and then

I’ll write. But you can’t tell the GPU:
“Please write to the page table register this”.

So the page tables don’t work, the GPU
can’t see any memory, so everything is broken.

Linux uses this, FreeBSD doesn’t. It uses
direct writes. And we think this is maybe

a Firewall somewhere in the Liverpool,
some kind of security thing they added.

We can directly write from the CPU.
But it like breaks the regular…

like it’s not asynchronous anymore. So
this could break things. And it’s a really

hacky solution. I would really like to fix
this. And I’m thinking: “Maybe the firewall

is in the firmware, right?”. But it’s
proprietary and undocumented firmware.

So let’s look at that firmware. It’s
a thing, it needs microcode, a CP thing.

It’s undocumented. But we take the blobs
out of FreeBSD. And that’s great because

we have don’t have to ship them. Let’s
dig deeper into those blobs. So how do you

reverse-engineer an unknown CPU
architecture? That’s really easy,

run an instruction and see what it did.
And then just keep doing that. Thankfully,

we upload custom firmwares, so it’s
actually really easy to just have like

a two-instruction firmware that does
something, and then writes a register

to a memory location. And that’s actually
really easy to find. If you first like

write the memory instruction, it’s really
easy to find in the binary because you see

like GPU register offsets that stand out
a bit in one column. So long story short,

we wrote F32DIS which is a disassembler
for the proprietary AMD F32 microcode.

I shamelessly stole the instruction
syntax from ARM. So you may recognize

that if you’ve ever seen an ARM disassembly.
And this is not complete but it can

disassemble every single instruction
in all the firmware in Liverpool for PFP,

ME, CE, MEC and RLC which are five
different blocks in the GPU. As far

as I notice that’s never been done before,
all the firmware was like in a voodoo

black magic thing that’s been shipped.
Not even the non-AMD kernel developers

know anything about this. So…
applause

ongoing applause

And you can disassemble the desktop
GPU stuff, too. So this could be good for

debugging strange GPU shenanigans
in non-PS4 stuff.

Alright. Alas, it’s not in the firmware.
It seems to be blocked in hardware.

I found a debug register that actually
says: “there was an access violation

in the bus when you try to write this
thing”. And I tried a bunch of workarounds

and I even bought an AMD APU system,
desktop. Dumped all the registers,

diff’ed them against the one I had on Linux
and tried setting every single value

from the other GPU and hoping I find some
magic bits somewhere, but… no.

They probably have a setting for this,
somewhere, but it’s a sea of ones and zeros,

good luck finding it. It does work with
a CPU Write, workaround, though.

So, hey, at least we get 3D! And it’s
actually pretty stable, so if there’s

a race condition I’m not really seeing it.
So – checklist! What works,

what doesn’t work. We have interrupts,
and timers – the core thing you need

to run any OS – we have a serial port,
we can shutdown the system and reboot,

and you’ll think that’s funny but actually
that goes through ICC, so again,

at least some interesting code there.
I actually just implemented that about

four hours ago. Because pulling the plug
was getting old. The Power button works.

USB works. There’s a funny story with USB
as it used not to work. And we said:

“Fix it later, there seems to be special
code missing.” And then someone

pulled a repo from the USB-not-working
branch, and tested it, and said:

“It’s working!” It seems we fixed it by
accident, by changing something else.

The hard disk works which is via the USB.
Blu-ray works, I wrote a driver for that,

also four hours ago. – Three hours ago
now? Yeah, something like that.

And I spent 20 minutes looking for someone
in the Hackcenter that had a DVD I could

stick in to try. Apparently I’m from
he past if I ask for DVDs.

But it does work. So that’s good. Wi-Fi
and Bluetooth works.

Ethernet works, except only at GBit speeds.
Frame buffer works. HDMI works.

It’s currently hard-coded to 1080p so…
It does work. We can fix that

by improving the encoder implementation.
3D works with the ugly register write hack.

And SPDIF audio works. So that’s good.
HDMI audio doesn’t work. Mostly because

I only got audio grossly working, in
general, recently, and I haven’t had

a chance to program the encoder to support
the audio stuff yet. Because, again,

new more annoying hacks there. And the
real-time clock doesn’t work and everything.

That’s simple, the clock, that device is
simple. But ever since the PS2 the way

Sony has implemented real-time clocks
is that instead of reading and writing

the time on the clock, which is what you
would think is the normal thing to do,

they never write the time on the clock.
Instead, they store an offset from the clock

to the real time, in some kind of storage
location. And there’s a giant mess of…

…registry it’s called, in the PS4, and
I don’t even know where it’s stored.

It might be on the hard drive, it might be
encrypted. So basically, getting

the real-time clock to actually show the
right time involves a pile of nonsense

that I haven’t had the chance to look at
yet. But… we have NTP, right?

So it’s good enough. – Oh, and we have
Blinkenlights! Important! The Power LED

does some interesting things, if you’re
on Linux. So that’s good.

So – the code: you can get the ps4-kexec
code on our Github page. That has

the kexec and the hardware configuration,
and the bootloader Linux stuff.

You can get the ps4 Linux branch which is
the… our fork of the kernel,

rebased on 4.9 which is the latest (?)
version, I think.

You can get our Radeon patches which are
three, I think, really tiny patches for

user space libraries just to support this
new chip. Really simple stuff, the NOP

thing, and a couple of commands. And the
RAI and F32DIS thing I mentioned.

You can get Radeon tools at that Github
repo. Just push that right before the stock.

So if you’re interested – there you go.
And if you going at the RAI file, well,

we wanna put you on a run before the guys
at that website realize they really should

take that down! But I’m sure the internet
wayback machine has it somewhere.

Okay! That’s everything for the story of
how we got Linux running on the PS4.

And you can reach us at that website
or fail0verflow on Twitter.

applause
Thank you!

ongoing applause

I hope that wasn’t too fast, sorry, I had
to rush through my 89 slides a little bit

because I really wanted to do a demo.
I think this kind of is the demo, right.

But we can try something else.
So maybe I can shut this –

so I can aim with my controller.

This is really not meant as a mouse!
That’s not Right Button.

Come on! Yeah, I think it is…

Close? Close! Maybe…

So we have this little icon here.
I wonder what happens if it works.

Do we have internet access? Hopefully
Wi-Fi works, let’s then just check real quick.

keyboard typing sounds

This could bork really badly if we don’t.

keyboard typing sounds

mumbles ping 8.8.8.8

Yeah, we have internet access.
So, Wi-Fi works!

Okay. I wonder what happens
if we click that!

It takes a while to load.
This is not optimized for…

laughter and applause
marcan laughs

So the CPUs on this thing are
a little bit slow. But…

sounds of the machine
Hey, it works!

And now it’s a real game console!

laughter and applause

And this is… there we go, okay.

So I think we can probably take some Q&A
because this is a little bit slow to load.

But we can try a game, maybe.

Herald: If you are for Q&A I think
there will be some questions.

So shall we start with one
from the internet.

Signal Angel: Hey! The internet wants to
know if most of your research will be

published, or if stuff’s
going to stay private.

marcan: All of this… the publishing is
basically the code which… and you know

the explanation I just gave… I said that
everything’s on Github. So all the drivers

we wrote, all the… I mean… and in this
case also the spec is the code.

If you really want to I could write some
Wiki pages on this. But roughly speaking,

what’s in the drivers is what we found
out. The really interesting bit,

I think, is that F32 stuff from the AMD
GPU stuff. And that we have a repo for.

But if you have any general questions, or
name a particular device, or any details,

feel free to ask. I don’t know… again, it
would be nice if we wrote a bunch

of docs and everything. But it’s not really
a matter of not wanting to write them,

it’s lazy engineers not wanting to write
documentation. But the code is at least…

the things we have on Github are fairly
clean. So.

Herald: Okay, so, someone is piling up
on 4. Guys, if you have questions

you see the microphones over here.
Just pile up over there

and I’m gonna point… 4 please!

Question: Just a small question.
How likely is it that you upstream

some of that stuff. Because… I mean…

marcan: So there’s two sides to that.
One side is that we need to actually

get together and upstream it. The code…
some of it has horrible hacks, some of it

isn’t too bad. So we want to upstream it.

We have to sit down and actually do it.
I think most of the custom x86 based

machine stuff and the kernel is doable.
The drivers are probably doable.

Some people might scream at the interrupt
hacks. But it’s probably not terrible.

And if they have a better way of doing it
I’m all ears, there are other kernel devs.

The Radeon stuff is quite fishy because of
the encoder thing that is like (?) non-standard.

And also understandably
AMD GPU driver developers

that work for AMD may want to have nothing
to do with this. And in fact I know

for a fact that at least
one of them doesn’t. But

they can’t really stop us from upstreaming
things into the Linux kernel, right?

So I think as long as we get to come
to a state where it’s doable it’s fine.

But most likely I think…
laughter

…I think most likely the non-GPU stuff
will go in first if we have a chance

to do that. And of course, if you wanna
try upstreaming it go ahead!

It’s open source, right? So.

Herald: Over to microphone 1, please.

Question: Hi. First I think I should
employ you to try and find trouble Hudson. (?)

And control him into using your FreeBSD
kexec implementation in heads.

Instead of having to run all of Linux in it,
as a joke. But my real question is:

if the reason you used Gentoo was
because systemd was yet another hurdle

in getting this to run?

laughter
marcan laughs

marcan: I run Gentoo on my main machine,
I run Gentoo on most of the machines

I care about. I do run Arch on a few of
the others and then I’d live with systemd.

But the reason why I run Gentoo is, first
it’s what I like and use. And second it’s

super easy to use patches on Gentoo.
You get those things we put onto Github,

which are just patch files, it’s not really
a repo. Because they’re so easy

it’s not worth cloning everything. Just
get those patch files, stick them on

/etc/portage/patches/, have a little hook to patch,
and that’s all you need. So it’s really

easy to patch packages in Gentoo,
that’s one of the main reasons.

laughs about something in audience

Herald: No. 3 please!

Question: Will there be new exploits,
new way to boot Linux

on PS3 with modern firmwares
because finding one

with firmware 1.76 is really rare.

marcan: That was 4.05!

Question: Ah, okay.
marcan: But again, our goal is to focus

on… I just told you the story of the
pre-exploit thing because I think

that’s good like a hacker story, a good
knowledge suite trying new platforms.

And the Linux thing we’re working on.
The reason why we don’t want to publish

the exploit or really get involved in the
whole exploit scene is that there is

a lot of drama, it’s not rocket science
in that it’s like super custom code,

this is WebKit and FreeBSD. It’s actually not
that hard. And we know for a fact

that several people have reproduced this
on various firmwares. So there’s no need

for us to be the exploit provider. And
we don’t want to get into that because

it’s a giant drama fest as we all know,
anyway. Please DIY it this time!

Question: Okay. Thanks.

Herald: And what is the internet saying?

Signal Angel: The internet wants to know
if you ever had fun with the BSD

on the second processor.

marcan: Oh, that’s a very good question.
I myself haven’t. I don’t know if anyone

else has looked at it briefly. One of the
commands for rebooting will boot

that CPU into FreeBSD. And there’s
probably fun to be had there.

But we haven’t really looked into it.

Herald: And over to 5, please.

Question: I was wondering if any of that
stuff was applicable to the PS4 VR edition

or whatever it’s called, the new one?
Did you ever test it?

marcan: Sorry, say it again!

Question: Sony brought up a new PS4
I thought.

marcan: Oh, the Pro you mean,
the PS4 Pro?

Question: Yes.
marcan: So Linux boots on the Pro,

we got that far. GPU is broken. So we
would like to get this ported to the Pro

and also working. It’s basically an
incremental update, so it’s not that hard,

but the GPU needs a new definition,
new jBullet(?) stuff.

Yeah, you get a lot of C frames
down-burned (?), yeah…

So, as you can see, 3D works,
and, there you go!

synth speech from game
applause

I only have to look up and down in this game!

continued synth speech from game

Herald: Well, then number 3, please.

Question: I want to ask you if you want to
port these Radeon patches to the new

amdgpu driver because AMD now supports
the Southern Island GPUs?

marcan: Yes, that’s a very good question.
Actually, the first attempt we made

at writing this driver was with amdgpu.
And at the time it wasn’t working at all.

And there was a big concern about its
freshness at the time and it was

experimentally supporting this GPU
generation. I’m told it should work.

So I would like to port this… move to
amdgpu and we have a working

implementation, and we got to clean up
code much better, we know where all

the nits are, I want to try again with
amdgpu and see if that works.

That’s a very good question because the
newer gen might require the driver maybe, so …

Question: Thank you.
Herald: Well then I’m gonna guess we ask

the internet again.

Signal Angel: Okay, the internet states
that about a year ago you argued

with someone on twitter that the PS4 wasn’t
a PC and now you’re saying that kind of

is something. And what’s about that?

marcan: So again, the reason of saying
it’s not a PC is that it’s not an IBM

Personal Computer compatible device.
It’s an x86 device that happens to

be structured roughly like a current PC
but if you look at the details

so many things are completely different.
It really isn’t a PC. Like on Linux I had

to define “sub arch PS4”. It’s an x86
but it’s not a PC. And that’s actually

a very important distinction because
there’s a lot of things you have

never heard of that are x86 but not PC.
It’s like e.g. there’s a high chance

your monitor at home has
an 8186 CPU in it. So, yeah.

Herald: So nobody’s piling at the
microphones any more.

Is there one last question
from the internet?

Signal Angel: Yes, there is.

The question is…

…if there was any
decryption needed.

marcan: No. So this is purely… you
exploit WebKit, you get user mode,

you exploit the kernel, you got kernel
mode. You jump Linux…

there’s no security like… there’s nothing
like stopping you from doing

all that stuff. There’s a sand box in
FreeBSD but obviously you exploit

around the sand box. There’s nothing…
there’s no hypervisor, there’s no monitoring,

there’s nothing like saying: “Oh this code
should not be running.” There’s no

like integrity checking. They have a security
architecture but as it’s tradition for Sony

you can just walk around it.
laughter

applause

The PS3 was notable for the fact that
the PS Jailbreak which is a USB…

it’s effectively a piracy device
that was released by someone

that basically used a USB exploit
in the kernel and only a USB exploit

in the kernel to effectively enable piracy.
So when you have like a stack of security

and you break one thing and you get
piracy that’s a fail! This is basically

the same idea. Except I have no idea what
you do to do piracy and I don’t care.

But Sony doesn’t really know how to
architecture secure systems.

That’s it.

Herald: That’s it, here we go,
that’s your applause!

applause

postroll music

subtitles created by c3subtitles.de
in the year 2017. Join, and help us!