preroll music
Camille Gay: Hello, everyone. My name is Camille. I am
a researcher at Toyota Motor Corporation,
and this is a presentation about RAMN, a
platform that we developed to make
education and research in the automotive
systems more accessible. The automotive
industry can be inaccessible to many
people because automotive projects involve
prohibitive costs and be tied to NDAs that
everybody is going to sign. What we want
to propose with this project is an
inexpensive test bed to study and research
automotive systems, which is both open
source and developed with open source
tools so that at least anyone can get
access to a good alternative for education
and research. The main focus of this test
bed is security, but you will see that the
usage of the test bed is not limited to
security, and I will keep the security
talk mostly for the end of the
presentation. I will start by giving a
short introduction about automotive
systems. Then I will present the design
details of the test bed besides
demonstrations and concrete details about
its hardware and software. As an example
of how the test bed can be used, I will
spend some time experimenting with cruise
control and by that I mean I will go
through the whole development process,
starting from evaluating the differential
equations of a simple mechanical model. I
will experiment with various control
strategies, implement them in C and make
measurements in a driving simulator using
only data from the CAN bus. And I will do
all that using only open source tools.
This is to demonstrate how the test bed
can be used, but also to have a concrete
project that I can use as a reference to
explain after what concretely would have
been different if we were experimenting
with a real electronic control unit. I
will also explain how we can get close to
automotive hardware and software without
signing NDAs. So the second part of the
talk is mainly here to give you more
information about the automotive industry
in case you are not familiar with it.
Before I start, let me just clarify that
this is not an advertisement. We are not
selling anything we present here and do
not profit financially from it. We're
simply showing design files whithout
permising licenses and without
royalties. OK, so first, let me give you a
very quick introduction about automotive
systems. You can see your car as a
collection of systems divided in four
different domains, the powertrain domain,
which includes the engine and the
transmission. The chassis domain, which
includes the steering column and
suspensions. The body domain, which
includes the lights, the doors and
heating, and the infotainment domain,
which includes navigation and
connectivity. Many of the different
systems that can be found in the car are
controlled by electronic control units or
ECUs for short. There are many kinds of
ECUs in the car. Sometimes hundreds of
them. And usually it's a lot hard to
understand. They have a limited number of
inputs, generally data from sensors and
actuators. And they have a limited number
of outputs generally to control actuators.
So, for example, an airbag ECU may use an
accelerometer as its input and an airbag
trigger as its output. The role of the ECU
would be to use data from the
accelerometer to detect a shock and output
a signal as actuator to detonate an airbag
when the shock is detected. It is very
common for ECUs to read data from other
ECUs. Most of the time ECUs will need to
share data with other ECUs on the same
domain. In the case of [unclear],
for example the transmission control unit
gets input from the engine ECU to
determine the correct gear. If the data is
critical that connection may even be
redundant. ECUs may also need to
communicate with ECUs on a different
domain. For example the brake system,
usually in the chassis domain, willl need to
communicate its data to store them, usually
in the body domain. Most of the time the
technology that is used for communication
is CAN. And CAN technology uses a bus
topology, which means a CAN message will
be received by all ECUs on the same CAN
bus, there is no authentication or
encryption at the link layer. So any
message can be sent by any ECU. And
security features need to be implemented
at higher layers. A standard CAN frame
consist mainly of an arbitration ID of 11
bits and a payload of 8 bytes. CAN-FD is a
recent evolution of CAN where the payload
size may be extended up to 64 bytes. For
an ECU network manufacturers will assign a
meaning to each arbitration ID and each
bit in the payload, so the file that
determines the traffic on the CAN bus is
often referred to as a DBC file. For
example, assuming a lamp controller and
two lamps on the CAN bus. The manufacturer
may decide that ID 123 is used by the lamp
controller to communicate the command of
both lamps, that ID 124 is used by the
left lamp to give feedback about the
status and that ID 125 is used by the
right lamp to give feedback about its
status. Each of those messages will be
broadcasted periodically by the assigned
ECU on the CAN bus and will serve as a
basis for most data exchange between ECUs.
So that's it for the introduction, there
are many reasons why people would be
interested in automotive systems and ECU
networks. The opportunity that gets by far
most attention from the media is
vulnerability research. But there are also
other reasons. For example, owners: They
want to check their car's compliance with
regulations such as emissions regulations
and privacy regulations. For example,
GDPR. Other owners may want to exercise
their rights to repair as they are
guaranteed by the country they live in.
And finally, some owners may want to
experiment and innovative DIY features or
simply satisfy their curiosity and educate
others. And while those may be valid
reasons to experiment with the car,
manufacturers are typically against people
tinkering with a car because they may be
worried about their intellectual property
being stolen, about vulnerabilities being
exploited or people hurting themselves and
others while tinkering. And what probably
suffers the most from this delicate
situation is education and research in
automotive security, because people can
not easily get access to safe equipment or
access the information that they would
need. In the long term, this may mean that
manufacturers will have less security
technologies to choose from to secure
their cars and that less talents would be
available to develop and evaluate them. As
the development, of course, involves many
people from many companies, so it is
important to make sure that everyone
involved is competent in automotive
security. And some people are pushing for
more open sourcing cars and who knows,
maybe one day the car will be 100%
open source, but if it happens, it is
going to take a long time. And
manufacturers themselves do not have
access to a hundred percent of the source
code of the cars they make because ECUs
contain intellectual property from other
companies. So this is mostly a political
topic. So there is not much we can
contribute to as researchers. However what
we can contribute technically right now is
to try the other way around and use what
is publicly available to make it
accessible, to learn and research
automotive systems. And this is what we
try to do with RAMN, which is the topic of
this presentation. The objective is to
provide a platform for research that is:
Open, and by that we mean it should be
easy to modify the source code, and
reprogram the ECUs; Accessible, and by
that we mean inexpensive, small and
requiring no prior skills in automotive
systems; Safe - with no risk of accidents or
legal repercussions; and motivating,
something that you can interact with so
that you get the same kind of experience
as you do when you experiment with a real
car. So already some solutions are
available if you want to experiment with
an ECU network. Besides, of course, a real
car, the first one is making your own test
bed from real ECUs so we can see many
hackers sharing the test bed at security
conferences. And usually if you see
something like this, you stop and you
immediately get interested. So it is a
nice way to motivate people to learn.
Unfortunately, those are not easy to
reprogram because manufacturers do not
share information about the ECUs and they
require a lot of skills to build. So it's
not accessible to everyone. Another option
is to use development boards such as
Arduino, and that is what you can see
mostly on academic papers. They have the
advantage of being reproducible and you
can modify the source code as you want. So
they can be used in many cases for
research. But they lack many safety
features offered on an actual ECU hardware
and software. Even if you are able to
simulate a CAN bus, you don't get the same
level of interaction as you do with a real
car. So it's not something that motivates
people and makes them want to learn more.
And the third option is to use a
professional test bed such as Pasta -
another work from our team. This is a good
option because you get access to the
source code and you can reprogram the ECUs
and the CAN bus network is already
simulating a full car. So the groundwork
is already done. So major drawback are
that it is very expensive. So it is not
accessible to everyone. So there are some
options to study and research automotive
systems, but none of them seem to be both
accessible and motivating at the same
time. So many people don't even think of
running automotive systems because it
never occurred to them that they could
like it. And in comparison with other
industries, you have so many ways to get
started, if you want to learn about Linux,
you can start with a Raspberry Pi. If you
want to learn about electronics you can
start with an Arduino and so on. So we
wanted something that would give a similar
experience, but for automotive systems. So
we noticed that most of the test beds that
people are using to experiment with ECUs
are made of 4 ECUs. So the ECUs are often
communicating with each other using a CAN
bus. So we tried to fit all that in a PCB
the size of a credit card. And we named
that PCB RAMN. It features four ECUs
connected over a common CAN bus or CAN-FD
bus, which is accessible from outside by a
terminal block. One of the ECUs is also
connected to USB, which is also the power
supply. PIN-sockets can be used to
connect sensors, actuators and additional
hardware and the board features many
probes to easily access important electric
signals. The 4 ECUs simulate a CAN network
with messages identical to Pasta. The name
RAMN is obviously a reference to Pasta as
this is a cheap alternative, mostly aimed
at students. In real cars CAN messages
typically have different payload sizes,
but by default we operate with maximum
payload size, to demonstrate heavy
traffic, so the basic format is like this:
Arbitration ID, 2 bytes for the data,
2 bytes for a counter and 4 bytes of
additional data, random data used as a
placeholder for additional data such as
checksum or MAC. You can easily modify the
CAN bus's arbitration ID end format. And
here we are assuming a full by-wire
vehicle, which means all physical
functions of a car are accessible to the
CAN bus, which is usually not the case on
current cars. The block diagram of RAMN
looks like this. And as I explained
earlier, all ECUs are periodically
exchanging messages on the CAN bus. If you
connect a CAN adapter and have a look at
the traffic, it will typically look like
this. So the board itself is enough to
simulate an ECU network, but it does not
look very motivating. What we wanted on
top of this was some sensors and actuators
to make it more interactive. So we created
4 expansion boards for sensors and
actuators. To simulate the infotainment
domain, we simply use a screen. For the
body domain, we use an engine key and some
LEDs. For the chassis domain we mainly use
a slide switch to simulate a side brake
and a rotating potentiometer to simulate
the steering wheel. And for the power
train domain we use slide potentiometers
for the brake and accelerator and a
joystick for the shift lever. The EC
connected to USB implements a standard CAN
or CAN-FD interface, either of a standard
serial port using Slcan or over a native
interface on Linux thanks to the
counterlight firmware projects. If you
connect the board to a USB port on the
computer, it should be recognized as a USB
to CAN adapter. So it is not necessary to
own an external CAN adapter to get
started. This is a demo of what it looks
like to use RAMN. Just connect it over
USB, if you use Linux, you can get it to be
recognized as a standard CAN network
interface. So it will show up in ifconfig.
Then you can use various tools available
on Linux to observe the traffic, for
example CAN sniffer. Here you can see the
traffic explained earlier, the CAN IDs on
the left and the payload on the right. So
with this, we can simulate a ECU network
with sensors and actuators, which is
enough for basic interactions. But it
still doesn't feel like you are actually
experimenting with a car. Ideally the ECUs
should be performing realistic ECU
functions, not just lighting up LEDs based
on some switches and potentiometers. And
for this we thought of a connection in a
closed loop with an open source driving
simulator which is an affordable and safe solution.
To feel like you are driving the ECU network.
Fortunately, there is a great open source
driving simulator for autonomous driving
research. It is called CARLA. It features
a Python API. So it is very easy to
interact with it and it also comes with an
example self-driving algorithm. So you can
immediately start experimenting with your
virtual self-driving car. So we wrote some
scripts so that most sensors values for
example speed and altitude would be
simulated on the computer running CARLA.
Then broadcasted on the CAN bus. On the
other side we made it so that all controls
of CARLA such as throttle, steering and
brakes will be decided by the ECU network.
And this is what you could refer to as
HILs or Hardware-In-The-Loop simulation in
the automotive industry, RAMN is not as
advanced as professional tools, but at
least it is accessible. So to achieve
manual control, it is not complicated with
CARLA. On the manual control example
provided with the API there is a while
true game loop that with events of the
keyboard, applies controls, then updates
the physics simulation. So CARLA does not
simulate the CAN bus by default so we
created a Python class to easily interact
with the CAN bus using the CAN message
specifications of RAMN. To integrate it
with CARLA we just need to replace
keyboard controls with actuator control
data read from the CAN bus. To close the
loop we broadcast sensor data using CAN
messages based on data retrieved from the
Python API of the physics simulator. And
with this we are able to control the car
manually, the potentiometers on the board.
Here I gently press accelerator, release
the handbrake, and I can control the car
using the steering wheel
on the expansion board.
Video stops
So manual control is nice, but
automatic control is better because
ultimately we want to focus on working on
the CAN bus. So on the original CARLA
project, there was also an example script
for automatic control. Again there is a
while true loop where the code basically
simulates the physics, lets the self-
driving AI make a decision, then applies
the controls to the physics simulation. To
integrate RAMN in this certain algorithm,
again, we need to replace the apply
control part with the controls from the
ECU network. We also need to send CAN
messages with sensor data retrieved from
the Python API of the physics simulation.
That is identical to having manual
control. What we need to do more is also
send a message to broadcast the status of
the AI to the ECU Network, so that the ECU
network knows what controls the AI
algorithm is requesting. So periodically
the AI would broadcast its status by
sending messages over USB, which are
converted into CAN messages by the ECU
input to USB. All ECUs on the network will
receive those messages and will decide the
actual controls of the car based on their
own algorithm. For example, the power
train ECU may decide to apply brakes
depending on the highest value between the
AI brakes and the brake potentiometer. All
ECUs will receive the CAN message and
apply the control, some ECUs may filter
that message if they do not need it. Some
ECUs like the body ECU may process it and
take action. For example, if the brakes
are engaged, the body ECU will light up
the stop lamp on the expansion board.
Finally, the ECU connected to USB will
forward the brake controls to the
simulator that will apply the brakes in
the physics simulation. So this is what it
actually looks like, all I need to do
again is release the handbrake and the car
will drive itself. The car is controlled
by the ECU network, so when the car stops
in the simulation, it is because the
controls were applied by the power train
ECU. You can see that the stop lamp lights
up because the body ECU also received and
processed the break CAN message. Since the
ECU is in charge of the controls, I can
force the car to stop any time by forcing
brakes on the power train ECU potentiometer.
So if you connect an external
CAN adapter to the CAN bus,
you'll be able to send and receive
messages. So using RAMN you can experiment
with the CAN bus of the self-driving
virtual car. When you want to reprogram an
ECU in the real world, you have two
options. Either you interact with the
hardware bootloader of the ECUs
microcontroller, which depends on the
microcontrollers model and manufacturer,
or you interact with diagnostics and
calibration software, which depends on the
car model and manufacturer. Diagnostic and
calibration are often done on the CAN bus,
but other options may be available.
Protocols are defined by standard
documents. You often hear about UDS and
OBD-II which both rely on the same
transport layer ISO-TP. But there are also
other protocols, such as Keyword Protocol
2000 or XCP. All those protocols can often
also be implemented over other layers. For
the hardware bootloaders it depends on the
microcontroller manufacturer. For example
for an STM 32 microcontroller you can
reprogram the firmware by interacting over
CAN according to the application note
3154, over USB according to the
application note 3156. Or using other
protocols defined in other application
notes. In the case of RAMN, we use the
hardware bootloader to reprogram the ECUs
and you can also use a UDS protocol over
CAN. And in the future we may consider
adding XCP and KWD 2000. How to reprogram
an ECU using calibration and diagnostic
software is a topic that is already
heavily discussed at automotive security
talks, I will not spend time on this
topic. You can find the definition of the
standards online. Usually you need to pay
for them. But you can also find summaries
for free on some websites. For example,
Wikipedia. What is more interesting to
discuss here is the hardware bootloader.
The various ways to force a microcontroller
into bootloader mode are
described in the application note 2606.
The format of CAN messages we need to send
to interact with the bootloader is defined
in the application note 3154. And it's not
complicated. The format is simply common
byte plus argument. All that within the
same CAN message payload. So we wrote some
script to make it easier to interact with
the bootloader. So here I am showing the
script that we use to interact with the
bootloader. First thing I do is retrieve
all information from the bootloader,
including the version, the supported
commands and the Chip ID. I then use a
read memory command to dump all known
sections of memories. This includes the
ECU firmware. So it can be a good start to
start experimenting with reverse
engineering. We can also activate memory
without protection and see what happens
when you try to dump the memory again. And
in this case is not allowing memory dumps
anymore. You can remove the memory
protection, which will wipe out the
memory, and after you do that, you can use
the bootloader to reprogram the ECUs. For
the hardware we also designed additional
expansion boards with various features.
First one is an expansion to connect a
JTAG debugger. Second one is about to add
extra QuadSPI Memories, which is why you
see packages such as EEPROM, FRAM, NAND,
SRAM and so on. The third one is a board
to add a trusted platform module. And the
last one is an interface to easily connect
a chip whisperer. All those expansions are
compatible with each other, so you can
stack them and create a very advanced ECU
network with all ECUs between a TPM and
one gigabit of memory. And it looks better
when the expansions are stacked on top of
each other. Since we would like to use
RAMN for education, we try to vary the
type of electrical signals using the
expansions. And we added many external
probes to easily connect external tools. You
can use one of the many excellent probes
to connect an oscilloscope and have a look
at analog signals. For example, here's the
brake potentiometer. Or connect a logic
analyzer and have a look at digital
signals, for example, to analyze SPI
communication between the ECU and the
screen. For the hardware design we kept it
simple using only two layers and keeping
all components on the same side. We
designed the hardware with flash
tolerances which should be easy to produce
with most PCB fabrication services. We use
components with large size. So if you have
some skills in soldering, you should be
able to solder one yourself. We designed
the hardware using KiCAD, which is an open
source tool for designing hardware. You
can easily modify the schematics, layout
and even generate nice 3D views. So
firmware is designed using STM32CubeIDE,
it is accessible to beginners, and you can
easily reconfigure peripherals and clocks
from the graphical interface. You will
even get some statistics, such as the
estimated power consumption. But of
course, you do not need to use the
graphical interface and libraries if you
would rather program everything yourself
by interacting directly with registers. We
use free RTOS as a Real-Time operating
system. The basic software of ECUs has
several tasks running in parallel, one for
receiving CAN messages, one for sending
CAN messages and several periodic tasks to
take care of the ECU control loops. Those
tasks receive data from different
interrupt services within, using queues and
DMA memory overwrites. The tasks can also
exchange data between each other using
queues or memory overwrites. Again, to
keep the project accessible to beginners,
we did the OS configuration using the
graphical interface where you can see all
tasks and their configuration, for example
the priority. You can add or remove
interrupts. And you can also configure the
various queues in memory. There is still a
lot of memory left even though the memory
controller is less performant. So you can
easily add your own application on top of
this. That's all for the hardware and
software section, you can find more
details on GitHub. I would like to move to
the next section and show you a usage
example. There are usually two budgets for
people working in automotive security
Either they are automotive engineers who
learn about security or they are security
engineers who wrote about automotive
systems. Since this is a security
conference and I assume most people do not
need me to explain how the platform can be
used to study and research basic security
topics, I will focus on the automotive
side. So diagnostics and calibration topic
is already covered by many security
talks. I will not spend time on this. So
I will spend time on something
that is not often mentioned at security
conferences: Control algorithms and safety
critical hardware and software. And for
this, I would like to demonstrate the
design of a PID controller for cruise
control as an example. I will show you how
being knowledgeable in control systems may
be relevant to security engineers, and how
many of the activities that are done by
engineers in the automotive industry can
also be simulated using open source tools,
including RAMN. Once we have an
implementation in C that works, I would
then use it as a reference to talk about
safety critical systems and the
differences with real ECU hardware and
software. So let's get started. Cruise
control is very simple: When a human is
controlling the throttle with the
accelerator pedal depending on the skill
the car may have an uneven speed. What we
want is an ECU that optimizes the control
of the throttle so that we can maintain a
steady speed. An automatic car can be very
easy to model. If you press the
accelerator pedal, which opens the
throttle, you will get speed out of your
car. But the relationship between speed
and throttle is not as simple as a
multiplication. No, we have to follow the
laws of dynamics. In this case that's the
sum of forces on the car is equal to its
mass times its acceleration. We can
consider that there is a force pushing the
car that is proportional to the throttle,
and that there is an opposing force
proportional to the speed due to friction.
When we solve this diversal equation, what we
expect to see is that the speed follows an
exponential curve. And a simple way to
control the speed that may come to your
mind is open loop control. You make some
measurements on the flat road of the
relationship between throttle and maximum
speed, and you keep in a lookup table.
When the user asks the ECU to reach a
certain speed, the ECU can use the lookup
table to find what throttle controls
should be applied based on past
measurements. And this may work in some
situations, but according to the laws of
dynamics, as soon as we reach an upward
slope, the car will lose speed because of
gravity. So at least that is what we
expect, but we should verify that on the
CAN bus. This is something we can use RAMN
for. Here again, using an external CAN
adapter connected to a second PC. On that
PC I simply receive data from the physical
CAN bus. For the rest of the section I
will only be modifying the firmware on the
power train ECU. I will not change the
simulator, I will not even reboot it.
So in the simulator, I drove around the city
to try to find a nice place to experiment.
More precisely, I looked for a place with
flat road followed by an upward slope.
Then I programmed the power train ECU to a
apply a constant throttle, which is only
one line of code. And after reprogramming
the ECU I let the car drive straight and I
recorded data from the CAN bus. I used an
open source tool for CAN bus analysis
called Bus Master. Bus Master allows you
to load DBC files or to input manually the
format of your CAN frames. Here I simply
told Bus Master what were the CAN
arbitration IDs of the throttle control
message and the speed sensor message. And
what was the format of the payload. Once I
do that, I can plot the evolution of the
throttle and speed over time. And the
results we get are exactly what we
expected from our differential equations.
The speed of the vehicle is following an
exponential curve and as soon as we reach
the upward slope, we start loosing speed
because the throttle is constant. There
are some noise and non-linearities at low
speed but overall it seems our model of
the car is correct. We are approaching the
problem correctly. What we can see here is
that it takes about 40 seconds for one
test-drive. And 40 seconds is already too
long. So before testing the ECU in the
driving simulator, we want to use the
software that can simulate differential
equations so that we can see the impact of
the cruise control algorithm directly,
without having to wait for 40 seconds.
Most of the time this is done using a
professional tool such as Matlab and
Simulink. But here we will use open source
tool Scilab. It has a graphical interface
where we can connect inputs and outputs
from a block diagram. Differential
equations are a bit hard to deal with
because the relationship between inputs
and outputs is complicated. What you
typically do in control theory is use a
Laplace transform, which will change the
variable called t to a complex number
called s. This may be complicated with a
contradictory background, but you just
need to know that a differentiation is
equivalent to a multiplication by s and
that integration is equivalent to division
by s. In our system it is easier to model
the Laplace transform because we now have
a simple relationship between throttle and
speed. Speed equals transfer function of
car times throttle. And based on the
measurements from the CAN bus, we can
evaluate the transfer function of the car
to be equal to approximately 1 / (1 + 4s).
We can simulate the car in Scilab by
using a block function which has the exact
same transfer function. Using Scilab, we
can now test various scenarios and get the
results immediately. Here I am testing the
scenario in which we start from zero
speed, apply a constant throttle and after
20 seconds, we add a new force - gravity -
which corresponds to the slope. And this
is what we call here the disturbance. And
with Scilab simulation we can verify we
get waveforms similar to our measurements
on the CAN bus. With a constant throttle,
the speed follows an exponential curve
that is close to maximum speed after
around 14 seconds. And as soon as there is
a disturbance: the gravity here, showing
green, we can check that the car looses
speed because there is no reaction from
the throttle. So to fix that the solution
is obvious: The ECUs need to have
feedback. We need the ECU to make use of
the speed sensor data that it can find on
the CAN bus so that it can adapt the
throttle to the actual speed of the
vehicle. So first solution that may come
to mind to software engineers is Bang-Bang
Control. Bang-Bang Control is quite
simple. You measure the speed and if it is
above a certain threshold, you stop
applying the throttle. If it goes below a
certain threshold, you apply the throttle
again. This is extremely easy to implement
in C on the ECU. And once we reprogram the
ECU on RAMN, we can make measurements on
the CAN bus and verify that this time we
are not loosing speed anymore when we
reach a slope. But as you can see, there
are some oscillations. And as you can
imagine, the oscillations are not
something very nice for passengers.
Apparently people driving like this is the
reason cruise control was invented. I do
not know if that story is true, but I can
believe it. So Bang-Bang Control made a
good start, but it is not good enough for
cruise control. And the most famous
algorithm used in control theory is the
PID controller. You can find a lot of
resources online. The PID controller is
one of the control mechanism used in
software of the self-driving AI of CARLA.
In the PID controller you measure the
difference between the target speed and
the current speed. You call that different
error and you can control the throttle
using the sum of 3 control blocks. The
error multiplied by Kₚ. The integral of
the error multiplied by Kᵢ and the
derivative of the error multiplied by Kd.
Kₚ, Kᵢ and Kd are constant called gains.
And you need to have a very fine tuning of
those gains to achieve optimal control.
Here I can simulate the PID controller
using Scilab with different blocks.
Remember that the division by s is an
integration, and the multiplication by s
is a derivation. Thanks to the simulation
I am able to try many values without
having to actually drive the car. And when
we are able to find correct values for the
PID controller, we get this. The ECU is
able to reach a target quite quickly
without oscillations and without
overshooting the maximum speed. And when
there is a disturbance so gravity, it will
dynamically adapt the controls of the
throttle so that the target speed is
maintained. So this is good, because this
is what we want for cruise control. But is
a gain of only one control block is
correct, the speed of the vehicle may look
like something totally different,
potentially dangerous. And this is why the
integrity of calibration data is important
not only from a safety point of view, but
also from a security point of view.
Because an attacker should not be able to
make an ECU have a dangerous behavior with
only one small change. The last thing I
need to explain is how to implement these
algorithms in C, and this is not obvious
because we're dealing with integration and
derivations, which are not possible for
digital functions. So there are many ways
to implement this in a digital PID
controller in C. I will just explain two
approaches. The first approach is to stay
in the time domain and approximate the
derivation by the difference between two
successive errors divided by the sampling
time. And we can approximate the integral
operation with a Riemann sum which is a
running sum of errors so far multiplied by
the sampling time. This may look a bit
intimidating, but when you look at it
closely, you can see it is just a
combination of constants and values that
can be computed from current and past
sensor values from the CAN bus. The actual
implementation in C looks like this. We
need to define 2 variables. One to store
the running sum of errors and one to store
the error of the previous control loop
execution. In the control loop, we define
constant gains for each stage. We compute
the current error. We add the error to the
sum of all errors. We compute the
difference between current errors and
previous error. We then add all those
values with the respective gain to the
output variable and then we clamp that
output in case it goes out of bound. We
then apply the throttle control and save
the current error in the previous error
variable for use in the next iteration.
The second approach is to use a Laplace
transform of the PID controller. We first
need to convert it to a Z transform, the
equivalent of Laplace transform but for
digital signals. It looks a bit
complicated, but there are many tools to
do the conversion for you. If you want to
do the conversion by hand, one way is to
use the bilinear transformation in which
you replace s by this, to approximate the Z
transform of your ECU. And again, this may
all look a bit intimidating, but you can
actually compute the throttle output by
using the throttle output from 2
iterations ago, the current error, the
previous error and the error before that
which can all be computed from sensor
values on the CAN bus. And while this
control algorithm is equivalent to our
previous implementation, it looks totally
different. And what I would like to stress
here that identical control algorithms may
have very different software
implementations, which may be relevant for
reverse engineers. I only show the first
implementation not waste time, but you can
see now that the ECU in RAMN is able to
make the car maintain a constant speed and
for dynamic control of the throttle. So
that's it for the example. I just wanted
to show that RAMN can be used for
realistic activities of the CAN bus and
that the ECUs are not just doing some
random easy work. The control theory part
may be complicated and if that was going
too fast for you at least I hope it proves
there is a lot of things to discover and
experiment with. And that all that
learning can be done with open source
tools. Now I would like to discuss what
would have been different with real ECUs
because, as you can imagine, actual ECU
software is not as simple as this. I will
also show you what alternatives we have.
Technologies hidden behind NDAs, so that
we can get as close as we can to real
ECUs. We showed in RAMN that this cruise
control ECU worked, but we only tried it
on a single scenario, namely a flat road
followed by an upward slope. But what
about this scenario in which a car in
front is driving slowly? Or what about a
scenario in which we are in a downward
slope? In this case the ECU will not be
able to prevent a car from going too fast
because it does not have access to the
brakes, which could lead to an accident.
And the difficult problem here is not
whether you can think of one more
scenario. The difficult problem is can you
think of them all are you sure? And
thinking of all potential scenarios,
quantifying the risk and implementing
countermeasures is what makes automotive
systems really different. And to prove
that an ECU is reasonably safe, you
actually need to follow ISO 26262
standard, which is a standard for
functional safety. The standard defines
different requirements at many levels. Not
all ECUs are equally critical, so safety
relevant issues are assigned and
automotive safety integrity level or ASIL
for short. And ASIL A is a less critical
level and ASIL D is the most critical
level. So if you were to design a real
cruise control ECU for use in a real car,
you could not just connect some random ECU
to the CAN bus and try it on the highway.
You will need to go through a lot of
analysis, such as HAZOP, HARA, FMEA, STPA
and so on. Usually there are so many
requirements that they cannot be tracked
with only a human and a sheet of paper.
They are managed using dedicated tools
such as Rational Doors. Now let's discuss
how the software would be different. The
main contribution of automotive software
is to realize control algorithms safely.
But without countermeasures, many things
could go wrong. For example there could be
bugs in the ECU application code. And then
without any bug the software will be
robust enough when there are transient
errors in the hardware. There could also
be problems with the toolchains that
compile the firmware and problems with the
libraries and RTOS. And when we have a
close look at the PID controller
implementation which seemed good enough in
our testing, you can see it is actually a
terrible implementation. We are mixing
integers and unsigned integers and floats
without proper checks and type casting. We
are not checking for overflows and also
random software issues. And this is not
acceptable, both for safety and security.
And in this case, the problems were
obvious on purpose, but sometimes it can
be very hard to spot because they stem
from very subtle computing issues. And
those issues may lead to system failures.
So to avoid such scenarios, the automotive
industry usually mandates the use of a
language subset which restricts what
developers can do, but makes sure that
numerical errors are less likely to
happen. So usually in the automotive
industry, the standard that is used is
MISRA-C and it is very similar to CERT-C,
which is popular in the security industry.
Using a language subset is only one of the
requirements that are dictated by ISO
26262. There are many other requirements.
At a high level they try to enforce a low
complexity of the software. For example,
by restricting the size of components,
restricting the copying between software
components and making sure the scheduling
is appropriate. And that there are not too
many interrupts. But we also have more
concrete requirements, such as restricting
functions to 1 entry and 1 exit point,
forbid dynamic memory, avoid global
variables, limit the use of pointers and
so on. The other issues we have to deal
with, which is riddled with bugs is
transient errors. In the harsh
environment, data is not always reliable.
There may be a bit flip occurring outside
of memory, for example, because of noise
and communication lines, but also may be
bit flips occurring inside a microcontrollers
memory. For example, because of cosmic
rays. Those issues do not originate from
software, they originate from hardware,
but they do need to be addressed by
software because remember, in the case of
the example cruise control ECU, just one
bit flip could lead to unwanted behavior of
the ECU and the car. So to address those
issues, automotive controllers need
special countermeasures. For example
having redundant memory or having
redundant CPUs. In ECUs, you will
typically find microcontrollers that have
been designed specially for automotive
use. All those microcontrollers require
you to sign an NDA so you can not just buy
them and start programming them. So that
makes it a bit hard to study an actual ECU
microcontroller and real automotive
software. ISO 26262 is not the only
standard for safety critical systems. ISO
26262 is actually derived from IEC 61508,
so they are both similar in their
concepts. And IEC61508 microcontrollers
they do not require NDAs for most other
activities you may be interested in. And
more completely RAMN can be used with
STM32L4 or STM32L5 microcontrollers. And
for those microcontrollers, you do not
need an NDA to download guidelines on how
to implement safe software. For example,
you can find a list of features that are
required for safety applications, and you
can request more data that you would need
to actually achieve compliance with
IEC 61508, such as the FMEA and FMEDA. But to
obtain those data, you would need to sign an
NDA. No, I personally do not think that
those data are essential for education and
research. So using such microcontrollers
is a good alternative. But again, let me
stress that this is an alternative for
learning and researching, not for actual
use in a car. I don't have time to detail
all the safety features, let me just talk
about memory redundancy, since this one
impacts the code of the application of the
example cruise control ECU. So in the
example, we wrote the gain of each stage
in code memory defining them as constants.
For safer applications, this would not be
here. They belong to another section, the
data flash where ECC protection can be
activated. If possible, calibration data
should be stored twice with checksums and
preferably MACs. If you're not familiar
with ECC memory, it is a kind of memory
that can detect bit flips and sometimes
automatically corrects them. Another
memory is also available for the RAM,
but not at all addresses. So in the
application, we have to ensure that safety
critical variables are placed in a section
of RAM in which bit flips can be detected,
in this case in section SRAM2. For data in
RAM that are actually constants such the
gains you may also want to activate write
protection, a feature that is only
available in SRAM2. Last slide about
software. In the example cruise control,
we are using GCC toolchain in ISO 26262 it
is a requirement to have what is called a
tool confidence lever to ensure that
poor chains will not introduce errors as it
could be the case with some optimizations.
So normally you cannot use GCC. Realtime
operating systems and libraries may also
have problems. That is why they need to be
certified. Both STM32-HAL and FreeRTOS are
compliant with MISRA-C which is nice, but
they are not compliant with ISO 26262.
However, it looks like ST is bringing
Azure RTOS into the ecosystem and that one
is actually precertified ISO 26262. Maybe
in the future it would be an option to
experiment with with an actual ISO 26262
operating system. So now let's talk a bit
about hardware. In case it was not clear
to you, you cannot use commercial
electronics to implement an ECU.
Smartphone vendors will often warn you not
to let a device in your car because a
parked car can reach extreme temperatures
that commercial electronics are not
designed to resist. And if you think that
the life inside the cabin is hard, you
should think about an ECU which has to
stay in the engine compartment and operate
without failures. And you would not think
of putting a smartphone or an Arduino here
and trust your life with it. And extreme
temperatures are just one of the many
environmental factors that make it
difficult for an ECU to stay reliable. The
ECU also need to resist high humidity,
corrosive gases, vibrations, micro cuts,
load dumps, electrostatic discharges,
electromagnetic noise and so on. And when
subjected to such a harsh environment,
many things could go wrong with
electronics. You probably know about
corrosion, but many other physical
phenomena are at risk of happening to the
components. Solder cracs, intermetallic
growth, whiskers, dendrites,
electromigration, etc. For example,
whiskers are metal growing out of
electrical components and dendrites are
metals leaving the plus side towards the
minus side. And many other phenomena may
result in a dangerous failure. So
obviously, ECUs need to be designed to
resist harsh environments and have
countermeasures against all those
potential failures. ECUs need to pass
various tests that simulate harsh
environments. Those tests are usually
defined by manufacturers and the test
specifications are not made public. What
is made public, however, is the test
specifications for individual electronic
components. And those tests are usually
defined by AEC. So the Automotive
Electronic Council, and you can have a
look at them online. For RAMN, we tried to
follow design guidelines similar to those
of real ECUs. But of course, we cannot
follow actual rules as it to be much less
accessible. Completely, we selected
AEC-Q100 grade zero components for
everything except connectors and
microcontrollers, because those may
require NDAs or not be easily accessible.
Depending on the part reference, ECU
microcontrollers may be usable from -40 to
125°C. RAMN tried to stay close to
automotive grade, but it is still not
automotive grade, especially in the
reliability department, so it can not be
used as a real ECU. As a result, we try to
stay close to automotive hardware is to
help researchers evaluate the impact of
manufacturing tolerances and environments
because remember, manufacturers are making
millions of cars that need to operate on a
large temperature range. So if you are
developing, for example, a security
technology that relies on hardware
characteristics such as the clocks of the
ECUs, you will need to prove that the
technology works despite manufacturing
tolerances and harsh environments. And
with RAMN it is easy to have a large
sample of ECU networks and since they are
small they can easily fit in various
testing equipment. And now let's move on
to the last section, security. In the
automotive industry, we just can not apply
the same reasoning as you do in many other
industries. For example, a credit card, if
you detect the temperature is too cold, it
may think that it is a tampering attack
and decides to shut down because it is not
safety critical. On the other hand, the
car needs to start quickly because the
user should not be left out in the cold
and also credit cards have an expiration
date. So they do not need to guarantee
security for more than a few years. But
cars do not have an expiration date. If
they are well maintained they may be used
for several decades and the security
technologies should keep on working. So in
the end, automotive security technologies
have different requirements.
Unfortunately, according to past research,
a security technology is often less
reliable when you extend its operating
temperature range and its lifetime. For
example, at low temperatures, they may be
liable to cold boot attacks. At high
temperatures, it has also been shown that
electronics tend to be less reliable
concerning glitching attacks and in those
papers high-temperature means something
like 60 or 100°C, far from the maximum
temperature required for some ECUs. Also,
it has been shown that the higher age for
electronics usually results in different
security properties. And you may think
that the safety features of automotive
microcontrollers would prevent some
attacks such as glitching attacks, but it
has been shown that ECC memories are also
susceptible to glitching attacks. And that
even ISO 26262 ASIL-D microcontrollers,
which is the highest level of safety may
be susceptible to glitching. So safety
features often help, but there aren't
really enough to ensure security. What is
also different with automotive is that you
need to rethink the strategy in case of
security problems, for example, with
credit cards, it is not uncommon for
authentication to fail randomly. When the
credit card fails to work usually you just
need to try it once more and it will
probably work and everything stays again.
No life is at risk. But the car cannot
have the same strategy. If you add
authentication to the brake CAN message
and you start receiving break request that
fail authentication what should the car
really do. Because it may be a cyber
attacks, which you want to avoid, but you
should not relay out the possibility of a
random malfunction or a false positive for
an intrusion detection system. And by
adding complexity into the system, you
always increase the odds of a problem. And
which one would be worse between breaking
because of a cyber attack or not breaking
because of a malfunction? And there is no
easy way to answer that question. But what
I want to stress here is that many
security countermeasures that people
suggest for cars such as encrypting the
CAN bus, permanently disabling debug ports
or obfuscating the firmware, they may not
necessarily be the best ideas, because if
you suspect a malfunction with an ECU, you
need to investigate the problem seriously
because it may harm people. You cannot
just replace a car as you would with a
credit card or a smartphone.
So technologies that can truly take into
account both automotive requirements and
security requirements are better. And we
should make sure that education and
research in these areas are accessible to
many researchers without NDAs or
prohibitive costs. Now, of course, you can
use RAMN to try out different attacks. The
first obvious one is to inject CAN
messages to alter the behavior of the car.
Here for example the breaks. Another kind
of security that I did not mention in this
presentation is physical security for
sensors and actuators. Here I am
demonstrating what happens when it
overtake the control of the steering wheel
actuator. A human will probably break in
this situation. The self driving algorithm
in CARLA here does not realize it has lost
control of the steering wheel and is still
trying to correct the trajectory when a
better decision would be to stop the car.
So this is the end of the presentation. We
developed an inexpensive, safe and
interactive platform to study and research
automotive systems. The platform is
accessible to beginners. It is not
automotive grade, but is close enough for
research and educational purposes. The
project is open source and with permissive
licenses. If you have questions or ideas,
do not hesitate to contact us, especially
if you are involved with education,
research, training, CTF, etc. And thank
you for watching.
Herald: Camille, thanks for this
comprehensive talk, this was amazing. We
unfortunately don't have much time for
the questions or answers, but there is
one question that popped up, which is
about the hardware and your PCB. How did
you design it? How much does it cost
actually, how can you get hold of that
thing?
Camille: So I designed everything with
KiCAD. And I mean, I think a few years ago
it was very hard to design hardware, but
now you have footprint libraries available
online. It has become very easy. The board
was between 50 to 100€ for a quantity of
one. So microcontrollers are obviously the
expensive parts and the PCB assembling
parts - it is up to the PCB Fabrication
Service. If you have questions just ask me
on GitHub, I would be happy to answer.
rC3 postroll music
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!