33c3 intro music
Herald: Let me present Felix Domke with
the Software Defined Emissions: A Hacker's
Review of Dieselgate.
applause
Felix Domke: Yeah, hey everyone. Thank you
for coming here. I saw there are a lot of
interesting talks at the same time in the
other rooms, so thank you for coming here
and listening to me about software-defined
emissions. "A Hacker's Review of
Dieselgate" is the subtitle. I'm Felix
Domke. I usually do embedded software,
mainly security. I'm definitely not on
cars and definitely not on things that
have combustion thingies, so I only got
dragged into car software last year when
my own Volkswagen car was accused of
cheating and I wanted to know what exactly
was going on. I held the talk last year
about some of the details of the
Volkswagen Sharan defeat device. For the
details you can take a look at that talk.
This time I want to look more at the
process of finding or analyzing car
software. I want to look at whether this
process scales to more cars. The first
step when having a piece of software that
does not always do what people think it
does is, well obtain a firmware image,
obtain a binary image of the firmware, and
in the case of my car I knew it was a
Bosch EDC17, which is a Bosch ECU that a
lot of cars use, including my Volkswagen
car. So, I didn't know anything about
ECUs, dumping software and so on, so I
asked Google "Hey, what do I need to do to
dump an EDC 17," and Google had a lot of
answers for this, but usually those were
people that wanted to sell me some device.
Those were chip tuners that built their
own devices where you can plug in the ECU
and then it extracts the image, usually by
exploiting some bugs in the software. But
I didn't really want to buy something and
it takes like a lot of time until I get in
my hands. I wanted to start. So I was
looking to do this on my own. What these
sites usually tell you without paying is
how you wire up your given ECU for their
device, so they tell you where to connect
12 volt, where to connect the CAN bus,
which is the serial communication bus that
the ECU uses to communicate with the rest
of the car
devices. Usually it's pretty easy, so...
when analyzing ECU it makes a lot of sense
to reproduce the scenario on your desk and
not in your car, so in order to make an
ECU boot all you need is ground, 12 volts,
there's usually an ignition pin that you
also have to supply 12 volt to it, and
then it boots. So on my desk it looked
something like this, and then once we have
the setup we can boot the ECU, we can use
Python to talk to the ECU, which is great,
and then we can use socket CAN, which is
the Linux CAN support that's really great,
and we can even use MicroPython if we
want to have a smaller device that we can
put in a car. So we can talk with the ECU.
Talking with the ECU in modern cars,
there's a protocol called UDS. Basically,
I simplified this slightly, you can ask
the ECU "Hey, I want to read memory by
address," you give it an address and you
ask it to read four bytes in this case of
that address, and then it returns it to
you. So I thought "Hey, maybe I can use
this to dump the software." In my case the
device responds with a Security Access
Denied, so I looked into what I need to
do. You actually have to do a Security
Access command. You send a command that's
called Request Seed. You get back
basically a 32 bit random number and then
what you have to do is to, have to
process this seed through a super-secret
function and then return it in a response
call. The question is how do we know this
super secret function. There are multiple
methods. We can look at the ECU software
itself, if the algorithms in there, to
verify it. We can reverse diagnostic
software that uses this mechanism. So for
example the Volkswagen software they use
for car shops, or maybe someone else
already reversed this and put it in their
own tools which may be easier to get it
from in terms of third-party diagnostic
software. And in the case of my Bosch ECU,
the super secret function was this. I
basically had to add this number to it.
The mechanism is called pin code.
It's in... I mean it's not super
secret. Anyway, once I know this...
applause
Yeah, thank you. I mean, it... once you
do this, you send back the result and hey,
then you can read... you can send the read
command again and, hey you're getting back
data, so this is great, right? We can read
memory at runtime of the ECU, and we can
even do this while the car is operating.
However it turns out that for the Bosch
ECU you can only dump specific regions. You
can dump most of memory, some memory areas
are excluded, but most of the interesting
stuff you can read. But you can't read any
code. You cannot read anything in flash.
But we are hackers, of course, so we find
a way. The CPU used in these ECUs is a
Infineon TriCore CPU, and it's used, at
least this particular one, was used in the
ECU I cared about, and the security model
for this chip is that you can always enter
a specific bootloader mode and execute
your own code, so you can strap a few
lines they... the chip tuners tell you
that, right, they tell you what's high and
low, which pins you have to connect your
ground and 3.3 volt, and then it enters
this bootloader mode, you can upload some
piece of code. However, you can't read the
flash, because the flash is locked. When
you start in bootloader mode, the flash is
not readable until you write a specific
password to a register. That was not so
great, so I looked into what else I could
do. The datasheet is very specific on how
to operate this chip. For example there's
this one flash supply pin. So, even though
the flash is in the same package as the
rest of the CPU, it has a dedicated supply
pin and it tells you which parameters not
to exceed to ensure correct operation, but
I really don't want the correct operation
which is in this case preventing me from
dumping the flash. So, what can we do? We
can violate the requirements. The
requirements for 3.3 volts. Let's see what
happens outside of that range. And turns
out, down to a certain voltage level,
roughly 1.6 volt, everything just works as
normal. That doesn't help us.
And below that voltage the device hangs in
the bootloader, so that doesn't help us
either. The interesting parts happened
when you are at the very specific voltage
level, and this is a little bit
unscientific, because it's really just the
voltage level I tried, and then most of
the time the device comes up and flash is
protected, and then the remaining times
the device comes up and hangs in the
bootloader. But one in 10 times something
interesting happened. The device came up
and the flash was not protected, so I
could dump it out.
applause
So having the image now in my hands, I
could start in actually reversing the
defeat device and what I found was - I
don't want to duplicate a lot of what I
talked about in the last talk - I found a
function called "acoustic function," or
"Akustikfunktion" in German. It's a
function that senses vehicle speed, the
duration of the engine operation, and some
other things, and then controls emission
related functionality, or in short you can
say that this is the test cycle detection
that enables the defeat device. And I
verified it to exist on my Sharan device
by driving through the test cycle and
logging data. And during the last year I
verified that it's actually the same
defeat device, more or less, that exists
on a lot of other Volkswagen cars. All
these Volkswagen defeat devices that we
talked about for the Euro 5 cars, they use
more or less the same acoustic function.
Basically, to remind you, there are a few
curves stored in the software that look
like this. This is the NEDC. This is the
test cycle you have to drive a car
through. They exactly define how fast you
have to drive for a given time in seconds,
so it's speed over time. If we draw this
as distance over time it looks like this.
So this is the distance you got. You're
not really moving the car, because you're
doing this in a lab on a dynamometer, but
what the car thinks it has moved to, and
if we overlay this with the curves we
found in a software there's a perfect
match. So this is the way how they
describe the test cycle. So this was for
my Sharan. So I looked into, what do the
other cars do, especially what do the
cars in North America do, because they're
not using the NEDC. And I found something
interesting, or some someone sent an
interesting document to me, that was this.
It was an emission service action. It
basically describes how there was a recall
for some vehicles, that required a
software update in the shop.
So this is basically the document that
informs the car shop what they have to do,
and it had something very interesting in
it. By the way, this was in December 2014,
so this was way before the whole
Dieselgate was public, but this was
already while the EPA was already talking
with Volkswagen, already demanding
explanations. All that investigation was
already proceeding. Volkswagen knew about
this, that people figured out about the
defeat device, and it had something very
interesting in it, that said "in addition,
the vehicle's engine management software
had been improved to ensure the vehicle's
tailpipe emissions are optimized and
operating efficiently." That sounds really
fishy to me, so I was curious, what
exactly did they change in the software
update? And luckily they tell you the old
and the new software versions, and you can
then go and look them up on a firmware
DVD, that you can download on the
Volkswagen website, and it turned out that
it's an ECU software similar to the the
Bosch ECU software I looked at before. So
there's an acoustic function again there,
and the curve stored there, they match the
US test cycles. This is one of them. There
are many more test cycles in the US, so
there's another curve that matches this,
and this is the curve stored in the
software and this is the corresponding
test cycle. And there are a lot of them.
But I noticed something really
interesting, and some of the curves...
they were much wider open than the other
ones. So for example this one... there's
really a nonzero probability that if you
just have your morning commute, through, I
don't know, some streets or something,
that you accidentally match this driving
cycle every time you start driving in the
morning, so the car would, every time you
drive this, think it is in test cycle
mode, and would operate with the
optimized... in the optimized emission
mode and apparently this caused problems
and what I saw, what Volkswagen added in
the software that was part of this recall,
was this function. So, this is from it
from a disassembly. In pseudocode, this is
this. So they started looking at the
steering wheel angle, and if they figured
out that you move the steering wheel
angle, then they ignored the curves, the
more open curves. So the idea is "Yeah, if
you move the steering wheel, you're
definitely not in a test cycle, so at that
point we do not try to operate in this
emission optimized mode. And it's a
little bit of speculation, but it matches
up pretty well with with all the facts
that I read, is that because those cars
operated in the test cycle mode too often,
that eventually caused the the particulate
filters to clog, and their solution for
Volkswagen, and again, this was while they
were already investigated by the EPA, was
to add the steering wheel angle detection.
For more details, I worked with this with
the NDR, and they produced a feature on
that, so there are some more details.
So this is Volkswagen, but there are more
cars, and if we look at this... this is a
meta-study based on something that the the
Ministry of Transport... they tested a lot
of diesel cars and what they found was
this. This is actually a representation by
the ICCT. So the orange line is the
emission limit, and the bars have an upper
and lower end, and the lower end is how
much emissions the cars have. This is just
for nitrogen oxides, for NOx emissions,
what they had in the lab, when you're
driving the test cycle. So and you can see
all of these cars managed to stay under
the orange line, so they get their
certification, but when driving them on a
real street, they produce the emissions
corresponding to the upper end of that
bar, which is for some cars significantly
higher. It's off by a factor of 10 and
more. When you're driving the car on a
street. And this is interesting, because
the cars, they can meet the emission
goals. The question is why don't they
always meet the emission goals?
Why do they operate so differently in the
test cycle than on the street? And I try
to give you a partial answer. And let's
look at how a car can optimize emissions.
The first thing they do... so this is a
very simplified diesel engine. So fresh
air goes in, fuel goes out, and there's an
exhaust pipe, right? And a lot of nitrogen
oxide, a lot of NOx, goes out as well, and
we don't want that. So we added an EGR
valve, which is basically a valve that
causes a part... a fraction of the
outgoing air to recirculate again through
the engine and burn again. And what this
causes is that the flame temperature goes
down, and if we look at the relationship,
it's very simplified here, but with a
lower flame temperature you get fewer NOx
concentrations, so you improve emissions
by lowering the flame temperature, however
at the same time you're increasing the
soot level, or the particulate matter, and
there is this trade-off - if you do too
much of EGR, too much of the exhaust gas
recirculation, you're getting too much
soot, and the other hand if you do it too
few, you get too much NOx, so you can
argue that the green area isn't really
great, because there's no point where both
of them are great. And here we see the
result of a clogged EGR valve. If there's
too much soot it will clog. EGR, as the
conclusion, is the least cost solution. It
doesn't really work at higher loads. It
works at low loads, and it does not
require exhaust... high exhaust
temperatures, which is great, but
excessive use of that clogs particulate
filters, affects the combustion, the
drivability goes down, and there are
trade-offs with this. It's also not very
useful for higher engine loads, for
example when you're accelerating you have
to disable EGR at high speeds. So a better
method, that was added on top of this, is
called "selective catalytic reduction". I
am... so, basically the idea is you have
an SCR catalyst in your exhaust pipe...
there are more catalysts there, but let's
talk about NOx, and in there, this
happens. We can simplify this, somehow,
and say if you put ammonia into this
catalyst, the NOx is converted to nitrogen
and water.
And Nitrogen and water is great, it's
harmless. It's already part of the air.
The only issue is ammonia is this and this
is not something you want the driver to
refill in your car. So instead this
solution is we can create ammonia in the
car from using from something that's less
dangerous, and we have the reaction there.
We can simplify this again and say we take
urea - Harnstoff auf Deutsch - and heat,
and we create ammonia. Urea or urea
solution is this. It's called AdBlue or
DEF - diesel exhaust fluid - it's not
dangerous. You can buy it, you can
transport it... it's relatively cheap. The
idea is, we have this reaction, it
requires ammonia in the catalyst, and we
put AdBlue into it, or urea, and using the
heat that we have from the exhaust pipe,
we create the ammonia that we need to
reduce the nitrogen oxides back to
nitrogen and water. There's a great
property of this, that some of the ammonia
that's produced in the catalyst stays
there until it's used up, so
there's some storage there. So the the
requirement for creating ammonia is heat,
and if you don't have heat, but for
example because you just started up your
engine... if there is still ammonia from
the last usage in your catalyst you can
still use that, and use that up, and by
the time you have used it up, maybe the
heat is enough to supply more AdBlue and
then fill up that storage. The downside is
you need a pump to dose the AdBlue, and
you need lots of software to control this
process. And you need a heater because the
AdBlue freezes at some point, and it's an
expensive solution, it adds roughly $500
to a car, which can be significant amount
of money for a small car, and it requires
a large AdBlue tank for long service
intervals, so you don't have to refill it
every few thousand kilometers or
something. The great thing about SCR is
that it's efficient at higher loads.
There's a third method called LNT, Lean
NOx Trap, it's cheaper than SCR for
smaller engine, it doesn't require
anything,
however the bad thing is it requires
frequent re-generation, which decreases
fuel efficiency, so it's kind of a stop-
gap solution. And it's not efficient for
continuous high engine load, for example
if you're driving on the German Autobahn
at full speed, then LNT is not going to
help you much. For the sake of this talk,
let's keep in mind that EGR is exhaust gas
recirculation, that's the thing that
operates within the engine and then we
have the SCR, the selective catalytic
reduction, that uses AdBlue and is after
the engine. We also saw that all these
technologies have significant trade-offs
for NOx compliance, so we can kind of see
the motivation for a defeat device here,
because it would be the solution to all of
these trade-offs. You get no downsides
during regular driving, because nobody can
measure your emissions and while
maintaining conformance because during a
test cycle you have perfect emissions.
That kind of explains why there are defeat
devices. OK, let's get back to the bigger
picture and see what other cars do. So
this is an Opel car, it's a Zafira car,
it's a Euro 6 car, it's a pretty modern
car, it has an SCR catalyst. In theory it
should have really great, low emissions,
especially at higher speeds because that's
where SCR is good at. But quite
surprisingly it doesn't. If we look again
at this report, we can see that this
Zafira exceeds the limit by up to 12x
compared to the Euro 6 limit. This is
especially interesting because there's
this Opel advertisement where they
advertise their diesel technology applying
to the Insignia and the Zafira and they
say a lot of diesel fun without regrets,
the new diesel generation of Opel achieves
best emission values and gasoline levels.
Yeah, after they got sued for this they
had to change it slightly and they had to
add this to the sentence.
scattered laughter
So, during this testing, they had
this 12x emission
limits. For example, one particular test
was to drive the test cycle in the same
way, but at a different temperature,
at 10°C,
and the car exceeded the
values by a factor of 6 even though the
car would be operated in the very same
way. It was just that the ambient air
temperature was 10°C instead
of 25. So they asked Opel why this was the
case during their investigation, and Opel
responded, saying that the EGR and the SCR
injection, they work to the full extent in
the temperature range of 20 to 30°C.
It's what they call "normal use".
So our question was, is it really just the
temperature window? So we got a car and
investigated. The ECU in that car is a
General Motors ECU. It's developed in-
house, Opel is a GM daughter. It uses an
automotive PowerPC, yay PowerPC! It uses
somewhat obscure variable length
instruction extension to PowerPC. So how
do we start? Again, we need a firmware
image. So let's ask the Internet, "How do
I dump this ECU?" Luckily, someone in some
chip tuning forum already uploaded their
stock ECU, which means they're what they
dumped from their ECU using some chip
tuning tool. It's not the same ECU, but
it's very similar and I hope they shared
some code so I can analyze the software,
maybe find a way to dump it. So the dump
was made with this tool and the tool did
not let me export the binary. And the tool
is free, however to use any of the
features in that software, you have to buy
their expensive hardware, that then
connects to the car, which I didn't want.
This is how their software looks like. I
loaded the image I found on the internet,
and I couldn't save it or anything without
having the device attached. However, I can
just use a memory debugger and just dump
it from the address space, and that gave
me a first firmware image to start with. I
threw it in a disassembler and I found the
UDS function and, yeah, it implements read
memory by address, so that's good. Most of
the RAM was readable without a security
challenge. That is good, so I didn't even
need a security challenge to read RAM.
However, the flash, it is readable, but
only with the security challenge. So let's
take a look at the security challenge.
Maybe it's as simple as the Bosch one. So
their way of doing this is, they store
16-bit input and output value in the
firmware, and it's different for every
device, and they don't store the algorithm
to compute the output from the input,
instead they just store the pair, and,
well, it's just 16 bit, right, so let's
brute force it. The issue is, you can only
try every 15 seconds, so it's kind of
lame. The question is, how do the GM
tools, the original factory tools, get
access to that? Luckily, chip tuners had
reversed that and then obfuscated it into
their own tools. But that can be de-
obfuscated and eventually it's just a
little bit amount of bit shifting and so
that was easy to fix. And also the the GM
repair manuals tell you how to wire up the
ECU. They tell you where to put 12 volts,
the CAN bus again, and ground, and the
ignition pin, and with all of that in
place, I can do the security challenge. I
can now read all of flash memory and read
the four megabytes of PowerPC code, which
mostly consists of mathematical functions.
There are no strings or anything, it's
really hard to find what a function does,
what... There are thousands of variables.
It's really hard to find what they mean,
right, so I need to know some entry
points, some known data values, and then I
could refer to... one thing I could find
are real-world constants, for example
there's the density of diesel fuel stored,
which allows me to understand that this is
something related to fuel, an amount of
fuel. Or more useful are the OBD2 calls.
So there are some standardized things you
can ask an ECU, that's engine rpm, vehicle
speed, and things like that, and I would
find that table in the firmware and then I
had a first start of things like RPM,
speed, and so on. That was a good start.
It's not much more than what you can see
here. So there's a lot of stuff not
included in these.
The next thing I did was, I drove the car
for a few weeks and I let a device
attached that would constantly log all
memory using the rebuy address thing, and
every few minutes I would get one memory
dump, basically. It's a few hundred
kilobytes of RAM, and then I put this into
my disassembly, and that allowed me to
understand more of what individual
variables do there. And I found some
interesting things. So the first thing,
one of the first things I found, was
basically something that resembles this.
So there was something that look at the
ambient temperature, and this basically
checks for range, right, and it did...
this was for controlling the SCR systems,
and it's interesting to know that the NEDC
requires the temperature between 20
and 30°C, and this is right
centered around this, when they check from
17.5 to 33°C. But this was of
course nothing new. I found something
similar, however, another temperature
check. And as you can see it's written in
a different way. It effectively achieves
the same thing, but it's a separate piece
of code, and this time it was for the EGR
system. So we have these two exhaust
treatment or optimization mechanisms
there, EGR and SCR, and they don't share
code. They have their own temperature
window. So we found the temperature
window, which was known to exist. The
question was, is there more? And one thing
we found was this: It's basically reading
the vehicle speed and comparing to a fixed
number, and it turned out it's something
like this. So it would check the
vehicle speed and if it's above 145 km/h
it would set a flag and then under 140 it
would clear. Keep in mind that the NEDC
maximum speed
120 km/h, so during a test cycle that
would never happen. So let's see if this,
what we found in software, if this really
translate something the car does in the
real world, and it's getting slightly
technical here, I apologize, but we need
to log some variables and a useful value
to to know is, how much NOx is there after
the engine, and after the SCR catalyst,
and luckily there are 2 NOx sensors in the
car. One before and one after the
catalyst, and they give you basically the
NOx concentration in ppm. So we log that,
and we also log the signal of how much
AdBlue is dosed into the system, and we
log the catalyst temperature. And one
thing to keep in mind is that there's also
this amount of ammonia that's stored in
the catalyst. We don't have this as a
value, but just keep this in mind. And
this is how we've driven the car. The
blue line is the vehicle speed. You can
see that it goes from 0 to 150 km/h, and
the critical point here is the 145 km/h
that we found in the firmware. The green
one is the catalyst temperature, which we
see between ambient level, and then up to
380 degrees. The critical point here is
200 degrees Celsius, where this urea to
ammonia process starts to work. We logged
something that is the SCR strategy. So it
turns out there are multiple ways how the
ECU computes how much AdBlue to dose and I
call them strategy. So 0 means off, no
AdBlue is dosed. 1 means the regular way
that keeps into account the storage
mechanism, and then 2 is a special reduced
way. And then also we log the actual
dosing value. And then we also had the
sensor data from between the engine and
the catalyst, and between the catalyst
and the exhaust. The first thing that
happens... or, actually nothing happens
until the point where we reach 200°C
at the catalyst. You can...
until that point, as I said the required
temperature is not... does not allow
AdBlue dosing, and then it starts dosing
quite a few amount of AdBlue. But then,
when we cross the 145 km/h, the SCR
strategy changes, and no further AdBlue is
dosed until basically this point, and this
point is exactly 120 seconds after we go
lower than 140 km/h. So this matches what
we found
in the software, right, this was what we
found in the
software. So we can see that this was
actually true. This is real behavior of
the ECU. And to look at the effect of
this, you have to check the difference
between the blue and the green line in the
lower diagram, between basically the
amount of NOx that is removed by the SCR
catalyst. And you can see during the
regular operation quite a lot of NOx is
removed. The blue line goes up because
we're driving faster and faster, and the
green line goes down almost to zero, and
this this works for quite a while, and it
even keeps working a while until the
catalyst runs out of ammonia, and then it
would need more AdBlue to operate, but
because we're in the reduced mode it does
not put any more AdBlue into the system.
So the SCR basically stops working, and
the emission levels reach the engine
emissions, so no further... The SCR system
does not work in this red area. And here
we see this again, so here are the
sequences of active dosing. Here we see
where the catalyst temperature is too low
for dosing. We see the regular operation.
We see where it's still working,
because there's still ammonia stored and
then until we run out of ammonia and no
refill happens, until exactly 120 seconds
after going below 140 kilometers an hour.
So our conclusion after this is that the
SCR is programmed to stop working at 145
km/h. The efficiency goes to 0. Opel
offered a hand-waving explanation, one of
the press releases why this is necessary.
They argued with some physical details,
and we presented these physical details to
some experts, professors that work with
combustion engines for a long time. Most
of them disagree with these explanations,
I mean, yeah. But more importantly, other
cars including my Volkswagen Sharan
device, which is Euro 5 car, so one
generation older, and it's known to have a
defeat device, and it performs
significantly better than this car. OK, we
continue to look. We found something here
that is a... that looks like this. It
takes a... there's a
barometric pressure sensor that sends us
the pressure of the air and compares it
with a value and if we look at how
pressure is related to height we see that
what they check with is 91.5 kilopascal
and which corresponds to 850 meter. And
apparently Europe's highest test center's
at 800 meter, which may be a coincidence
or not. But above that point they reduce
their CR system as well. Now, the
interesting thing is... yeah barometric
pressure is something very important to
know for an ECU. There's a good reason to
have the sensor for all of the combustion
process. You need to know how much air
there is. So for EGR it makes a lot of
sense to have this, but for SCR, which is
the system after the engine, no combustion
is happening. We are not aware of any
effect that the outside air pressure has
on the SCR system, and also other SC
equipped cars don't have this mechanism,
so... for us it does not make
physically... it doesn't seem to be
physically required. So far we looked at
SCR. Let's look at EGR. What we saw was,
when we drove the car during the test
cycle, so we put it in the lab and drive
the
test cycle, we consistently saw much
higher EGR values, much higher than
compared to driving on the street,
compared to all kind of scenarios that we
drove on the street. So higher EGR value
here means, that the EGR valve, that I
showed you earlier, is more open, more
exhaust gas recirculates to the engine. It
causes lower NOx emissions before the SCR
catalyst, and we really... we're curious
why did the car behave so differently when
running on a street than running in a test
cycle. And we already took into account
temperature, so the temperature was not
the issue anymore. And thankfully the car,
when it computes the reason for reducing
EGR, it stores a reason in some variable
that we can log, and it looks like this.
There is a number of things that can
happen, that causes the ECU to switch to
some low EGR mode, and a few of them
make sense, for example if something is
broken, fault flags are set, or if the, I
don't know, the coolant temperature is out
of range, it makes sense to just keep the
device running at all cost. But when none
of these reasons apply, the value stored
is 2, and 2 basically means, that the full
EGR operation is used, so it's basically
the NOx-optimized mode with the fewest
emissions. And then we looked at some
real-world driving, you can see this in
the background - the vehicle speed is in
the background - and we saw that... the
red graph shows you the reason to go to
this limited EGR mode and what we saw is
that most of the time the reason is 13 and
only a few times it's 2, which means that
it's not limited. And looking into this
more details, we see it sometimes drops
back to 2, to the unlimited mode, to the
optimized... emission optimized mode, but
any acceleration, or almost any
acceleration switches it back to 13, and
then it stays there for a long time. And
13, if we look it up, is what I call load
limit. And then, interestingly, if we run
it through the NEDC, we never saw a 13. So
the engine stays in mode 2 all the time,
and 16 just means that the engine is off.
But we never see 13. So this explains why
the EGR values were so different in a test
cycle. So, let's look into this load limit
function that we found. It's basically
defined by curves, by five curves. For
every gear there's a curve, or for a
bucket of gears. It's basically that they
look up RPM, they get a value for that
curve, and if you exceed that value, they
switch to the reduced EGR mode. What they
compare this threshold with is the amount
of fuel injected per cylinder per
revolution, but you can also say this is
torque, just with a
constant factor. And then once you are
outside of one of these curves, it
switches to the non-optimized mode where
it emits a lot more emissions, and then
you have to go back into the green area to
switch back to the optimized mode. So
let's see what this means in practice. So
here we
have a car, and the traffic light is red,
so the car stops, and then the traffic
light goes green and the car accelerates,
and accelerates, and accelerates, gets
faster and faster, and then it's at the
highest speed here, and drives for a
while. And this is a typical city cycle,
this is there to... how you drive in a
city, and then the next traffic light
turns red and the car brakes and stops in
front of the traffic light. Let's take a
look at this again with one more variable,
the RPM. We can see that when the car
starts moving, the RPM goes up. And then
at some point there is a drop in RPM, and
this is because it's a manual transmission
and the driver switched to the next gear.
Now it's switched to again the next gear,
and this causes the RPM to drop, but the
speed to remain almost
constant, and it drives for a long time in
the same gear, and then the traffic light
goes red, the driver presses the clutch,
the engine goes back to idle state, there
is no connection anymore to the wheels,
between the engines and the wheels, and
the car gets slower. OK, one more
variable. It's the last one, I promise. It
is torque. The engine power in kilowatt or
something is not just a function of RPM,
it's a function of RPM and torque. so RPM
and torque together are very useful to
characterize engine behavior. And a very
good way to do this is to have a graph
where we put RPM on the one axis and
torque we put on the other axis, and then
we draw this in two dimensions, and so we
get this, basically. This is the operating
points we go through when driving the
cycle we saw. So the green dot here
indicates where we are. And so we restart
the car, the car accelerates, sorry, the
car idles for a while, so the green dot
stays there. It idles at around 800 RPM,
almost no torque, because there's nothing
to move, and then the driver accelerates
and the torque goes up, the RPM goes up
more slowly, and then at some point, the
driver presses the clutch, which
disconnects the engine, the
torque goes down, the RPM adjusts to the
speed of the next gear, and then the
driver releases the clutch and now the
engine again has to move the car, so the
torque goes up until reaching the the
highest RPM value and then that the driver
again switches to the next gear, so the
whole thing repeats, and then while the
car is driving, the majority of this the
cycle, the engine spends in this one
operating point. We're currently at 1800
RPM or something, and 80 Newton meter or
so torque. And then at some point the
driver presses the clutch, the engine goes
back to idle and stays there, basically.
So this is how you read this diagram. And
now what we found in the firmware was that
overlaid basically on this representation
we see a mask, or a limit. If we go over
this curve, those are the same curves that
I showed you earlier, just laid on top of
this. If we go over this curve,
then we switch to the worse emission mode,
we switch to the mode where the EGR value
is limited. So we can see in our driving
that this happens basically at this point,
the point where the driver
accelerates above a certain point, that
causes it to go over the load limit and
the engine basically switches or
significantly reduces EGR. And that's fine
because EGR doesn't work when you need a
lot of engine power, so it make sense that
that's at that point, and what we would
think is that it switches back once we
leave this load envelope, once we go below
the limit again, once we are inside the
limit, we would expect the ECU to switch
back to the full EGR operation. But what
we see instead is that this does not
happen, and the reason is that you don't
have to go under the maximum, the load
limit, you have to go into this green
area. You have to go back to idling at a
very low RPM to switch back to the full
EGR mode and this only happens at the very
end. When the driving cycle is almost
done, when the driver presses the clutch
and lets the engine idle. So especially
this long sequence where the driver... the
car was driving at the same speed, we were
technically in ... within the load limit,
where we're not exceeding the load limit,
but because we previously exceeded the
load limit and it doesn't matter for how
long you exceeded it, and we did not go to
the green area before, we were still in
this low EGR, high emission mode, even
though we're still within the load limit
imposed by the software. So let's take a
look at how often this actually happens in
real-world data. So here's us driving
through a city, and we can see we
constantly exceed these load limits. And
this is driving on the Autobahn, and yeah
we constantly exceed those. But they look
interesting. They look as if they had been
designed according to something, right,
they have the specific form and it's not
just... yeah, I... I don't know... and it
turns out if you do something really
strange, you can stay within these limits,
so we tried that and we managed to stay
within the limit by doing something, and
we... it was reproducible, we could do
this a lot of time it would always stay in
this limit and the answer is: If you drive
the test cycle you're staying in this
limit.
applause
So yeah, these curves basically defined...
they closely correlate to the limits that
you need to pass the NEDC. Okay, to be
clear it is fully acceptable that the EGR
rate is reduced when... for higher engine
loads. It's natural, you have to do this.
For example, when you accelerate the EGR
rate will decrease up to zero probably,
when you do it ... when you're running at
high speeds, all of that is great. So this
method of having a load limit ... well,
you can argue if really having the load
limit exactly where the NEDC is makes
sense, but having a load limit is okay,
right? However, what we think is not okay
is that, if you only exceeded the limit
once ... um... you would stay in this high
emissions mode for potentially a long time
until you get back to low speed idle the
next time. And we think that is the
problem. We ... so far this was all based
on what we saw in the software, so let's
see if this translates to something that
happens in reality. So to repro this we...
the car... drive at constantly... or we
let it idle, then we accelerate it to
2,000 RPM, we let it drive there for a
while and then we quickly exceeded the
load limit by going to 3,000 and then
going back and then after doing that we
would again stay at 2,000 RPM. So it looks
like this and we would naturally expect
the engine to operate in the same way on
the left and on the right side because the
engine is doing the same thing there, it's
the same torque level, it's the same RPM,
everything is the same. So we would expect
the same emissions, right, um ... and it
turns out it isn't. And ... this is a
slightly convoluted diagram. So if you
look at the green and red bars in the
middle you can see what happens before and
after exceeding the limit for just once.
And in the middle you can see the EGR
position, the EGR valve position, and you
can see that we get pretty high values
between... 6... maybe 65 percent or
something before exceeding the load limit
once.
And after we exceeded it once even though
the engine again is operating in the same
exact operating point, we see much lower
EGR valve positions, around 50% or
something. And if we look at the bottom we
see what the engine NOx emissions and we
see that they are significantly higher on
the right side than they are on the left
side. So this... for me, this does not
sound like this is truly optimized for
emissions because the engine is doing the
same thing, in both cases the emissions
should be low. So going back to this quote
that it works, the EGR and SCR injection
work to the full extent in a temperature
range of 20 to 30°C. Okay,
but what about the EGR load limit and what
about the the barometric pressure limit
for SCR and what about the SCR speed
limit? That would not be "to the full
extent", right? And the Opel answer is
really interesting. Of course, they denied
doing a test cycle detection, they say
they don't do that. And what they said is,
when asked whether they lied to the KBA
when saying that it works to the full
extent they said "The statement 'fully'
was really related to the NEDC test
schedule, right, which... it went on and
further... the Opel CEO had to say this.
He said: "The recent
accusations based on the findings of
hacker Mr. Felix Domke" - hey, that's me -
"are misleading oversimplifications and
misinterpretations of the complicated
interrelationships of a modern emission
control system of a diesel engine.
Emission control devices are highly
sophisticated integrated systems which
cannot be broken down into isolated
parameters." Especially not by a hacker,
right?
applause
That was kind of funny. There was another
funny thing. Sorry, I only have a German
quote and I didn't want to translate it,
but when Opel basically ... they
repeatedly say they don't have a cycle
detection, right, and they say it's not a
cycle detection because, if you use the
car on the street in the same way as you
would do them during the test cycle, the
car would behave in the same way, so it's
not...,right? applause ... and ... okay.
But what is with Volkswagen, right, they
have the same thing, if you drive the NEDC
on the street the car will go to test mode
they have the same thing. I don't see how
this does not represent a cycle detection.
That was a lot of things to say about
Opel, but on the bright side, they also
said that they will - even though all that
was incorrect, what we found - they said
"We will further improve the efficiency of
emissions after treatment of our SCR
diesel engines and so on as far as the
laws of physics allow. This includes a
voluntary service action" - and this
basically means a software update for your
car - "for the cars that are already on
the road starting in June." So that is
great. They're actually improving
something. Question's in which year,
because this statement is from May 2016
and it's not out yet, but... Opel actually
provided a new software already in July
and I think they already worked on this
for quite a while and in July 16 the German
KBA, the Kraftfahrtbundesamt, the Federal
Motor Transport Authority, they are pretty
nice actually, and they do know about what
they do, they are bit limited by the
resources they have, and by the manpower
they have, but they know about cars and
they know how to do these investigations.
I mean, they're a little bit bound, but
what they should do and what they should
not do, but they asked me to review a new
ECU software that was given to them by
Opel for the Zafira in question and
Insignia, which had a similar ECU and I
looked at that software and I dumped the
firmware and I looked at basically all the
code sequences that I looked at before and
I was positively surprised because they
removed... they addressed each of our
concerns. All of them, within the physical
limitations of course. So they improved
the temperature window and everything, so
there was a significant improvement. They
were able to improve the software and they
let the DUH, which is the German
Environmental Aid, they used a PEMS system
- PEMS is a portable
emissions measurement system. It's
something you put on the exhaust pipe on
your car and then you can measure the
exhaust during real-world driving, and
Opel gave them a car with the new ECU
software. Otherwise the car was identical
to the old software, and the results are
this, right, so on the left side you see
the old software, that has all these
things that we criticized, and on the
right side you see the same car with a new
ECU software and it's significantly
better. It's only slightly above the
limit, right, but it's much better than
before and to put this in relation,
before they were on the list pretty bad -
so this is sorted by worst to best - so
they are in the, well, upper half at
least, and now they are almost one of the
best cars, just by switching the ECU
software. And I mean this is great news,
right, they actually improved their cars.
Let's just hope they get this out to the
cars soon. Let's just hope it doesn't have
side effects and something, but I'm sure
Opel knows how to test for this. Going
back to these, we worked on the Opel
thing... I think the Opel case, it....
once they actually upgrade the cars, and
once the cars really show these great
values that the preliminary software
showed, I think we can close the Opel
case, but there's a lot of other cars
still to look at, and really, I mean...
the effort to do this does not scale to so
many cars, so we need to do something more
fundamentally to improve the situation.
What I found out is that digital control
systems, they are black boxes. The
manufacturers have designed them to be
black boxes. They even boast to you that
they are 7,000 parameter in there and no
hacker can understand this and it's a very
sophisticated problem. They are designed
to be black box, and this is not just true
for Opel, this is true for all car
manufacturers. Nobody wants anyone to look
into their ECUs, and people seem to be ok
with that. Like they think "Oh this is so
complicated, there are so many German
engineers working on
this problem, they must have found a great
solution." So we are trusting these black
boxes and we are not able to review the
black boxes that we put into our cars and
we have to trust the manufacturer to do
the right thing and currently, the
investigation to do this without
assistance from the manufacturer, it does
not scale. We can do it but... the
manufacturers can put more security on
their ECUs... it probably can be broken,
but it takes a lot more time, so it simply
does not scale sufficiently. The issue is
black boxes are really powerful, right.
Black boxes can hurt people with, for
example, excessive emissions. They can
kill people if we think think about
autonomous cars that do mistakes. So what
we do need, I think, is more transparency.
A system that can kill people needs to be
reviewable by the people. I think this is
a very important thing.
applause
So, to have a system that can kill
people... to have it reviewable by the
people, we need to do things. For example,
we need... we want access to source code
for reviews. It doesn't necessarily mean
we want open source, but we don't ask at
all the car manufacturers to open source
all the software. That's not what I'm
talking about. What we need is... think
about how Microsoft is sharing source code
of Windows with universities or other
countries. We need experts to look at the
source code, and we want control software
that is reviewable by design, that has a
lot of documentation, that has good
comments, that is human readable code. I
don't want to see a disassembly, I want to
see the source, the MATLAB, or whatever
they are using to define the functionality
source, and read that. And I want to
understand why did they choose that
curve of that map in this way? What was
the design criteria? That needs to be
reviewed. And we need transparency for
control software decisions, which means
that if a car operates in a certain way,
if I'm driving that car, I want to choose
that I can log what the car is doing, for
example by putting
in, I don't know, a USB stick or something
if it's my car, and then the car will log
all the data to that. That is... in the
end that allows me to reconstruct any
decision that the software does. I think
this is required to have the necessary
transparency, that allows us to un-
blackbox these devices. All right.
Thank you very much.
applause
Okay, I actually finished five minutes
early. I didn't think this would happen,
so...
Herald: I'm so surprised.
F: I am surprised too.
Herald: You are on time. You have five
minutes left
F: Wow, what do I do with these five
minutes
Herald: We can walk around the stage or...
Maybe people have some questions?
F: I think so!
Herald: Well, let's ask the Internet! Is
the Internet ready?
Signal Angel: Yes. Our first question:
What dou you think is the responsibility
of Bosch as a supplier for having their
software and hardware used for this?
F: So the question was: What's the
responsibility for Bosch, who built the
software for Volkswagen? It's a good
question and I have to be careful in what
I answer. My personal opinion, and let's
take this aside from Volkswagen and Bosch,
is that if you build software that you
know is used to be illegally it should...
it must be your responsibility to not do
that. And I'm not sure if this is
something that is legally enforceable, but
it should be something that's enforceable
ethically or for all of us programmers,
that we don't build software that is
designed to break the law.
applause
Herald: We quickly hop over to microphone
1 please.
Microphone: Thank you for a wonderful
talk. I'm just wondering if you're aware
of some cases of Volkswagen cars in
Australia, which was suffering from sudden
and rapid power loss. This was happening
about five years ago and there was a case
where a Volkswagen suffered rapid power
loss on a motorway. The driver was Mrs.
Melissa Ryan and she was rear-ended by a
truck and killed. So when you say that
these things can cause death, were you...
are you aware that any sort of Volkswagen
software has been leading to power loss in
the vehicles and affecting
performance on the road, now I don't know
whether Australian driving conditions are
different to European driving conditions,
and how that might affect that. Have you
done any tests that might indicate that
could be happening in normal driving?
F: Yeah, so... the question was whether
I'm aware of, I think an Australian
incident, right, where...
M1: Can I...
F: Yeah.
M1: There were many reported cases. One of
them was fatal, but there were many
reported cases of that happening.
F: Of a sudden power loss, is that right?
M1: Sudden and rapid power loss in the
engine.
F: Yeah, of the engine. I'm not aware of
these incidents and I what I do know
and... is that the the personal safety is
the number one design criteria for ECUs.
That does not mean that they are perfect,
of course, that could mean that rare
bugs... that there could be malfunctions.
I don't know about this, but at least it's
the first design principle to provide the
safety for the people driving the car,
which i think is a good thing, right. It's
not the profit or anything, or at least we
can hope so. I'm not aware of this
particular incidence, and so I can't
really say anything more about this. It
would be great if... Are you aware of any
additional details that were found in the
investigation, please sent them to me.
M1: Volkswagen was claiming that this was
a gearbox problem on automatic cars, but
then it started happening on manual cars
as well, so that excuse went out of the
window.
F: The issue with the problems is that
most of them are very complex, so they
probably involve more than just the engine
ECU, so they're very... but it's a
good example of where we need to
understand exactly what is happening, and
where we may not want to rely on
Volkswagen or any other manufacturer alone
to assist in figuring out what happens. We
need more transparency there so that we
can have definitely neutral accident
investigations.
Herald: This was a long question and
really detailed answer. Thank you very
much.
F: Sorry, I will be short
Herald: Felix, that's your applause
applause
music
subtitles created by c3subtitles.de
in the year 2018. Join, and help us!