Software Defined Emissions (33c3)

33c3 intro music

Herald: Let me present Felix Domke with

the Software Defined Emissions: A Hacker's
Review of Dieselgate.

applause

Felix Domke: Yeah, hey everyone. Thank you

for coming here. I saw there are a lot of
interesting talks at the same time in the

other rooms, so thank you for coming here
and listening to me about software-defined

emissions. "A Hacker's Review of
Dieselgate" is the subtitle. I'm Felix

Domke. I usually do embedded software,
mainly security. I'm definitely not on

cars and definitely not on things that
have combustion thingies, so I only got

dragged into car software last year when
my own Volkswagen car was accused of

cheating and I wanted to know what exactly
was going on. I held the talk last year

about some of the details of the
Volkswagen Sharan defeat device. For the

details you can take a look at that talk.
This time I want to look more at the

process of finding or analyzing car
software. I want to look at whether this

process scales to more cars. The first
step when having a piece of software that

does not always do what people think it
does is, well obtain a firmware image,

obtain a binary image of the firmware, and
in the case of my car I knew it was a

Bosch EDC17, which is a Bosch ECU that a
lot of cars use, including my Volkswagen

car. So, I didn't know anything about
ECUs, dumping software and so on, so I

asked Google "Hey, what do I need to do to
dump an EDC 17," and Google had a lot of

answers for this, but usually those were
people that wanted to sell me some device.

Those were chip tuners that built their
own devices where you can plug in the ECU

and then it extracts the image, usually by
exploiting some bugs in the software. But

I didn't really want to buy something and
it takes like a lot of time until I get in

my hands. I wanted to start. So I was
looking to do this on my own. What these

sites usually tell you without paying is
how you wire up your given ECU for their

device, so they tell you where to connect
12 volt, where to connect the CAN bus,

which is the serial communication bus that
the ECU uses to communicate with the rest

of the car
devices. Usually it's pretty easy, so...

when analyzing ECU it makes a lot of sense
to reproduce the scenario on your desk and

not in your car, so in order to make an
ECU boot all you need is ground, 12 volts,

there's usually an ignition pin that you
also have to supply 12 volt to it, and

then it boots. So on my desk it looked
something like this, and then once we have

the setup we can boot the ECU, we can use
Python to talk to the ECU, which is great,

and then we can use socket CAN, which is
the Linux CAN support that's really great,

and we can even use MicroPython if we
want to have a smaller device that we can

put in a car. So we can talk with the ECU.
Talking with the ECU in modern cars,

there's a protocol called UDS. Basically,
I simplified this slightly, you can ask

the ECU "Hey, I want to read memory by
address," you give it an address and you

ask it to read four bytes in this case of
that address, and then it returns it to

you. So I thought "Hey, maybe I can use
this to dump the software." In my case the

device responds with a Security Access
Denied, so I looked into what I need to

do. You actually have to do a Security
Access command. You send a command that's

called Request Seed. You get back
basically a 32 bit random number and then

what you have to do is to, have to
process this seed through a super-secret

function and then return it in a response
call. The question is how do we know this

super secret function. There are multiple
methods. We can look at the ECU software

itself, if the algorithms in there, to
verify it. We can reverse diagnostic

software that uses this mechanism. So for
example the Volkswagen software they use

for car shops, or maybe someone else
already reversed this and put it in their

own tools which may be easier to get it
from in terms of third-party diagnostic

software. And in the case of my Bosch ECU,
the super secret function was this. I

basically had to add this number to it.
The mechanism is called pin code.

It's in... I mean it's not super
secret. Anyway, once I know this...

applause
Yeah, thank you. I mean, it... once you

do this, you send back the result and hey,
then you can read... you can send the read

command again and, hey you're getting back
data, so this is great, right? We can read

memory at runtime of the ECU, and we can
even do this while the car is operating.

However it turns out that for the Bosch
ECU you can only dump specific regions. You

can dump most of memory, some memory areas
are excluded, but most of the interesting

stuff you can read. But you can't read any
code. You cannot read anything in flash.

But we are hackers, of course, so we find
a way. The CPU used in these ECUs is a

Infineon TriCore CPU, and it's used, at
least this particular one, was used in the

ECU I cared about, and the security model
for this chip is that you can always enter

a specific bootloader mode and execute
your own code, so you can strap a few

lines they... the chip tuners tell you
that, right, they tell you what's high and

low, which pins you have to connect your
ground and 3.3 volt, and then it enters

this bootloader mode, you can upload some
piece of code. However, you can't read the

flash, because the flash is locked. When
you start in bootloader mode, the flash is

not readable until you write a specific
password to a register. That was not so

great, so I looked into what else I could
do. The datasheet is very specific on how

to operate this chip. For example there's
this one flash supply pin. So, even though

the flash is in the same package as the
rest of the CPU, it has a dedicated supply

pin and it tells you which parameters not
to exceed to ensure correct operation, but

I really don't want the correct operation
which is in this case preventing me from

dumping the flash. So, what can we do? We
can violate the requirements. The

requirements for 3.3 volts. Let's see what
happens outside of that range. And turns

out, down to a certain voltage level,
roughly 1.6 volt, everything just works as

normal. That doesn't help us.
And below that voltage the device hangs in

the bootloader, so that doesn't help us
either. The interesting parts happened

when you are at the very specific voltage
level, and this is a little bit

unscientific, because it's really just the
voltage level I tried, and then most of

the time the device comes up and flash is
protected, and then the remaining times

the device comes up and hangs in the
bootloader. But one in 10 times something

interesting happened. The device came up
and the flash was not protected, so I

could dump it out.
applause

So having the image now in my hands, I
could start in actually reversing the

defeat device and what I found was - I
don't want to duplicate a lot of what I

talked about in the last talk - I found a
function called "acoustic function," or

"Akustikfunktion" in German. It's a
function that senses vehicle speed, the

duration of the engine operation, and some
other things, and then controls emission

related functionality, or in short you can
say that this is the test cycle detection

that enables the defeat device. And I
verified it to exist on my Sharan device

by driving through the test cycle and
logging data. And during the last year I

verified that it's actually the same
defeat device, more or less, that exists

on a lot of other Volkswagen cars. All
these Volkswagen defeat devices that we

talked about for the Euro 5 cars, they use
more or less the same acoustic function.

Basically, to remind you, there are a few
curves stored in the software that look

like this. This is the NEDC. This is the
test cycle you have to drive a car

through. They exactly define how fast you
have to drive for a given time in seconds,

so it's speed over time. If we draw this
as distance over time it looks like this.

So this is the distance you got. You're
not really moving the car, because you're

doing this in a lab on a dynamometer, but
what the car thinks it has moved to, and

if we overlay this with the curves we
found in a software there's a perfect

match. So this is the way how they
describe the test cycle. So this was for

my Sharan. So I looked into, what do the
other cars do, especially what do the

cars in North America do, because they're
not using the NEDC. And I found something

interesting, or some someone sent an
interesting document to me, that was this.

It was an emission service action. It
basically describes how there was a recall

for some vehicles, that required a
software update in the shop.

So this is basically the document that
informs the car shop what they have to do,

and it had something very interesting in
it. By the way, this was in December 2014,

so this was way before the whole
Dieselgate was public, but this was

already while the EPA was already talking
with Volkswagen, already demanding

explanations. All that investigation was
already proceeding. Volkswagen knew about

this, that people figured out about the
defeat device, and it had something very

interesting in it, that said "in addition,
the vehicle's engine management software

had been improved to ensure the vehicle's
tailpipe emissions are optimized and

operating efficiently." That sounds really
fishy to me, so I was curious, what

exactly did they change in the software
update? And luckily they tell you the old

and the new software versions, and you can
then go and look them up on a firmware

DVD, that you can download on the
Volkswagen website, and it turned out that

it's an ECU software similar to the the
Bosch ECU software I looked at before. So

there's an acoustic function again there,
and the curve stored there, they match the

US test cycles. This is one of them. There
are many more test cycles in the US, so

there's another curve that matches this,
and this is the curve stored in the

software and this is the corresponding
test cycle. And there are a lot of them.

But I noticed something really
interesting, and some of the curves...

they were much wider open than the other
ones. So for example this one... there's

really a nonzero probability that if you
just have your morning commute, through, I

don't know, some streets or something,
that you accidentally match this driving

cycle every time you start driving in the
morning, so the car would, every time you

drive this, think it is in test cycle
mode, and would operate with the

optimized... in the optimized emission
mode and apparently this caused problems

and what I saw, what Volkswagen added in
the software that was part of this recall,

was this function. So, this is from it
from a disassembly. In pseudocode, this is

this. So they started looking at the
steering wheel angle, and if they figured

out that you move the steering wheel
angle, then they ignored the curves, the

more open curves. So the idea is "Yeah, if
you move the steering wheel, you're

definitely not in a test cycle, so at that
point we do not try to operate in this

emission optimized mode. And it's a
little bit of speculation, but it matches

up pretty well with with all the facts
that I read, is that because those cars

operated in the test cycle mode too often,
that eventually caused the the particulate

filters to clog, and their solution for
Volkswagen, and again, this was while they

were already investigated by the EPA, was
to add the steering wheel angle detection.

For more details, I worked with this with
the NDR, and they produced a feature on

that, so there are some more details.
So this is Volkswagen, but there are more

cars, and if we look at this... this is a
meta-study based on something that the the

Ministry of Transport... they tested a lot
of diesel cars and what they found was

this. This is actually a representation by
the ICCT. So the orange line is the

emission limit, and the bars have an upper
and lower end, and the lower end is how

much emissions the cars have. This is just
for nitrogen oxides, for NOx emissions,

what they had in the lab, when you're
driving the test cycle. So and you can see

all of these cars managed to stay under
the orange line, so they get their

certification, but when driving them on a
real street, they produce the emissions

corresponding to the upper end of that
bar, which is for some cars significantly

higher. It's off by a factor of 10 and
more. When you're driving the car on a

street. And this is interesting, because
the cars, they can meet the emission

goals. The question is why don't they
always meet the emission goals?

Why do they operate so differently in the
test cycle than on the street? And I try

to give you a partial answer. And let's
look at how a car can optimize emissions.

The first thing they do... so this is a
very simplified diesel engine. So fresh

air goes in, fuel goes out, and there's an
exhaust pipe, right? And a lot of nitrogen

oxide, a lot of NOx, goes out as well, and
we don't want that. So we added an EGR

valve, which is basically a valve that
causes a part... a fraction of the

outgoing air to recirculate again through
the engine and burn again. And what this

causes is that the flame temperature goes
down, and if we look at the relationship,

it's very simplified here, but with a
lower flame temperature you get fewer NOx

concentrations, so you improve emissions
by lowering the flame temperature, however

at the same time you're increasing the
soot level, or the particulate matter, and

there is this trade-off - if you do too
much of EGR, too much of the exhaust gas

recirculation, you're getting too much
soot, and the other hand if you do it too

few, you get too much NOx, so you can
argue that the green area isn't really

great, because there's no point where both
of them are great. And here we see the

result of a clogged EGR valve. If there's
too much soot it will clog. EGR, as the

conclusion, is the least cost solution. It
doesn't really work at higher loads. It

works at low loads, and it does not
require exhaust... high exhaust

temperatures, which is great, but
excessive use of that clogs particulate

filters, affects the combustion, the
drivability goes down, and there are

trade-offs with this. It's also not very
useful for higher engine loads, for

example when you're accelerating you have
to disable EGR at high speeds. So a better

method, that was added on top of this, is
called "selective catalytic reduction". I

am... so, basically the idea is you have
an SCR catalyst in your exhaust pipe...

there are more catalysts there, but let's
talk about NOx, and in there, this

happens. We can simplify this, somehow,
and say if you put ammonia into this

catalyst, the NOx is converted to nitrogen
and water.

And Nitrogen and water is great, it's
harmless. It's already part of the air.

The only issue is ammonia is this and this
is not something you want the driver to

refill in your car. So instead this
solution is we can create ammonia in the

car from using from something that's less
dangerous, and we have the reaction there.

We can simplify this again and say we take
urea - Harnstoff auf Deutsch - and heat,

and we create ammonia. Urea or urea
solution is this. It's called AdBlue or

DEF - diesel exhaust fluid - it's not
dangerous. You can buy it, you can

transport it... it's relatively cheap. The
idea is, we have this reaction, it

requires ammonia in the catalyst, and we
put AdBlue into it, or urea, and using the

heat that we have from the exhaust pipe,
we create the ammonia that we need to

reduce the nitrogen oxides back to
nitrogen and water. There's a great

property of this, that some of the ammonia
that's produced in the catalyst stays

there until it's used up, so
there's some storage there. So the the

requirement for creating ammonia is heat,
and if you don't have heat, but for

example because you just started up your
engine... if there is still ammonia from

the last usage in your catalyst you can
still use that, and use that up, and by

the time you have used it up, maybe the
heat is enough to supply more AdBlue and

then fill up that storage. The downside is
you need a pump to dose the AdBlue, and

you need lots of software to control this
process. And you need a heater because the

AdBlue freezes at some point, and it's an
expensive solution, it adds roughly $500

to a car, which can be significant amount
of money for a small car, and it requires

a large AdBlue tank for long service
intervals, so you don't have to refill it

every few thousand kilometers or
something. The great thing about SCR is

that it's efficient at higher loads.
There's a third method called LNT, Lean

NOx Trap, it's cheaper than SCR for
smaller engine, it doesn't require

anything,
however the bad thing is it requires

frequent re-generation, which decreases
fuel efficiency, so it's kind of a stop-

gap solution. And it's not efficient for
continuous high engine load, for example

if you're driving on the German Autobahn
at full speed, then LNT is not going to

help you much. For the sake of this talk,
let's keep in mind that EGR is exhaust gas

recirculation, that's the thing that
operates within the engine and then we

have the SCR, the selective catalytic
reduction, that uses AdBlue and is after

the engine. We also saw that all these
technologies have significant trade-offs

for NOx compliance, so we can kind of see
the motivation for a defeat device here,

because it would be the solution to all of
these trade-offs. You get no downsides

during regular driving, because nobody can
measure your emissions and while

maintaining conformance because during a
test cycle you have perfect emissions.

That kind of explains why there are defeat
devices. OK, let's get back to the bigger

picture and see what other cars do. So
this is an Opel car, it's a Zafira car,

it's a Euro 6 car, it's a pretty modern
car, it has an SCR catalyst. In theory it

should have really great, low emissions,
especially at higher speeds because that's

where SCR is good at. But quite
surprisingly it doesn't. If we look again

at this report, we can see that this
Zafira exceeds the limit by up to 12x

compared to the Euro 6 limit. This is
especially interesting because there's

this Opel advertisement where they
advertise their diesel technology applying

to the Insignia and the Zafira and they
say a lot of diesel fun without regrets,

the new diesel generation of Opel achieves
best emission values and gasoline levels.

Yeah, after they got sued for this they
had to change it slightly and they had to

add this to the sentence.
scattered laughter

So, during this testing, they had
this 12x emission

limits. For example, one particular test
was to drive the test cycle in the same

way, but at a different temperature,
at 10°C,

and the car exceeded the
values by a factor of 6 even though the

car would be operated in the very same
way. It was just that the ambient air

temperature was 10°C instead
of 25. So they asked Opel why this was the

case during their investigation, and Opel
responded, saying that the EGR and the SCR

injection, they work to the full extent in
the temperature range of 20 to 30°C.

It's what they call "normal use".
So our question was, is it really just the

temperature window? So we got a car and
investigated. The ECU in that car is a

General Motors ECU. It's developed in-
house, Opel is a GM daughter. It uses an

automotive PowerPC, yay PowerPC! It uses
somewhat obscure variable length

instruction extension to PowerPC. So how
do we start? Again, we need a firmware

image. So let's ask the Internet, "How do
I dump this ECU?" Luckily, someone in some

chip tuning forum already uploaded their
stock ECU, which means they're what they

dumped from their ECU using some chip
tuning tool. It's not the same ECU, but

it's very similar and I hope they shared
some code so I can analyze the software,

maybe find a way to dump it. So the dump
was made with this tool and the tool did

not let me export the binary. And the tool
is free, however to use any of the

features in that software, you have to buy
their expensive hardware, that then

connects to the car, which I didn't want.
This is how their software looks like. I

loaded the image I found on the internet,
and I couldn't save it or anything without

having the device attached. However, I can
just use a memory debugger and just dump

it from the address space, and that gave
me a first firmware image to start with. I

threw it in a disassembler and I found the
UDS function and, yeah, it implements read

memory by address, so that's good. Most of
the RAM was readable without a security

challenge. That is good, so I didn't even
need a security challenge to read RAM.

However, the flash, it is readable, but
only with the security challenge. So let's

take a look at the security challenge.
Maybe it's as simple as the Bosch one. So

their way of doing this is, they store
16-bit input and output value in the

firmware, and it's different for every
device, and they don't store the algorithm

to compute the output from the input,
instead they just store the pair, and,

well, it's just 16 bit, right, so let's
brute force it. The issue is, you can only

try every 15 seconds, so it's kind of
lame. The question is, how do the GM

tools, the original factory tools, get
access to that? Luckily, chip tuners had

reversed that and then obfuscated it into
their own tools. But that can be de-

obfuscated and eventually it's just a
little bit amount of bit shifting and so

that was easy to fix. And also the the GM
repair manuals tell you how to wire up the

ECU. They tell you where to put 12 volts,
the CAN bus again, and ground, and the

ignition pin, and with all of that in
place, I can do the security challenge. I

can now read all of flash memory and read
the four megabytes of PowerPC code, which

mostly consists of mathematical functions.
There are no strings or anything, it's

really hard to find what a function does,
what... There are thousands of variables.

It's really hard to find what they mean,
right, so I need to know some entry

points, some known data values, and then I
could refer to... one thing I could find

are real-world constants, for example
there's the density of diesel fuel stored,

which allows me to understand that this is
something related to fuel, an amount of

fuel. Or more useful are the OBD2 calls.
So there are some standardized things you

can ask an ECU, that's engine rpm, vehicle
speed, and things like that, and I would

find that table in the firmware and then I
had a first start of things like RPM,

speed, and so on. That was a good start.
It's not much more than what you can see

here. So there's a lot of stuff not
included in these.

The next thing I did was, I drove the car

for a few weeks and I let a device
attached that would constantly log all

memory using the rebuy address thing, and
every few minutes I would get one memory

dump, basically. It's a few hundred
kilobytes of RAM, and then I put this into

my disassembly, and that allowed me to
understand more of what individual

variables do there. And I found some
interesting things. So the first thing,

one of the first things I found, was
basically something that resembles this.

So there was something that look at the
ambient temperature, and this basically

checks for range, right, and it did...
this was for controlling the SCR systems,

and it's interesting to know that the NEDC
requires the temperature between 20

and 30°C, and this is right
centered around this, when they check from

17.5 to 33°C. But this was of
course nothing new. I found something

similar, however, another temperature
check. And as you can see it's written in

a different way. It effectively achieves
the same thing, but it's a separate piece

of code, and this time it was for the EGR
system. So we have these two exhaust

treatment or optimization mechanisms
there, EGR and SCR, and they don't share

code. They have their own temperature
window. So we found the temperature

window, which was known to exist. The
question was, is there more? And one thing

we found was this: It's basically reading
the vehicle speed and comparing to a fixed

number, and it turned out it's something
like this. So it would check the

vehicle speed and if it's above 145 km/h
it would set a flag and then under 140 it

would clear. Keep in mind that the NEDC
maximum speed

120 km/h, so during a test cycle that
would never happen. So let's see if this,

what we found in software, if this really
translate something the car does in the

real world, and it's getting slightly
technical here, I apologize, but we need

to log some variables and a useful value
to to know is, how much NOx is there after

the engine, and after the SCR catalyst,
and luckily there are 2 NOx sensors in the

car. One before and one after the
catalyst, and they give you basically the

NOx concentration in ppm. So we log that,
and we also log the signal of how much

AdBlue is dosed into the system, and we
log the catalyst temperature. And one

thing to keep in mind is that there's also
this amount of ammonia that's stored in

the catalyst. We don't have this as a
value, but just keep this in mind. And

this is how we've driven the car. The
blue line is the vehicle speed. You can

see that it goes from 0 to 150 km/h, and
the critical point here is the 145 km/h

that we found in the firmware. The green
one is the catalyst temperature, which we

see between ambient level, and then up to
380 degrees. The critical point here is

200 degrees Celsius, where this urea to
ammonia process starts to work. We logged

something that is the SCR strategy. So it
turns out there are multiple ways how the

ECU computes how much AdBlue to dose and I
call them strategy. So 0 means off, no

AdBlue is dosed. 1 means the regular way
that keeps into account the storage

mechanism, and then 2 is a special reduced
way. And then also we log the actual

dosing value. And then we also had the
sensor data from between the engine and

the catalyst, and between the catalyst
and the exhaust. The first thing that

happens... or, actually nothing happens
until the point where we reach 200°C

at the catalyst. You can...
until that point, as I said the required

temperature is not... does not allow
AdBlue dosing, and then it starts dosing

quite a few amount of AdBlue. But then,
when we cross the 145 km/h, the SCR

strategy changes, and no further AdBlue is
dosed until basically this point, and this

point is exactly 120 seconds after we go
lower than 140 km/h. So this matches what

we found
in the software, right, this was what we

found in the
software. So we can see that this was

actually true. This is real behavior of
the ECU. And to look at the effect of

this, you have to check the difference
between the blue and the green line in the

lower diagram, between basically the
amount of NOx that is removed by the SCR

catalyst. And you can see during the
regular operation quite a lot of NOx is

removed. The blue line goes up because
we're driving faster and faster, and the

green line goes down almost to zero, and
this this works for quite a while, and it

even keeps working a while until the
catalyst runs out of ammonia, and then it

would need more AdBlue to operate, but
because we're in the reduced mode it does

not put any more AdBlue into the system.
So the SCR basically stops working, and

the emission levels reach the engine
emissions, so no further... The SCR system

does not work in this red area. And here
we see this again, so here are the

sequences of active dosing. Here we see
where the catalyst temperature is too low

for dosing. We see the regular operation.
We see where it's still working,

because there's still ammonia stored and
then until we run out of ammonia and no

refill happens, until exactly 120 seconds
after going below 140 kilometers an hour.

So our conclusion after this is that the
SCR is programmed to stop working at 145

km/h. The efficiency goes to 0. Opel
offered a hand-waving explanation, one of

the press releases why this is necessary.
They argued with some physical details,

and we presented these physical details to
some experts, professors that work with

combustion engines for a long time. Most
of them disagree with these explanations,

I mean, yeah. But more importantly, other
cars including my Volkswagen Sharan

device, which is Euro 5 car, so one
generation older, and it's known to have a

defeat device, and it performs
significantly better than this car. OK, we

continue to look. We found something here
that is a... that looks like this. It

takes a... there's a
barometric pressure sensor that sends us

the pressure of the air and compares it
with a value and if we look at how

pressure is related to height we see that
what they check with is 91.5 kilopascal

and which corresponds to 850 meter. And
apparently Europe's highest test center's

at 800 meter, which may be a coincidence
or not. But above that point they reduce

their CR system as well. Now, the
interesting thing is... yeah barometric

pressure is something very important to
know for an ECU. There's a good reason to

have the sensor for all of the combustion
process. You need to know how much air

there is. So for EGR it makes a lot of
sense to have this, but for SCR, which is

the system after the engine, no combustion
is happening. We are not aware of any

effect that the outside air pressure has
on the SCR system, and also other SC

equipped cars don't have this mechanism,
so... for us it does not make

physically... it doesn't seem to be
physically required. So far we looked at

SCR. Let's look at EGR. What we saw was,
when we drove the car during the test

cycle, so we put it in the lab and drive
the

test cycle, we consistently saw much
higher EGR values, much higher than

compared to driving on the street,
compared to all kind of scenarios that we

drove on the street. So higher EGR value
here means, that the EGR valve, that I

showed you earlier, is more open, more
exhaust gas recirculates to the engine. It

causes lower NOx emissions before the SCR
catalyst, and we really... we're curious

why did the car behave so differently when
running on a street than running in a test

cycle. And we already took into account
temperature, so the temperature was not

the issue anymore. And thankfully the car,
when it computes the reason for reducing

EGR, it stores a reason in some variable
that we can log, and it looks like this.

There is a number of things that can
happen, that causes the ECU to switch to

some low EGR mode, and a few of them
make sense, for example if something is

broken, fault flags are set, or if the, I
don't know, the coolant temperature is out

of range, it makes sense to just keep the
device running at all cost. But when none

of these reasons apply, the value stored
is 2, and 2 basically means, that the full

EGR operation is used, so it's basically
the NOx-optimized mode with the fewest

emissions. And then we looked at some
real-world driving, you can see this in

the background - the vehicle speed is in
the background - and we saw that... the

red graph shows you the reason to go to
this limited EGR mode and what we saw is

that most of the time the reason is 13 and
only a few times it's 2, which means that

it's not limited. And looking into this
more details, we see it sometimes drops

back to 2, to the unlimited mode, to the
optimized... emission optimized mode, but

any acceleration, or almost any
acceleration switches it back to 13, and

then it stays there for a long time. And
13, if we look it up, is what I call load

limit. And then, interestingly, if we run
it through the NEDC, we never saw a 13. So

the engine stays in mode 2 all the time,
and 16 just means that the engine is off.

But we never see 13. So this explains why
the EGR values were so different in a test

cycle. So, let's look into this load limit
function that we found. It's basically

defined by curves, by five curves. For
every gear there's a curve, or for a

bucket of gears. It's basically that they
look up RPM, they get a value for that

curve, and if you exceed that value, they
switch to the reduced EGR mode. What they

compare this threshold with is the amount
of fuel injected per cylinder per

revolution, but you can also say this is
torque, just with a

constant factor. And then once you are
outside of one of these curves, it

switches to the non-optimized mode where
it emits a lot more emissions, and then

you have to go back into the green area to
switch back to the optimized mode. So

let's see what this means in practice. So
here we

have a car, and the traffic light is red,
so the car stops, and then the traffic

light goes green and the car accelerates,
and accelerates, and accelerates, gets

faster and faster, and then it's at the
highest speed here, and drives for a

while. And this is a typical city cycle,
this is there to... how you drive in a

city, and then the next traffic light
turns red and the car brakes and stops in

front of the traffic light. Let's take a
look at this again with one more variable,

the RPM. We can see that when the car
starts moving, the RPM goes up. And then

at some point there is a drop in RPM, and
this is because it's a manual transmission

and the driver switched to the next gear.
Now it's switched to again the next gear,

and this causes the RPM to drop, but the
speed to remain almost

constant, and it drives for a long time in
the same gear, and then the traffic light

goes red, the driver presses the clutch,
the engine goes back to idle state, there

is no connection anymore to the wheels,
between the engines and the wheels, and

the car gets slower. OK, one more
variable. It's the last one, I promise. It

is torque. The engine power in kilowatt or
something is not just a function of RPM,

it's a function of RPM and torque. so RPM
and torque together are very useful to

characterize engine behavior. And a very
good way to do this is to have a graph

where we put RPM on the one axis and
torque we put on the other axis, and then

we draw this in two dimensions, and so we
get this, basically. This is the operating

points we go through when driving the
cycle we saw. So the green dot here

indicates where we are. And so we restart
the car, the car accelerates, sorry, the

car idles for a while, so the green dot
stays there. It idles at around 800 RPM,

almost no torque, because there's nothing
to move, and then the driver accelerates

and the torque goes up, the RPM goes up
more slowly, and then at some point, the

driver presses the clutch, which
disconnects the engine, the

torque goes down, the RPM adjusts to the
speed of the next gear, and then the

driver releases the clutch and now the
engine again has to move the car, so the

torque goes up until reaching the the
highest RPM value and then that the driver

again switches to the next gear, so the
whole thing repeats, and then while the

car is driving, the majority of this the
cycle, the engine spends in this one

operating point. We're currently at 1800
RPM or something, and 80 Newton meter or

so torque. And then at some point the
driver presses the clutch, the engine goes

back to idle and stays there, basically.
So this is how you read this diagram. And

now what we found in the firmware was that
overlaid basically on this representation

we see a mask, or a limit. If we go over
this curve, those are the same curves that

I showed you earlier, just laid on top of
this. If we go over this curve,

then we switch to the worse emission mode,
we switch to the mode where the EGR value

is limited. So we can see in our driving
that this happens basically at this point,

the point where the driver
accelerates above a certain point, that

causes it to go over the load limit and
the engine basically switches or

significantly reduces EGR. And that's fine
because EGR doesn't work when you need a

lot of engine power, so it make sense that
that's at that point, and what we would

think is that it switches back once we
leave this load envelope, once we go below

the limit again, once we are inside the
limit, we would expect the ECU to switch

back to the full EGR operation. But what
we see instead is that this does not

happen, and the reason is that you don't
have to go under the maximum, the load

limit, you have to go into this green
area. You have to go back to idling at a

very low RPM to switch back to the full
EGR mode and this only happens at the very

end. When the driving cycle is almost
done, when the driver presses the clutch

and lets the engine idle. So especially
this long sequence where the driver... the

car was driving at the same speed, we were
technically in ... within the load limit,

where we're not exceeding the load limit,
but because we previously exceeded the

load limit and it doesn't matter for how
long you exceeded it, and we did not go to

the green area before, we were still in
this low EGR, high emission mode, even

though we're still within the load limit
imposed by the software. So let's take a

look at how often this actually happens in
real-world data. So here's us driving

through a city, and we can see we
constantly exceed these load limits. And

this is driving on the Autobahn, and yeah
we constantly exceed those. But they look

interesting. They look as if they had been
designed according to something, right,

they have the specific form and it's not
just... yeah, I... I don't know... and it

turns out if you do something really
strange, you can stay within these limits,

so we tried that and we managed to stay
within the limit by doing something, and

we... it was reproducible, we could do
this a lot of time it would always stay in

this limit and the answer is: If you drive
the test cycle you're staying in this

limit.
applause

So yeah, these curves basically defined...
they closely correlate to the limits that

you need to pass the NEDC. Okay, to be
clear it is fully acceptable that the EGR

rate is reduced when... for higher engine
loads. It's natural, you have to do this.

For example, when you accelerate the EGR
rate will decrease up to zero probably,

when you do it ... when you're running at
high speeds, all of that is great. So this

method of having a load limit ... well,
you can argue if really having the load

limit exactly where the NEDC is makes
sense, but having a load limit is okay,

right? However, what we think is not okay
is that, if you only exceeded the limit

once ... um... you would stay in this high
emissions mode for potentially a long time

until you get back to low speed idle the
next time. And we think that is the

problem. We ... so far this was all based
on what we saw in the software, so let's

see if this translates to something that
happens in reality. So to repro this we...

the car... drive at constantly... or we
let it idle, then we accelerate it to

2,000 RPM, we let it drive there for a
while and then we quickly exceeded the

load limit by going to 3,000 and then
going back and then after doing that we

would again stay at 2,000 RPM. So it looks
like this and we would naturally expect

the engine to operate in the same way on
the left and on the right side because the

engine is doing the same thing there, it's
the same torque level, it's the same RPM,

everything is the same. So we would expect
the same emissions, right, um ... and it

turns out it isn't. And ... this is a
slightly convoluted diagram. So if you

look at the green and red bars in the
middle you can see what happens before and

after exceeding the limit for just once.
And in the middle you can see the EGR

position, the EGR valve position, and you
can see that we get pretty high values

between... 6... maybe 65 percent or
something before exceeding the load limit

once.
And after we exceeded it once even though

the engine again is operating in the same
exact operating point, we see much lower

EGR valve positions, around 50% or
something. And if we look at the bottom we

see what the engine NOx emissions and we
see that they are significantly higher on

the right side than they are on the left
side. So this... for me, this does not

sound like this is truly optimized for
emissions because the engine is doing the

same thing, in both cases the emissions
should be low. So going back to this quote

that it works, the EGR and SCR injection
work to the full extent in a temperature

range of 20 to 30°C. Okay,
but what about the EGR load limit and what

about the the barometric pressure limit
for SCR and what about the SCR speed

limit? That would not be "to the full
extent", right? And the Opel answer is

really interesting. Of course, they denied
doing a test cycle detection, they say

they don't do that. And what they said is,
when asked whether they lied to the KBA

when saying that it works to the full
extent they said "The statement 'fully'

was really related to the NEDC test
schedule, right, which... it went on and

further... the Opel CEO had to say this.
He said: "The recent

accusations based on the findings of
hacker Mr. Felix Domke" - hey, that's me -

"are misleading oversimplifications and
misinterpretations of the complicated

interrelationships of a modern emission
control system of a diesel engine.

Emission control devices are highly
sophisticated integrated systems which

cannot be broken down into isolated
parameters." Especially not by a hacker,

right?
applause

That was kind of funny. There was another
funny thing. Sorry, I only have a German

quote and I didn't want to translate it,
but when Opel basically ... they

repeatedly say they don't have a cycle
detection, right, and they say it's not a

cycle detection because, if you use the
car on the street in the same way as you

would do them during the test cycle, the
car would behave in the same way, so it's

not...,right? applause ... and ... okay.
But what is with Volkswagen, right, they

have the same thing, if you drive the NEDC
on the street the car will go to test mode

they have the same thing. I don't see how
this does not represent a cycle detection.

That was a lot of things to say about
Opel, but on the bright side, they also

said that they will - even though all that
was incorrect, what we found - they said

"We will further improve the efficiency of
emissions after treatment of our SCR

diesel engines and so on as far as the
laws of physics allow. This includes a

voluntary service action" - and this
basically means a software update for your

car - "for the cars that are already on
the road starting in June." So that is

great. They're actually improving
something. Question's in which year,

because this statement is from May 2016
and it's not out yet, but... Opel actually

provided a new software already in July
and I think they already worked on this

for quite a while and in July 16 the German
KBA, the Kraftfahrtbundesamt, the Federal

Motor Transport Authority, they are pretty
nice actually, and they do know about what

they do, they are bit limited by the
resources they have, and by the manpower

they have, but they know about cars and
they know how to do these investigations.

I mean, they're a little bit bound, but
what they should do and what they should

not do, but they asked me to review a new
ECU software that was given to them by

Opel for the Zafira in question and
Insignia, which had a similar ECU and I

looked at that software and I dumped the
firmware and I looked at basically all the

code sequences that I looked at before and
I was positively surprised because they

removed... they addressed each of our
concerns. All of them, within the physical

limitations of course. So they improved
the temperature window and everything, so

there was a significant improvement. They
were able to improve the software and they

let the DUH, which is the German
Environmental Aid, they used a PEMS system

- PEMS is a portable
emissions measurement system. It's

something you put on the exhaust pipe on
your car and then you can measure the

exhaust during real-world driving, and
Opel gave them a car with the new ECU

software. Otherwise the car was identical
to the old software, and the results are

this, right, so on the left side you see
the old software, that has all these

things that we criticized, and on the
right side you see the same car with a new

ECU software and it's significantly
better. It's only slightly above the

limit, right, but it's much better than
before and to put this in relation,

before they were on the list pretty bad -
so this is sorted by worst to best - so

they are in the, well, upper half at
least, and now they are almost one of the

best cars, just by switching the ECU
software. And I mean this is great news,

right, they actually improved their cars.
Let's just hope they get this out to the

cars soon. Let's just hope it doesn't have
side effects and something, but I'm sure

Opel knows how to test for this. Going
back to these, we worked on the Opel

thing... I think the Opel case, it....
once they actually upgrade the cars, and

once the cars really show these great
values that the preliminary software

showed, I think we can close the Opel
case, but there's a lot of other cars

still to look at, and really, I mean...
the effort to do this does not scale to so

many cars, so we need to do something more
fundamentally to improve the situation.

What I found out is that digital control
systems, they are black boxes. The

manufacturers have designed them to be
black boxes. They even boast to you that

they are 7,000 parameter in there and no
hacker can understand this and it's a very

sophisticated problem. They are designed
to be black box, and this is not just true

for Opel, this is true for all car
manufacturers. Nobody wants anyone to look

into their ECUs, and people seem to be ok
with that. Like they think "Oh this is so

complicated, there are so many German
engineers working on

this problem, they must have found a great
solution." So we are trusting these black

boxes and we are not able to review the
black boxes that we put into our cars and

we have to trust the manufacturer to do
the right thing and currently, the

investigation to do this without
assistance from the manufacturer, it does

not scale. We can do it but... the
manufacturers can put more security on

their ECUs... it probably can be broken,
but it takes a lot more time, so it simply

does not scale sufficiently. The issue is
black boxes are really powerful, right.

Black boxes can hurt people with, for
example, excessive emissions. They can

kill people if we think think about
autonomous cars that do mistakes. So what

we do need, I think, is more transparency.
A system that can kill people needs to be

reviewable by the people. I think this is
a very important thing.

applause

So, to have a system that can kill

people... to have it reviewable by the
people, we need to do things. For example,

we need... we want access to source code
for reviews. It doesn't necessarily mean

we want open source, but we don't ask at
all the car manufacturers to open source

all the software. That's not what I'm
talking about. What we need is... think

about how Microsoft is sharing source code
of Windows with universities or other

countries. We need experts to look at the
source code, and we want control software

that is reviewable by design, that has a
lot of documentation, that has good

comments, that is human readable code. I
don't want to see a disassembly, I want to

see the source, the MATLAB, or whatever
they are using to define the functionality

source, and read that. And I want to
understand why did they choose that

curve of that map in this way? What was
the design criteria? That needs to be

reviewed. And we need transparency for
control software decisions, which means

that if a car operates in a certain way,
if I'm driving that car, I want to choose

that I can log what the car is doing, for
example by putting

in, I don't know, a USB stick or something
if it's my car, and then the car will log

all the data to that. That is... in the
end that allows me to reconstruct any

decision that the software does. I think
this is required to have the necessary

transparency, that allows us to un-
blackbox these devices. All right.

Thank you very much.
applause

Okay, I actually finished five minutes
early. I didn't think this would happen,

so...
Herald: I'm so surprised.

F: I am surprised too.
Herald: You are on time. You have five

minutes left
F: Wow, what do I do with these five

minutes
Herald: We can walk around the stage or...

Maybe people have some questions?
F: I think so!

Herald: Well, let's ask the Internet! Is
the Internet ready?

Signal Angel: Yes. Our first question:
What dou you think is the responsibility

of Bosch as a supplier for having their
software and hardware used for this?

F: So the question was: What's the
responsibility for Bosch, who built the

software for Volkswagen? It's a good
question and I have to be careful in what

I answer. My personal opinion, and let's
take this aside from Volkswagen and Bosch,

is that if you build software that you
know is used to be illegally it should...

it must be your responsibility to not do
that. And I'm not sure if this is

something that is legally enforceable, but
it should be something that's enforceable

ethically or for all of us programmers,
that we don't build software that is

designed to break the law.
applause

Herald: We quickly hop over to microphone
1 please.

Microphone: Thank you for a wonderful
talk. I'm just wondering if you're aware

of some cases of Volkswagen cars in
Australia, which was suffering from sudden

and rapid power loss. This was happening
about five years ago and there was a case

where a Volkswagen suffered rapid power
loss on a motorway. The driver was Mrs.

Melissa Ryan and she was rear-ended by a
truck and killed. So when you say that

these things can cause death, were you...
are you aware that any sort of Volkswagen

software has been leading to power loss in
the vehicles and affecting

performance on the road, now I don't know
whether Australian driving conditions are

different to European driving conditions,
and how that might affect that. Have you

done any tests that might indicate that
could be happening in normal driving?

F: Yeah, so... the question was whether
I'm aware of, I think an Australian

incident, right, where...
M1: Can I...

F: Yeah.
M1: There were many reported cases. One of

them was fatal, but there were many
reported cases of that happening.

F: Of a sudden power loss, is that right?
M1: Sudden and rapid power loss in the

engine.
F: Yeah, of the engine. I'm not aware of

these incidents and I what I do know
and... is that the the personal safety is

the number one design criteria for ECUs.
That does not mean that they are perfect,

of course, that could mean that rare
bugs... that there could be malfunctions.

I don't know about this, but at least it's
the first design principle to provide the

safety for the people driving the car,
which i think is a good thing, right. It's

not the profit or anything, or at least we
can hope so. I'm not aware of this

particular incidence, and so I can't
really say anything more about this. It

would be great if... Are you aware of any
additional details that were found in the

investigation, please sent them to me.
M1: Volkswagen was claiming that this was

a gearbox problem on automatic cars, but
then it started happening on manual cars

as well, so that excuse went out of the
window.

F: The issue with the problems is that
most of them are very complex, so they

probably involve more than just the engine
ECU, so they're very... but it's a

good example of where we need to
understand exactly what is happening, and

where we may not want to rely on
Volkswagen or any other manufacturer alone

to assist in figuring out what happens. We
need more transparency there so that we

can have definitely neutral accident
investigations.

Herald: This was a long question and
really detailed answer. Thank you very

much.
F: Sorry, I will be short

Herald: Felix, that's your applause
applause

music

subtitles created by c3subtitles.de
in the year 2018. Join, and help us!