Daniel Lange (DLange), Felix "tmbinc" Domke: The exhaust emissions scandal („Dieselgate“)

♪ (preroll music) ♪

Angel: After half a year, Volkswagen committed
to tweaks to their emission readings.

Those two boys, Daniel Lange and
Felix Domke here on my left,

will share some insights with us.

Daniel will not only focus on the ECUs,

which is the acronym for the
Electronic Control Unit,

and I think we're seeing one
over here already,

whereby Felix will show us some tricks
to extract and tweak the firmware.

On both sides we will see how many people
have been involved in the entire process

and we would get an idea what everything
is involved in there.

So, you applause and I'm gonna take over
the Bildschirm.

Good luck!

Felix: Alright. Hello? Okay.

Hey, so, I'm Felix Domke.

Do we see the video output yet?
No.

Anyway, I'm Felix Domke.

I'm here on my own
because I was personally interested

in how Volkswagen is cheating
on their emission control.

And maybe we get video at some point.

I want to stress that I was self-funded.

I did this with my own money because I was
personally interested in this.

So I did not do this on behalf of anyone else.

Daniel: Let's start the Keynote again and
see whether that one works better then.

F: I am sure we figure this out.

D: Oh, it worked before?

Yes, that's because one of us two wanted
to use a Mac.

audience laughs

F: But I wanted to use Keynote,
I don't care which operating system.

D: This one works.

F: Anyway. So I will now hand off to Daniel
which will give the first part of this talk

and after that I will give
the second part of this talk.

D: Okay, thanks Felix.
My name is Daniel.

I used to work for a big Bavarian auto
manufacturer which is not Audi...

audience laughing

...for 14 years.

I've been running the IT strategy,
I've been doing IT architecture.

And most relevant to this talk,

I've been responsible for the
process chain electronics and electric.

I've done the rollout of
Connected Drive in China.

So I kind of have quite deep insight into
how the automotive industry works

and I'd like to share a bit with you.

I'm an Engineer by training,
I guess many of you are.

And I want to share
how the engineers think

inside such a big corporation
like Volkswagen,

and what pressures, what boundary
conditions they are working on.

I have my own company now
which makes my life a bit easier

than Felix's, as you see
in the legal disclaimer.

Those are folks from the UK.

They're called "Brandalism", I hope you
notice the "McDonald's"-M at the end.

Those are folks who started a few years
ago to reclaim the public space.

They were just annoyed by
all of that advertising.

And when the Paris negotiations
for the eco treatment came,

they just felt a big invitation to
use the opportunity

that Volkswagen has created
for all of us

and make advertising in their style, but
perhaps not in the message

they would usually have conveyed.

I'm a strategist.

So what is the thing that defines how the
automotive indusry works today?

We are in a saturated market.

In the developed countries, so everywhere
in Europe, in the North Americas,

everybody has a car that wants one.
Some have two.

So when you want to sell another car

you're basically talking about replacing
an existing car with another one.

The only growth you have
is in the BRIC states.

BRIC is: Brazil, Russia, India, and China.

And, here, especially in China.

You have a big overcapacity. There's just
too many automotive manufacturers

and there's too many plants they have.

So all of them basically struggle
to get the loads on the plants,

to produce enough cars and to have
those cars sold at some point in time.

Because the queueing in between
production and sales

is actually the big parking spaces you see
in Bremerhaven or so

where there's ten thousands, in some
countries even hundreds of thousands of cars

basically stored in between
production and sales.

Ten years ago, fifteen years ago,
that didn't exist.

The cars were basically sold off the factory.

But people have been moving away
very, very slowly from cars.

They have, as I said, a saturated market.

It's just not that easy
to sell a car anymore.

That is because of social shifts.

When I was young, there was
"The Dukes of Hazzard",

there was "General Lee", this car
that basically is the star of the show.

There was "Knight Rider" and nobody watched
it for David Hasselhoff, not even the girls.

They watched it for KITT.

When I was young, I wanted to own a car.

I wanted to have KITT, possibly.

And when I grew old enough, I found out
I can get a car that looks like KITT

but, you know, all the
fun stuff is not in there,

so I turned to computers.

The next thing is organization,
megacities.

We live in very condensed spaces
in those cities.

If you talk about a place like Beijing
where there's like 21 million people

in an area that is one city,
where there's nothing big inbetween,

there's no river, there's no forest,

it's just like one city.

If you go to Tokyo, Yokohama,
you can drive on the motorway

for nearly three hours when you enter
the city before you leave the city.

And you're driving on the motorway,
you're driving on an elevated road

for which you paid toll
so you actually can drive.

But it's three hours before
you leave the city again.

And in these cities owning a car
and operating a car

is about the worst thing you can do.
You just don't want to do that.

The average speed of a car in Beijing
these days is 12 km/h.

If you're a good runner you can beat that.

And incidentally this is exactly the speed
that a horse carriage makes.

audience laughs and applauds

We have managed to undo
all of the innovation of the last 200 years,

it's just the interior
is a little bit more comfortable.

The actual getting from A to B is the same
as with a horse carriage these days.

And then there's technology shifts.

The problem is, there are big things,
big visions that everybody follows,

like electric mobility.
Electric mobility means:

You buy a car that's one and a half times
the price of your standard car,

you lug around 300 kg of batteries for
no apparent reason to do so,

and you now need to install something
in your garage —

which you most probably don't have,
look at megacities —

to be able to recharge the car
because it only goes a hundred miles.

So it's currently not a very compelling
thing to sell to the end customer.

There's self-driving cars, which is kind of
a great, big vision.

But I would really call that a "vision".

A "vision" is something that's not being
implemented in my lifetime.

And then there's downsizing.

Downsizing means ...

Everybody wanted to have
the biggest engine,

everybody wanted to have the
biggest car, let's say, 10 years ago.

You wanted to have that six cylinder
that was giving you status.

But now the automotive industry
has an overall cap

on how much emissions the average
new car fleet my have.

And they can only reach that if they manage
to sell smaller engines to you.

Because for everybody who buys
a really big engine

that will never ever make that emission cap,

they need somebody whom they've
sold a small car to —

preferrably an electric car, because
they even have statistical advantages

to make them a bit more attractive —

to set that off.

So very literally the poor guy
with the small car needs to exist

for the rich guy that drives the
eight cylinder and doesn't give a shit.

Strategy-wise, there's only two things
that an automotive car company is driven in.

And that's really everything there is.

There is "reach a target ROCE".

ROCE is: Return on Capital Employed.
That is just two numbers:

your EBIT, which is your Earnings
Before Income Tax,

and the amount of money you have
in your company, the employed capital,

which you got from people
that lent it to you

or from your stakeholders, from your investors.

And that is what the company
is measured against.

Every automotive company
basically runs like this.

Just this one figure,
it's a percentage like "30%."

"30%" means: On the money you have
you made a 30% return during that year.

The downside of measuring in ROCE is that
everytime you use that Euro or Dollar

it counts again because
the money works for you.

That means what you're looking at

is a company that gradually moves
from a very industrial type of application

to something that tries to move faster,

that tries to be quick and
regain money faster.

And then there's
"outperform the competition."

You have to understand the situation

that there's a good dozen companies

and everybody has the
same strategic position:

"We will outperform the competition."

So statistically, you will know that
half of them are going to fail

because that won't happen, right?
Somebody has to be the lower half.

But the only thing I have seen
in about five or six companies

where I know the strategy in detail,
is: the sequence.

Is the first or is the latter
the more important one?

And sometimes that depends on markets.

There's this new emerging market and
you want to outperform the competition,

you want to grow more.

And then there's this laggard market somewhere
in the European Union

where you just look at the money, you know,
how much money are we making on this.

But that's all, that is how an engineer is
basically steered, that is the strategy.

And that means when you break that down
through the levels of hierarchy, what is counting is:

How much money do you
need to make this?

How much money are you
gonna make on this?

Those two divided will be
contributing to the ROCE.

And do you deliver anything that can
help us outperform the competition?

You notice that there is a lack, which is,
you know: What does the customer want?

Or: What is good for associates?
Or something like that.

Just in case you hadn't noticed before.

Okay, I'd like to do a bit of a quiz with
you before you all fall asleep after lunch.

Eleven million.

"Eleven million" in the context of the
exhaust emission scandal.

What is that number?

Audience: Cars affected!

Correct! Cars affected.

Eleven million is actually the
Volkswagen cars

which need to be recalled world-wide

to get this little filter thing fixed
and their software updated

to meet the emission targets which they
had been produced against.

1500 ...?

A: Number of engineers!

Number of engineers?
No, not correct.

Number of engineers would be above 10000
for a car in Volkswagen Group. Sorry?

A: Cost for fixing it per car?

Cost for fixing it per car? No, that's
maximum 600, we're gonna see later.

unintelligible suggestion from audience

No? Well that was too difficult then,
and that was a bit intentional.

That's the amount of hard disks they
collected from the associates.

audience laughs and applauds

Now the thing is, we've had in the press
that there is maximum 13 managers

which are responsible for this
emission scandal within Volkswagen.

But then they collect 1500 hard disks and
USB sticks from 380 associates,

and that number is a month old because
they haven't reported newer numbers.

So something is mismatching there, right?
Something is mismatching there.

So the first number we have is
for how many associates

are actually somehow
affected by this is 380.

Because you come to work somewhere in
Wolfsburg I think, right?

And then there's this nice chap coming up
and telling you,

"Uhm, actually we took
the hard disk off your PC,

you're gonna get a new one from IT,
we guess tomorrow,

they're a bit behind with, you know ..."

6.7 billion ...?

Just shout!

unintelligible suggestions from audience

Fine? No that will be less, much less.

unintelligible suggestions from audience

Yes, you're getting close.
It's the money they put back,

they set aside to actually pay
for the recall and the legal fees.

Now if you divide that by 11 million
you get about €600 per car.

So it's not that much money per car.

In Europe, the plan is basically that you
go to the dealer and get a software update.

In the States, people already got $1000
in cash and in coupons

as a goodwill measure.

So something I learnt from Martin Haase
here going to the CCC Congress all the time

is that we need to read text really well.

So the upper one is the original in German,
the lower one is my English translation.

The English translation is as accurate as
possible, so it's not good English.

Please excuse that, it is so you get the gist
in case you can only read the English.

So that is Mr. Pötsch, he's the president
of the Volkswagen supervisory board.

He is the poor guy that now
has to sort it all out.

He used to be the CFO. We're gonna see why
that is important a little bit later.

And he has made this analysis:

It was "individual misbehaviour",
so it's not an organizational problem,

it's "weaknesses in particular processes",

and it's "the attitude in
particular sub-partitions" ...

"Teilbereiche des Unternehmens",
it's impossible to translate in English,

it's actually impossible in German, but,
you know, the legal team came up with that.

So the "attitude in particular sub-partitions
of the company to tolerate rule violations."

Now, if we go through this very quickly:
It's not a rule violation,

you violated the fucking law.

The other thing is, if you have particular
processes, you have particular associates,

and you have particular sub-partitions
of the company,

That tells you something, right?
That just tells you something.

This was probably two days' work of
somebody in the legal team,

and I guess you noticed, right?
I guess you notice.

"Legal team" is probably these people.

Jones Day is a big American lawyer company

and they've asked them to help
with sorting out this.

Now the funny thing is, there's public prosecutors
all over the planet interested in Volkswagen

but Volkswagen thinks it's
not really clever

to have those people come in
and find all the info,

it's better to have Jones Day,
their own kind of bought-in legal team,

ask the associates first.

Now the problem is, whenever the let's say
German prosecutors wake up and go in there

and say, like, "We would like
to see what has happened,

so please hand over the material,
please hand over the hard disks,"

they would get a very, very nice reception,
be greeted with coffee and shown a room

where all of the hard disks
and everything is stored,

"We collected it for you."

I have no idea whether they gonna show
everything to them

I have no idea whether there may be
some material lost in between.

We've heard from Anna earlier
in Germany it seems to be

that things hit the shredder and
hard disks get lost and everything.

So if it works like that in the government
I have no idea how it works in companies.

But if I was on the prosecutors I'd
probably see that I speed up a little

because otherwise you'll get
all pre-prepared material.

And because Jones Day
can't do all of that —

you have to interview all of those people,
and you have to look through the hard disks —

they asked Deloitte
to come in and help them.

Now Deloitte are a very good company,
they have very, very good forensic teams,

so that's a very good choice.
But the important thing here is:

Out of the four big consulting companies
that do finance analysis and stuff

those are the only Americans. The others
are headquartered somewhere else.

So what it tells you here —
American legal teams, American auditors —

that's where Volkswagen looks.
Volkswagen is actually afraid of America.

They are not that afraid of Europe or some
other country in some other continent.

Now, let's talk text a bit again.

"We have no findings on the involvement
of the supervisory board

or the board of management presented."

Now, again, "no findings", okay,
"presented", right?

It's not "we don't have any findings"
or "there is nothing",

it says "we have no findings presented."

And the other thing is "involvement",
that's an odd term.

In German, "Involvierung",
that's not even German, right?

If you look it up, "Involvierung", nobody
of you talks of "Involvierung"

when you talk to your family or
when you do something at work.

The trick here is, the supervisory board
has a reason for existing: supervision.

audience laughs and applauds

The board of management has a reason for
existing and that is: decision.

They are the deciding body.

None of them are ever "involved", right?

When you work on something
in a big hierarchical company

there is no "involvement"
of your board member,

there is no "involvement"
of your supervisory board member.

So per definition, they cannot
have an involvement, right?

If he wanted to be straight
he would have said:

"I, as a former board of management director
and now as the head of the supervisory board,

I guarantee there was no involvement
of my or my colleagues in this.

And if there was, I would pay back my
salary, I will go to jail, I will ... whatever."

Right, sacrifice a goat?

But that would have been
straight communication.

But this is not straight communication,
this is... bullshit.

Okay, quiz time!
10 ...?

You remember, this guy here told us there's
no involvement in anything fishy, right?

It's all those small engineers,
all those bad, bad people down there.

But they are gonna hunt 'em down, right?

So there's no involvement
with anything fishy here.

So, in that context, what is "10"?

A: Board members!

10 board members?
Close, they have a little more.

A: Levels of hierarchy.

Levels of hierarchy — quite good. It's,
I think, eight or so, but you're quite close.

No, it's actually the amount of planes
that Volkswagen owns.

All of them are jet planes.

Because if you're a board member
you have to, you know, fly in style.

And because there's nothing
ever fishy at Volkswagen

it's run by Lion Air Services
out of the Braunschweig airport.

And obviously, Lion Air Servives is registered
in Georgetown on the Cayman Islands.

applause and laughter

Nothing fishy ever in that company.

Okay, let's get back to topic. I have
about another ten minutes

before I want to get Felix the chance to
show you what he has done on the ECUs.

So I need to get you up to speed
about how all of this context here works.

And this here is called the NEDC,
it's the New European Driving Cycle.

This is what your car is tested
against for emissions.

It works like that: You condition
the vehicle a day before.

Which means you really
drive it hard on the Autobahn

so the exhaust is really free
and everything.

And then you do these cycles here where
you basically accelerate the vehicle,

slow down, accelerate the vehicle,
slow down, accelerate the vehicle,

slow a bit down, slow a bit more down,
and then you cycle again.

And the last, cycle 5, is an optional one,
depending on what you measure,

that is actually going to the autobahn

and you're going up to a top speed of
120 km/h for a very short period of time.

The people that have detected the
tweaked emissions

in the VW Jetta and Passat they looked at,
they have called this

"a very light usage cycle,"
and they called it "unrealistic".

Because basically nobody drives the car
like this, it's a very artificial thing.

And that is the problem
for the engineer, right?

The engineer looks at this and says,
"Yeah, you know, it's a standard.

It's something we do to measure against."

But nobody drives like this.
It's not realistic, right?

So if you fake the data in this we're not
actually faking something our customer uses

because no customer drives like this,
it's very artificial.

And there's a very good report by ICCT
which is, "Mind the Gap".

Which is what you hear in London when you
go into the Tube.

And what they mean is the gap between what
gets out when you measure emissions like this

and what gets out when you
actually drive the car.

And that gap is widening
year by year by year.

Because engineers get better and better
at optimizing for this cycle.

The cars on the street? Phhh, they do get
better as well, but less, right?

That's why the gap widens.

And trickery on those tests
is very common.

I'm sorry you can't probably
read that in the stream

and probably can't read that
when you're back down there.

But that's an original slide I had to take
from Transport & Environment,

from that report which I just named.

And what it says there is
what tricks people are doing

to actually drive down the emissions.

For example, they blow up the tyres

by 3 bars more than you could
actually use them on the road.

Now when you do, the bottom of the tyre
looks like this, right?

So that means you only have a very, very
small portion of the tyre

that still touches the ground,
so your resistance gets reduced.

They put diesel into the oil beause diesel
is lighter than the oil which you are using

inside the vehicle, so friction gets reduced.

They take off the mirror, the side mirror
on the passenger side

because that is not legally
required to be existing,

so, you know, it's resistance,
so get away with it.

They tape close all of the
openings of the vehicle

because obviously when the
wind goes over it

it goes much smoother
once you have everything taped.

Now all of these things are either okay or
they are kind of borderline grey area.

And they do this. This is how
actually emissions are tested.

So this is why an engineer,
when he looks at this, says,

"Yeah, it's an optimization problem. They
want me to get a low number

and I have pretty clever ideas, which involve
diesel and sticky tape and everything,

to reduce the number."

sighs

The results are this.

That's from a 2012 report from — a 2013
report, I'm sorry — from ADAC,

the German MRT company.

And what you see, the lighter blue ones
are actually the emissions

which the car produces in this cycle.

The darker blue ones are the ones
which are produced

when you just go on the motorway
and drive them.

And you see that there is a discrepancy

which is ten times, twenty times, thirty
times what is the measured data.

So what you need to understand is that
even in the past nobody ever thought,

nobody in the industry ever thought that
the data which was measured

had any real connection with reality, right?

The only connection was, you knew that
what you're measuring

within the duty cycle NEDC
is definitely less

than what you would ever see
in any realtime use.

But that's it, that's it.
That's no secret, right?

It's something that has been
out there for years.

Now the folks at Deutsche Umwelthilfe,
which are actually people that helped

find out what Volkswagen did, they wanted
to see that others do it as well.

And because I wanted to give you as much
information as possible

we are going to look at this product here now,
which is not a Volkswagen as you may see.

And when you measure this car
it actually looks like this.

So that means when the car is thinking it
is running an NEDC —

because it is conditioned to do so,
it is the right temperature,

it is the right setup —
it actually delivers the blue bars.

And if you run it because you just run it
and you don't do the conditioning

it delivers the grey bars.

Now there's many things you can say
about how they measured this

because, obviously, this is not science to
the best level of accuracy.

But you do see a pattern here,
and you do see the pattern

of the 30-, 35-fold emissions.
And that is what you always see

because this is what an engine
like the one in this car —

a 1.6 l diesel engine
if I remember correctly —

actually does when it's just
operated normally.

And the lower ones are the ones which you
get when the engineers did all the good tweaking.

Now why has all of this ...
Oh sorry, so this is just one test, right?

And you see that this test,
when the vehicle is cold,

you get fresh air with a nice rose smell
out of the exhaust.

And when the vehicle is operated normally
you basically get what you expect,

you get the combustion products
out of burning diesel.

Now why is all of this now a problem?

This is now a problem because of the
American legal system.

The American legal system is
very, very different

from what people in the European Union
are used to.

In America, there are two things which are
a bit strange perhaps

to somebody who's accustomed with a
German legal system.

The first thing is, there's jurys.

So there's common people that actually
decide about what's right or what's wrong.

And that means, what they
award as compensation

to people that have had a disadvantage
are often astronomic figures.

Now these figures are sometimes
reduced again by the judges,

but it's not uncommon that if something
hurt you or you got into an accident

you're awarded million dollar sums.

In Germany, if somebody shoots your
eye out, you may be getting €100,000.

So there's a huge discrepancy there.

And the other thing is, in America
there are punitive damages.

"Punitive damages" means: You did
something wrong, you did it on purpose,

and you're punished for it.

In Europe, a company basically is,
you did something wrong

so now you have to compensate the
disadvantage somebody else had.

So to a certain extent,
a company that doesn't try to trick

actually kind of loses an opportunity because
if they are not detected to be tricking

they have just saved money.

There's no punitive element, there's no
"You will go to jail for this."

At least in this context of
environmental regulation.

Now in case you couldn't read that, that's
actually a sign I took in california.

You go into a store and it tells you
that basically everything you see there

and touch there is giving you cancer and
your unborn children will be damaged.

This is what it says there: Belts, shoes,
jewellery, handbags,

all products with metal, and everything

causes cancer, birth defects,
and other reproducive damages.

So this is America, right?

Their view of protecting the consumer is
completely different from Europe.

And this is why Volkswagen goes and says,
"We will show good faith.

We will give you, American Volkswagen
owner, a thousand dollars

because we just wanna make sure that
you at least know we care."

It's important that you care because
the jury will say,

"Well at least they awarded $1000,
maybe a little too little,

but at least they did something."

The jury would say that. A professional
judge in Germany would say, "Pshh, why?"

So this is why as a European customer
you actually go to the dealership,

and if that guy is really nice
you may be getting a coffee

while you wait the hour
that he flashes your car.

So that's the only thing you're currently
supposedly getting in Europe.

Okay, now the problem is:
What they did hurts.

And it hurts because,
if you do the statistics ...

Very nice people have published a
publication here, a real scientific publication

where they did the maths,
and they say:

59 people may be dying
earlier in the United States

because of the additional emissions
in the environment

which they took in and which may
damage their body.

The social cost of treating those people —
because they may be developing cancer,

they may be going to a hospital,
and so on —

is about 450 million Euros.
Now that's statistics, right?

"Lies, damn lies, and statistics."
Mark Twain is often quoted with that.

But the problem is: That is a real cost,
it is a real damage.

If you do violate emission laws it is
something that is damaging people's health.

It may be something that is difficult
to prove statistically,

but it is something which you don't only
do to save money here or there,

it is something which you do
to actually hurt people.

Okay, I need to speed up a bit. Very
sorry, skip this, that's the next quiz.

15.9 million is actually the salary
of this guy here.

That's a lady from BMW,
I just wanted to put that out there.

She says, "It shouldn't be called
Dieselgate, it's Volkswagen-Gate.

We never did anything wrong at BMW."

And the SZ, actually, yay, they follow,
right? In November, it was "Abgasskandal",

in December, it's "Volkswagen-
Abgasskandal".

The only problem is that even in 2000,
BMW was cought cheating on the Motorrad.

So this is 15 years ago. 15 years ago
BMW actually put the same code

which we are now seeing in Volkswagen
into their ECUs for the F 650 motorcycle.

And we will see again here the same 34,
in this case, -fold increase

in between real use and test bench use.

Now, honestly, they've been caught, they've
been caught earlier, and they fixed it.

So in 2001, they actually
brought a new version

and apparantly that didn't have
this cheat code anymore.

But here we see a pattern again:
too little time for development,

too little money willing
to be spent on this,

so engineers try to trick.

When you get caught,
and you get caught early

nobody probably of you
remember this here.

It's fine, it kinda fades away into history.

If you're Volkswagen, if you have
11 million cars out of there,

you have a big problem.

Okay, I'll skip this one, it's really
nice, you can see it in the slides.

But I have to go to this here
to give Felix enough time.

So how does component development work?

There's a huge set of legal frameworks.
It's a very structured top-down process.

You get requirements from the people
that represent the market in the company,

you get requirements from the CFO,
from the finance director.

And these are broken down into documents
which are more than a thousand pages long.

And there's every single detail that
could exist in this ECU written out.

There's a piece of paper
for everything it does.

Everything. There's not a bit in this thing
which is not pushed down

into a very hard set of requirements.

This is then put into a tool,
often Rational DOORS by IBM or something,

and then every time something changes
this is documented.

There's a complete paper trail, right?

So that means unless there
will be a cover-up,

unless we're not given all the information
as a public,

there's no way Volkswagen cannot find out
who did exactly what at what point in time,

which level of management was involved.

Because every step of the development goes
through a Q-Gate, a Quality Gate.

There's managers sitting there and they're
approving everything it does,

every progress that has been made,
and they're getting reports,

at least bi-weekly, on the progress.

And these reports go up the ladder, they
are copied to the next levels of management.

So this is a fully transparent process and
this is a fully top-down driven process.

It is completely impossible that you have
an engineer that sits there and says, like,

"Well, I wanna cheat," and does the code.
There's no motivation for him to do either.

He doesn't get any money for it, he would
only be risking his career, so he won't do.

And this is why we have paper trails,
and this is why engineers have written down,

"I'm doing this because my
manager told me to do this."

And this is why you have Bosch sending
a letter in 2007 to Volkswagen which says,

"We delivered you this code you
requested. We're your supplier, we do.

But if you send it into production
it will be illegal."

And they did.

So this is how actually this
exhaust system works.

And this is a little bit important to
understand what Felix is now doing

and showing you how the ECU
that manages this all works.

To the left would be the engine,
to the right is the exhaust,

the end of the exhaust
where the remainders come out.

And the first thing is,
you have diesel oxid cathalytic

and it basically takes out ...
The interesting stuff here is CO,

so carbon oxide, and PM, the
particle mass, through 98%, 50%.

The hydrocarbonides before that,
they just kind of don't go through

the rest of the process anymore.

Then you have a filter that basically
traps all of the diesel particles,

the stuff that causes
cancer in your lungs.

But you have to burn them out at some
point in time, about every 700 km,

when there have been enough collected.

So it's a bit a trick, right?

The trick is: You collect them so they
don't exit the exhaust

but at some point in time you have to burn
them again, so they do exit the exhaust.

Now the positive thing here is,
they get larger, and the larger they are,

the less risk they — at least as much
as we know — cause as a health hazard.

So this is the DPF here. And then at the end,
this is the really interesting thing,

this is what most of the
scandal now focuses on:

There's a selective catalytic reduction.

And what this thing does is,
it does reduce the particle mass,

it does reduce the particles.
That's nice.

But the interesting thing is NOx.
It goes against this to about 90%.

So this is what it is made for.

It basically injects urea into the airflow
and helps to reduce the NOx content

by creating by-products
which are mostly water

that comes out the end of the exhaust.

And this is the system, this is a very
complex technical system

that has to be managed,
and this is managed by an ECU.

This ECU which they selected to do this,
and everybody does, is the engine ECU.

Because to the left of the diagram before
was this big engine, you didn't see it,

it fell off the diagram, but that's
actually the fan blowing into the system.

So this is what you want to manage to
actually control what happens there.

Now this thing is quite
a sophisticated processor,

it's about the most complex device
outside multimedia and entertainment

which we find in the car, and it is a
very proprietory thing

because it contains a
physical model of engines.

So there have been hundreds, if not
thousands of engineers sitting there

and modelling how an engine works,
really physically modelling it.

And the things that an OEM —
an original equipment manufacturer,

a car maker — can actually tweak
are variables.

They can say,
"My engine has this and this size,

my combustion cycle looks like this and that."

But the code itself is opaque to the OEM.

It's a proprietory product which you can
buy from Continental, or Bosch, or so.

And there's about 20,000 variables
which you can tune.

And this thing is simulated and tested
to death. Because it is hugely important.

Because you have this machine here
that has like 100, 200 horsepowers

and if you steer it wrong it will blow up,
and it will blow up really hard.

So this is why this thing is about the best
tested piece of software you will ever find.

Which also again means there's everything
documented, everything is written down,

everything is seen by everybody
who's working with these,

whether it's in development,
whether it's in integration,

whether it's in the plants that
flash these things, and so on.

There's nothing secret here in this, right?

The functions which are there are
actually there to be seen,

well, seen if they are named apparantly, and
that is something that Felix will talk about.

audience applauds

F: Thank you. Hey, okay.
So I will do the second part of this talk.

I'm Felix, by the way.

So my motivation with this
was a little bit different.

I'm curious, and, I mean, we can find a lot
of source material for this whole scandal.

We can find a lot of
information in the press,

a lot of information in the
Volkswagen press releases.

However, it should be easier
because all the cars are there,

the 11 million cars are out there
that have the cheat code in them.

And we are hackers, and we know code,
and the truth is in the code.

So my approach was, well,
let's take a car, let's take it apart,

let's take the firmware out of it,
let's throw it in a disassembler,

maybe get some measurements, and
then look at what the car is actually doing

instead of relying on all of this
second-hand, third-hand information.

So what do we need for this approach?

So first of all, we need a car
that's affected.

You need to drive that car somehow,
and driving a car on an open road

can be dangerous if you have to follow
particular driving cycles.

So there's a "dyno" you can put the car on
and then you can just drive

without the car physically moving.

The wheels are moving,
but the car isn't moving.

And this is what other people have done,

and they have taken very interesting
measurements out of this.

However, we as hackers,
we can go one step further.

We can take a look at the ECU itself.

And not only that, we can also ask
other people who worked with these things

and may be able to get
more information about them.

I will talk about this in a minute.

So first of all, this is my car, luckily that
car was affected by the recall.

So I was very happy when I got the letter
telling me I have to go to the shop in January

and get a firmware update because
firmware updates are exciting, right?

I love updating things,
so updating a car seems great.

Yeah, it sucked that my car was putting out
more emissions than it should have,

but otherwise, it gave me the chance
to actually look at the car.

I mean, I could have rented a car or
something, but that makes it much easier.

I also went on a dyno with my car.
On a dyno, there are no speed limits

or no people to run over when you just
have to keep a constant speed or something,

so it makes things much easier.

And I talked about ripping apart
my car and disassembling it.

I didn't really want to do that,
so what I did instead was what I always do:

I go to eBay and I bought an extra ECU.

Here it is, maybe you can show it?

You can go here after the talk
and take a look at it.

This is the ECU. This here is the main CPU
that also includes the flash.

On the other side there are the power drivers
that drive the actual stuff in the car.

And then there's other
watchdog circuits and so on.

Okay, thank you.

So, the ECU was built by Bosch,
it's an EDC17C46,

that's the name of the hardware.

And it can easily be obtained on eBay,
and you can put it on your desk,

you apply 12 volt to it and then it boots.

It will complain about a lot of
sensors being missing and so on

but you can see it executing code.

And it doesn't have the very same
firmware as my car, but it's very close.

The flash chip is unfortunately in the
same pakage as the main CPU,

which is an Infineon TriCore chip,

which is apparantly only used
in automotive equipment,

or at least I'm only aware of it
being used there.

And I was able to dump the flash by
attacking the hardware

and exploiting a bug in the hardware
that I haven't found documented anywhere,

but it was not that complicated.

And then I had a firmware dump, I had a
2 megabit binary,

and I throw it in a disassembler.

And what we see is interesting because
the code is written very different

from other code that we know.

So usually, code has
a lot of flow control

and usually more or less
resembles spaghetti code.

This was the exact opposite.

It's more like someone took electrical
schematics and put them into code.

There's a set of input signals,
there's a set of processing on it,

and there's a set of output signals.

That gets updated every 10 ms or
once per rotation depending on processoids.

Really interesting way of writing software
and building this.

Also it's very data-driven, so a large
part of the firmware is not code but is data.

All of the computations,
they don't use constants at all,

they always refer to something
from the data section.

As Daniel said, Bosch writes this code,
the code is not directly visible to Volkswagen,

but they have visibility into this data,
and they know what the data does.

They have tools to change the data.

Volkswagen and other companies
can customize this,

really they cannot just customize it,

they can change the whole
behaviour of this ECU

by changing just the data, not the code.

The ECU really is a small embedded machine
in your car that takes care of the engine,

it's an Engine Electronic Control Unit,
there are multiple names for it.

The most important thing that it does is
that it takes sensor input,

for example the throttle, and then it
applies control to the system.

For example it calculates the amount of fuel
to inject, the amount of air to inject

to make the motor running at the speed
you want it to run.

These days it's much more complicated.

One important thing the ECU does
these days is emission control.

This is why we would expect to find the
"cheat code", the code that cheats

that Volkswagen used to
cheat in the whole thing,

we would expect to find it in the ECU.

Now taking a look at
two megabyte firmware binaries

that doesn't have any visible strings in it,

it's kind of painful if you're just suscepting
a code analysis.

So what I did was to do realtime logging.

You can actually read data from your ECU
by plugging into this OBD-II port

which is next to your steering wheel.

And while the engine is running you can
read out certain data.

Usually you can read out boring data
like RPM, and speed,

and some things that the
vendor wants you to see.

But there's also a mode that's
a little bit hidden,

but you can get pretty easily into it,

where you can read by address,
where you can just read the whole memory.

Well, not everything.
Some security data is locked out.

But the data we are interested in,
we can read that memory.

Now we still need to understand
where the interesting stuff is.

We can disassemble the firmware,
and that's all fine.

We can also get a little help
from something called "A2L files".

The chip tuners use them extensively
when they change the mappings,

they want to optimize an engine
for a different goal,

for example for more power instead of
long lifetime, or something.

They change things in the ECU firmware.

They do reverse engineer a lot,
but they also got these files.

And I'm not sure how they got them,
but they are out there.

And if you use the right Google terms
you will find them.

They are specific to each firmware.

I wasn't able to find one for
my actual firmware

but I was able to find one for
firmware that is close to mine.

And if you look into this file,
what you see is the symbol names,

it's basically a fancy map file.

You see the symbol names, you see a
mostly German description of that symbol,

you see a real-use unit, and you see the
adress in memory that we can read at.

So with the help of these files we can read
out almost any internal state in the ECU.

We still have to make sense out of that,
but at least we know where the data is

and what to look for.

It's surprising how complex an ECU is.
For example, this thing, what does it display?

Everybody would say it's a function of RPM,
it shows you how fast the engine is running.

Well, it's not quite the case,
and if we look careful we see that

this code is post-processing
the RPM signal.

It's 12 kilobyte of densely written code
that has a lot of internal state

that tries to make the RPM value,

convert it to something
that the customer wants to see.

For example, you want your idle speed
to be stuck at 780, you don't want it to oscillate.

But in reality it does,
and this code takes away all of that

and makes it flat 780.

You realize probably at this point that there
is a lot of cheating that could go on here

without most people noticing.

You don't really believe that the speedometer
in your car displays your actual speed, right?

It displays something related to speed ...

But let's get back to topic.

Selective Catalytic Reduction is the process
of, well, if you don't have it

you get a lot of NOx, of nitrogen oxides
at the end of the exhaust.

That's bad, you don't want that.

There is one way of getting rid of this,
is to add an SCR catalyst.

And the SCR catalyst —
I simplified this a lot,

you can find a lot more information
about this —

SCR is a process that reduces the NOx
using something called DEF,

or AdBlue is a term for it.
It's some fluid that you put in there.

Basically it's an Urea/water solution.

And the AdBlue, at a high temperature,
converts to Ammonia

and then it reacts with the NOx
to nitrogen and water.

Which is great because that's not
in any way harmful to us.

However, there's a problem here because
the dosage of the AdBlue needs to be correct

and it's very hard to do.

If we dose too little of that
the conversion is not perfect

and we will still get
a lot of NOx at the output.

Which is better than not doing anything.

It's not perfect,
but it's not more harmful than before.

However, if you put in
too much of the AdBlue

what you get at the output is ammonia,
and you really don't want that.

So the primary goal of emission control
is, if you have the SCR system,

is to eliminate as much
as possible of the NOx

and minimize the amount of ammonia
that comes out of the exhaust pipe.

Ammonia is NH3.

Calculating the right dosage works
with a model again.

They modeled everything that happens
in the exhaust process,

they have a model of the catalyst,
they have a model of the internal state,

they do have a number of sensors and
outputs from the other models

that tell them a lot of values.

And the model uses this with a lot of
internal storage, internal state.

And the model then calculates
the amount of AdBlue to dose

to convert as much NOx as possible
without leaking any ammonia.

The way things usually work in an ECU is,
there's one system that controls things

and there's another system
that monitors things.

It's independent from the main system,
it tries to be as independent as possible.

It's still running on the same hardware
but it's not sharing a lot of code.

There is an efficiency monitoring scheme that,
if the conversion is not good enough anymore,

it will flag this as an OBD-II error

and you will see your
"check engine" light going on,

and then you go to the shop, and the shop
will diagnose your car and will fix this,

for example if your catalyst is broken.

Based on the test results we would have
expected this efficiency monitoring

to actually flag the inefficiencies.
But it didn't.

It turns out the main model
doesn't always work.

There are some operating conditions
where the main model is not sufficient,

it has certain bounds where it works,
and outside of these conditions —

for example if the engine is too hot or if
the exhaust mass is too large —

the model doesn't produce
meaningful results.

It may overdose the AdBlue,
and we don't want that.

There's an alternative model
which is much, much simpler,

and takes only a few sensory inputs,

and doesn't rely on as many variables
to be perfect.

It will still calculate an AdBlue dosage.

However, the main goal of this alternative
model is to make the exhaust processing work

in all situations without ever
overdosing the NH3.

They're calculating both of these models and
then they are selecting one of the models.

The output of the selection then controls
the AdBlue dosage,

the pump that injects the AdBlue
into the exhaust.

There's code that controls
which of the models to use.

There's also a statistics model that counts
how often each mode is selected.

Again, all of this model selection
depends on the data.

It's code that does the selection
but it depends on a lot of data,

there are parameters tought of this.

Let's take a look at the selection criteria
for this alternative model.

We see that a lot of these parameters
are dummy variables,

things that can never happen.

For example, the athmospheric pressure
can't be negative, that can never happen.

Or the air temperature ...
I hope it's never larger than that,

or smaller than 0.1K, right?

However, one thing stuck out,

and that was a check if the engine condition
is larger than negative temperature.

Which does not exist,
the temperature is always positive.

That last one is always true,

so the model that would be selected would
always be the alternative model.

That sounded weird and
I was looking at the firmware.

Maybe I understood it incorrectly,
or maybe I looked at the wrong place

when looking at these parameters?

But if we look at the intermediate results
there is a bit at a certain location

that tells us which model was selected,
and that bit is indeed always set.

That is weird, it sounds fishy.

Let's take a look at the statistics,
the car counts what model you're in.

20% of the cases
my car does not do dosing at all.

So I drove some time and then
looked at the values.

And the 20% where it doesn't do anything
is mostly the warm-up cycle.

But everytime it does something,
it's actually the alternative model

which we know does underdose NH3
because it doesn't want to leak ammonia.

And that makes sense because my car
uses much less than expected of the AdBlue.

The expected value is roughly 2.5 liters
per 1000 kilometers, of the AdBlue.

In my case it only used 0.6 liters
per 1000 kilometers.

Which is great for me because I don't have
to refill this tank very often.

In fact, I never had to do it,
the shop always does it when I'm there.

But this is fishy,
and let's take a look at this.

What we also see is that sometimes
the regular model is active,

so there must be something more.

If we look at the selection criteria we find
that there's an additional term there

that I haven't found before.

There's an additional condition
that has to be true

in order to go to the alternative model
that underdoses.

We look at the particular conditions
and we find a lot of stuff

that is related to diagnostics,
things they can do in the shop.

So that's definitely not
happening on the street.

But one of the criteria,
that really was weird

because it looks if the engine and fuel
temperature is larger than 50°C,

it looks at the athmospheric pressure
and if it's lower than 750m,

that must be satisfied.

If all of these conditions are satisfied
it will move back to the main model

that does the proper exhaust processing.
And one thing was really weird.

There were seven curves,
not all of them used,

that define an upper and a lower bound

on the distance driven
after a certain amount of time.

This is how it looks in disassembly.
I'm not sure if you can read this.

But the comments are from this A2L file
and they call it "acoustic function".

I'm not sure if this has anything
to do with acoustics.

I tried to find all the usages, and there
was nothing related to sound or anything.

I think it's just a name for it.

Now if we go and take a look at these
upper and lower bounds, we see this:

These are three curves that are defined,
each of them has an upper and a lower bound.

It's basically the distance

that you need to have driven
after a certain amount of time.

And if you ever fall out of one of these curves
we're switching back to the alternative model

that underdoses NH3
and causes the inefficiencies.

This is weird,
and I didn't really know what this is.

Let's get back to something
completely different, which is the NEDC.

We've seen this slide before,
the NEDC mandates you how to drive.

One thing is also interesting:
It mandates you that ...

You want this test at "cold-start",
and what's better for a cold start

than heating the car to 20°C
and keep it that warm until you start.

That's the "cold-start", that's the
cold start as defined in the law: 20°C.

This is speed over time, so to get
distance over time we need to integrate this.

And we get this graph.

And if we overlay what we found in the
firmware we get this.

audience laughs and applauds

What we can see here is that if you drive
the driving cycle correctly

you will exactly be in the bounds
of one of these curves.

And you can do this on the street,
you can do this everywhere.

As long as you satisfy the distance over
time and your car is warm enough

it will detect this in some way.

Well, you can drive this on a street,
but it's really dangerous

because you have to follow
a given speed pattern.

So i did this on a dyno,
I put my laptop in there,

I logged the data in real-time
and then displayed it.

Basically, this is what it looks like.
In the middle you see a bar.

You have to drive and keep this
middle bar in the middle,

which means you are well within this upper
and lower bound, and not try to escape it.

And as long as you do, one of the
other green boxes will tell you

that the car is still detecting this
as being in this cycle.

Then what I did in the end, I stayed in the
cycle for a while and I logged all the data.

At the end I would just hit
a constant speed

which would eventually get me
out of the conditions.

This is the log that I made.

On the first graph you see
the vehicle speed,

you see how I tried to follow the NEDC
more or less successfully.

On the second graph you see
the distance over time,

you see that I stay within the bounds
enforced by the firmware.

You an also see on the third graph—

this is the actual signal at the AdBlue pump—

that it actually doses
quite a lot of AdBlue.

It calculates the amount of AdBlue to dose
based on the model output

which you see in graph 5 and 6.

By the way, graph 4 is the actual NOx
emitted by the engine based on their model.

That's the RML, their mission model then
calculates the amout of the dosing to happen.

As we see, as long as we stay within the
limits enforced that match the NEDC

everthing is good
and a lot of AdBlue is dosed.

And then, in the end, I drove too fast.

And you can see in the second graph
that I crossed the upper bar,

the blue line goes
over the red line, right?

You can see that the car
immediately detects this,

that I'm no longer in the driving cycle.

The interesting part you see here is the
effect on the AdBlue dosing, which is here.

It immediately stops doing the dosing.
And you can see in the model below

the model still calculates that
AdBlue should be dosed.

But after they have the max,
after they switch the model

and switch to the alternative model,
the alternative model just outputs zeroes,

it doesn't dose anything.

This shows that when we're
following the cycle

everything is fine,
enough Urea is dosed,

and then once we leave the cycle,
there's a severe reduction in the dosing.

And it's all based on
detecting this driving cycle.

Two more slides.
A: Two more slides.

F: Two more slides.
A: Two more slides, here we go!

audience laughs and applauds

I have to be clear
on the limitations here.

All of this was looking at
disassembled code and so on,

I could have done something wrong here,
so take this with a grain of salt.

We couldn't do NOx measurements
on the dyno, unfortunately.

And I have to stress: We looked at one
particular car that uses SCR processing,

not all of the affected cars are doing this,

there are some other
mechanisms in the other cars.

And I looked at a car
for the German market,

at least the curves have to be different
for the other markets.

Let's reenumerate the results—
and this is my last slide.

Most of the time, on a regular car,
a nonstandard treatment mode is active

that is not as efficient
as the real mode that is implemented.

We can show the code
that is responsible for this:

This is this negative temperature limit
that they look at

which doesn't make any sense and
always selects the alternative mode.

And we can see, in the logs,
the state selection bit,

we can see the counters that count
that the alternative model is active.

We can see that there's an AdBlue
underdosing in this state

which causes the inefficient
NOx conversions,

that's what we've seen before when
people put the car on the dyno.

We know that the efficiency checks
are only enabled in the main mode

and the car does exceed the limits.

This shows how the alternate model is
selected where it doses too little AdBlue

and causes the inefficient conversion.

We can see that if we
follow the driving cylce,

the minimum temperature and
the distance over time,

we will see that it switches
to the main model

that should have been active
all the time.

We can show the code
that's responsible for that,

the driving cycle detection that uses
the upper bound and the lower bound.

We can extract the exact limits, overlay
the NEDC data and see that there's a match.

We can, if we do this actually on a dyno,
we can see how it switches the SCR state.

We can show the effect on the DEF dosing,
on the AdBlue dosing.

As you've seen on the slide before,
as soon as we switch out of the driving cycle

into the street mode,
the dosing will get close to zero.

Once you're back in the main model
all the efficiency checks are enabled,

for example to take better Urea.

So the efficiency checks are there,
but they are not active

because the car is forced to run
in the alternative model.

These results are all in line
with the Volkswagen press releases.

These are basically just the details
as extracted from the firmware

to show you the background.

Thank you.

audience applauds

A: Wow!
Thank you very much, Daniel and Felix.

audience applauds

I'm really sorry,
but we have to clear the stage.

There is not going to be time
for the Q&A session.

Do that down there. I'm sure that a few
people just come down,

grab you and ask questions.
Unfortunately, we can't do that.

I have to close it in exactly four seconds
over here because we have to go off the stream.

Thank you very much Felix,
thank you very much Daniel.

F: Thank you.

♪ postroll music ♪

subtitles created by c3subtitles.de
Join, and help us!