33c3 intro music Herald: Let me present Felix Domke with the Software Defined Emissions: A Hacker's Review of Dieselgate. applause Felix Domke: Yeah, hey everyone. Thank you for coming here. I saw there are a lot of interesting talks at the same time in the other rooms, so thank you for coming here and listening to me about software-defined emissions. "A Hacker's Review of Dieselgate" is the subtitle. I'm Felix Domke. I usually do embedded software, mainly security. I'm definitely not on cars and definitely not on things that have combustion thingies, so I only got dragged into car software last year when my own Volkswagen car was accused of cheating and I wanted to know what exactly was going on. I held the talk last year about some of the details of the Volkswagen Sharan defeat device. For the details you can take a look at that talk. This time I want to look more at the process of finding or analyzing car software. I want to look at whether this process scales to more cars. The first step when having a piece of software that does not always do what people think it does is, well obtain a firmware image, obtain a binary image of the firmware, and in the case of my car I knew it was a Bosch EDC17, which is a Bosch ECU that a lot of cars use, including my Volkswagen car. So, I didn't know anything about ECUs, dumping software and so on, so I asked Google "Hey, what do I need to do to dump an EDC 17," and Google had a lot of answers for this, but usually those were people that wanted to sell me some device. Those were chip tuners that built their own devices where you can plug in the ECU and then it extracts the image, usually by exploiting some bugs in the software. But I didn't really want to buy something and it takes like a lot of time until I get in my hands. I wanted to start. So I was looking to do this on my own. What these sites usually tell you without paying is how you wire up your given ECU for their device, so they tell you where to connect 12 volt, where to connect the CAN bus, which is the serial communication bus that the ECU uses to communicate with the rest of the car devices. Usually it's pretty easy, so... when analyzing ECU it makes a lot of sense to reproduce the scenario on your desk and not in your car, so in order to make an ECU boot all you need is ground, 12 volts, there's usually an ignition pin that you also have to supply 12 volt to it, and then it boots. So on my desk it looked something like this, and then once we have the setup we can boot the ECU, we can use Python to talk to the ECU, which is great, and then we can use socket CAN, which is the Linux CAN support that's really great, and we can even use MicroPython if we want to have a smaller device that we can put in a car. So we can talk with the ECU. Talking with the ECU in modern cars, there's a protocol called UDS. Basically, I simplified this slightly, you can ask the ECU "Hey, I want to read memory by address," you give it an address and you ask it to read four bytes in this case of that address, and then it returns it to you. So I thought "Hey, maybe I can use this to dump the software." In my case the device responds with a Security Access Denied, so I looked into what I need to do. You actually have to do a Security Access command. You send a command that's called Request Seed. You get back basically a 32 bit random number and then what you have to do is to, have to process this seed through a super-secret function and then return it in a response call. The question is how do we know this super secret function. There are multiple methods. We can look at the ECU software itself, if the algorithms in there, to verify it. We can reverse diagnostic software that uses this mechanism. So for example the Volkswagen software they use for car shops, or maybe someone else already reversed this and put it in their own tools which may be easier to get it from in terms of third-party diagnostic software. And in the case of my Bosch ECU, the super secret function was this. I basically had to add this number to it. The mechanism is called pin code. It's in... I mean it's not super secret. Anyway, once I know this... applause Yeah, thank you. I mean, it... once you do this, you send back the result and hey, then you can read... you can send the read command again and, hey you're getting back data, so this is great, right? We can read memory at runtime of the ECU, and we can even do this while the car is operating. However it turns out that for the Bosch ECU you can only dump specific regions. You can dump most of memory, some memory areas are excluded, but most of the interesting stuff you can read. But you can't read any code. You cannot read anything in flash. But we are hackers, of course, so we find a way. The CPU used in these ECUs is a Infineon TriCore CPU, and it's used, at least this particular one, was used in the ECU I cared about, and the security model for this chip is that you can always enter a specific bootloader mode and execute your own code, so you can strap a few lines they... the chip tuners tell you that, right, they tell you what's high and low, which pins you have to connect your ground and 3.3 volt, and then it enters this bootloader mode, you can upload some piece of code. However, you can't read the flash, because the flash is locked. When you start in bootloader mode, the flash is not readable until you write a specific password to a register. That was not so great, so I looked into what else I could do. The datasheet is very specific on how to operate this chip. For example there's this one flash supply pin. So, even though the flash is in the same package as the rest of the CPU, it has a dedicated supply pin and it tells you which parameters not to exceed to ensure correct operation, but I really don't want the correct operation which is in this case preventing me from dumping the flash. So, what can we do? We can violate the requirements. The requirements for 3.3 volts. Let's see what happens outside of that range. And turns out, down to a certain voltage level, roughly 1.6 volt, everything just works as normal. That doesn't help us. And below that voltage the device hangs in the bootloader, so that doesn't help us either. The interesting parts happened when you are at the very specific voltage level, and this is a little bit unscientific, because it's really just the voltage level I tried, and then most of the time the device comes up and flash is protected, and then the remaining times the device comes up and hangs in the bootloader. But one in 10 times something interesting happened. The device came up and the flash was not protected, so I could dump it out. applause So having the image now in my hands, I could start in actually reversing the defeat device and what I found was - I don't want to duplicate a lot of what I talked about in the last talk - I found a function called "acoustic function," or "Akustikfunktion" in German. It's a function that senses vehicle speed, the duration of the engine operation, and some other things, and then controls emission related functionality, or in short you can say that this is the test cycle detection that enables the defeat device. And I verified it to exist on my Sharan device by driving through the test cycle and logging data. And during the last year I verified that it's actually the same defeat device, more or less, that exists on a lot of other Volkswagen cars. All these Volkswagen defeat devices that we talked about for the Euro 5 cars, they use more or less the same acoustic function. Basically, to remind you, there are a few curves stored in the software that look like this. This is the NEDC. This is the test cycle you have to drive a car through. They exactly define how fast you have to drive for a given time in seconds, so it's speed over time. If we draw this as distance over time it looks like this. So this is the distance you got. You're not really moving the car, because you're doing this in a lab on a dynamometer, but what the car thinks it has moved to, and if we overlay this with the curves we found in a software there's a perfect match. So this is the way how they describe the test cycle. So this was for my Sharan. So I looked into, what do the other cars do, especially what do the cars in North America do, because they're not using the NEDC. And I found something interesting, or some someone sent an interesting document to me, that was this. It was an emission service action. It basically describes how there was a recall for some vehicles, that required a software update in the shop. So this is basically the document that informs the car shop what they have to do, and it had something very interesting in it. By the way, this was in December 2014, so this was way before the whole Dieselgate was public, but this was already while the EPA was already talking with Volkswagen, already demanding explanations. All that investigation was already proceeding. Volkswagen knew about this, that people figured out about the defeat device, and it had something very interesting in it, that said "in addition, the vehicle's engine management software had been improved to ensure the vehicle's tailpipe emissions are optimized and operating efficiently." That sounds really fishy to me, so I was curious, what exactly did they change in the software update? And luckily they tell you the old and the new software versions, and you can then go and look them up on a firmware DVD, that you can download on the Volkswagen website, and it turned out that it's an ECU software similar to the the Bosch ECU software I looked at before. So there's an acoustic function again there, and the curve stored there, they match the US test cycles. This is one of them. There are many more test cycles in the US, so there's another curve that matches this, and this is the curve stored in the software and this is the corresponding test cycle. And there are a lot of them. But I noticed something really interesting, and some of the curves... they were much wider open than the other ones. So for example this one... there's really a nonzero probability that if you just have your morning commute, through, I don't know, some streets or something, that you accidentally match this driving cycle every time you start driving in the morning, so the car would, every time you drive this, think it is in test cycle mode, and would operate with the optimized... in the optimized emission mode and apparently this caused problems and what I saw, what Volkswagen added in the software that was part of this recall, was this function. So, this is from it from a disassembly. In pseudocode, this is this. So they started looking at the steering wheel angle, and if they figured out that you move the steering wheel angle, then they ignored the curves, the more open curves. So the idea is "Yeah, if you move the steering wheel, you're definitely not in a test cycle, so at that point we do not try to operate in this emission optimized mode. And it's a little bit of speculation, but it matches up pretty well with with all the facts that I read, is that because those cars operated in the test cycle mode too often, that eventually caused the the particulate filters to clog, and their solution for Volkswagen, and again, this was while they were already investigated by the EPA, was to add the steering wheel angle detection. For more details, I worked with this with the NDR, and they produced a feature on that, so there are some more details. So this is Volkswagen, but there are more cars, and if we look at this... this is a meta-study based on something that the the Ministry of Transport... they tested a lot of diesel cars and what they found was this. This is actually a representation by the ICCT. So the orange line is the emission limit, and the bars have an upper and lower end, and the lower end is how much emissions the cars have. This is just for nitrogen oxides, for NOx emissions, what they had in the lab, when you're driving the test cycle. So and you can see all of these cars managed to stay under the orange line, so they get their certification, but when driving them on a real street, they produce the emissions corresponding to the upper end of that bar, which is for some cars significantly higher. It's off by a factor of 10 and more. When you're driving the car on a street. And this is interesting, because the cars, they can meet the emission goals. The question is why don't they always meet the emission goals? Why do they operate so differently in the test cycle than on the street? And I try to give you a partial answer. And let's look at how a car can optimize emissions. The first thing they do... so this is a very simplified diesel engine. So fresh air goes in, fuel goes out, and there's an exhaust pipe, right? And a lot of nitrogen oxide, a lot of NOx, goes out as well, and we don't want that. So we added an EGR valve, which is basically a valve that causes a part... a fraction of the outgoing air to recirculate again through the engine and burn again. And what this causes is that the flame temperature goes down, and if we look at the relationship, it's very simplified here, but with a lower flame temperature you get fewer NOx concentrations, so you improve emissions by lowering the flame temperature, however at the same time you're increasing the soot level, or the particulate matter, and there is this trade-off - if you do too much of EGR, too much of the exhaust gas recirculation, you're getting too much soot, and the other hand if you do it too few, you get too much NOx, so you can argue that the green area isn't really great, because there's no point where both of them are great. And here we see the result of a clogged EGR valve. If there's too much soot it will clog. EGR, as the conclusion, is the least cost solution. It doesn't really work at higher loads. It works at low loads, and it does not require exhaust... high exhaust temperatures, which is great, but excessive use of that clogs particulate filters, affects the combustion, the drivability goes down, and there are trade-offs with this. It's also not very useful for higher engine loads, for example when you're accelerating you have to disable EGR at high speeds. So a better method, that was added on top of this, is called "selective catalytic reduction". I am... so, basically the idea is you have an SCR catalyst in your exhaust pipe... there are more catalysts there, but let's talk about NOx, and in there, this happens. We can simplify this, somehow, and say if you put ammonia into this catalyst, the NOx is converted to nitrogen and water. And Nitrogen and water is great, it's harmless. It's already part of the air. The only issue is ammonia is this and this is not something you want the driver to refill in your car. So instead this solution is we can create ammonia in the car from using from something that's less dangerous, and we have the reaction there. We can simplify this again and say we take urea - Harnstoff auf Deutsch - and heat, and we create ammonia. Urea or urea solution is this. It's called AdBlue or DEF - diesel exhaust fluid - it's not dangerous. You can buy it, you can transport it... it's relatively cheap. The idea is, we have this reaction, it requires ammonia in the catalyst, and we put AdBlue into it, or urea, and using the heat that we have from the exhaust pipe, we create the ammonia that we need to reduce the nitrogen oxides back to nitrogen and water. There's a great property of this, that some of the ammonia that's produced in the catalyst stays there until it's used up, so there's some storage there. So the the requirement for creating ammonia is heat, and if you don't have heat, but for example because you just started up your engine... if there is still ammonia from the last usage in your catalyst you can still use that, and use that up, and by the time you have used it up, maybe the heat is enough to supply more AdBlue and then fill up that storage. The downside is you need a pump to dose the AdBlue, and you need lots of software to control this process. And you need a heater because the AdBlue freezes at some point, and it's an expensive solution, it adds roughly $500 to a car, which can be significant amount of money for a small car, and it requires a large AdBlue tank for long service intervals, so you don't have to refill it every few thousand kilometers or something. The great thing about SCR is that it's efficient at higher loads. There's a third method called LNT, Lean NOx Trap, it's cheaper than SCR for smaller engine, it doesn't require anything, however the bad thing is it requires frequent re-generation, which decreases fuel efficiency, so it's kind of a stop- gap solution. And it's not efficient for continuous high engine load, for example if you're driving on the German Autobahn at full speed, then LNT is not going to help you much. For the sake of this talk, let's keep in mind that EGR is exhaust gas recirculation, that's the thing that operates within the engine and then we have the SCR, the selective catalytic reduction, that uses AdBlue and is after the engine. We also saw that all these technologies have significant trade-offs for NOx compliance, so we can kind of see the motivation for a defeat device here, because it would be the solution to all of these trade-offs. You get no downsides during regular driving, because nobody can measure your emissions and while maintaining conformance because during a test cycle you have perfect emissions. That kind of explains why there are defeat devices. OK, let's get back to the bigger picture and see what other cars do. So this is an Opel car, it's a Zafira car, it's a Euro 6 car, it's a pretty modern car, it has an SCR catalyst. In theory it should have really great, low emissions, especially at higher speeds because that's where SCR is good at. But quite surprisingly it doesn't. If we look again at this report, we can see that this Zafira exceeds the limit by up to 12x compared to the Euro 6 limit. This is especially interesting because there's this Opel advertisement where they advertise their diesel technology applying to the Insignia and the Zafira and they say a lot of diesel fun without regrets, the new diesel generation of Opel achieves best emission values and gasoline levels. Yeah, after they got sued for this they had to change it slightly and they had to add this to the sentence. scattered laughter So, during this testing, they had this 12x emission limits. For example, one particular test was to drive the test cycle in the same way, but at a different temperature, at 10°C, and the car exceeded the values by a factor of 6 even though the car would be operated in the very same way. It was just that the ambient air temperature was 10°C instead of 25. So they asked Opel why this was the case during their investigation, and Opel responded, saying that the EGR and the SCR injection, they work to the full extent in the temperature range of 20 to 30°C. It's what they call "normal use". So our question was, is it really just the temperature window? So we got a car and investigated. The ECU in that car is a General Motors ECU. It's developed in- house, Opel is a GM daughter. It uses an automotive PowerPC, yay PowerPC! It uses somewhat obscure variable length instruction extension to PowerPC. So how do we start? Again, we need a firmware image. So let's ask the Internet, "How do I dump this ECU?" Luckily, someone in some chip tuning forum already uploaded their stock ECU, which means they're what they dumped from their ECU using some chip tuning tool. It's not the same ECU, but it's very similar and I hope they shared some code so I can analyze the software, maybe find a way to dump it. So the dump was made with this tool and the tool did not let me export the binary. And the tool is free, however to use any of the features in that software, you have to buy their expensive hardware, that then connects to the car, which I didn't want. This is how their software looks like. I loaded the image I found on the internet, and I couldn't save it or anything without having the device attached. However, I can just use a memory debugger and just dump it from the address space, and that gave me a first firmware image to start with. I threw it in a disassembler and I found the UDS function and, yeah, it implements read memory by address, so that's good. Most of the RAM was readable without a security challenge. That is good, so I didn't even need a security challenge to read RAM. However, the flash, it is readable, but only with the security challenge. So let's take a look at the security challenge. Maybe it's as simple as the Bosch one. So their way of doing this is, they store 16-bit input and output value in the firmware, and it's different for every device, and they don't store the algorithm to compute the output from the input, instead they just store the pair, and, well, it's just 16 bit, right, so let's brute force it. The issue is, you can only try every 15 seconds, so it's kind of lame. The question is, how do the GM tools, the original factory tools, get access to that? Luckily, chip tuners had reversed that and then obfuscated it into their own tools. But that can be de- obfuscated and eventually it's just a little bit amount of bit shifting and so that was easy to fix. And also the the GM repair manuals tell you how to wire up the ECU. They tell you where to put 12 volts, the CAN bus again, and ground, and the ignition pin, and with all of that in place, I can do the security challenge. I can now read all of flash memory and read the four megabytes of PowerPC code, which mostly consists of mathematical functions. There are no strings or anything, it's really hard to find what a function does, what... There are thousands of variables. It's really hard to find what they mean, right, so I need to know some entry points, some known data values, and then I could refer to... one thing I could find are real-world constants, for example there's the density of diesel fuel stored, which allows me to understand that this is something related to fuel, an amount of fuel. Or more useful are the OBD2 calls. So there are some standardized things you can ask an ECU, that's engine rpm, vehicle speed, and things like that, and I would find that table in the firmware and then I had a first start of things like RPM, speed, and so on. That was a good start. It's not much more than what you can see here. So there's a lot of stuff not included in these. The next thing I did was, I drove the car for a few weeks and I let a device attached that would constantly log all memory using the rebuy address thing, and every few minutes I would get one memory dump, basically. It's a few hundred kilobytes of RAM, and then I put this into my disassembly, and that allowed me to understand more of what individual variables do there. And I found some interesting things. So the first thing, one of the first things I found, was basically something that resembles this. So there was something that look at the ambient temperature, and this basically checks for range, right, and it did... this was for controlling the SCR systems, and it's interesting to know that the NEDC requires the temperature between 20 and 30°C, and this is right centered around this, when they check from 17.5 to 33°C. But this was of course nothing new. I found something similar, however, another temperature check. And as you can see it's written in a different way. It effectively achieves the same thing, but it's a separate piece of code, and this time it was for the EGR system. So we have these two exhaust treatment or optimization mechanisms there, EGR and SCR, and they don't share code. They have their own temperature window. So we found the temperature window, which was known to exist. The question was, is there more? And one thing we found was this: It's basically reading the vehicle speed and comparing to a fixed number, and it turned out it's something like this. So it would check the vehicle speed and if it's above 145 km/h it would set a flag and then under 140 it would clear. Keep in mind that the NEDC maximum speed 120 km/h, so during a test cycle that would never happen. So let's see if this, what we found in software, if this really translate something the car does in the real world, and it's getting slightly technical here, I apologize, but we need to log some variables and a useful value to to know is, how much NOx is there after the engine, and after the SCR catalyst, and luckily there are 2 NOx sensors in the car. One before and one after the catalyst, and they give you basically the NOx concentration in ppm. So we log that, and we also log the signal of how much AdBlue is dosed into the system, and we log the catalyst temperature. And one thing to keep in mind is that there's also this amount of ammonia that's stored in the catalyst. We don't have this as a value, but just keep this in mind. And this is how we've driven the car. The blue line is the vehicle speed. You can see that it goes from 0 to 150 km/h, and the critical point here is the 145 km/h that we found in the firmware. The green one is the catalyst temperature, which we see between ambient level, and then up to 380 degrees. The critical point here is 200 degrees Celsius, where this urea to ammonia process starts to work. We logged something that is the SCR strategy. So it turns out there are multiple ways how the ECU computes how much AdBlue to dose and I call them strategy. So 0 means off, no AdBlue is dosed. 1 means the regular way that keeps into account the storage mechanism, and then 2 is a special reduced way. And then also we log the actual dosing value. And then we also had the sensor data from between the engine and the catalyst, and between the catalyst and the exhaust. The first thing that happens... or, actually nothing happens until the point where we reach 200°C at the catalyst. You can... until that point, as I said the required temperature is not... does not allow AdBlue dosing, and then it starts dosing quite a few amount of AdBlue. But then, when we cross the 145 km/h, the SCR strategy changes, and no further AdBlue is dosed until basically this point, and this point is exactly 120 seconds after we go lower than 140 km/h. So this matches what we found in the software, right, this was what we found in the software. So we can see that this was actually true. This is real behavior of the ECU. And to look at the effect of this, you have to check the difference between the blue and the green line in the lower diagram, between basically the amount of NOx that is removed by the SCR catalyst. And you can see during the regular operation quite a lot of NOx is removed. The blue line goes up because we're driving faster and faster, and the green line goes down almost to zero, and this this works for quite a while, and it even keeps working a while until the catalyst runs out of ammonia, and then it would need more AdBlue to operate, but because we're in the reduced mode it does not put any more AdBlue into the system. So the SCR basically stops working, and the emission levels reach the engine emissions, so no further... The SCR system does not work in this red area. And here we see this again, so here are the sequences of active dosing. Here we see where the catalyst temperature is too low for dosing. We see the regular operation. We see where it's still working, because there's still ammonia stored and then until we run out of ammonia and no refill happens, until exactly 120 seconds after going below 140 kilometers an hour. So our conclusion after this is that the SCR is programmed to stop working at 145 km/h. The efficiency goes to 0. Opel offered a hand-waving explanation, one of the press releases why this is necessary. They argued with some physical details, and we presented these physical details to some experts, professors that work with combustion engines for a long time. Most of them disagree with these explanations, I mean, yeah. But more importantly, other cars including my Volkswagen Sharan device, which is Euro 5 car, so one generation older, and it's known to have a defeat device, and it performs significantly better than this car. OK, we continue to look. We found something here that is a... that looks like this. It takes a... there's a barometric pressure sensor that sends us the pressure of the air and compares it with a value and if we look at how pressure is related to height we see that what they check with is 91.5 kilopascal and which corresponds to 850 meter. And apparently Europe's highest test center's at 800 meter, which may be a coincidence or not. But above that point they reduce their CR system as well. Now, the interesting thing is... yeah barometric pressure is something very important to know for an ECU. There's a good reason to have the sensor for all of the combustion process. You need to know how much air there is. So for EGR it makes a lot of sense to have this, but for SCR, which is the system after the engine, no combustion is happening. We are not aware of any effect that the outside air pressure has on the SCR system, and also other SC equipped cars don't have this mechanism, so... for us it does not make physically... it doesn't seem to be physically required. So far we looked at SCR. Let's look at EGR. What we saw was, when we drove the car during the test cycle, so we put it in the lab and drive the test cycle, we consistently saw much higher EGR values, much higher than compared to driving on the street, compared to all kind of scenarios that we drove on the street. So higher EGR value here means, that the EGR valve, that I showed you earlier, is more open, more exhaust gas recirculates to the engine. It causes lower NOx emissions before the SCR catalyst, and we really... we're curious why did the car behave so differently when running on a street than running in a test cycle. And we already took into account temperature, so the temperature was not the issue anymore. And thankfully the car, when it computes the reason for reducing EGR, it stores a reason in some variable that we can log, and it looks like this. There is a number of things that can happen, that causes the ECU to switch to some low EGR mode, and a few of them make sense, for example if something is broken, fault flags are set, or if the, I don't know, the coolant temperature is out of range, it makes sense to just keep the device running at all cost. But when none of these reasons apply, the value stored is 2, and 2 basically means, that the full EGR operation is used, so it's basically the NOx-optimized mode with the fewest emissions. And then we looked at some real-world driving, you can see this in the background - the vehicle speed is in the background - and we saw that... the red graph shows you the reason to go to this limited EGR mode and what we saw is that most of the time the reason is 13 and only a few times it's 2, which means that it's not limited. And looking into this more details, we see it sometimes drops back to 2, to the unlimited mode, to the optimized... emission optimized mode, but any acceleration, or almost any acceleration switches it back to 13, and then it stays there for a long time. And 13, if we look it up, is what I call load limit. And then, interestingly, if we run it through the NEDC, we never saw a 13. So the engine stays in mode 2 all the time, and 16 just means that the engine is off. But we never see 13. So this explains why the EGR values were so different in a test cycle. So, let's look into this load limit function that we found. It's basically defined by curves, by five curves. For every gear there's a curve, or for a bucket of gears. It's basically that they look up RPM, they get a value for that curve, and if you exceed that value, they switch to the reduced EGR mode. What they compare this threshold with is the amount of fuel injected per cylinder per revolution, but you can also say this is torque, just with a constant factor. And then once you are outside of one of these curves, it switches to the non-optimized mode where it emits a lot more emissions, and then you have to go back into the green area to switch back to the optimized mode. So let's see what this means in practice. So here we have a car, and the traffic light is red, so the car stops, and then the traffic light goes green and the car accelerates, and accelerates, and accelerates, gets faster and faster, and then it's at the highest speed here, and drives for a while. And this is a typical city cycle, this is there to... how you drive in a city, and then the next traffic light turns red and the car brakes and stops in front of the traffic light. Let's take a look at this again with one more variable, the RPM. We can see that when the car starts moving, the RPM goes up. And then at some point there is a drop in RPM, and this is because it's a manual transmission and the driver switched to the next gear. Now it's switched to again the next gear, and this causes the RPM to drop, but the speed to remain almost constant, and it drives for a long time in the same gear, and then the traffic light goes red, the driver presses the clutch, the engine goes back to idle state, there is no connection anymore to the wheels, between the engines and the wheels, and the car gets slower. OK, one more variable. It's the last one, I promise. It is torque. The engine power in kilowatt or something is not just a function of RPM, it's a function of RPM and torque. so RPM and torque together are very useful to characterize engine behavior. And a very good way to do this is to have a graph where we put RPM on the one axis and torque we put on the other axis, and then we draw this in two dimensions, and so we get this, basically. This is the operating points we go through when driving the cycle we saw. So the green dot here indicates where we are. And so we restart the car, the car accelerates, sorry, the car idles for a while, so the green dot stays there. It idles at around 800 RPM, almost no torque, because there's nothing to move, and then the driver accelerates and the torque goes up, the RPM goes up more slowly, and then at some point, the driver presses the clutch, which disconnects the engine, the torque goes down, the RPM adjusts to the speed of the next gear, and then the driver releases the clutch and now the engine again has to move the car, so the torque goes up until reaching the the highest RPM value and then that the driver again switches to the next gear, so the whole thing repeats, and then while the car is driving, the majority of this the cycle, the engine spends in this one operating point. We're currently at 1800 RPM or something, and 80 Newton meter or so torque. And then at some point the driver presses the clutch, the engine goes back to idle and stays there, basically. So this is how you read this diagram. And now what we found in the firmware was that overlaid basically on this representation we see a mask, or a limit. If we go over this curve, those are the same curves that I showed you earlier, just laid on top of this. If we go over this curve, then we switch to the worse emission mode, we switch to the mode where the EGR value is limited. So we can see in our driving that this happens basically at this point, the point where the driver accelerates above a certain point, that causes it to go over the load limit and the engine basically switches or significantly reduces EGR. And that's fine because EGR doesn't work when you need a lot of engine power, so it make sense that that's at that point, and what we would think is that it switches back once we leave this load envelope, once we go below the limit again, once we are inside the limit, we would expect the ECU to switch back to the full EGR operation. But what we see instead is that this does not happen, and the reason is that you don't have to go under the maximum, the load limit, you have to go into this green area. You have to go back to idling at a very low RPM to switch back to the full EGR mode and this only happens at the very end. When the driving cycle is almost done, when the driver presses the clutch and lets the engine idle. So especially this long sequence where the driver... the car was driving at the same speed, we were technically in ... within the load limit, where we're not exceeding the load limit, but because we previously exceeded the load limit and it doesn't matter for how long you exceeded it, and we did not go to the green area before, we were still in this low EGR, high emission mode, even though we're still within the load limit imposed by the software. So let's take a look at how often this actually happens in real-world data. So here's us driving through a city, and we can see we constantly exceed these load limits. And this is driving on the Autobahn, and yeah we constantly exceed those. But they look interesting. They look as if they had been designed according to something, right, they have the specific form and it's not just... yeah, I... I don't know... and it turns out if you do something really strange, you can stay within these limits, so we tried that and we managed to stay within the limit by doing something, and we... it was reproducible, we could do this a lot of time it would always stay in this limit and the answer is: If you drive the test cycle you're staying in this limit. applause So yeah, these curves basically defined... they closely correlate to the limits that you need to pass the NEDC. Okay, to be clear it is fully acceptable that the EGR rate is reduced when... for higher engine loads. It's natural, you have to do this. For example, when you accelerate the EGR rate will decrease up to zero probably, when you do it ... when you're running at high speeds, all of that is great. So this method of having a load limit ... well, you can argue if really having the load limit exactly where the NEDC is makes sense, but having a load limit is okay, right? However, what we think is not okay is that, if you only exceeded the limit once ... um... you would stay in this high emissions mode for potentially a long time until you get back to low speed idle the next time. And we think that is the problem. We ... so far this was all based on what we saw in the software, so let's see if this translates to something that happens in reality. So to repro this we... the car... drive at constantly... or we let it idle, then we accelerate it to 2,000 RPM, we let it drive there for a while and then we quickly exceeded the load limit by going to 3,000 and then going back and then after doing that we would again stay at 2,000 RPM. So it looks like this and we would naturally expect the engine to operate in the same way on the left and on the right side because the engine is doing the same thing there, it's the same torque level, it's the same RPM, everything is the same. So we would expect the same emissions, right, um ... and it turns out it isn't. And ... this is a slightly convoluted diagram. So if you look at the green and red bars in the middle you can see what happens before and after exceeding the limit for just once. And in the middle you can see the EGR position, the EGR valve position, and you can see that we get pretty high values between... 6... maybe 65 percent or something before exceeding the load limit once. And after we exceeded it once even though the engine again is operating in the same exact operating point, we see much lower EGR valve positions, around 50% or something. And if we look at the bottom we see what the engine NOx emissions and we see that they are significantly higher on the right side than they are on the left side. So this... for me, this does not sound like this is truly optimized for emissions because the engine is doing the same thing, in both cases the emissions should be low. So going back to this quote that it works, the EGR and SCR injection work to the full extent in a temperature range of 20 to 30°C. Okay, but what about the EGR load limit and what about the the barometric pressure limit for SCR and what about the SCR speed limit? That would not be "to the full extent", right? And the Opel answer is really interesting. Of course, they denied doing a test cycle detection, they say they don't do that. And what they said is, when asked whether they lied to the KBA when saying that it works to the full extent they said "The statement 'fully' was really related to the NEDC test schedule, right, which... it went on and further... the Opel CEO had to say this. He said: "The recent accusations based on the findings of hacker Mr. Felix Domke" - hey, that's me - "are misleading oversimplifications and misinterpretations of the complicated interrelationships of a modern emission control system of a diesel engine. Emission control devices are highly sophisticated integrated systems which cannot be broken down into isolated parameters." Especially not by a hacker, right? applause That was kind of funny. There was another funny thing. Sorry, I only have a German quote and I didn't want to translate it, but when Opel basically ... they repeatedly say they don't have a cycle detection, right, and they say it's not a cycle detection because, if you use the car on the street in the same way as you would do them during the test cycle, the car would behave in the same way, so it's not...,right? applause ... and ... okay. But what is with Volkswagen, right, they have the same thing, if you drive the NEDC on the street the car will go to test mode they have the same thing. I don't see how this does not represent a cycle detection. That was a lot of things to say about Opel, but on the bright side, they also said that they will - even though all that was incorrect, what we found - they said "We will further improve the efficiency of emissions after treatment of our SCR diesel engines and so on as far as the laws of physics allow. This includes a voluntary service action" - and this basically means a software update for your car - "for the cars that are already on the road starting in June." So that is great. They're actually improving something. Question's in which year, because this statement is from May 2016 and it's not out yet, but... Opel actually provided a new software already in July and I think they already worked on this for quite a while and in July 16 the German KBA, the Kraftfahrtbundesamt, the Federal Motor Transport Authority, they are pretty nice actually, and they do know about what they do, they are bit limited by the resources they have, and by the manpower they have, but they know about cars and they know how to do these investigations. I mean, they're a little bit bound, but what they should do and what they should not do, but they asked me to review a new ECU software that was given to them by Opel for the Zafira in question and Insignia, which had a similar ECU and I looked at that software and I dumped the firmware and I looked at basically all the code sequences that I looked at before and I was positively surprised because they removed... they addressed each of our concerns. All of them, within the physical limitations of course. So they improved the temperature window and everything, so there was a significant improvement. They were able to improve the software and they let the DUH, which is the German Environmental Aid, they used a PEMS system - PEMS is a portable emissions measurement system. It's something you put on the exhaust pipe on your car and then you can measure the exhaust during real-world driving, and Opel gave them a car with the new ECU software. Otherwise the car was identical to the old software, and the results are this, right, so on the left side you see the old software, that has all these things that we criticized, and on the right side you see the same car with a new ECU software and it's significantly better. It's only slightly above the limit, right, but it's much better than before and to put this in relation, before they were on the list pretty bad - so this is sorted by worst to best - so they are in the, well, upper half at least, and now they are almost one of the best cars, just by switching the ECU software. And I mean this is great news, right, they actually improved their cars. Let's just hope they get this out to the cars soon. Let's just hope it doesn't have side effects and something, but I'm sure Opel knows how to test for this. Going back to these, we worked on the Opel thing... I think the Opel case, it.... once they actually upgrade the cars, and once the cars really show these great values that the preliminary software showed, I think we can close the Opel case, but there's a lot of other cars still to look at, and really, I mean... the effort to do this does not scale to so many cars, so we need to do something more fundamentally to improve the situation. What I found out is that digital control systems, they are black boxes. The manufacturers have designed them to be black boxes. They even boast to you that they are 7,000 parameter in there and no hacker can understand this and it's a very sophisticated problem. They are designed to be black box, and this is not just true for Opel, this is true for all car manufacturers. Nobody wants anyone to look into their ECUs, and people seem to be ok with that. Like they think "Oh this is so complicated, there are so many German engineers working on this problem, they must have found a great solution." So we are trusting these black boxes and we are not able to review the black boxes that we put into our cars and we have to trust the manufacturer to do the right thing and currently, the investigation to do this without assistance from the manufacturer, it does not scale. We can do it but... the manufacturers can put more security on their ECUs... it probably can be broken, but it takes a lot more time, so it simply does not scale sufficiently. The issue is black boxes are really powerful, right. Black boxes can hurt people with, for example, excessive emissions. They can kill people if we think think about autonomous cars that do mistakes. So what we do need, I think, is more transparency. A system that can kill people needs to be reviewable by the people. I think this is a very important thing. applause So, to have a system that can kill people... to have it reviewable by the people, we need to do things. For example, we need... we want access to source code for reviews. It doesn't necessarily mean we want open source, but we don't ask at all the car manufacturers to open source all the software. That's not what I'm talking about. What we need is... think about how Microsoft is sharing source code of Windows with universities or other countries. We need experts to look at the source code, and we want control software that is reviewable by design, that has a lot of documentation, that has good comments, that is human readable code. I don't want to see a disassembly, I want to see the source, the MATLAB, or whatever they are using to define the functionality source, and read that. And I want to understand why did they choose that curve of that map in this way? What was the design criteria? That needs to be reviewed. And we need transparency for control software decisions, which means that if a car operates in a certain way, if I'm driving that car, I want to choose that I can log what the car is doing, for example by putting in, I don't know, a USB stick or something if it's my car, and then the car will log all the data to that. That is... in the end that allows me to reconstruct any decision that the software does. I think this is required to have the necessary transparency, that allows us to un- blackbox these devices. All right. Thank you very much. applause Okay, I actually finished five minutes early. I didn't think this would happen, so... Herald: I'm so surprised. F: I am surprised too. Herald: You are on time. You have five minutes left F: Wow, what do I do with these five minutes Herald: We can walk around the stage or... Maybe people have some questions? F: I think so! Herald: Well, let's ask the Internet! Is the Internet ready? Signal Angel: Yes. Our first question: What dou you think is the responsibility of Bosch as a supplier for having their software and hardware used for this? F: So the question was: What's the responsibility for Bosch, who built the software for Volkswagen? It's a good question and I have to be careful in what I answer. My personal opinion, and let's take this aside from Volkswagen and Bosch, is that if you build software that you know is used to be illegally it should... it must be your responsibility to not do that. And I'm not sure if this is something that is legally enforceable, but it should be something that's enforceable ethically or for all of us programmers, that we don't build software that is designed to break the law. applause Herald: We quickly hop over to microphone 1 please. Microphone: Thank you for a wonderful talk. I'm just wondering if you're aware of some cases of Volkswagen cars in Australia, which was suffering from sudden and rapid power loss. This was happening about five years ago and there was a case where a Volkswagen suffered rapid power loss on a motorway. The driver was Mrs. Melissa Ryan and she was rear-ended by a truck and killed. So when you say that these things can cause death, were you... are you aware that any sort of Volkswagen software has been leading to power loss in the vehicles and affecting performance on the road, now I don't know whether Australian driving conditions are different to European driving conditions, and how that might affect that. Have you done any tests that might indicate that could be happening in normal driving? F: Yeah, so... the question was whether I'm aware of, I think an Australian incident, right, where... M1: Can I... F: Yeah. M1: There were many reported cases. One of them was fatal, but there were many reported cases of that happening. F: Of a sudden power loss, is that right? M1: Sudden and rapid power loss in the engine. F: Yeah, of the engine. I'm not aware of these incidents and I what I do know and... is that the the personal safety is the number one design criteria for ECUs. That does not mean that they are perfect, of course, that could mean that rare bugs... that there could be malfunctions. I don't know about this, but at least it's the first design principle to provide the safety for the people driving the car, which i think is a good thing, right. It's not the profit or anything, or at least we can hope so. I'm not aware of this particular incidence, and so I can't really say anything more about this. It would be great if... Are you aware of any additional details that were found in the investigation, please sent them to me. M1: Volkswagen was claiming that this was a gearbox problem on automatic cars, but then it started happening on manual cars as well, so that excuse went out of the window. F: The issue with the problems is that most of them are very complex, so they probably involve more than just the engine ECU, so they're very... but it's a good example of where we need to understand exactly what is happening, and where we may not want to rely on Volkswagen or any other manufacturer alone to assist in figuring out what happens. We need more transparency there so that we can have definitely neutral accident investigations. Herald: This was a long question and really detailed answer. Thank you very much. F: Sorry, I will be short Herald: Felix, that's your applause applause music subtitles created by c3subtitles.de in the year 2018. Join, and help us!