1 00:00:00,149 --> 00:00:14,099 prerol music 2 00:00:14,099 --> 00:00:23,599 Herald: So a very warm welcome to Thomas Roth. He is a security researcher and his 3 00:00:23,599 --> 00:00:28,980 specialty is exploiting techniques and reverse engineering and industrial 4 00:00:28,980 --> 00:00:37,590 security. And the talk today will be about out SCADA the gateway to shell. 5 00:00:37,590 --> 00:00:45,364 applause 6 00:00:45,364 --> 00:00:50,390 And just one little notice: this talk will be in English and will be translated 7 00:00:50,390 --> 00:00:53,580 in German as well. Thomas Roth: Thank you. 8 00:00:53,580 --> 00:00:55,090 Herald: Yes. 9 00:00:55,090 --> 00:00:59,290 Thomas Roth: Awesome, thank you. OK, yeah. Welcome to my talk gateway to shell. Who 10 00:00:59,290 --> 00:01:03,850 am I? He already introduced me, but still my name is Thomas Roth. I'm a security 11 00:01:03,850 --> 00:01:08,850 researcher. I do a lot of low level security, so a lot of ARM reverse 12 00:01:08,850 --> 00:01:13,321 engineering, Coldfire and so on. And yeah, you can find me on Twitter or if you 13 00:01:13,321 --> 00:01:20,730 want to write me an email. Feel free to send me one to thomas@stacksmashing.net. 14 00:01:20,730 --> 00:01:25,720 Before we start a short introduction to the background of this talk, so, this year 15 00:01:25,720 --> 00:01:30,830 I did some SCADA penetration tests and I found that while the PLC sensors 16 00:01:30,830 --> 00:01:35,210 are pretty well covered in the security research area, I found that all the small 17 00:01:35,210 --> 00:01:39,810 devices that surround SCADA environments are not really well covered. So basically 18 00:01:39,810 --> 00:01:44,060 we have the big Siemens PLCs and so on, and there's a lot of research going on 19 00:01:44,060 --> 00:01:48,760 about them. But there are also a ton of other small Ethernet devices involved in 20 00:01:48,760 --> 00:01:56,700 industrial networks that are not really researched very well yet. And all devices 21 00:01:56,700 --> 00:02:00,570 that we're going to talk about are running their latest respective firmware. 22 00:02:00,570 --> 00:02:07,310 Unfortunately, there will be zero days and these are not theoretical attacks. Like if 23 00:02:07,310 --> 00:02:12,489 you go to Shodan or similar search engine, you can find tens of thousands of these 24 00:02:12,489 --> 00:02:18,359 devices vulnerable and open in the Internet. So let me give you a quick 25 00:02:18,359 --> 00:02:24,779 introduction into the terminology in SCADA, because I know in the title I say 26 00:02:24,779 --> 00:02:29,079 SCADA, but actually it should be ICS, which stands for industrial control 27 00:02:29,079 --> 00:02:36,900 systems, because basically ICS describes the whole system from your supervision, 28 00:02:36,900 --> 00:02:42,069 the big room with all the big screens up to your PLCs the sensors, the actors and 29 00:02:42,069 --> 00:02:46,930 so on that you will find in your installation. And the term SCADA just 30 00:02:46,930 --> 00:02:50,959 describes the supervision and control centers. So the big screens that you might 31 00:02:50,959 --> 00:02:55,040 know from movies and so on, where when the bad guy comes, suddenly all the lights 32 00:02:55,040 --> 00:03:02,400 turn red. Then there's something called a PLC, which is programable logic 33 00:03:02,400 --> 00:03:06,889 controller. It's basically like an Arduino, just for industrial applications 34 00:03:06,889 --> 00:03:11,909 and they are really easy to program and you can get them from Siemens or Schneider 35 00:03:11,909 --> 00:03:17,610 and so on and so forth. Then there is something called an RTU, a remote terminal 36 00:03:17,610 --> 00:03:22,279 unit, which is a small device that generally are, well, back in the day, was 37 00:03:22,279 --> 00:03:27,029 only used for monitoring. But today you can actually program a lot of RTUs. So 38 00:03:27,029 --> 00:03:33,280 it's kind of a mix between a PLC and an RTU. So it's basically a PLC in a remote 39 00:03:33,280 --> 00:03:41,269 location. Alrighty, to the actual topic, industrial control gateways. So when you 40 00:03:41,269 --> 00:03:45,870 look at industrial control network, you'll find that there are a lot of different 41 00:03:45,870 --> 00:03:50,281 sensors and actors and a lot of them speak different protocols. So, for example, some 42 00:03:50,281 --> 00:03:56,470 might be serial, some might be IP, some might be Modbus and so on. And so you can 43 00:03:56,470 --> 00:04:01,459 buy these small gateways that connect all these different protocols to an IP 44 00:04:01,459 --> 00:04:06,539 network. So, for example, via Ethernet or even via GPRS or Wi-Fi and so on. And I've 45 00:04:06,539 --> 00:04:11,829 seen them in almost any industrial installation that I've seen. So, for 46 00:04:11,829 --> 00:04:16,440 example, they're used in power plants. They are used in water dam control 47 00:04:16,440 --> 00:04:22,880 systems. They are used to control the power grid and so on. And the security 48 00:04:22,880 --> 00:04:27,080 concept is, "Hey, but these devices are airgapped!", so it doesn't matter really 49 00:04:27,080 --> 00:04:31,599 if they are vulnerable or not fully up to date and so on, but that's not really true 50 00:04:31,599 --> 00:04:34,919 because a lot of these devices, while they might be airgapped, they also have 51 00:04:34,919 --> 00:04:42,650 antennas and they are interconnected by a ton of different wireless protocols such 52 00:04:42,650 --> 00:04:50,970 as Wi-Fi, LoRa or GSM or even proprietary radio links. So, yeah, and even the 53 00:04:50,970 --> 00:04:54,940 case studies show that basically in this case, you would have a monitoring network 54 00:04:54,940 --> 00:04:59,650 that's connected via the cellular network to control the water mains and so on and 55 00:04:59,650 --> 00:05:04,889 check the pressure. Or even worse, they even recommend that you connect the actors 56 00:05:04,889 --> 00:05:10,430 like valves and water level gotchas and so on over GPS, which we know is not a secure 57 00:05:10,430 --> 00:05:17,830 protocol to do anything that could be critical. Or you have stuff like 58 00:05:17,830 --> 00:05:24,160 water storage tanks that are controlled via Wi-Fi and so on or even public in the 59 00:05:24,160 --> 00:05:33,349 Internet. So, yeah, these devices are airgapped? Nope. So attacking in the field 60 00:05:33,349 --> 00:05:37,889 I already mentioned, if you go to Shodan, you will find a ton of different 61 00:05:37,889 --> 00:05:42,970 devices reachable via the Internet and even via GPS. So if you live 62 00:05:42,970 --> 00:05:49,090 close to, for example, a dam or something, it's kind of interesting to look at an SDR 63 00:05:49,090 --> 00:05:52,290 or similar radio equipment to see what's going over the airwaves, because you will 64 00:05:52,290 --> 00:05:59,090 find a ton of interesting stuff and sometimes, you can even very trivially get 65 00:05:59,090 --> 00:06:03,930 a physical access to the in field devices because they might just be in a white box 66 00:06:03,930 --> 00:06:07,569 somewhere hidden. And if you break into it, you can pull out the SIM card and it 67 00:06:07,569 --> 00:06:12,419 will put you directly into the SCADA network, if you're lucky. Don't do that, 68 00:06:12,419 --> 00:06:13,760 by the way. 69 00:06:13,760 --> 00:06:17,169 laughter 70 00:06:17,169 --> 00:06:24,539 So, yeah, let's let's hack some gateways. So the equipment you will need to and 71 00:06:24,539 --> 00:06:29,039 everything in this talk was done on this desk, just using these devices here, you 72 00:06:29,039 --> 00:06:33,000 really just need a laptop, you need an oscilloscope or similar measurement 73 00:06:33,000 --> 00:06:37,320 equipment just to ensure that you don't burn out your logic analyzer. You need a 74 00:06:37,320 --> 00:06:43,230 logic analyzer, a soldering iron, a multimeter and a power supply. And that's 75 00:06:43,230 --> 00:06:48,360 really basically it, because you can hack almost any embedded device that's using 76 00:06:48,360 --> 00:06:56,530 these devices and to find potential targets. I have this kind of map where 77 00:06:56,530 --> 00:07:02,139 first try to understand, can I get the firmware of the device or do I have to 78 00:07:02,139 --> 00:07:07,251 somehow, for example, use J-Tech to get it out of the device? Can I actually buy the 79 00:07:07,251 --> 00:07:12,460 devices at a sensible price? Because some of these devices cost like 600 € or so, 80 00:07:12,460 --> 00:07:18,360 and if you buy ten of them, that gets expensive very quickly. And so, uh, I need 81 00:07:18,360 --> 00:07:24,360 to check eBay and see what devices can I actually buy. And they should be half what 82 00:07:24,360 --> 00:07:29,449 current, because if you look at all the devices, like 10 years old or so, they are 83 00:07:29,449 --> 00:07:33,940 completely broken. You don't even have to look to start to look at their security. 84 00:07:33,940 --> 00:07:40,819 So, yeah, the first device that I that I choose to really look at was the moxa 85 00:07:40,819 --> 00:07:50,689 W2150A, which is this small device, which is also mounted on the board right here, 86 00:07:50,689 --> 00:07:54,319 mainly because I found the phone was available and it looked like an 87 00:07:54,319 --> 00:07:58,669 interesting device because it has Wi-Fi and so if I managed to break into it, I 88 00:07:58,669 --> 00:08:07,930 can jump an airgap potentially. And the W2150A is just a simple device server. So 89 00:08:07,930 --> 00:08:14,939 you can connect any serial device, any RS485 device simply to it and it will be 90 00:08:14,939 --> 00:08:20,669 exposed via Ethernet or even via Wi-Fi. And you can download the firmware publicly 91 00:08:20,669 --> 00:08:29,270 and it's available on eBay relatively cheap. So like 150 bucks or something. So 92 00:08:29,270 --> 00:08:33,290 I downloaded the firmware and I looked at the entropy of the firmware and 93 00:08:33,290 --> 00:08:37,090 I immediately saw that the entropy is very high, which means either it's very 94 00:08:37,090 --> 00:08:41,590 compressed or it's encrypted, unfortunately, using a tool called 95 00:08:41,590 --> 00:08:46,510 binwalk, which is really useful for looking into firmwares I saw that there's 96 00:08:46,510 --> 00:08:51,510 no compression detected. And so it was very likely that this firmware image is 97 00:08:51,510 --> 00:08:59,940 encrypted. But I noticed on the Web page that before you upgrade to version 2.0 or 98 00:08:59,940 --> 00:09:08,650 2.1 of the firmware, you must upgrade to the firmware version 1.11. And I thought, 99 00:09:08,650 --> 00:09:13,540 that's interesting. Let's look at the release notes for version 1.11. And it 100 00:09:13,540 --> 00:09:22,330 turns out that 1.11 adds the support for the encrypted firmware. So I downloaded 101 00:09:22,330 --> 00:09:28,093 the one point eleven firmware and sure enough, it's unencrypted. And if you've 102 00:09:28,093 --> 00:09:33,710 ever done anything with ARM before, if you just look into a firmware hex dump, you 103 00:09:33,710 --> 00:09:39,930 can immediately recognize whether it's ARM or not, because the first four bits of each 104 00:09:39,930 --> 00:09:45,580 instructions are the conditional bits and those are almost always E. So if 105 00:09:45,580 --> 00:09:50,320 you see a Hexdump and roughly every fourth byte is an E, you know, this is an ARM 106 00:09:50,320 --> 00:09:57,550 firmware and it's not encrypted or anything else. And so, yeah, sure enough, 107 00:09:57,550 --> 00:10:02,640 I ran binwalk on this image. This time we see there is a huge drop in entropy, which 108 00:10:02,640 --> 00:10:08,570 is the bootloader and so on, and then a high entropy, which is basically the all 109 00:10:08,570 --> 00:10:15,280 the compressed filesystems and so on. And binwalk was able to detect the SquashFS 110 00:10:15,280 --> 00:10:22,060 filesystem and extract it for me very, very easy. And so my goal was to extract 111 00:10:22,060 --> 00:10:27,250 the firmware, find the firmware upgrade code and somehow try to decipher the new 112 00:10:27,250 --> 00:10:34,250 firmware. And so I was browsing through the files and sure enough, found the file 113 00:10:34,250 --> 00:10:40,810 that was helpfully called libupgradeFirmware.so and if we look into 114 00:10:40,810 --> 00:10:45,010 the symbols, which they luckily didn't remove or anything, there is a beautiful 115 00:10:45,010 --> 00:10:48,066 symbol called firmware decrypt. 116 00:10:48,066 --> 00:10:51,150 laughter 117 00:10:51,150 --> 00:10:56,430 So we load the whole thing into disassembler and we see that 118 00:10:56,430 --> 00:11:03,870 there's some fancy XORing going on in the bottom left corner. And I'm 119 00:11:03,870 --> 00:11:08,000 going to walk you through what's, what exactly is happening in this code. 120 00:11:08,000 --> 00:11:13,310 So basically, first, there's a variable called password loaded into the registar 121 00:11:13,310 --> 00:11:21,790 R2 and then a second count variable is basically set and it starts looping and 122 00:11:21,790 --> 00:11:35,540 increasing always by four and goes through this whole xor shebang and it turns out 123 00:11:35,540 --> 00:11:41,200 that this is the obfuscation method for the AES Key. So, in password, in memory, 124 00:11:41,200 --> 00:11:45,950 we have an obfuscated key and we can be obfusciated by just implementing the code 125 00:11:45,950 --> 00:11:53,890 we see here in C or in the emulator. And sure enough, eventually this 126 00:11:53,890 --> 00:12:03,330 will be used as the key into the ECB 128 AES decryption. And so I implemented the 127 00:12:03,330 --> 00:12:08,760 whole thing in C, it was almost a copy paste from the decompiler, so you can in 128 00:12:08,760 --> 00:12:14,500 IAD Pro, you just hit F5, copy the C code at the bit, fix the memory offsets and so 129 00:12:14,500 --> 00:12:20,020 on. And you have the whole key obfuscation method basically reverse engineered almost 130 00:12:20,020 --> 00:12:25,630 automatically. And so I compile it. And sure enough, Moxa key extration, it turns 131 00:12:25,630 --> 00:12:31,200 out that the key is two eight eight seven Conn seven five six four. I build a short 132 00:12:31,200 --> 00:12:39,260 script to decrypt the 2.1 firmware and this time Binwalk finds all the files and 133 00:12:39,260 --> 00:12:41,740 we can start reverse engineering the actual firmware. 134 00:12:41,740 --> 00:12:48,519 applause 135 00:12:48,519 --> 00:12:54,180 The scripts for this are available on my github. I'll push the actual decrypts stuff 136 00:12:54,180 --> 00:12:59,810 after the talk because this is the first time this has been released. And so after 137 00:12:59,810 --> 00:13:03,470 I was at this point, I knew that the firmware is.. I can decrypted the firmware 138 00:13:03,470 --> 00:13:07,930 I can look into it. By the way, it's not signed or anything. The only verification 139 00:13:07,930 --> 00:13:14,480 method is CRC32. And so at this point I knew, OK, I can buy this device and 140 00:13:14,480 --> 00:13:19,980 start playing with it. And so I went to eBay, I bought one. I got it. I screwed it 141 00:13:19,980 --> 00:13:24,140 open. And sure enough, there's an ARM processor in there. It's an Freescale 142 00:13:24,140 --> 00:13:28,640 i.MX25, which is just a regular ARM processor. It's like 400 MHz or something, 143 00:13:28,640 --> 00:13:34,880 I don't know. And I started probing all the all the small pins inside of the 144 00:13:34,880 --> 00:13:43,040 device to try to find JTAG or serial or anything. And so I actually hooked up 145 00:13:43,040 --> 00:13:47,320 my power supply to foot pedal so that I can probe and just press with my foot to 146 00:13:47,320 --> 00:13:54,250 reset the device. And sure enough, I found that there's a full serial console 147 00:13:54,250 --> 00:14:00,660 available inside of the device on these pins. And if you boot the device, it even 148 00:14:00,660 --> 00:14:05,160 tells you, please press enter to activate this console, and so you do that and you 149 00:14:05,160 --> 00:14:07,463 are root on the device. 150 00:14:07,463 --> 00:14:14,820 applause 151 00:14:14,820 --> 00:14:18,660 So that's kind of cool, but that means that you require physical access, so 152 00:14:18,660 --> 00:14:23,530 that's not really a vulnerability, but it's very nice to have when doing security 153 00:14:23,530 --> 00:14:29,420 research because it means you can suddenly debug all the code on there. And so if you 154 00:14:29,420 --> 00:14:35,050 write an exploit, you can just touch GDB to the binary and start very, very simply, 155 00:14:35,050 --> 00:14:40,420 writing the exploit. So at this point, I was trying to look at the available 156 00:14:40,420 --> 00:14:46,010 services on the device. So for example, there is a web interface, there's a 157 00:14:46,010 --> 00:14:52,530 proprietary configuration protocol, there's telnet, there's snmp, there is a 158 00:14:52,530 --> 00:14:58,910 serial driver protocol and so on. And I started looking at the web interface and 159 00:14:58,910 --> 00:15:03,740 there was cross site scripting that was Cross site request forgery, there was 160 00:15:03,740 --> 00:15:07,440 insecure authentication where they basically hash on the client. So they have 161 00:15:07,440 --> 00:15:12,520 some JavaScript that hashes your password and then locks you in. Then there's a 162 00:15:12,520 --> 00:15:17,720 command injection which lets you execute code as root, there are stack overflows. 163 00:15:17,720 --> 00:15:23,675 And just a week ago there was a zero day released for the web server. So yeah, demo 164 00:15:23,675 --> 00:15:36,820 time. So just let me open up the Moxa Pitch right here. And so this one is 165 00:15:36,820 --> 00:15:41,240 authenticated, so I'll just enter the default password, which, by the way, in 166 00:15:41,240 --> 00:15:46,050 the field will 90 percent of the time these devices will be configured with 167 00:15:46,050 --> 00:15:54,950 default credentials. But still, so, if we just start browsing through this thing and 168 00:15:54,950 --> 00:16:00,140 go to the basic settings, we can start with a simple cross site scripting just in 169 00:16:00,140 --> 00:16:08,810 the device name. One sec, so just for example we just paste in some JavaScript. 170 00:16:08,810 --> 00:16:15,017 Submit the whole thing, and hello 34c3. 171 00:16:15,017 --> 00:16:19,560 applause 172 00:16:19,560 --> 00:16:23,770 I know what you're thinking, like cross site scripting, come on, that's not a 173 00:16:23,770 --> 00:16:28,530 vulnerability, that's just nothing. So let's look at the ping test that's 174 00:16:28,530 --> 00:16:33,910 integrated into this device. And funilly, a different device from Moxa that runs an 175 00:16:33,910 --> 00:16:39,570 entirely different firmware had the same vulnerability in the past. But if I just 176 00:16:39,570 --> 00:16:46,390 paste in my ping, so my IP address, a semicolon and then, for example, I cut 177 00:16:46,390 --> 00:16:51,970 /etc/passwd and activate enter. Here we go. 178 00:16:51,970 --> 00:17:00,060 applause 179 00:17:00,060 --> 00:17:08,199 Kind of funny, but, yes, for sure not intended. All righty, but I know what 180 00:17:08,199 --> 00:17:12,740 you're thinking, right, these are authenticated bugs in the web interface, 181 00:17:12,740 --> 00:17:17,460 so we need something unauthenticated. We want something that's like cool and a real 182 00:17:17,460 --> 00:17:23,420 exploit. Right? And so I decided to look at the.. this custom TCP protocol, which 183 00:17:23,420 --> 00:17:29,430 runs on Port 4900. And my goal was to reverse engineer the whole protocol and 184 00:17:29,430 --> 00:17:34,030 build a fuzzer for it, to find vulnerabilities, that turned out not to be 185 00:17:34,030 --> 00:17:40,990 necessary. So during some testing, I just sent a lot of bytes onto this thing and 186 00:17:40,990 --> 00:17:49,140 enabled crash debugging via the serial console. And sure enough, it crashed and 187 00:17:49,140 --> 00:17:58,740 put my program countdown right to 0x41414140. Wonderful. Thank you, Moxa. 188 00:17:58,740 --> 00:18:04,370 applause 189 00:18:04,370 --> 00:18:21,550 So, Demo time. So let's increase the size of this a bit. So I built a small script. 190 00:18:21,550 --> 00:18:34,490 Just called moxa_pown and I'll just supply the IP address to it. Let's see. Opening a 191 00:18:34,490 --> 00:18:43,620 second shell to connect to it via netcat. Here we go, we have a root shell on the 192 00:18:43,620 --> 00:18:44,600 device. 193 00:18:44,600 --> 00:18:54,263 applause 194 00:18:54,263 --> 00:19:01,540 So, yeah, that was the Moxa w21508, basically rolls of the tongue. And so the 195 00:19:01,540 --> 00:19:08,690 next device I decided to look at was the Advantech EKI-1522 which you can find 196 00:19:08,690 --> 00:19:17,460 right here. And it's, again, just a simple serial device server this time without 197 00:19:17,460 --> 00:19:21,410 Wi-Fi, even though they are available with Wi-Fi. It comes with two Ethernet ports 198 00:19:21,410 --> 00:19:26,060 two serial ports and so on. And I basically followed the same steps again. 199 00:19:26,060 --> 00:19:31,170 So I looked at the.. I downloaded the firmware. I looked at the edit using 200 00:19:31,170 --> 00:19:35,800 binwalk. And this time we see almost no entropy. So there is.. this guy is 201 00:19:35,800 --> 00:19:40,280 basically completely unencrypted. And again, we saw some ARM 32 bit it runs a 202 00:19:40,280 --> 00:19:51,010 Linux kernel, 2.6.31 and a BOA Web server where the last update was in 2005. And the 203 00:19:51,010 --> 00:19:56,770 firmware, I think, is from 2017. So these are kind of outdated. And I found 204 00:19:56,770 --> 00:20:01,230 during the initial analysis just of the firmware that the main binary to look at 205 00:20:01,230 --> 00:20:07,180 will be this edgserver binary. And so I loaded it into IDA pro and looked at the 206 00:20:07,180 --> 00:20:12,780 different things that calls. And there are a lot of calls to functions like 207 00:20:12,780 --> 00:20:18,340 string copy, to system, to sprintf and so on that are generally kind of considered 208 00:20:18,340 --> 00:20:25,660 unsecure. And sure enough, I am doing static analysis. I found that there's some 209 00:20:25,660 --> 00:20:33,630 code for sending an email as an alert, for example, when the system reboots. And 210 00:20:33,630 --> 00:20:39,250 the full command invocation is mailx -s blah blah blah, and we have control over 211 00:20:39,250 --> 00:20:46,160 some parts in the string because we can configure the two address in the UI. And if 212 00:20:46,160 --> 00:20:51,040 we look at what's happening here, it basically just sets up this 213 00:20:51,040 --> 00:20:56,500 format string. Then it goes to include the subject and then it gets some arguments 214 00:20:56,500 --> 00:21:04,260 from the stack and basically calls into system. And so there's no filtering 215 00:21:04,260 --> 00:21:09,930 going on at all. So we have an unfiltered part of the system, invocation, code 216 00:21:09,930 --> 00:21:15,380 execution. And this was before I had the device in my hand. And this is kind of a 217 00:21:15,380 --> 00:21:19,470 funny story because I first bought because it was just 40 bucks, I bought this 218 00:21:19,470 --> 00:21:24,770 device, which in the firmware has the same bug, but the mail functionality is broken, 219 00:21:24,770 --> 00:21:33,780 so I couldn't test it. So I had to go to eBay again, buy another one and buy the 220 00:21:33,780 --> 00:21:38,950 bigger one. And so I ordered the bigger one on eBay. Looks like this. It comes 221 00:21:38,950 --> 00:21:45,660 with a Cavium CNS C.P.U. It has JTAG exposed on the bottom there and serial 222 00:21:45,660 --> 00:21:50,940 console is available again without any authentication. So beautiful. You just 223 00:21:50,940 --> 00:21:57,809 connect your bus pirate or your UART adapter to it and you have full serial 224 00:21:57,809 --> 00:22:06,740 console. So, again, we had to look at finding vulnerabilities for this device 225 00:22:06,740 --> 00:22:11,559 and there, again, a ton of different services, there's like a Web interface 226 00:22:11,559 --> 00:22:15,670 available. There is a proprietary configuration protocol that's based on 227 00:22:15,670 --> 00:22:22,760 UDP. There is Telnet, there's snmp, there's a serial driver protocol and so 228 00:22:22,760 --> 00:22:28,380 on. And again, looked at the website and again, cross site scripting cross side 229 00:22:28,380 --> 00:22:33,280 request forgery, command injection, broken authentication, which basically if you log 230 00:22:33,280 --> 00:22:38,710 in from one computer, it uses, I think http digest authentication, you can 231 00:22:38,710 --> 00:22:42,690 connect from a completely different computer and it doesn't ask for a 232 00:22:42,690 --> 00:22:49,700 password. I don't know why that is, but.. Yeah. So I was thinking I was doing 233 00:22:49,700 --> 00:22:52,130 something wrong, but it turned out it was just broken. 234 00:22:52,130 --> 00:22:54,855 laughter 235 00:22:54,855 --> 00:23:03,170 So, yeah, and there's, again, a stack overflow in another protocol. So I guess, 236 00:23:03,170 --> 00:23:13,980 again, demo time. Let's first look at the device itself, so, you know the 237 00:23:13,980 --> 00:23:22,830 password, firstly, we have a nice device description here. This is just a basic web 238 00:23:22,830 --> 00:23:29,320 interface. Right. And we can, again, just copy in some basic JavaScript 239 00:23:29,320 --> 00:23:38,620 hit the save button. Reload and there we go, cross site scripting yet again, OK, 240 00:23:38,620 --> 00:23:49,129 again, not really interesting. Right. So, um, let's look at the stack overflow. 241 00:23:49,129 --> 00:24:04,070 Again, I have a small script advantech_pown. For the IP there. And we have netcat 242 00:24:04,070 --> 00:24:12,090 running on there. Sure enough, there we go, that's root on the Advantech device 243 00:24:12,090 --> 00:24:13,810 again, via stack overflow. 244 00:24:13,810 --> 00:24:25,516 applause 245 00:24:25,516 --> 00:24:31,500 Yeah, so two of three devices have basically broken already. Let's look 246 00:24:31,500 --> 00:24:38,150 at the next one. This one is a Lantronix EDS2100. And this one is kind of 247 00:24:38,150 --> 00:24:43,770 interesting because it's not ARM. I normally I almost exclusively do ARMs. So 248 00:24:43,770 --> 00:24:48,500 this one was kind of interesting. And this device, which is mounted somewhere right 249 00:24:48,500 --> 00:24:57,390 here. Yeah. This device comes with a serial to ethernet secure device server. 250 00:24:57,390 --> 00:25:01,929 It has two serial ports. It has Ethernet and you can buy it in two 251 00:25:01,929 --> 00:25:07,830 variants. One comes with Linux and one is Evolution OS, which is I guess, a 252 00:25:07,830 --> 00:25:14,880 proprietary operating system from Lantronics. And I'm using the EvolutionOS 253 00:25:14,880 --> 00:25:22,120 variant in this talk. Looking at the firmware it turns out it's unencrypted and 254 00:25:22,120 --> 00:25:28,240 it's coldfire architecture, which I've never done really anything with before, 255 00:25:28,240 --> 00:25:32,630 and there are no obvious external software components. So if you go through this, 256 00:25:32,630 --> 00:25:37,440 through the firmware, you'll find there's an SSH implementation, there's an SSL 257 00:25:37,440 --> 00:25:42,810 implementation, but it's not openSSL and it's not anything very well known. And the 258 00:25:42,810 --> 00:25:47,490 same is true for the web server and so on. It's not really anything that's well 259 00:25:47,490 --> 00:25:56,500 known. And this time, while probing the device, I did not really find anything 260 00:25:56,500 --> 00:26:01,580 interesting in terms of serial consoles or so, but it just found a potential debugger 261 00:26:01,580 --> 00:26:05,730 port, but it didn't have a fitting debugger unfortunately. The CPU is from 262 00:26:05,730 --> 00:26:14,760 NXP runs at 160MHz or something. Yeah. This time we actually have a web 263 00:26:14,760 --> 00:26:21,660 interface, we have Telnet SSL and it even has a file system, so you have like FTP 264 00:26:21,660 --> 00:26:26,210 and TFTP which allows you to download the configuration, upload the configuration 265 00:26:26,210 --> 00:26:30,980 and so on. And it's kind of hard to secure it correctly because there are so many 266 00:26:30,980 --> 00:26:37,000 protocols and it's not really clear what's set up by default. But yeah, you get 267 00:26:37,000 --> 00:26:44,350 the idea. And this time the web interface was surprisingly secure. So there was no 268 00:26:44,350 --> 00:26:50,230 cross site scripting. There was no command injection, because there's also not really 269 00:26:50,230 --> 00:26:55,440 a shell that you could execute commands into. But I still found some stuff. 270 00:26:55,440 --> 00:27:01,540 One is the configuration injection, which allows you basically to change the format 271 00:27:01,540 --> 00:27:06,630 of the configuration using a different field. And I found an authentication 272 00:27:06,630 --> 00:27:11,970 bypass, so I was able to write a small piece of code that takes a while and then 273 00:27:11,970 --> 00:27:23,650 completely removes the password from the device. Demo time. So if we connect to the 274 00:27:23,650 --> 00:27:29,750 Lantronics device, it will currently ask for a password, which in theory we don't 275 00:27:29,750 --> 00:27:44,580 have. Let's clean up here a bit. I know it's just. And let's run Lantronix_pown, 276 00:27:44,580 --> 00:27:51,300 oh, that was fast. That worked. Yeah, sure enough, the password is gone. 277 00:27:51,300 --> 00:27:59,980 applause 278 00:27:59,980 --> 00:28:07,230 Awesome. To be honest, I didn't expect the demos to go so smoothly, so I put in an 279 00:28:07,230 --> 00:28:13,710 hour for the talk for this went very well so far, so that's good. So before we 280 00:28:13,710 --> 00:28:22,170 finish already, some other devices are even worse. So, for example, as I 281 00:28:22,170 --> 00:28:26,549 mentioned, I bought some other devices, for example, this Advantaech device and 282 00:28:26,549 --> 00:28:31,610 this Moxa device and this Lantronix device, which are basically the 283 00:28:31,610 --> 00:28:38,940 predecessors of the other devices. And those guys are really interesting to look 284 00:28:38,940 --> 00:28:45,850 at, one could say. So, some of those are running eCos, which is an embedded Linux 285 00:28:45,850 --> 00:28:52,390 platform, which was last released in 2009, and some devices run a Linux kernel with 286 00:28:52,390 --> 00:28:57,570 the 2.4 version and you see Linux without any memory protection whatsoever. So even 287 00:28:57,570 --> 00:29:03,640 if they, so even a small stack overflow in one of the userspace applications gives 288 00:29:03,640 --> 00:29:08,640 you full root access to the device because you can directly exploit the kernel and 289 00:29:08,640 --> 00:29:12,840 there are unfixed public vulnerabilities. So in the first penetration test that I 290 00:29:12,840 --> 00:29:19,290 did, that included actually this device and Moxa and part of a small one. I found 291 00:29:19,290 --> 00:29:25,170 that using SNMPWwalk, it gives you back the administration password via SNMP. 292 00:29:25,170 --> 00:29:26,780 laughing 293 00:29:26,780 --> 00:29:31,500 And so I went online. I tried to report it. And it turns out it's well known 294 00:29:31,500 --> 00:29:34,160 there's a metasploit module for this 295 00:29:34,160 --> 00:29:36,830 laughing 296 00:29:36,830 --> 00:29:41,690 and it's unfixed, OK? And these devices are still in support. So I don't know why 297 00:29:41,690 --> 00:29:50,950 the vendor is not patching this. Yeah. So the summary with trivial vulnerabilities 298 00:29:50,950 --> 00:29:56,520 in most devices, or at least all that I've looked at, there are no security 299 00:29:56,520 --> 00:30:00,571 mitigations whatsoever. So they don't even enable like the compiler flags that you 300 00:30:00,571 --> 00:30:05,850 just set and then you have at least some kind of stack protection and some like 301 00:30:05,850 --> 00:30:11,070 stack cookies and whatnot. And some vendors are really bad at responding to 302 00:30:11,070 --> 00:30:18,429 vulnerability reports. So, yeah, I'm not going to name the vendor, but not even, on 303 00:30:18,429 --> 00:30:22,180 Twitter I asked them to please give me a security contact and they responded, 304 00:30:22,180 --> 00:30:26,840 please use our contact form. I said I did, three times. I send you emails, you're not 305 00:30:26,840 --> 00:30:30,600 responding to me. And so they stopped responding to me on Twitter too. 306 00:30:30,600 --> 00:30:40,809 laughing applause 307 00:30:40,809 --> 00:30:47,200 So how to mitigate? Well, the only way that I would see to mitigate against this, 308 00:30:47,200 --> 00:30:53,380 and I'm more on the deconstructive side of the story, is defense in depth. So never 309 00:30:53,380 --> 00:30:56,850 directly expose any of these devices to the Internet, even if they say they 310 00:30:56,850 --> 00:31:02,490 support VPN, even if they say they are a secure device of whatever, just don't do 311 00:31:02,490 --> 00:31:08,780 it. Get a real VPN gateway and make sure that you never rely on a single level of, 312 00:31:08,780 --> 00:31:16,169 for example, encryption. So, for example, WPA2 was broken by the crack attack and 313 00:31:16,169 --> 00:31:20,799 they actually released a patch for it after two months. And these are these are 314 00:31:20,799 --> 00:31:26,370 still two months where you are exposed to vulnerability on your potentially mission- 315 00:31:26,370 --> 00:31:33,760 critical system. Also never use GPRS for these devices without VPN because it just, 316 00:31:33,760 --> 00:31:41,480 it will go wrong. Okay. Yeah, thank you. I guess now we have time for Q&A. Thank you 317 00:31:41,480 --> 00:31:43,282 all for coming. 318 00:31:43,282 --> 00:31:49,170 applause 319 00:31:49,170 --> 00:31:57,990 Herald: Thank you very much for the talk. So we have very much time for Q&A. So 320 00:31:57,990 --> 00:32:03,980 please line up to the microphones and we have someone at microphone 4 already. 321 00:32:03,980 --> 00:32:09,220 Mic 4: Yes, hello. Hello. Thanks for your talk. This is.. obviously this is a 322 00:32:09,220 --> 00:32:14,792 problem. This is a part of the bigger problem of security in IT. Right. In 323 00:32:14,792 --> 00:32:18,950 anything related to any kind of technology. And this is only going to go 324 00:32:18,950 --> 00:32:25,230 worse with time, right. Internet of shit, internet of things and so and so on, so 325 00:32:25,230 --> 00:32:31,740 forth. So my question is, you gave some ideas how to mitigate this in this very 326 00:32:31,740 --> 00:32:36,540 specific area that use VPN, et cetera, et cetera. But my question is, so hacker 327 00:32:36,540 --> 00:32:42,462 community is not very, let's say, interested in regulation. Right? And when 328 00:32:42,462 --> 00:32:46,610 we see, when we see a government trying to do something with technology that usually 329 00:32:46,610 --> 00:32:51,580 goes bad, we have this idea in our head that, OK, this can only go like this can 330 00:32:51,580 --> 00:32:56,741 only go bad. Right. But so my question is: do you think that perhaps there is some 331 00:32:56,741 --> 00:33:00,810 space for regulation here? T: There's definitely space for 332 00:33:00,810 --> 00:33:07,450 regulation, but I think regulation does not solve the underlying technical issues. 333 00:33:07,450 --> 00:33:13,611 So these devices, it's 2017 and these devices are using C-code. I think that's 334 00:33:13,611 --> 00:33:18,580 just asking for trouble, basically. And so we really need to see this shift, even in 335 00:33:18,580 --> 00:33:22,690 the embedded world, to switch to memory safe languages, for example Rust or 336 00:33:22,690 --> 00:33:28,129 something similar, and really to stop using C in this kind of context. I don't 337 00:33:28,129 --> 00:33:35,729 think there's anyone who can .. Thank you. applause 338 00:33:35,729 --> 00:33:39,072 T: But there's definitely space for regulation. 339 00:33:39,072 --> 00:33:43,173 Herald: Since there was a question from the Internet. 340 00:33:43,173 --> 00:33:47,530 Signal Angel: OK, yeah, the Internet wants to know why you are not naming the bad 341 00:33:47,530 --> 00:33:51,980 vendor, because it looks like it's the only option basically if they don't 342 00:33:51,980 --> 00:33:57,990 respond to you. Let's say I asked them on Twitter and my Twitter is right there. And 343 00:33:57,990 --> 00:34:02,640 if you click on Tweets and Replies.. laughter 344 00:34:02,640 --> 00:34:05,850 Signal Angel: Yeah, somebody just posted the link on IRC. 345 00:34:05,850 --> 00:34:10,869 laughter T: I did not name them, just for the 346 00:34:10,869 --> 00:34:13,309 record. laughter 347 00:34:13,309 --> 00:34:17,029 applause Herald: So we have a question from 348 00:34:17,029 --> 00:34:23,369 microphone number 2. Mic 2: So you shown an exploit for the 349 00:34:23,369 --> 00:34:29,559 last device that disabled authentication. What did you use to achieve that? 350 00:34:29,559 --> 00:34:35,529 T: So this one is unpatched and not yet fixed, so I would rather not disclose the 351 00:34:35,529 --> 00:34:38,720 details yet. Mic 2: OK. 352 00:34:38,720 --> 00:34:42,919 Herald: Microphone number 1, please. Mic 1: I wonder if you've also been 353 00:34:42,919 --> 00:34:47,729 looking at a building automation system, control systems, or just industrial 354 00:34:47,729 --> 00:34:53,510 automation control systems? T: So you can use these devices basically 355 00:34:53,510 --> 00:35:00,609 wherever you want. And I think some of the Moxa ones are used in home automation. But 356 00:35:00,609 --> 00:35:05,920 I've looked at I guess Crestron, it's called? But not in a lot of detail. So I'm 357 00:35:05,920 --> 00:35:09,509 more on the industrial side at the moment. Mic 1: Thanks. 358 00:35:09,509 --> 00:35:15,079 Herald: Microphone number 3. Mic 3: Any field experience or even just 359 00:35:15,079 --> 00:35:21,259 opinions on using industrial strength Raspberry Pi hardware with community 360 00:35:21,259 --> 00:35:25,559 supported Linux distributions or something like OpenBC whatever on them. 361 00:35:25,559 --> 00:35:30,869 T: Yeah. So I guess the big trouble there is support, right? There are some, some 362 00:35:30,869 --> 00:35:34,579 German companies and so on that provide support for industrial Raspberry Pis and 363 00:35:34,579 --> 00:35:40,789 even like nice casing and so on. But I'm not sure if really Raspberry Pi is the way 364 00:35:40,789 --> 00:35:45,240 to go here. I think there are boards that are.. the problem is not the 365 00:35:45,240 --> 00:35:49,720 underlying stack, right? It's not the hardware. Really, that's the issue. It's 366 00:35:49,720 --> 00:35:55,950 the software. And you will have the same issues on on the Raspberry Pi. So, yeah, I 367 00:35:55,950 --> 00:36:00,880 guess you could buy these devices, which are like industrial grade shockproof and 368 00:36:00,880 --> 00:36:07,460 whatnot, and put some Linux on it and do it better. But I don't think that 369 00:36:07,460 --> 00:36:11,650 the hardware or platform will change anything at the moment. 370 00:36:11,650 --> 00:36:16,319 Herald: There is another question from microphone number 4. 371 00:36:16,319 --> 00:36:21,749 Mic 4: Hi, more a social question, did you get in contact with any development team, 372 00:36:21,749 --> 00:36:25,849 software development team in any of these companies, or might it be that there is no 373 00:36:25,849 --> 00:36:33,080 one behind the emails and everything? T: So I guess some of these companies are 374 00:36:33,080 --> 00:36:37,349 really so big, that they don't reply to you if you don't have a support contract 375 00:36:37,349 --> 00:36:45,049 with them. But, for example, the support of the ones that are not on my Twitter is 376 00:36:45,049 --> 00:36:49,730 kind of decent when it comes to two security reports. And so my next steps 377 00:36:49,730 --> 00:36:57,220 will be to go via the ICS Cert, but, you know, to report them. So, yes, there are 378 00:36:57,220 --> 00:37:03,737 development teams that will get in contact with you, just not from all vendors. 379 00:37:03,737 --> 00:37:06,670 Herald: Thank you. We have another question from the Internet. 380 00:37:06,670 --> 00:37:13,960 Signal Angel: Hello? OK. The Internet wants to know what to do about, because 381 00:37:13,960 --> 00:37:18,259 there are a lot of old devices in the field, how do you propose a vendor should 382 00:37:18,259 --> 00:37:24,200 deal with legacy devices and updates? T: Yeah, so keeping legacy devices 383 00:37:24,200 --> 00:37:29,680 supported is very expensive because, for example, if you buy a Qualcomm chip, they 384 00:37:29,680 --> 00:37:35,089 will eventually drop support for the Linux kernel for it and so on. But if you buy 385 00:37:35,089 --> 00:37:39,619 like a Freescale automotive chip, they guarantee you a certain time of support. 386 00:37:39,619 --> 00:37:43,490 But then you actually have to invest the money to regularly provide the updates and 387 00:37:43,490 --> 00:37:48,859 ensure that your devices are secure. The problem is that the lifetime of industrial 388 00:37:48,859 --> 00:37:55,470 installations currently is much larger than the lifetime of this processors' supports 389 00:37:55,470 --> 00:38:00,819 and so on. So I guess we'll have to get used to upgrading our hardware regularly 390 00:38:00,819 --> 00:38:07,400 or switch to, or figure out a different way of deploying secure software onto 391 00:38:07,400 --> 00:38:11,259 them. But I really think the underlying problem is, that we are still using 392 00:38:11,259 --> 00:38:16,229 memory unsafe languages. And I guess the fact that there's cross site scripting 393 00:38:16,229 --> 00:38:20,150 just shows that there's no security awareness really at those vendors 394 00:38:20,150 --> 00:38:29,395 whatsoever. At some of the vendors. Herald: So, microphone number 2, please. 395 00:38:29,395 --> 00:38:34,349 Mic 2: I was wondering, you mentioned that some of these facilities use GPRS. 396 00:38:34,349 --> 00:38:36,390 T: Yeah. Mic 2: Do you know if they have mostly 397 00:38:36,390 --> 00:38:40,749 their own closed infrastructure, or if they're using general consumer telecom 398 00:38:40,749 --> 00:38:44,849 stuff? T: So they will use commercial 399 00:38:44,849 --> 00:38:50,479 networks mostly, and then they have custom EPNs which have an IPSec tunnel or 400 00:38:50,479 --> 00:38:55,700 something similar to their premises. But there's also there's also a company that 401 00:38:55,700 --> 00:39:02,589 sells industrial control SIM cards which give you a public IP and you don't 402 00:39:02,589 --> 00:39:08,100 want to search on Shodan for that vendor. Mic 2: Yeah. Thank you. 403 00:39:08,100 --> 00:39:11,050 Herald: There is a question from microphone number 3. 404 00:39:11,050 --> 00:39:14,999 Mic 3: Hi there, isn't economics meant to solve some of these problems? We're not 405 00:39:14,999 --> 00:39:20,359 talking about dirt cheap devices. How surely at 300 bucks you should better have 406 00:39:20,359 --> 00:39:24,539 someone who's read security one and one. How long before a large organization gets 407 00:39:24,539 --> 00:39:28,200 the result of their security audit and goes to the aforementioned vendors and 408 00:39:28,200 --> 00:39:32,960 says, provide us something that's not trivially hackable, otherwise we stop 409 00:39:32,960 --> 00:39:37,839 buying your rubbish? T: Well, I mean, it's the same in all of 410 00:39:37,839 --> 00:39:45,329 IT, right? So everything has vulnerabilities. And yes, there should be 411 00:39:45,329 --> 00:39:50,400 market pressure. But that's why I'm trying to raise awareness for the issues that 412 00:39:50,400 --> 00:39:53,270 these devices have. Mic 3: Thanks. 413 00:39:53,270 --> 00:39:55,729 Herald: There's another question from the Internet. 414 00:39:55,729 --> 00:40:01,339 Signal Angel: Yep. The Internet wants to know how and if it's a good idea to raise 415 00:40:01,339 --> 00:40:06,549 the level of awareness in public, because they think it's a good approach to make 416 00:40:06,549 --> 00:40:11,869 people, the public know that, well, infrastructure in the cities is at risk. 417 00:40:11,869 --> 00:40:16,000 T: Uh, sorry. Could you repeat the first part of the question? 418 00:40:16,000 --> 00:40:21,339 Signal Angel: Yeah. They want to know how to raise awareness for this in the public? 419 00:40:21,339 --> 00:40:27,789 T: Good question. I guess we need some news articles or something about this in 420 00:40:27,789 --> 00:40:32,800 regular paper, but I personally think it's just an accident waiting to happen. So 421 00:40:32,800 --> 00:40:37,999 eventually someone will turn off the lights in a city or wherever, will open a 422 00:40:37,999 --> 00:40:44,773 flood valve or something. And that's when the awareness will start. 423 00:40:44,773 --> 00:40:47,813 Herald: There's another question from microphone number 4. 424 00:40:47,813 --> 00:40:51,680 Mic 4: OK, for what kind of industrial processes are these devices you just 425 00:40:51,680 --> 00:40:57,108 demoed used? T: So I've seen them in power utility. I 426 00:40:57,108 --> 00:41:02,350 know they're used in water dam control systems. They are used and in 427 00:41:02,350 --> 00:41:07,039 serial connecting a CNC machine to the network, they are used in connecting all 428 00:41:07,039 --> 00:41:10,690 kinds of stuff. Because if you have a big plant, you have a ton of different 429 00:41:10,690 --> 00:41:15,719 sensors. So you might, you might need the water level sensor. And for whatever 430 00:41:15,719 --> 00:41:20,680 reason, you only can get it with a modbus and then you need to convert the modbus to 431 00:41:20,680 --> 00:41:25,119 TCP and then you need one of these gateways. And so, I've seen in one 432 00:41:25,119 --> 00:41:28,529 cabinet, 20 of them. So they're really used a lot I guess. 433 00:41:28,529 --> 00:41:31,869 Mic 4: OK, thank you. I just retweeted your tweet to Star Alliance. 434 00:41:31,869 --> 00:41:37,979 T: Huh. laughs Thank you. laughs Herald: So there's another question from 435 00:41:37,979 --> 00:41:41,260 the Internet. Signal Angel: Yeah, the Internet wants to 436 00:41:41,260 --> 00:41:50,749 know if you did any research on MQTT for example from like Beckhoff uses? 437 00:41:50,749 --> 00:41:54,489 T: I actually talked to someone who recommended me to look at Beckhoff 438 00:41:54,489 --> 00:41:58,249 yesterday, but I've not looked at them whatsoever yet. 439 00:41:58,249 --> 00:42:01,900 Herald: And there's another question from microphone 3. 440 00:42:01,900 --> 00:42:07,450 Mic 3: OK, could you show the Moxa web panel, because I would like to double 441 00:42:07,450 --> 00:42:16,619 check, which proves that they and they would like you to see their Web page. And 442 00:42:16,619 --> 00:42:24,050 I think this browser isn't very secure. T: OK, let's take a look. 443 00:42:24,050 --> 00:42:29,160 Mic 3: Yeah, and under gohead the webserver small print. 444 00:42:29,160 --> 00:42:41,527 laughter Herald: Nice finding. 445 00:42:41,527 --> 00:42:47,859 T: That's probably the issue here. laughs 446 00:42:47,859 --> 00:42:55,658 Herald: Are there any more questions? Any questions from the Internet? 447 00:42:55,658 --> 00:43:02,009 Signal Angel: The internet wants to know how a memory safe language would prevent 448 00:43:02,009 --> 00:43:08,750 the authentication bypasses you showed? T: Not one would not be protected against 449 00:43:08,750 --> 00:43:13,130 but it protects against a ton of other stuff. It's just one example of where the 450 00:43:13,130 --> 00:43:18,420 industry needs to change. We need to stop using memory unsafe languages. We need to 451 00:43:18,420 --> 00:43:23,910 start really thinking about security design from the start, and we must not in 452 00:43:23,910 --> 00:43:28,319 2017, there's no excuse for having cross site scripting or anything on the web 453 00:43:28,319 --> 00:43:35,720 page. That's also if we in the Lantronics website, if you click logout, 454 00:43:35,720 --> 00:43:39,479 it tells you logout is not supported in your browser. 455 00:43:39,479 --> 00:43:43,290 laughter T: Probably because I'm not using Internet 456 00:43:43,290 --> 00:43:48,130 Explorer five. Herald: So there's another question from 457 00:43:48,130 --> 00:43:53,239 microphone number 3. Mic 3: Any remote part of the exploit 458 00:43:53,239 --> 00:43:57,750 where you did a buffer overflow - I think. 459 00:43:57,750 --> 00:44:01,490 T: Yeah? Mic 3: What I'm wondering is, are 460 00:44:01,490 --> 00:44:07,180 there.. isn't it like very standard to have ALSR on these devices? 461 00:44:07,180 --> 00:44:10,239 T: No! laughts It should be, but it isn't. 462 00:44:10,239 --> 00:44:16,199 Mic 3: Okay. Thank you though. That was pretty much my question. 463 00:44:16,199 --> 00:44:23,428 Herald: Is there another question from the Internet? It doesn't seem like it? 464 00:44:23,428 --> 00:44:36,052 Signal Angel: So, one just came in, OK, if you want to hear it. Ok, nope. 465 00:44:36,052 --> 00:44:41,329 laughter Herald: So, all right, give a very warm 466 00:44:41,329 --> 00:44:43,329 applause to Thomas Roth again! 467 00:44:43,329 --> 00:44:46,779 applause 468 00:44:46,779 --> 00:44:59,882 postroll music 469 00:44:59,882 --> 00:45:08,000 Subtitles created by c3subtitles.de in the year 2021. Join, and help us!