0:00:00.149,0:00:14.099
prerol music
0:00:14.099,0:00:23.599
Herald: So a very warm welcome to Thomas[br]Roth. He is a security researcher and his
0:00:23.599,0:00:28.980
specialty is exploiting techniques and[br]reverse engineering and industrial
0:00:28.980,0:00:37.590
security. And the talk today will be[br]about out SCADA the gateway to shell.
0:00:37.590,0:00:45.364
applause
0:00:45.364,0:00:50.390
And just one little notice: this talk[br]will be in English and will be translated
0:00:50.390,0:00:53.580
in German as well.[br]Thomas Roth: Thank you.
0:00:53.580,0:00:55.090
Herald: Yes.
0:00:55.090,0:00:59.290
Thomas Roth: Awesome, thank you. OK, yeah.[br]Welcome to my talk gateway to shell. Who
0:00:59.290,0:01:03.850
am I? He already introduced me, but still[br]my name is Thomas Roth. I'm a security
0:01:03.850,0:01:08.850
researcher. I do a lot of low level[br]security, so a lot of ARM reverse
0:01:08.850,0:01:13.321
engineering, Coldfire and so on. And[br]yeah, you can find me on Twitter or if you
0:01:13.321,0:01:20.730
want to write me an email. Feel free to[br]send me one to thomas@stacksmashing.net.
0:01:20.730,0:01:25.720
Before we start a short introduction to[br]the background of this talk, so, this year
0:01:25.720,0:01:30.830
I did some SCADA penetration tests and I[br]found that while the PLC sensors
0:01:30.830,0:01:35.210
are pretty well covered in the security[br]research area, I found that all the small
0:01:35.210,0:01:39.810
devices that surround SCADA environments[br]are not really well covered. So basically
0:01:39.810,0:01:44.060
we have the big Siemens PLCs and so on,[br]and there's a lot of research going on
0:01:44.060,0:01:48.760
about them. But there are also a ton of[br]other small Ethernet devices involved in
0:01:48.760,0:01:56.700
industrial networks that are not really[br]researched very well yet. And all devices
0:01:56.700,0:02:00.570
that we're going to talk about are running[br]their latest respective firmware.
0:02:00.570,0:02:07.310
Unfortunately, there will be zero days and[br]these are not theoretical attacks. Like if
0:02:07.310,0:02:12.489
you go to Shodan or similar search engine,[br]you can find tens of thousands of these
0:02:12.489,0:02:18.359
devices vulnerable and open in the[br]Internet. So let me give you a quick
0:02:18.359,0:02:24.779
introduction into the terminology in[br]SCADA, because I know in the title I say
0:02:24.779,0:02:29.079
SCADA, but actually it should be ICS,[br]which stands for industrial control
0:02:29.079,0:02:36.900
systems, because basically ICS describes[br]the whole system from your supervision,
0:02:36.900,0:02:42.069
the big room with all the big screens up[br]to your PLCs the sensors, the actors and
0:02:42.069,0:02:46.930
so on that you will find in your[br]installation. And the term SCADA just
0:02:46.930,0:02:50.959
describes the supervision and control[br]centers. So the big screens that you might
0:02:50.959,0:02:55.040
know from movies and so on, where when the[br]bad guy comes, suddenly all the lights
0:02:55.040,0:03:02.400
turn red. Then there's something called a[br]PLC, which is programable logic
0:03:02.400,0:03:06.889
controller. It's basically like an[br]Arduino, just for industrial applications
0:03:06.889,0:03:11.909
and they are really easy to program and[br]you can get them from Siemens or Schneider
0:03:11.909,0:03:17.610
and so on and so forth. Then there is[br]something called an RTU, a remote terminal
0:03:17.610,0:03:22.279
unit, which is a small device that[br]generally are, well, back in the day, was
0:03:22.279,0:03:27.029
only used for monitoring. But today you[br]can actually program a lot of RTUs. So
0:03:27.029,0:03:33.280
it's kind of a mix between a PLC and an[br]RTU. So it's basically a PLC in a remote
0:03:33.280,0:03:41.269
location. Alrighty, to the actual topic,[br]industrial control gateways. So when you
0:03:41.269,0:03:45.870
look at industrial control network, you'll[br]find that there are a lot of different
0:03:45.870,0:03:50.281
sensors and actors and a lot of them speak[br]different protocols. So, for example, some
0:03:50.281,0:03:56.470
might be serial, some might be IP, some[br]might be Modbus and so on. And so you can
0:03:56.470,0:04:01.459
buy these small gateways that connect all[br]these different protocols to an IP
0:04:01.459,0:04:06.539
network. So, for example, via Ethernet or[br]even via GPRS or Wi-Fi and so on. And I've
0:04:06.539,0:04:11.829
seen them in almost any industrial[br]installation that I've seen. So, for
0:04:11.829,0:04:16.440
example, they're used in power plants.[br]They are used in water dam control
0:04:16.440,0:04:22.880
systems. They are used to control the[br]power grid and so on. And the security
0:04:22.880,0:04:27.080
concept is, "Hey, but these devices are[br]airgapped!", so it doesn't matter really
0:04:27.080,0:04:31.599
if they are vulnerable or not fully up to[br]date and so on, but that's not really true
0:04:31.599,0:04:34.919
because a lot of these devices, while they[br]might be airgapped, they also have
0:04:34.919,0:04:42.650
antennas and they are interconnected by a[br]ton of different wireless protocols such
0:04:42.650,0:04:50.970
as Wi-Fi, LoRa or GSM or even proprietary[br]radio links. So, yeah, and even the
0:04:50.970,0:04:54.940
case studies show that basically in this[br]case, you would have a monitoring network
0:04:54.940,0:04:59.650
that's connected via the cellular network[br]to control the water mains and so on and
0:04:59.650,0:05:04.889
check the pressure. Or even worse, they[br]even recommend that you connect the actors
0:05:04.889,0:05:10.430
like valves and water level gotchas and so[br]on over GPS, which we know is not a secure
0:05:10.430,0:05:17.830
protocol to do anything that could [br]be critical. Or you have stuff like
0:05:17.830,0:05:24.160
water storage tanks that are controlled[br]via Wi-Fi and so on or even public in the
0:05:24.160,0:05:33.349
Internet. So, yeah, these devices are[br]airgapped? Nope. So attacking in the field
0:05:33.349,0:05:37.889
I already mentioned, if you go to [br]Shodan, you will find a ton of different
0:05:37.889,0:05:42.970
devices reachable via the Internet [br]and even via GPS. So if you live
0:05:42.970,0:05:49.090
close to, for example, a dam or something,[br]it's kind of interesting to look at an SDR
0:05:49.090,0:05:52.290
or similar radio equipment to see what's[br]going over the airwaves, because you will
0:05:52.290,0:05:59.090
find a ton of interesting stuff and[br]sometimes, you can even very trivially get
0:05:59.090,0:06:03.930
a physical access to the in field devices[br]because they might just be in a white box
0:06:03.930,0:06:07.569
somewhere hidden. And if you break into[br]it, you can pull out the SIM card and it
0:06:07.569,0:06:12.419
will put you directly into the SCADA[br]network, if you're lucky. Don't do that,
0:06:12.419,0:06:13.760
by the way.
0:06:13.760,0:06:17.169
laughter
0:06:17.169,0:06:24.539
So, yeah, let's let's hack some gateways.[br]So the equipment you will need to and
0:06:24.539,0:06:29.039
everything in this talk was done on this[br]desk, just using these devices here, you
0:06:29.039,0:06:33.000
really just need a laptop, you need an[br]oscilloscope or similar measurement
0:06:33.000,0:06:37.320
equipment just to ensure that you don't[br]burn out your logic analyzer. You need a
0:06:37.320,0:06:43.230
logic analyzer, a soldering iron, a[br]multimeter and a power supply. And that's
0:06:43.230,0:06:48.360
really basically it, because you can hack[br]almost any embedded device that's using
0:06:48.360,0:06:56.530
these devices and to find potential[br]targets. I have this kind of map where
0:06:56.530,0:07:02.139
first try to understand, can I get the[br]firmware of the device or do I have to
0:07:02.139,0:07:07.251
somehow, for example, use J-Tech to get it[br]out of the device? Can I actually buy the
0:07:07.251,0:07:12.460
devices at a sensible price? Because some[br]of these devices cost like 600 € or so,
0:07:12.460,0:07:18.360
and if you buy ten of them, that gets[br]expensive very quickly. And so, uh, I need
0:07:18.360,0:07:24.360
to check eBay and see what devices can I[br]actually buy. And they should be half what
0:07:24.360,0:07:29.449
current, because if you look at all the[br]devices, like 10 years old or so, they are
0:07:29.449,0:07:33.940
completely broken. You don't even have to[br]look to start to look at their security.
0:07:33.940,0:07:40.819
So, yeah, the first device that I that I[br]choose to really look at was the moxa
0:07:40.819,0:07:50.689
W2150A, which is this small device, which[br]is also mounted on the board right here,
0:07:50.689,0:07:54.319
mainly because I found the phone [br]was available and it looked like an
0:07:54.319,0:07:58.669
interesting device because it has Wi-Fi[br]and so if I managed to break into it, I
0:07:58.669,0:08:07.930
can jump an airgap potentially. And the[br]W2150A is just a simple device server. So
0:08:07.930,0:08:14.939
you can connect any serial device, any[br]RS485 device simply to it and it will be
0:08:14.939,0:08:20.669
exposed via Ethernet or even via Wi-Fi.[br]And you can download the firmware publicly
0:08:20.669,0:08:29.270
and it's available on eBay relatively[br]cheap. So like 150 bucks or something. So
0:08:29.270,0:08:33.290
I downloaded the firmware and I[br]looked at the entropy of the firmware and
0:08:33.290,0:08:37.090
I immediately saw that the entropy is very[br]high, which means either it's very
0:08:37.090,0:08:41.590
compressed or it's encrypted,[br]unfortunately, using a tool called
0:08:41.590,0:08:46.510
binwalk, which is really useful for[br]looking into firmwares I saw that there's
0:08:46.510,0:08:51.510
no compression detected. And so it was[br]very likely that this firmware image is
0:08:51.510,0:08:59.940
encrypted. But I noticed on the Web page[br]that before you upgrade to version 2.0 or
0:08:59.940,0:09:08.650
2.1 of the firmware, you must upgrade to[br]the firmware version 1.11. And I thought,
0:09:08.650,0:09:13.540
that's interesting. Let's look at the[br]release notes for version 1.11. And it
0:09:13.540,0:09:22.330
turns out that 1.11 adds the support for[br]the encrypted firmware. So I downloaded
0:09:22.330,0:09:28.093
the one point eleven firmware and sure[br]enough, it's unencrypted. And if you've
0:09:28.093,0:09:33.710
ever done anything with ARM before, if you[br]just look into a firmware hex dump, you
0:09:33.710,0:09:39.930
can immediately recognize whether it's ARM[br]or not, because the first four bits of each
0:09:39.930,0:09:45.580
instructions are the conditional bits [br]and those are almost always E. So if
0:09:45.580,0:09:50.320
you see a Hexdump and roughly every fourth[br]byte is an E, you know, this is an ARM
0:09:50.320,0:09:57.550
firmware and it's not encrypted or[br]anything else. And so, yeah, sure enough,
0:09:57.550,0:10:02.640
I ran binwalk on this image. This time we[br]see there is a huge drop in entropy, which
0:10:02.640,0:10:08.570
is the bootloader and so on, and then a[br]high entropy, which is basically the all
0:10:08.570,0:10:15.280
the compressed filesystems and so on. And[br]binwalk was able to detect the SquashFS
0:10:15.280,0:10:22.060
filesystem and extract it for me very,[br]very easy. And so my goal was to extract
0:10:22.060,0:10:27.250
the firmware, find the firmware upgrade[br]code and somehow try to decipher the new
0:10:27.250,0:10:34.250
firmware. And so I was browsing through[br]the files and sure enough, found the file
0:10:34.250,0:10:40.810
that was helpfully called[br]libupgradeFirmware.so and if we look into
0:10:40.810,0:10:45.010
the symbols, which they luckily didn't[br]remove or anything, there is a beautiful
0:10:45.010,0:10:48.066
symbol called firmware decrypt.
0:10:48.066,0:10:51.150
laughter
0:10:51.150,0:10:56.430
So we load the whole thing into[br]disassembler and we see that
0:10:56.430,0:11:03.870
there's some fancy XORing going[br]on in the bottom left corner. And I'm
0:11:03.870,0:11:08.000
going to walk you through what's, what[br]exactly is happening in this code.
0:11:08.000,0:11:13.310
So basically, first, there's a variable[br]called password loaded into the registar
0:11:13.310,0:11:21.790
R2 and then a second count variable is[br]basically set and it starts looping and
0:11:21.790,0:11:35.540
increasing always by four and goes through[br]this whole xor shebang and it turns out
0:11:35.540,0:11:41.200
that this is the obfuscation method for[br]the AES Key. So, in password, in memory,
0:11:41.200,0:11:45.950
we have an obfuscated key and we can be[br]obfusciated by just implementing the code
0:11:45.950,0:11:53.890
we see here in C or in the emulator. [br]And sure enough, eventually this
0:11:53.890,0:12:03.330
will be used as the key into the ECB 128[br]AES decryption. And so I implemented the
0:12:03.330,0:12:08.760
whole thing in C, it was almost a copy[br]paste from the decompiler, so you can in
0:12:08.760,0:12:14.500
IAD Pro, you just hit F5, copy the C code[br]at the bit, fix the memory offsets and so
0:12:14.500,0:12:20.020
on. And you have the whole key obfuscation[br]method basically reverse engineered almost
0:12:20.020,0:12:25.630
automatically. And so I compile it. And[br]sure enough, Moxa key extration, it turns
0:12:25.630,0:12:31.200
out that the key is two eight eight seven[br]Conn seven five six four. I build a short
0:12:31.200,0:12:39.260
script to decrypt the 2.1 firmware and[br]this time Binwalk finds all the files and
0:12:39.260,0:12:41.740
we can start reverse engineering the[br]actual firmware.
0:12:41.740,0:12:48.519
applause
0:12:48.519,0:12:54.180
The scripts for this are available on my[br]github. I'll push the actual decrypts stuff
0:12:54.180,0:12:59.810
after the talk because this is the first[br]time this has been released. And so after
0:12:59.810,0:13:03.470
I was at this point, I knew that the[br]firmware is.. I can decrypted the firmware
0:13:03.470,0:13:07.930
I can look into it. By the way, it's not[br]signed or anything. The only verification
0:13:07.930,0:13:14.480
method is CRC32. And so at this point I[br]knew, OK, I can buy this device and
0:13:14.480,0:13:19.980
start playing with it. And so I went to[br]eBay, I bought one. I got it. I screwed it
0:13:19.980,0:13:24.140
open. And sure enough, there's an ARM[br]processor in there. It's an Freescale
0:13:24.140,0:13:28.640
i.MX25, which is just a regular ARM[br]processor. It's like 400 MHz or something,
0:13:28.640,0:13:34.880
I don't know. And I started probing all[br]the all the small pins inside of the
0:13:34.880,0:13:43.040
device to try to find JTAG or serial[br]or anything. And so I actually hooked up
0:13:43.040,0:13:47.320
my power supply to foot pedal so that I[br]can probe and just press with my foot to
0:13:47.320,0:13:54.250
reset the device. And sure enough, I found[br]that there's a full serial console
0:13:54.250,0:14:00.660
available inside of the device on these[br]pins. And if you boot the device, it even
0:14:00.660,0:14:05.160
tells you, please press enter to activate[br]this console, and so you do that and you
0:14:05.160,0:14:07.463
are root on the device.
0:14:07.463,0:14:14.820
applause
0:14:14.820,0:14:18.660
So that's kind of cool, but that means[br]that you require physical access, so
0:14:18.660,0:14:23.530
that's not really a vulnerability, but[br]it's very nice to have when doing security
0:14:23.530,0:14:29.420
research because it means you can suddenly[br]debug all the code on there. And so if you
0:14:29.420,0:14:35.050
write an exploit, you can just touch GDB[br]to the binary and start very, very simply,
0:14:35.050,0:14:40.420
writing the exploit. So at this point,[br]I was trying to look at the available
0:14:40.420,0:14:46.010
services on the device. So for example,[br]there is a web interface, there's a
0:14:46.010,0:14:52.530
proprietary configuration protocol,[br]there's telnet, there's snmp, there is a
0:14:52.530,0:14:58.910
serial driver protocol and so on. And I[br]started looking at the web interface and
0:14:58.910,0:15:03.740
there was cross site scripting that was[br]Cross site request forgery, there was
0:15:03.740,0:15:07.440
insecure authentication where they[br]basically hash on the client. So they have
0:15:07.440,0:15:12.520
some JavaScript that hashes your password[br]and then locks you in. Then there's a
0:15:12.520,0:15:17.720
command injection which lets you execute[br]code as root, there are stack overflows.
0:15:17.720,0:15:23.675
And just a week ago there was a zero day[br]released for the web server. So yeah, demo
0:15:23.675,0:15:36.820
time. So just let me open up the Moxa[br]Pitch right here. And so this one is
0:15:36.820,0:15:41.240
authenticated, so I'll just enter the[br]default password, which, by the way, in
0:15:41.240,0:15:46.050
the field will 90 percent of the time[br]these devices will be configured with
0:15:46.050,0:15:54.950
default credentials. But still, so, if we[br]just start browsing through this thing and
0:15:54.950,0:16:00.140
go to the basic settings, we can start[br]with a simple cross site scripting just in
0:16:00.140,0:16:08.810
the device name. One sec, so just for[br]example we just paste in some JavaScript.
0:16:08.810,0:16:15.017
Submit the whole thing, and hello 34c3.
0:16:15.017,0:16:19.560
applause
0:16:19.560,0:16:23.770
I know what you're thinking, like cross [br]site scripting, come on, that's not a
0:16:23.770,0:16:28.530
vulnerability, that's just nothing. So[br]let's look at the ping test that's
0:16:28.530,0:16:33.910
integrated into this device. And funilly,[br]a different device from Moxa that runs an
0:16:33.910,0:16:39.570
entirely different firmware had the same[br]vulnerability in the past. But if I just
0:16:39.570,0:16:46.390
paste in my ping, so my IP address, a[br]semicolon and then, for example, I cut
0:16:46.390,0:16:51.970
/etc/passwd and activate enter. [br]Here we go.
0:16:51.970,0:17:00.060
applause
0:17:00.060,0:17:08.199
Kind of funny, but, yes, for sure not[br]intended. All righty, but I know what
0:17:08.199,0:17:12.740
you're thinking, right, these are[br]authenticated bugs in the web interface,
0:17:12.740,0:17:17.460
so we need something unauthenticated. We[br]want something that's like cool and a real
0:17:17.460,0:17:23.420
exploit. Right? And so I decided to look[br]at the.. this custom TCP protocol, which
0:17:23.420,0:17:29.430
runs on Port 4900. And my goal was to[br]reverse engineer the whole protocol and
0:17:29.430,0:17:34.030
build a fuzzer for it, to find[br]vulnerabilities, that turned out not to be
0:17:34.030,0:17:40.990
necessary. So during some testing, I just[br]sent a lot of bytes onto this thing and
0:17:40.990,0:17:49.140
enabled crash debugging via the serial[br]console. And sure enough, it crashed and
0:17:49.140,0:17:58.740
put my program countdown right to[br]0x41414140. Wonderful. Thank you, Moxa.
0:17:58.740,0:18:04.370
applause
0:18:04.370,0:18:21.550
So, Demo time. So let's increase the size[br]of this a bit. So I built a small script.
0:18:21.550,0:18:34.490
Just called moxa_pown and I'll just supply[br]the IP address to it. Let's see. Opening a
0:18:34.490,0:18:43.620
second shell to connect to it via netcat.[br]Here we go, we have a root shell on the
0:18:43.620,0:18:44.600
device.
0:18:44.600,0:18:54.263
applause
0:18:54.263,0:19:01.540
So, yeah, that was the Moxa w21508,[br]basically rolls of the tongue. And so the
0:19:01.540,0:19:08.690
next device I decided to look at was the[br]Advantech EKI-1522 which you can find
0:19:08.690,0:19:17.460
right here. And it's, again, just a simple[br]serial device server this time without
0:19:17.460,0:19:21.410
Wi-Fi, even though they are available with[br]Wi-Fi. It comes with two Ethernet ports
0:19:21.410,0:19:26.060
two serial ports and so on. And I[br]basically followed the same steps again.
0:19:26.060,0:19:31.170
So I looked at the.. I downloaded the[br]firmware. I looked at the edit using
0:19:31.170,0:19:35.800
binwalk. And this time we see almost no[br]entropy. So there is.. this guy is
0:19:35.800,0:19:40.280
basically completely unencrypted. And[br]again, we saw some ARM 32 bit it runs a
0:19:40.280,0:19:51.010
Linux kernel, 2.6.31 and a BOA Web server[br]where the last update was in 2005. And the
0:19:51.010,0:19:56.770
firmware, I think, is from 2017. So these[br]are kind of outdated. And I found
0:19:56.770,0:20:01.230
during the initial analysis just of the[br]firmware that the main binary to look at
0:20:01.230,0:20:07.180
will be this edgserver binary. And so I[br]loaded it into IDA pro and looked at the
0:20:07.180,0:20:12.780
different things that calls. And there[br]are a lot of calls to functions like
0:20:12.780,0:20:18.340
string copy, to system, to sprintf and so[br]on that are generally kind of considered
0:20:18.340,0:20:25.660
unsecure. And sure enough, I am doing[br]static analysis. I found that there's some
0:20:25.660,0:20:33.630
code for sending an email as an alert, for[br]example, when the system reboots. And
0:20:33.630,0:20:39.250
the full command invocation is mailx -s[br]blah blah blah, and we have control over
0:20:39.250,0:20:46.160
some parts in the string because we can[br]configure the two address in the UI. And if
0:20:46.160,0:20:51.040
we look at what's happening[br]here, it basically just sets up this
0:20:51.040,0:20:56.500
format string. Then it goes to include the[br]subject and then it gets some arguments
0:20:56.500,0:21:04.260
from the stack and basically calls[br]into system. And so there's no filtering
0:21:04.260,0:21:09.930
going on at all. So we have an unfiltered[br]part of the system, invocation, code
0:21:09.930,0:21:15.380
execution. And this was before I had the[br]device in my hand. And this is kind of a
0:21:15.380,0:21:19.470
funny story because I first bought because[br]it was just 40 bucks, I bought this
0:21:19.470,0:21:24.770
device, which in the firmware has the same[br]bug, but the mail functionality is broken,
0:21:24.770,0:21:33.780
so I couldn't test it. So I had to go to[br]eBay again, buy another one and buy the
0:21:33.780,0:21:38.950
bigger one. And so I ordered the bigger[br]one on eBay. Looks like this. It comes
0:21:38.950,0:21:45.660
with a Cavium CNS C.P.U. It has JTAG[br]exposed on the bottom there and serial
0:21:45.660,0:21:50.940
console is available again without any[br]authentication. So beautiful. You just
0:21:50.940,0:21:57.809
connect your bus pirate or your UART[br]adapter to it and you have full serial
0:21:57.809,0:22:06.740
console. So, again, we had to look at[br]finding vulnerabilities for this device
0:22:06.740,0:22:11.559
and there, again, a ton of different[br]services, there's like a Web interface
0:22:11.559,0:22:15.670
available. There is a proprietary[br]configuration protocol that's based on
0:22:15.670,0:22:22.760
UDP. There is Telnet, there's snmp,[br]there's a serial driver protocol and so
0:22:22.760,0:22:28.380
on. And again, looked at the website and[br]again, cross site scripting cross side
0:22:28.380,0:22:33.280
request forgery, command injection, broken[br]authentication, which basically if you log
0:22:33.280,0:22:38.710
in from one computer, it uses, I think[br]http digest authentication, you can
0:22:38.710,0:22:42.690
connect from a completely different[br]computer and it doesn't ask for a
0:22:42.690,0:22:49.700
password. I don't know why that is, but..[br]Yeah. So I was thinking I was doing
0:22:49.700,0:22:52.130
something wrong, but it turned out it was[br]just broken.
0:22:52.130,0:22:54.855
laughter
0:22:54.855,0:23:03.170
So, yeah, and there's, again, a stack[br]overflow in another protocol. So I guess,
0:23:03.170,0:23:13.980
again, demo time. Let's first look at[br]the device itself, so, you know the
0:23:13.980,0:23:22.830
password, firstly, we have a nice device[br]description here. This is just a basic web
0:23:22.830,0:23:29.320
interface. Right. And we can, again, just[br]copy in some basic JavaScript
0:23:29.320,0:23:38.620
hit the save button. Reload and there we[br]go, cross site scripting yet again, OK,
0:23:38.620,0:23:49.129
again, not really interesting. Right. So,[br]um, let's look at the stack overflow.
0:23:49.129,0:24:04.070
Again, I have a small script advantech_pown.[br]For the IP there. And we have netcat
0:24:04.070,0:24:12.090
running on there. Sure enough, there we[br]go, that's root on the Advantech device
0:24:12.090,0:24:13.810
again, via stack overflow.
0:24:13.810,0:24:25.516
applause
0:24:25.516,0:24:31.500
Yeah, so two of three devices have[br]basically broken already. Let's look
0:24:31.500,0:24:38.150
at the next one. This one is a Lantronix[br]EDS2100. And this one is kind of
0:24:38.150,0:24:43.770
interesting because it's not ARM. I[br]normally I almost exclusively do ARMs. So
0:24:43.770,0:24:48.500
this one was kind of interesting. And this[br]device, which is mounted somewhere right
0:24:48.500,0:24:57.390
here. Yeah. This device comes with a[br]serial to ethernet secure device server.
0:24:57.390,0:25:01.929
It has two serial ports. It has[br]Ethernet and you can buy it in two
0:25:01.929,0:25:07.830
variants. One comes with Linux and one is[br]Evolution OS, which is I guess, a
0:25:07.830,0:25:14.880
proprietary operating system from[br]Lantronics. And I'm using the EvolutionOS
0:25:14.880,0:25:22.120
variant in this talk. Looking at the[br]firmware it turns out it's unencrypted and
0:25:22.120,0:25:28.240
it's coldfire architecture, which I've[br]never done really anything with before,
0:25:28.240,0:25:32.630
and there are no obvious external software[br]components. So if you go through this,
0:25:32.630,0:25:37.440
through the firmware, you'll find there's[br]an SSH implementation, there's an SSL
0:25:37.440,0:25:42.810
implementation, but it's not openSSL and[br]it's not anything very well known. And the
0:25:42.810,0:25:47.490
same is true for the web server and so on.[br]It's not really anything that's well
0:25:47.490,0:25:56.500
known. And this time, while probing[br]the device, I did not really find anything
0:25:56.500,0:26:01.580
interesting in terms of serial consoles or[br]so, but it just found a potential debugger
0:26:01.580,0:26:05.730
port, but it didn't have a fitting[br]debugger unfortunately. The CPU is from
0:26:05.730,0:26:14.760
NXP runs at 160MHz or something. Yeah.[br]This time we actually have a web
0:26:14.760,0:26:21.660
interface, we have Telnet SSL and it even[br]has a file system, so you have like FTP
0:26:21.660,0:26:26.210
and TFTP which allows you to download the[br]configuration, upload the configuration
0:26:26.210,0:26:30.980
and so on. And it's kind of hard to secure[br]it correctly because there are so many
0:26:30.980,0:26:37.000
protocols and it's not really clear what's[br]set up by default. But yeah, you get
0:26:37.000,0:26:44.350
the idea. And this time the web interface[br]was surprisingly secure. So there was no
0:26:44.350,0:26:50.230
cross site scripting. There was no command[br]injection, because there's also not really
0:26:50.230,0:26:55.440
a shell that you could execute commands[br]into. But I still found some stuff.
0:26:55.440,0:27:01.540
One is the configuration injection, which[br]allows you basically to change the format
0:27:01.540,0:27:06.630
of the configuration using a different[br]field. And I found an authentication
0:27:06.630,0:27:11.970
bypass, so I was able to write a small[br]piece of code that takes a while and then
0:27:11.970,0:27:23.650
completely removes the password from the[br]device. Demo time. So if we connect to the
0:27:23.650,0:27:29.750
Lantronics device, it will currently ask[br]for a password, which in theory we don't
0:27:29.750,0:27:44.580
have. Let's clean up here a bit. I know[br]it's just. And let's run Lantronix_pown,
0:27:44.580,0:27:51.300
oh, that was fast. That worked. Yeah, sure[br]enough, the password is gone.
0:27:51.300,0:27:59.980
applause
0:27:59.980,0:28:07.230
Awesome. To be honest, I didn't expect the[br]demos to go so smoothly, so I put in an
0:28:07.230,0:28:13.710
hour for the talk for this went very well[br]so far, so that's good. So before we
0:28:13.710,0:28:22.170
finish already, some other devices are[br]even worse. So, for example, as I
0:28:22.170,0:28:26.549
mentioned, I bought some other devices,[br]for example, this Advantaech device and
0:28:26.549,0:28:31.610
this Moxa device and this Lantronix[br]device, which are basically the
0:28:31.610,0:28:38.940
predecessors of the other devices. And[br]those guys are really interesting to look
0:28:38.940,0:28:45.850
at, one could say. So, some of those are[br]running eCos, which is an embedded Linux
0:28:45.850,0:28:52.390
platform, which was last released in 2009,[br]and some devices run a Linux kernel with
0:28:52.390,0:28:57.570
the 2.4 version and you see Linux without[br]any memory protection whatsoever. So even
0:28:57.570,0:29:03.640
if they, so even a small stack overflow in[br]one of the userspace applications gives
0:29:03.640,0:29:08.640
you full root access to the device because[br]you can directly exploit the kernel and
0:29:08.640,0:29:12.840
there are unfixed public vulnerabilities.[br]So in the first penetration test that I
0:29:12.840,0:29:19.290
did, that included actually this device[br]and Moxa and part of a small one. I found
0:29:19.290,0:29:25.170
that using SNMPWwalk, it gives you back[br]the administration password via SNMP.
0:29:25.170,0:29:26.780
laughing
0:29:26.780,0:29:31.500
And so I went online. I tried to report[br]it. And it turns out it's well known
0:29:31.500,0:29:34.160
there's a metasploit module for this
0:29:34.160,0:29:36.830
laughing
0:29:36.830,0:29:41.690
and it's unfixed, OK? And these devices[br]are still in support. So I don't know why
0:29:41.690,0:29:50.950
the vendor is not patching this. Yeah. So[br]the summary with trivial vulnerabilities
0:29:50.950,0:29:56.520
in most devices, or at least all that I've[br]looked at, there are no security
0:29:56.520,0:30:00.571
mitigations whatsoever. So they don't even[br]enable like the compiler flags that you
0:30:00.571,0:30:05.850
just set and then you have at least some[br]kind of stack protection and some like
0:30:05.850,0:30:11.070
stack cookies and whatnot. And some[br]vendors are really bad at responding to
0:30:11.070,0:30:18.429
vulnerability reports. So, yeah, I'm not[br]going to name the vendor, but not even, on
0:30:18.429,0:30:22.180
Twitter I asked them to please give me a[br]security contact and they responded,
0:30:22.180,0:30:26.840
please use our contact form. I said I did,[br]three times. I send you emails, you're not
0:30:26.840,0:30:30.600
responding to me. And so they stopped[br]responding to me on Twitter too.
0:30:30.600,0:30:40.809
laughing[br]applause
0:30:40.809,0:30:47.200
So how to mitigate? Well, the only way[br]that I would see to mitigate against this,
0:30:47.200,0:30:53.380
and I'm more on the deconstructive side of[br]the story, is defense in depth. So never
0:30:53.380,0:30:56.850
directly expose any of these devices to[br]the Internet, even if they say they
0:30:56.850,0:31:02.490
support VPN, even if they say they are a[br]secure device of whatever, just don't do
0:31:02.490,0:31:08.780
it. Get a real VPN gateway and make sure[br]that you never rely on a single level of,
0:31:08.780,0:31:16.169
for example, encryption. So, for example,[br]WPA2 was broken by the crack attack and
0:31:16.169,0:31:20.799
they actually released a patch for it[br]after two months. And these are these are
0:31:20.799,0:31:26.370
still two months where you are exposed to[br]vulnerability on your potentially mission-
0:31:26.370,0:31:33.760
critical system. Also never use GPRS for[br]these devices without VPN because it just,
0:31:33.760,0:31:41.480
it will go wrong. Okay. Yeah, thank you. I[br]guess now we have time for Q&A. Thank you
0:31:41.480,0:31:43.282
all for coming.
0:31:43.282,0:31:49.170
applause
0:31:49.170,0:31:57.990
Herald: Thank you very much for the talk.[br]So we have very much time for Q&A. So
0:31:57.990,0:32:03.980
please line up to the microphones and we[br]have someone at microphone 4 already.
0:32:03.980,0:32:09.220
Mic 4: Yes, hello. Hello. Thanks for your[br]talk. This is.. obviously this is a
0:32:09.220,0:32:14.792
problem. This is a part of the bigger[br]problem of security in IT. Right. In
0:32:14.792,0:32:18.950
anything related to any kind of[br]technology. And this is only going to go
0:32:18.950,0:32:25.230
worse with time, right. Internet of shit,[br]internet of things and so and so on, so
0:32:25.230,0:32:31.740
forth. So my question is, you gave some[br]ideas how to mitigate this in this very
0:32:31.740,0:32:36.540
specific area that use VPN, et cetera, et[br]cetera. But my question is, so hacker
0:32:36.540,0:32:42.462
community is not very, let's say,[br]interested in regulation. Right? And when
0:32:42.462,0:32:46.610
we see, when we see a government trying to[br]do something with technology that usually
0:32:46.610,0:32:51.580
goes bad, we have this idea in our head[br]that, OK, this can only go like this can
0:32:51.580,0:32:56.741
only go bad. Right. But so my question is:[br]do you think that perhaps there is some
0:32:56.741,0:33:00.810
space for regulation here?[br]T: There's definitely space for
0:33:00.810,0:33:07.450
regulation, but I think regulation does[br]not solve the underlying technical issues.
0:33:07.450,0:33:13.611
So these devices, it's 2017 and these[br]devices are using C-code. I think that's
0:33:13.611,0:33:18.580
just asking for trouble, basically. And so[br]we really need to see this shift, even in
0:33:18.580,0:33:22.690
the embedded world, to switch to memory[br]safe languages, for example Rust or
0:33:22.690,0:33:28.129
something similar, and really to stop[br]using C in this kind of context. I don't
0:33:28.129,0:33:35.729
think there's anyone who can .. Thank you.[br]applause
0:33:35.729,0:33:39.072
T: But there's definitely space for[br]regulation.
0:33:39.072,0:33:43.173
Herald: Since there was a question from[br]the Internet.
0:33:43.173,0:33:47.530
Signal Angel: OK, yeah, the Internet wants[br]to know why you are not naming the bad
0:33:47.530,0:33:51.980
vendor, because it looks like it's the[br]only option basically if they don't
0:33:51.980,0:33:57.990
respond to you. Let's say I asked them on[br]Twitter and my Twitter is right there. And
0:33:57.990,0:34:02.640
if you click on Tweets and Replies..[br]laughter
0:34:02.640,0:34:05.850
Signal Angel: Yeah, somebody just posted[br]the link on IRC.
0:34:05.850,0:34:10.869
laughter[br]T: I did not name them, just for the
0:34:10.869,0:34:13.309
record.[br]laughter
0:34:13.309,0:34:17.029
applause[br]Herald: So we have a question from
0:34:17.029,0:34:23.369
microphone number 2.[br]Mic 2: So you shown an exploit for the
0:34:23.369,0:34:29.559
last device that disabled authentication.[br]What did you use to achieve that?
0:34:29.559,0:34:35.529
T: So this one is unpatched and not yet[br]fixed, so I would rather not disclose the
0:34:35.529,0:34:38.720
details yet.[br]Mic 2: OK.
0:34:38.720,0:34:42.919
Herald: Microphone number 1, please.[br]Mic 1: I wonder if you've also been
0:34:42.919,0:34:47.729
looking at a building automation system,[br]control systems, or just industrial
0:34:47.729,0:34:53.510
automation control systems?[br]T: So you can use these devices basically
0:34:53.510,0:35:00.609
wherever you want. And I think some of the[br]Moxa ones are used in home automation. But
0:35:00.609,0:35:05.920
I've looked at I guess Crestron, it's[br]called? But not in a lot of detail. So I'm
0:35:05.920,0:35:09.509
more on the industrial side at the moment.[br]Mic 1: Thanks.
0:35:09.509,0:35:15.079
Herald: Microphone number 3.[br]Mic 3: Any field experience or even just
0:35:15.079,0:35:21.259
opinions on using industrial strength[br]Raspberry Pi hardware with community
0:35:21.259,0:35:25.559
supported Linux distributions or something[br]like OpenBC whatever on them.
0:35:25.559,0:35:30.869
T: Yeah. So I guess the big trouble there[br]is support, right? There are some, some
0:35:30.869,0:35:34.579
German companies and so on that provide[br]support for industrial Raspberry Pis and
0:35:34.579,0:35:40.789
even like nice casing and so on. But I'm[br]not sure if really Raspberry Pi is the way
0:35:40.789,0:35:45.240
to go here. I think there are[br]boards that are.. the problem is not the
0:35:45.240,0:35:49.720
underlying stack, right? It's not the[br]hardware. Really, that's the issue. It's
0:35:49.720,0:35:55.950
the software. And you will have the same[br]issues on on the Raspberry Pi. So, yeah, I
0:35:55.950,0:36:00.880
guess you could buy these devices, which[br]are like industrial grade shockproof and
0:36:00.880,0:36:07.460
whatnot, and put some Linux on it and [br]do it better. But I don't think that
0:36:07.460,0:36:11.650
the hardware or platform will [br]change anything at the moment.
0:36:11.650,0:36:16.319
Herald: There is another question from[br]microphone number 4.
0:36:16.319,0:36:21.749
Mic 4: Hi, more a social question, did you[br]get in contact with any development team,
0:36:21.749,0:36:25.849
software development team in any of these[br]companies, or might it be that there is no
0:36:25.849,0:36:33.080
one behind the emails and everything?[br]T: So I guess some of these companies are
0:36:33.080,0:36:37.349
really so big, that they don't reply to[br]you if you don't have a support contract
0:36:37.349,0:36:45.049
with them. But, for example, the support[br]of the ones that are not on my Twitter is
0:36:45.049,0:36:49.730
kind of decent when it comes to two[br]security reports. And so my next steps
0:36:49.730,0:36:57.220
will be to go via the ICS Cert, but, you[br]know, to report them. So, yes, there are
0:36:57.220,0:37:03.737
development teams that will get in contact[br]with you, just not from all vendors.
0:37:03.737,0:37:06.670
Herald: Thank you. We have another[br]question from the Internet.
0:37:06.670,0:37:13.960
Signal Angel: Hello? OK. The Internet[br]wants to know what to do about, because
0:37:13.960,0:37:18.259
there are a lot of old devices in the[br]field, how do you propose a vendor should
0:37:18.259,0:37:24.200
deal with legacy devices and updates?[br]T: Yeah, so keeping legacy devices
0:37:24.200,0:37:29.680
supported is very expensive because, for[br]example, if you buy a Qualcomm chip, they
0:37:29.680,0:37:35.089
will eventually drop support for the Linux[br]kernel for it and so on. But if you buy
0:37:35.089,0:37:39.619
like a Freescale automotive chip, they[br]guarantee you a certain time of support.
0:37:39.619,0:37:43.490
But then you actually have to invest the[br]money to regularly provide the updates and
0:37:43.490,0:37:48.859
ensure that your devices are secure. The[br]problem is that the lifetime of industrial
0:37:48.859,0:37:55.470
installations currently is much larger[br]than the lifetime of this processors' supports
0:37:55.470,0:38:00.819
and so on. So I guess we'll have to get[br]used to upgrading our hardware regularly
0:38:00.819,0:38:07.400
or switch to, or figure out a different[br]way of deploying secure software onto
0:38:07.400,0:38:11.259
them. But I really think the underlying[br]problem is, that we are still using
0:38:11.259,0:38:16.229
memory unsafe languages. And I guess the[br]fact that there's cross site scripting
0:38:16.229,0:38:20.150
just shows that there's no security[br]awareness really at those vendors
0:38:20.150,0:38:29.395
whatsoever. At some of the vendors.[br]Herald: So, microphone number 2, please.
0:38:29.395,0:38:34.349
Mic 2: I was wondering, you mentioned that[br]some of these facilities use GPRS.
0:38:34.349,0:38:36.390
T: Yeah.[br]Mic 2: Do you know if they have mostly
0:38:36.390,0:38:40.749
their own closed infrastructure, or if[br]they're using general consumer telecom
0:38:40.749,0:38:44.849
stuff?[br]T: So they will use commercial
0:38:44.849,0:38:50.479
networks mostly, and then they have custom[br]EPNs which have an IPSec tunnel or
0:38:50.479,0:38:55.700
something similar to their premises. But[br]there's also there's also a company that
0:38:55.700,0:39:02.589
sells industrial control SIM cards[br]which give you a public IP and you don't
0:39:02.589,0:39:08.100
want to search on Shodan for that vendor.[br]Mic 2: Yeah. Thank you.
0:39:08.100,0:39:11.050
Herald: There is a question from[br]microphone number 3.
0:39:11.050,0:39:14.999
Mic 3: Hi there, isn't economics meant to[br]solve some of these problems? We're not
0:39:14.999,0:39:20.359
talking about dirt cheap devices. How[br]surely at 300 bucks you should better have
0:39:20.359,0:39:24.539
someone who's read security one and one.[br]How long before a large organization gets
0:39:24.539,0:39:28.200
the result of their security audit and[br]goes to the aforementioned vendors and
0:39:28.200,0:39:32.960
says, provide us something that's not[br]trivially hackable, otherwise we stop
0:39:32.960,0:39:37.839
buying your rubbish?[br]T: Well, I mean, it's the same in all of
0:39:37.839,0:39:45.329
IT, right? So everything has[br]vulnerabilities. And yes, there should be
0:39:45.329,0:39:50.400
market pressure. But that's why I'm trying[br]to raise awareness for the issues that
0:39:50.400,0:39:53.270
these devices have.[br]Mic 3: Thanks.
0:39:53.270,0:39:55.729
Herald: There's another question from the[br]Internet.
0:39:55.729,0:40:01.339
Signal Angel: Yep. The Internet wants to[br]know how and if it's a good idea to raise
0:40:01.339,0:40:06.549
the level of awareness in public, because[br]they think it's a good approach to make
0:40:06.549,0:40:11.869
people, the public know that, well,[br]infrastructure in the cities is at risk.
0:40:11.869,0:40:16.000
T: Uh, sorry. Could you repeat the first[br]part of the question?
0:40:16.000,0:40:21.339
Signal Angel: Yeah. They want to know how[br]to raise awareness for this in the public?
0:40:21.339,0:40:27.789
T: Good question. I guess we need some[br]news articles or something about this in
0:40:27.789,0:40:32.800
regular paper, but I personally think it's[br]just an accident waiting to happen. So
0:40:32.800,0:40:37.999
eventually someone will turn off the[br]lights in a city or wherever, will open a
0:40:37.999,0:40:44.773
flood valve or something. And that's when[br]the awareness will start.
0:40:44.773,0:40:47.813
Herald: There's another question from[br]microphone number 4.
0:40:47.813,0:40:51.680
Mic 4: OK, for what kind of industrial[br]processes are these devices you just
0:40:51.680,0:40:57.108
demoed used?[br]T: So I've seen them in power utility. I
0:40:57.108,0:41:02.350
know they're used in water dam[br]control systems. They are used and in
0:41:02.350,0:41:07.039
serial connecting a CNC machine to the[br]network, they are used in connecting all
0:41:07.039,0:41:10.690
kinds of stuff. Because if you have a big[br]plant, you have a ton of different
0:41:10.690,0:41:15.719
sensors. So you might, you might need the[br]water level sensor. And for whatever
0:41:15.719,0:41:20.680
reason, you only can get it with a modbus[br]and then you need to convert the modbus to
0:41:20.680,0:41:25.119
TCP and then you need one of these[br]gateways. And so, I've seen in one
0:41:25.119,0:41:28.529
cabinet, 20 of them. So they're [br]really used a lot I guess.
0:41:28.529,0:41:31.869
Mic 4: OK, thank you. I just retweeted[br]your tweet to Star Alliance.
0:41:31.869,0:41:37.979
T: Huh. laughs Thank you. laughs[br]Herald: So there's another question from
0:41:37.979,0:41:41.260
the Internet.[br]Signal Angel: Yeah, the Internet wants to
0:41:41.260,0:41:50.749
know if you did any research on MQTT[br]for example from like Beckhoff uses?
0:41:50.749,0:41:54.489
T: I actually talked to someone who[br]recommended me to look at Beckhoff
0:41:54.489,0:41:58.249
yesterday, but I've not looked at them[br]whatsoever yet.
0:41:58.249,0:42:01.900
Herald: And there's another question from[br]microphone 3.
0:42:01.900,0:42:07.450
Mic 3: OK, could you show the Moxa web[br]panel, because I would like to double
0:42:07.450,0:42:16.619
check, which proves that they and they[br]would like you to see their Web page. And
0:42:16.619,0:42:24.050
I think this browser isn't very secure.[br]T: OK, let's take a look.
0:42:24.050,0:42:29.160
Mic 3: Yeah, and under gohead the[br]webserver small print.
0:42:29.160,0:42:41.527
laughter[br]Herald: Nice finding.
0:42:41.527,0:42:47.859
T: That's probably the issue here.[br]laughs
0:42:47.859,0:42:55.658
Herald: Are there any more questions? Any[br]questions from the Internet?
0:42:55.658,0:43:02.009
Signal Angel: The internet wants to know[br]how a memory safe language would prevent
0:43:02.009,0:43:08.750
the authentication bypasses you showed?[br]T: Not one would not be protected against
0:43:08.750,0:43:13.130
but it protects against a ton of other[br]stuff. It's just one example of where the
0:43:13.130,0:43:18.420
industry needs to change. We need to stop[br]using memory unsafe languages. We need to
0:43:18.420,0:43:23.910
start really thinking about security[br]design from the start, and we must not in
0:43:23.910,0:43:28.319
2017, there's no excuse for having cross[br]site scripting or anything on the web
0:43:28.319,0:43:35.720
page. That's also if we in the [br]Lantronics website, if you click logout,
0:43:35.720,0:43:39.479
it tells you logout is not supported in[br]your browser.
0:43:39.479,0:43:43.290
laughter[br]T: Probably because I'm not using Internet
0:43:43.290,0:43:48.130
Explorer five.[br]Herald: So there's another question from
0:43:48.130,0:43:53.239
microphone number 3.[br]Mic 3: Any remote part of the exploit
0:43:53.239,0:43:57.750
where you did a buffer [br]overflow - I think.
0:43:57.750,0:44:01.490
T: Yeah?[br]Mic 3: What I'm wondering is, are
0:44:01.490,0:44:07.180
there.. isn't it like very standard to[br]have ALSR on these devices?
0:44:07.180,0:44:10.239
T: No! laughts It should be, but it[br]isn't.
0:44:10.239,0:44:16.199
Mic 3: Okay. Thank you though. That was[br]pretty much my question.
0:44:16.199,0:44:23.428
Herald: Is there another question from the[br]Internet? It doesn't seem like it?
0:44:23.428,0:44:36.052
Signal Angel: So, one just came in, OK, if[br]you want to hear it. Ok, nope.
0:44:36.052,0:44:41.329
laughter[br]Herald: So, all right, give a very warm
0:44:41.329,0:44:43.329
applause to Thomas Roth again!
0:44:43.329,0:44:46.779
applause
0:44:46.779,0:44:59.882
postroll music
0:44:59.882,0:45:08.000
Subtitles created by c3subtitles.de[br]in the year 2021. Join, and help us!