WEBVTT
00:00:00.149 --> 00:00:14.099
prerol music
00:00:14.099 --> 00:00:23.599
Herald: So a very warm welcome to Thomas
Roth. He is a security researcher and his
00:00:23.599 --> 00:00:28.980
specialty is exploiting techniques and
reverse engineering and industrial
00:00:28.980 --> 00:00:37.590
security. And the talk today will be
about out SCADA the gateway to shell.
00:00:37.590 --> 00:00:45.364
applause
00:00:45.364 --> 00:00:50.390
And just one little notice: this talk
will be in English and will be translated
00:00:50.390 --> 00:00:53.580
in German as well.
Thomas Roth: Thank you.
00:00:53.580 --> 00:00:55.090
Herald: Yes.
00:00:55.090 --> 00:00:59.290
Thomas Roth: Awesome, thank you. OK, yeah.
Welcome to my talk gateway to shell. Who
00:00:59.290 --> 00:01:03.850
am I? He already introduced me, but still
my name is Thomas Roth. I'm a security
00:01:03.850 --> 00:01:08.850
researcher. I do a lot of low level
security, so a lot of ARM reverse
00:01:08.850 --> 00:01:13.321
engineering, Coldfire and so on. And
yeah, you can find me on Twitter or if you
00:01:13.321 --> 00:01:20.730
want to write me an email. Feel free to
send me one to thomas@stacksmashing.net.
00:01:20.730 --> 00:01:25.720
Before we start a short introduction to
the background of this talk, so, this year
00:01:25.720 --> 00:01:30.830
I did some SCADA penetration tests and I
found that while the PLC sensors
00:01:30.830 --> 00:01:35.210
are pretty well covered in the security
research area, I found that all the small
00:01:35.210 --> 00:01:39.810
devices that surround SCADA environments
are not really well covered. So basically
00:01:39.810 --> 00:01:44.060
we have the big Siemens PLCs and so on,
and there's a lot of research going on
00:01:44.060 --> 00:01:48.760
about them. But there are also a ton of
other small Ethernet devices involved in
00:01:48.760 --> 00:01:56.700
industrial networks that are not really
researched very well yet. And all devices
00:01:56.700 --> 00:02:00.570
that we're going to talk about are running
their latest respective firmware.
00:02:00.570 --> 00:02:07.310
Unfortunately, there will be zero days and
these are not theoretical attacks. Like if
00:02:07.310 --> 00:02:12.489
you go to Shodan or similar search engine,
you can find tens of thousands of these
00:02:12.489 --> 00:02:18.359
devices vulnerable and open in the
Internet. So let me give you a quick
00:02:18.359 --> 00:02:24.779
introduction into the terminology in
SCADA, because I know in the title I say
00:02:24.779 --> 00:02:29.079
SCADA, but actually it should be ICS,
which stands for industrial control
00:02:29.079 --> 00:02:36.900
systems, because basically ICS describes
the whole system from your supervision,
00:02:36.900 --> 00:02:42.069
the big room with all the big screens up
to your PLCs the sensors, the actors and
00:02:42.069 --> 00:02:46.930
so on that you will find in your
installation. And the term SCADA just
00:02:46.930 --> 00:02:50.959
describes the supervision and control
centers. So the big screens that you might
00:02:50.959 --> 00:02:55.040
know from movies and so on, where when the
bad guy comes, suddenly all the lights
00:02:55.040 --> 00:03:02.400
turn red. Then there's something called a
PLC, which is programable logic
00:03:02.400 --> 00:03:06.889
controller. It's basically like an
Arduino, just for industrial applications
00:03:06.889 --> 00:03:11.909
and they are really easy to program and
you can get them from Siemens or Schneider
00:03:11.909 --> 00:03:17.610
and so on and so forth. Then there is
something called an RTU, a remote terminal
00:03:17.610 --> 00:03:22.279
unit, which is a small device that
generally are, well, back in the day, was
00:03:22.279 --> 00:03:27.029
only used for monitoring. But today you
can actually program a lot of RTUs. So
00:03:27.029 --> 00:03:33.280
it's kind of a mix between a PLC and an
RTU. So it's basically a PLC in a remote
00:03:33.280 --> 00:03:41.269
location. Alrighty, to the actual topic,
industrial control gateways. So when you
00:03:41.269 --> 00:03:45.870
look at industrial control network, you'll
find that there are a lot of different
00:03:45.870 --> 00:03:50.281
sensors and actors and a lot of them speak
different protocols. So, for example, some
00:03:50.281 --> 00:03:56.470
might be serial, some might be IP, some
might be Modbus and so on. And so you can
00:03:56.470 --> 00:04:01.459
buy these small gateways that connect all
these different protocols to an IP
00:04:01.459 --> 00:04:06.539
network. So, for example, via Ethernet or
even via GPRS or Wi-Fi and so on. And I've
00:04:06.539 --> 00:04:11.829
seen them in almost any industrial
installation that I've seen. So, for
00:04:11.829 --> 00:04:16.440
example, they're used in power plants.
They are used in water dam control
00:04:16.440 --> 00:04:22.880
systems. They are used to control the
power grid and so on. And the security
00:04:22.880 --> 00:04:27.080
concept is, "Hey, but these devices are
airgapped!", so it doesn't matter really
00:04:27.080 --> 00:04:31.599
if they are vulnerable or not fully up to
date and so on, but that's not really true
00:04:31.599 --> 00:04:34.919
because a lot of these devices, while they
might be airgapped, they also have
00:04:34.919 --> 00:04:42.650
antennas and they are interconnected by a
ton of different wireless protocols such
00:04:42.650 --> 00:04:50.970
as Wi-Fi, LoRa or GSM or even proprietary
radio links. So, yeah, and even the
00:04:50.970 --> 00:04:54.940
case studies show that basically in this
case, you would have a monitoring network
00:04:54.940 --> 00:04:59.650
that's connected via the cellular network
to control the water mains and so on and
00:04:59.650 --> 00:05:04.889
check the pressure. Or even worse, they
even recommend that you connect the actors
00:05:04.889 --> 00:05:10.430
like valves and water level gotchas and so
on over GPS, which we know is not a secure
00:05:10.430 --> 00:05:17.830
protocol to do anything that could
be critical. Or you have stuff like
00:05:17.830 --> 00:05:24.160
water storage tanks that are controlled
via Wi-Fi and so on or even public in the
00:05:24.160 --> 00:05:33.349
Internet. So, yeah, these devices are
airgapped? Nope. So attacking in the field
00:05:33.349 --> 00:05:37.889
I already mentioned, if you go to
Shodan, you will find a ton of different
00:05:37.889 --> 00:05:42.970
devices reachable via the Internet
and even via GPS. So if you live
00:05:42.970 --> 00:05:49.090
close to, for example, a dam or something,
it's kind of interesting to look at an SDR
00:05:49.090 --> 00:05:52.290
or similar radio equipment to see what's
going over the airwaves, because you will
00:05:52.290 --> 00:05:59.090
find a ton of interesting stuff and
sometimes, you can even very trivially get
00:05:59.090 --> 00:06:03.930
a physical access to the in field devices
because they might just be in a white box
00:06:03.930 --> 00:06:07.569
somewhere hidden. And if you break into
it, you can pull out the SIM card and it
00:06:07.569 --> 00:06:12.419
will put you directly into the SCADA
network, if you're lucky. Don't do that,
00:06:12.419 --> 00:06:13.760
by the way.
00:06:13.760 --> 00:06:17.169
laughter
00:06:17.169 --> 00:06:24.539
So, yeah, let's let's hack some gateways.
So the equipment you will need to and
00:06:24.539 --> 00:06:29.039
everything in this talk was done on this
desk, just using these devices here, you
00:06:29.039 --> 00:06:33.000
really just need a laptop, you need an
oscilloscope or similar measurement
00:06:33.000 --> 00:06:37.320
equipment just to ensure that you don't
burn out your logic analyzer. You need a
00:06:37.320 --> 00:06:43.230
logic analyzer, a soldering iron, a
multimeter and a power supply. And that's
00:06:43.230 --> 00:06:48.360
really basically it, because you can hack
almost any embedded device that's using
00:06:48.360 --> 00:06:56.530
these devices and to find potential
targets. I have this kind of map where
00:06:56.530 --> 00:07:02.139
first try to understand, can I get the
firmware of the device or do I have to
00:07:02.139 --> 00:07:07.251
somehow, for example, use J-Tech to get it
out of the device? Can I actually buy the
00:07:07.251 --> 00:07:12.460
devices at a sensible price? Because some
of these devices cost like 600 € or so,
00:07:12.460 --> 00:07:18.360
and if you buy ten of them, that gets
expensive very quickly. And so, uh, I need
00:07:18.360 --> 00:07:24.360
to check eBay and see what devices can I
actually buy. And they should be half what
00:07:24.360 --> 00:07:29.449
current, because if you look at all the
devices, like 10 years old or so, they are
00:07:29.449 --> 00:07:33.940
completely broken. You don't even have to
look to start to look at their security.
00:07:33.940 --> 00:07:40.819
So, yeah, the first device that I that I
choose to really look at was the moxa
00:07:40.819 --> 00:07:50.689
W2150A, which is this small device, which
is also mounted on the board right here,
00:07:50.689 --> 00:07:54.319
mainly because I found the phone
was available and it looked like an
00:07:54.319 --> 00:07:58.669
interesting device because it has Wi-Fi
and so if I managed to break into it, I
00:07:58.669 --> 00:08:07.930
can jump an airgap potentially. And the
W2150A is just a simple device server. So
00:08:07.930 --> 00:08:14.939
you can connect any serial device, any
RS485 device simply to it and it will be
00:08:14.939 --> 00:08:20.669
exposed via Ethernet or even via Wi-Fi.
And you can download the firmware publicly
00:08:20.669 --> 00:08:29.270
and it's available on eBay relatively
cheap. So like 150 bucks or something. So
00:08:29.270 --> 00:08:33.290
I downloaded the firmware and I
looked at the entropy of the firmware and
00:08:33.290 --> 00:08:37.090
I immediately saw that the entropy is very
high, which means either it's very
00:08:37.090 --> 00:08:41.590
compressed or it's encrypted,
unfortunately, using a tool called
00:08:41.590 --> 00:08:46.510
binwalk, which is really useful for
looking into firmwares I saw that there's
00:08:46.510 --> 00:08:51.510
no compression detected. And so it was
very likely that this firmware image is
00:08:51.510 --> 00:08:59.940
encrypted. But I noticed on the Web page
that before you upgrade to version 2.0 or
00:08:59.940 --> 00:09:08.650
2.1 of the firmware, you must upgrade to
the firmware version 1.11. And I thought,
00:09:08.650 --> 00:09:13.540
that's interesting. Let's look at the
release notes for version 1.11. And it
00:09:13.540 --> 00:09:22.330
turns out that 1.11 adds the support for
the encrypted firmware. So I downloaded
00:09:22.330 --> 00:09:28.093
the one point eleven firmware and sure
enough, it's unencrypted. And if you've
00:09:28.093 --> 00:09:33.710
ever done anything with ARM before, if you
just look into a firmware hex dump, you
00:09:33.710 --> 00:09:39.930
can immediately recognize whether it's ARM
or not, because the first four bits of each
00:09:39.930 --> 00:09:45.580
instructions are the conditional bits
and those are almost always E. So if
00:09:45.580 --> 00:09:50.320
you see a Hexdump and roughly every fourth
byte is an E, you know, this is an ARM
00:09:50.320 --> 00:09:57.550
firmware and it's not encrypted or
anything else. And so, yeah, sure enough,
00:09:57.550 --> 00:10:02.640
I ran binwalk on this image. This time we
see there is a huge drop in entropy, which
00:10:02.640 --> 00:10:08.570
is the bootloader and so on, and then a
high entropy, which is basically the all
00:10:08.570 --> 00:10:15.280
the compressed filesystems and so on. And
binwalk was able to detect the SquashFS
00:10:15.280 --> 00:10:22.060
filesystem and extract it for me very,
very easy. And so my goal was to extract
00:10:22.060 --> 00:10:27.250
the firmware, find the firmware upgrade
code and somehow try to decipher the new
00:10:27.250 --> 00:10:34.250
firmware. And so I was browsing through
the files and sure enough, found the file
00:10:34.250 --> 00:10:40.810
that was helpfully called
libupgradeFirmware.so and if we look into
00:10:40.810 --> 00:10:45.010
the symbols, which they luckily didn't
remove or anything, there is a beautiful
00:10:45.010 --> 00:10:48.066
symbol called firmware decrypt.
00:10:48.066 --> 00:10:51.150
laughter
00:10:51.150 --> 00:10:56.430
So we load the whole thing into
disassembler and we see that
00:10:56.430 --> 00:11:03.870
there's some fancy XORing going
on in the bottom left corner. And I'm
00:11:03.870 --> 00:11:08.000
going to walk you through what's, what
exactly is happening in this code.
00:11:08.000 --> 00:11:13.310
So basically, first, there's a variable
called password loaded into the registar
00:11:13.310 --> 00:11:21.790
R2 and then a second count variable is
basically set and it starts looping and
00:11:21.790 --> 00:11:35.540
increasing always by four and goes through
this whole xor shebang and it turns out
00:11:35.540 --> 00:11:41.200
that this is the obfuscation method for
the AES Key. So, in password, in memory,
00:11:41.200 --> 00:11:45.950
we have an obfuscated key and we can be
obfusciated by just implementing the code
00:11:45.950 --> 00:11:53.890
we see here in C or in the emulator.
And sure enough, eventually this
00:11:53.890 --> 00:12:03.330
will be used as the key into the ECB 128
AES decryption. And so I implemented the
00:12:03.330 --> 00:12:08.760
whole thing in C, it was almost a copy
paste from the decompiler, so you can in
00:12:08.760 --> 00:12:14.500
IAD Pro, you just hit F5, copy the C code
at the bit, fix the memory offsets and so
00:12:14.500 --> 00:12:20.020
on. And you have the whole key obfuscation
method basically reverse engineered almost
00:12:20.020 --> 00:12:25.630
automatically. And so I compile it. And
sure enough, Moxa key extration, it turns
00:12:25.630 --> 00:12:31.200
out that the key is two eight eight seven
Conn seven five six four. I build a short
00:12:31.200 --> 00:12:39.260
script to decrypt the 2.1 firmware and
this time Binwalk finds all the files and
00:12:39.260 --> 00:12:41.740
we can start reverse engineering the
actual firmware.
00:12:41.740 --> 00:12:48.519
applause
00:12:48.519 --> 00:12:54.180
The scripts for this are available on my
github. I'll push the actual decrypts stuff
00:12:54.180 --> 00:12:59.810
after the talk because this is the first
time this has been released. And so after
00:12:59.810 --> 00:13:03.470
I was at this point, I knew that the
firmware is.. I can decrypted the firmware
00:13:03.470 --> 00:13:07.930
I can look into it. By the way, it's not
signed or anything. The only verification
00:13:07.930 --> 00:13:14.480
method is CRC32. And so at this point I
knew, OK, I can buy this device and
00:13:14.480 --> 00:13:19.980
start playing with it. And so I went to
eBay, I bought one. I got it. I screwed it
00:13:19.980 --> 00:13:24.140
open. And sure enough, there's an ARM
processor in there. It's an Freescale
00:13:24.140 --> 00:13:28.640
i.MX25, which is just a regular ARM
processor. It's like 400 MHz or something,
00:13:28.640 --> 00:13:34.880
I don't know. And I started probing all
the all the small pins inside of the
00:13:34.880 --> 00:13:43.040
device to try to find JTAG or serial
or anything. And so I actually hooked up
00:13:43.040 --> 00:13:47.320
my power supply to foot pedal so that I
can probe and just press with my foot to
00:13:47.320 --> 00:13:54.250
reset the device. And sure enough, I found
that there's a full serial console
00:13:54.250 --> 00:14:00.660
available inside of the device on these
pins. And if you boot the device, it even
00:14:00.660 --> 00:14:05.160
tells you, please press enter to activate
this console, and so you do that and you
00:14:05.160 --> 00:14:07.463
are root on the device.
00:14:07.463 --> 00:14:14.820
applause
00:14:14.820 --> 00:14:18.660
So that's kind of cool, but that means
that you require physical access, so
00:14:18.660 --> 00:14:23.530
that's not really a vulnerability, but
it's very nice to have when doing security
00:14:23.530 --> 00:14:29.420
research because it means you can suddenly
debug all the code on there. And so if you
00:14:29.420 --> 00:14:35.050
write an exploit, you can just touch GDB
to the binary and start very, very simply,
00:14:35.050 --> 00:14:40.420
writing the exploit. So at this point,
I was trying to look at the available
00:14:40.420 --> 00:14:46.010
services on the device. So for example,
there is a web interface, there's a
00:14:46.010 --> 00:14:52.530
proprietary configuration protocol,
there's telnet, there's snmp, there is a
00:14:52.530 --> 00:14:58.910
serial driver protocol and so on. And I
started looking at the web interface and
00:14:58.910 --> 00:15:03.740
there was cross site scripting that was
Cross site request forgery, there was
00:15:03.740 --> 00:15:07.440
insecure authentication where they
basically hash on the client. So they have
00:15:07.440 --> 00:15:12.520
some JavaScript that hashes your password
and then locks you in. Then there's a
00:15:12.520 --> 00:15:17.720
command injection which lets you execute
code as root, there are stack overflows.
00:15:17.720 --> 00:15:23.675
And just a week ago there was a zero day
released for the web server. So yeah, demo
00:15:23.675 --> 00:15:36.820
time. So just let me open up the Moxa
Pitch right here. And so this one is
00:15:36.820 --> 00:15:41.240
authenticated, so I'll just enter the
default password, which, by the way, in
00:15:41.240 --> 00:15:46.050
the field will 90 percent of the time
these devices will be configured with
00:15:46.050 --> 00:15:54.950
default credentials. But still, so, if we
just start browsing through this thing and
00:15:54.950 --> 00:16:00.140
go to the basic settings, we can start
with a simple cross site scripting just in
00:16:00.140 --> 00:16:08.810
the device name. One sec, so just for
example we just paste in some JavaScript.
00:16:08.810 --> 00:16:15.017
Submit the whole thing, and hello 34c3.
00:16:15.017 --> 00:16:19.560
applause
00:16:19.560 --> 00:16:23.770
I know what you're thinking, like cross
site scripting, come on, that's not a
00:16:23.770 --> 00:16:28.530
vulnerability, that's just nothing. So
let's look at the ping test that's
00:16:28.530 --> 00:16:33.910
integrated into this device. And funilly,
a different device from Moxa that runs an
00:16:33.910 --> 00:16:39.570
entirely different firmware had the same
vulnerability in the past. But if I just
00:16:39.570 --> 00:16:46.390
paste in my ping, so my IP address, a
semicolon and then, for example, I cut
00:16:46.390 --> 00:16:51.970
/etc/passwd and activate enter.
Here we go.
00:16:51.970 --> 00:17:00.060
applause
00:17:00.060 --> 00:17:08.199
Kind of funny, but, yes, for sure not
intended. All righty, but I know what
00:17:08.199 --> 00:17:12.740
you're thinking, right, these are
authenticated bugs in the web interface,
00:17:12.740 --> 00:17:17.460
so we need something unauthenticated. We
want something that's like cool and a real
00:17:17.460 --> 00:17:23.420
exploit. Right? And so I decided to look
at the.. this custom TCP protocol, which
00:17:23.420 --> 00:17:29.430
runs on Port 4900. And my goal was to
reverse engineer the whole protocol and
00:17:29.430 --> 00:17:34.030
build a fuzzer for it, to find
vulnerabilities, that turned out not to be
00:17:34.030 --> 00:17:40.990
necessary. So during some testing, I just
sent a lot of bytes onto this thing and
00:17:40.990 --> 00:17:49.140
enabled crash debugging via the serial
console. And sure enough, it crashed and
00:17:49.140 --> 00:17:58.740
put my program countdown right to
0x41414140. Wonderful. Thank you, Moxa.
00:17:58.740 --> 00:18:04.370
applause
00:18:04.370 --> 00:18:21.550
So, Demo time. So let's increase the size
of this a bit. So I built a small script.
00:18:21.550 --> 00:18:34.490
Just called moxa_pown and I'll just supply
the IP address to it. Let's see. Opening a
00:18:34.490 --> 00:18:43.620
second shell to connect to it via netcat.
Here we go, we have a root shell on the
00:18:43.620 --> 00:18:44.600
device.
00:18:44.600 --> 00:18:54.263
applause
00:18:54.263 --> 00:19:01.540
So, yeah, that was the Moxa w21508,
basically rolls of the tongue. And so the
00:19:01.540 --> 00:19:08.690
next device I decided to look at was the
Advantech EKI-1522 which you can find
00:19:08.690 --> 00:19:17.460
right here. And it's, again, just a simple
serial device server this time without
00:19:17.460 --> 00:19:21.410
Wi-Fi, even though they are available with
Wi-Fi. It comes with two Ethernet ports
00:19:21.410 --> 00:19:26.060
two serial ports and so on. And I
basically followed the same steps again.
00:19:26.060 --> 00:19:31.170
So I looked at the.. I downloaded the
firmware. I looked at the edit using
00:19:31.170 --> 00:19:35.800
binwalk. And this time we see almost no
entropy. So there is.. this guy is
00:19:35.800 --> 00:19:40.280
basically completely unencrypted. And
again, we saw some ARM 32 bit it runs a
00:19:40.280 --> 00:19:51.010
Linux kernel, 2.6.31 and a BOA Web server
where the last update was in 2005. And the
00:19:51.010 --> 00:19:56.770
firmware, I think, is from 2017. So these
are kind of outdated. And I found
00:19:56.770 --> 00:20:01.230
during the initial analysis just of the
firmware that the main binary to look at
00:20:01.230 --> 00:20:07.180
will be this edgserver binary. And so I
loaded it into IDA pro and looked at the
00:20:07.180 --> 00:20:12.780
different things that calls. And there
are a lot of calls to functions like
00:20:12.780 --> 00:20:18.340
string copy, to system, to sprintf and so
on that are generally kind of considered
00:20:18.340 --> 00:20:25.660
unsecure. And sure enough, I am doing
static analysis. I found that there's some
00:20:25.660 --> 00:20:33.630
code for sending an email as an alert, for
example, when the system reboots. And
00:20:33.630 --> 00:20:39.250
the full command invocation is mailx -s
blah blah blah, and we have control over
00:20:39.250 --> 00:20:46.160
some parts in the string because we can
configure the two address in the UI. And if
00:20:46.160 --> 00:20:51.040
we look at what's happening
here, it basically just sets up this
00:20:51.040 --> 00:20:56.500
format string. Then it goes to include the
subject and then it gets some arguments
00:20:56.500 --> 00:21:04.260
from the stack and basically calls
into system. And so there's no filtering
00:21:04.260 --> 00:21:09.930
going on at all. So we have an unfiltered
part of the system, invocation, code
00:21:09.930 --> 00:21:15.380
execution. And this was before I had the
device in my hand. And this is kind of a
00:21:15.380 --> 00:21:19.470
funny story because I first bought because
it was just 40 bucks, I bought this
00:21:19.470 --> 00:21:24.770
device, which in the firmware has the same
bug, but the mail functionality is broken,
00:21:24.770 --> 00:21:33.780
so I couldn't test it. So I had to go to
eBay again, buy another one and buy the
00:21:33.780 --> 00:21:38.950
bigger one. And so I ordered the bigger
one on eBay. Looks like this. It comes
00:21:38.950 --> 00:21:45.660
with a Cavium CNS C.P.U. It has JTAG
exposed on the bottom there and serial
00:21:45.660 --> 00:21:50.940
console is available again without any
authentication. So beautiful. You just
00:21:50.940 --> 00:21:57.809
connect your bus pirate or your UART
adapter to it and you have full serial
00:21:57.809 --> 00:22:06.740
console. So, again, we had to look at
finding vulnerabilities for this device
00:22:06.740 --> 00:22:11.559
and there, again, a ton of different
services, there's like a Web interface
00:22:11.559 --> 00:22:15.670
available. There is a proprietary
configuration protocol that's based on
00:22:15.670 --> 00:22:22.760
UDP. There is Telnet, there's snmp,
there's a serial driver protocol and so
00:22:22.760 --> 00:22:28.380
on. And again, looked at the website and
again, cross site scripting cross side
00:22:28.380 --> 00:22:33.280
request forgery, command injection, broken
authentication, which basically if you log
00:22:33.280 --> 00:22:38.710
in from one computer, it uses, I think
http digest authentication, you can
00:22:38.710 --> 00:22:42.690
connect from a completely different
computer and it doesn't ask for a
00:22:42.690 --> 00:22:49.700
password. I don't know why that is, but..
Yeah. So I was thinking I was doing
00:22:49.700 --> 00:22:52.130
something wrong, but it turned out it was
just broken.
00:22:52.130 --> 00:22:54.855
laughter
00:22:54.855 --> 00:23:03.170
So, yeah, and there's, again, a stack
overflow in another protocol. So I guess,
00:23:03.170 --> 00:23:13.980
again, demo time. Let's first look at
the device itself, so, you know the
00:23:13.980 --> 00:23:22.830
password, firstly, we have a nice device
description here. This is just a basic web
00:23:22.830 --> 00:23:29.320
interface. Right. And we can, again, just
copy in some basic JavaScript
00:23:29.320 --> 00:23:38.620
hit the save button. Reload and there we
go, cross site scripting yet again, OK,
00:23:38.620 --> 00:23:49.129
again, not really interesting. Right. So,
um, let's look at the stack overflow.
00:23:49.129 --> 00:24:04.070
Again, I have a small script advantech_pown.
For the IP there. And we have netcat
00:24:04.070 --> 00:24:12.090
running on there. Sure enough, there we
go, that's root on the Advantech device
00:24:12.090 --> 00:24:13.810
again, via stack overflow.
00:24:13.810 --> 00:24:25.516
applause
00:24:25.516 --> 00:24:31.500
Yeah, so two of three devices have
basically broken already. Let's look
00:24:31.500 --> 00:24:38.150
at the next one. This one is a Lantronix
EDS2100. And this one is kind of
00:24:38.150 --> 00:24:43.770
interesting because it's not ARM. I
normally I almost exclusively do ARMs. So
00:24:43.770 --> 00:24:48.500
this one was kind of interesting. And this
device, which is mounted somewhere right
00:24:48.500 --> 00:24:57.390
here. Yeah. This device comes with a
serial to ethernet secure device server.
00:24:57.390 --> 00:25:01.929
It has two serial ports. It has
Ethernet and you can buy it in two
00:25:01.929 --> 00:25:07.830
variants. One comes with Linux and one is
Evolution OS, which is I guess, a
00:25:07.830 --> 00:25:14.880
proprietary operating system from
Lantronics. And I'm using the EvolutionOS
00:25:14.880 --> 00:25:22.120
variant in this talk. Looking at the
firmware it turns out it's unencrypted and
00:25:22.120 --> 00:25:28.240
it's coldfire architecture, which I've
never done really anything with before,
00:25:28.240 --> 00:25:32.630
and there are no obvious external software
components. So if you go through this,
00:25:32.630 --> 00:25:37.440
through the firmware, you'll find there's
an SSH implementation, there's an SSL
00:25:37.440 --> 00:25:42.810
implementation, but it's not openSSL and
it's not anything very well known. And the
00:25:42.810 --> 00:25:47.490
same is true for the web server and so on.
It's not really anything that's well
00:25:47.490 --> 00:25:56.500
known. And this time, while probing
the device, I did not really find anything
00:25:56.500 --> 00:26:01.580
interesting in terms of serial consoles or
so, but it just found a potential debugger
00:26:01.580 --> 00:26:05.730
port, but it didn't have a fitting
debugger unfortunately. The CPU is from
00:26:05.730 --> 00:26:14.760
NXP runs at 160MHz or something. Yeah.
This time we actually have a web
00:26:14.760 --> 00:26:21.660
interface, we have Telnet SSL and it even
has a file system, so you have like FTP
00:26:21.660 --> 00:26:26.210
and TFTP which allows you to download the
configuration, upload the configuration
00:26:26.210 --> 00:26:30.980
and so on. And it's kind of hard to secure
it correctly because there are so many
00:26:30.980 --> 00:26:37.000
protocols and it's not really clear what's
set up by default. But yeah, you get
00:26:37.000 --> 00:26:44.350
the idea. And this time the web interface
was surprisingly secure. So there was no
00:26:44.350 --> 00:26:50.230
cross site scripting. There was no command
injection, because there's also not really
00:26:50.230 --> 00:26:55.440
a shell that you could execute commands
into. But I still found some stuff.
00:26:55.440 --> 00:27:01.540
One is the configuration injection, which
allows you basically to change the format
00:27:01.540 --> 00:27:06.630
of the configuration using a different
field. And I found an authentication
00:27:06.630 --> 00:27:11.970
bypass, so I was able to write a small
piece of code that takes a while and then
00:27:11.970 --> 00:27:23.650
completely removes the password from the
device. Demo time. So if we connect to the
00:27:23.650 --> 00:27:29.750
Lantronics device, it will currently ask
for a password, which in theory we don't
00:27:29.750 --> 00:27:44.580
have. Let's clean up here a bit. I know
it's just. And let's run Lantronix_pown,
00:27:44.580 --> 00:27:51.300
oh, that was fast. That worked. Yeah, sure
enough, the password is gone.
00:27:51.300 --> 00:27:59.980
applause
00:27:59.980 --> 00:28:07.230
Awesome. To be honest, I didn't expect the
demos to go so smoothly, so I put in an
00:28:07.230 --> 00:28:13.710
hour for the talk for this went very well
so far, so that's good. So before we
00:28:13.710 --> 00:28:22.170
finish already, some other devices are
even worse. So, for example, as I
00:28:22.170 --> 00:28:26.549
mentioned, I bought some other devices,
for example, this Advantaech device and
00:28:26.549 --> 00:28:31.610
this Moxa device and this Lantronix
device, which are basically the
00:28:31.610 --> 00:28:38.940
predecessors of the other devices. And
those guys are really interesting to look
00:28:38.940 --> 00:28:45.850
at, one could say. So, some of those are
running eCos, which is an embedded Linux
00:28:45.850 --> 00:28:52.390
platform, which was last released in 2009,
and some devices run a Linux kernel with
00:28:52.390 --> 00:28:57.570
the 2.4 version and you see Linux without
any memory protection whatsoever. So even
00:28:57.570 --> 00:29:03.640
if they, so even a small stack overflow in
one of the userspace applications gives
00:29:03.640 --> 00:29:08.640
you full root access to the device because
you can directly exploit the kernel and
00:29:08.640 --> 00:29:12.840
there are unfixed public vulnerabilities.
So in the first penetration test that I
00:29:12.840 --> 00:29:19.290
did, that included actually this device
and Moxa and part of a small one. I found
00:29:19.290 --> 00:29:25.170
that using SNMPWwalk, it gives you back
the administration password via SNMP.
00:29:25.170 --> 00:29:26.780
laughing
00:29:26.780 --> 00:29:31.500
And so I went online. I tried to report
it. And it turns out it's well known
00:29:31.500 --> 00:29:34.160
there's a metasploit module for this
00:29:34.160 --> 00:29:36.830
laughing
00:29:36.830 --> 00:29:41.690
and it's unfixed, OK? And these devices
are still in support. So I don't know why
00:29:41.690 --> 00:29:50.950
the vendor is not patching this. Yeah. So
the summary with trivial vulnerabilities
00:29:50.950 --> 00:29:56.520
in most devices, or at least all that I've
looked at, there are no security
00:29:56.520 --> 00:30:00.571
mitigations whatsoever. So they don't even
enable like the compiler flags that you
00:30:00.571 --> 00:30:05.850
just set and then you have at least some
kind of stack protection and some like
00:30:05.850 --> 00:30:11.070
stack cookies and whatnot. And some
vendors are really bad at responding to
00:30:11.070 --> 00:30:18.429
vulnerability reports. So, yeah, I'm not
going to name the vendor, but not even, on
00:30:18.429 --> 00:30:22.180
Twitter I asked them to please give me a
security contact and they responded,
00:30:22.180 --> 00:30:26.840
please use our contact form. I said I did,
three times. I send you emails, you're not
00:30:26.840 --> 00:30:30.600
responding to me. And so they stopped
responding to me on Twitter too.
00:30:30.600 --> 00:30:40.809
laughing
applause
00:30:40.809 --> 00:30:47.200
So how to mitigate? Well, the only way
that I would see to mitigate against this,
00:30:47.200 --> 00:30:53.380
and I'm more on the deconstructive side of
the story, is defense in depth. So never
00:30:53.380 --> 00:30:56.850
directly expose any of these devices to
the Internet, even if they say they
00:30:56.850 --> 00:31:02.490
support VPN, even if they say they are a
secure device of whatever, just don't do
00:31:02.490 --> 00:31:08.780
it. Get a real VPN gateway and make sure
that you never rely on a single level of,
00:31:08.780 --> 00:31:16.169
for example, encryption. So, for example,
WPA2 was broken by the crack attack and
00:31:16.169 --> 00:31:20.799
they actually released a patch for it
after two months. And these are these are
00:31:20.799 --> 00:31:26.370
still two months where you are exposed to
vulnerability on your potentially mission-
00:31:26.370 --> 00:31:33.760
critical system. Also never use GPRS for
these devices without VPN because it just,
00:31:33.760 --> 00:31:41.480
it will go wrong. Okay. Yeah, thank you. I
guess now we have time for Q&A. Thank you
00:31:41.480 --> 00:31:43.282
all for coming.
00:31:43.282 --> 00:31:49.170
applause
00:31:49.170 --> 00:31:57.990
Herald: Thank you very much for the talk.
So we have very much time for Q&A. So
00:31:57.990 --> 00:32:03.980
please line up to the microphones and we
have someone at microphone 4 already.
00:32:03.980 --> 00:32:09.220
Mic 4: Yes, hello. Hello. Thanks for your
talk. This is.. obviously this is a
00:32:09.220 --> 00:32:14.792
problem. This is a part of the bigger
problem of security in IT. Right. In
00:32:14.792 --> 00:32:18.950
anything related to any kind of
technology. And this is only going to go
00:32:18.950 --> 00:32:25.230
worse with time, right. Internet of shit,
internet of things and so and so on, so
00:32:25.230 --> 00:32:31.740
forth. So my question is, you gave some
ideas how to mitigate this in this very
00:32:31.740 --> 00:32:36.540
specific area that use VPN, et cetera, et
cetera. But my question is, so hacker
00:32:36.540 --> 00:32:42.462
community is not very, let's say,
interested in regulation. Right? And when
00:32:42.462 --> 00:32:46.610
we see, when we see a government trying to
do something with technology that usually
00:32:46.610 --> 00:32:51.580
goes bad, we have this idea in our head
that, OK, this can only go like this can
00:32:51.580 --> 00:32:56.741
only go bad. Right. But so my question is:
do you think that perhaps there is some
00:32:56.741 --> 00:33:00.810
space for regulation here?
T: There's definitely space for
00:33:00.810 --> 00:33:07.450
regulation, but I think regulation does
not solve the underlying technical issues.
00:33:07.450 --> 00:33:13.611
So these devices, it's 2017 and these
devices are using C-code. I think that's
00:33:13.611 --> 00:33:18.580
just asking for trouble, basically. And so
we really need to see this shift, even in
00:33:18.580 --> 00:33:22.690
the embedded world, to switch to memory
safe languages, for example Rust or
00:33:22.690 --> 00:33:28.129
something similar, and really to stop
using C in this kind of context. I don't
00:33:28.129 --> 00:33:35.729
think there's anyone who can .. Thank you.
applause
00:33:35.729 --> 00:33:39.072
T: But there's definitely space for
regulation.
00:33:39.072 --> 00:33:43.173
Herald: Since there was a question from
the Internet.
00:33:43.173 --> 00:33:47.530
Signal Angel: OK, yeah, the Internet wants
to know why you are not naming the bad
00:33:47.530 --> 00:33:51.980
vendor, because it looks like it's the
only option basically if they don't
00:33:51.980 --> 00:33:57.990
respond to you. Let's say I asked them on
Twitter and my Twitter is right there. And
00:33:57.990 --> 00:34:02.640
if you click on Tweets and Replies..
laughter
00:34:02.640 --> 00:34:05.850
Signal Angel: Yeah, somebody just posted
the link on IRC.
00:34:05.850 --> 00:34:10.869
laughter
T: I did not name them, just for the
00:34:10.869 --> 00:34:13.309
record.
laughter
00:34:13.309 --> 00:34:17.029
applause
Herald: So we have a question from
00:34:17.029 --> 00:34:23.369
microphone number 2.
Mic 2: So you shown an exploit for the
00:34:23.369 --> 00:34:29.559
last device that disabled authentication.
What did you use to achieve that?
00:34:29.559 --> 00:34:35.529
T: So this one is unpatched and not yet
fixed, so I would rather not disclose the
00:34:35.529 --> 00:34:38.720
details yet.
Mic 2: OK.
00:34:38.720 --> 00:34:42.919
Herald: Microphone number 1, please.
Mic 1: I wonder if you've also been
00:34:42.919 --> 00:34:47.729
looking at a building automation system,
control systems, or just industrial
00:34:47.729 --> 00:34:53.510
automation control systems?
T: So you can use these devices basically
00:34:53.510 --> 00:35:00.609
wherever you want. And I think some of the
Moxa ones are used in home automation. But
00:35:00.609 --> 00:35:05.920
I've looked at I guess Crestron, it's
called? But not in a lot of detail. So I'm
00:35:05.920 --> 00:35:09.509
more on the industrial side at the moment.
Mic 1: Thanks.
00:35:09.509 --> 00:35:15.079
Herald: Microphone number 3.
Mic 3: Any field experience or even just
00:35:15.079 --> 00:35:21.259
opinions on using industrial strength
Raspberry Pi hardware with community
00:35:21.259 --> 00:35:25.559
supported Linux distributions or something
like OpenBC whatever on them.
00:35:25.559 --> 00:35:30.869
T: Yeah. So I guess the big trouble there
is support, right? There are some, some
00:35:30.869 --> 00:35:34.579
German companies and so on that provide
support for industrial Raspberry Pis and
00:35:34.579 --> 00:35:40.789
even like nice casing and so on. But I'm
not sure if really Raspberry Pi is the way
00:35:40.789 --> 00:35:45.240
to go here. I think there are
boards that are.. the problem is not the
00:35:45.240 --> 00:35:49.720
underlying stack, right? It's not the
hardware. Really, that's the issue. It's
00:35:49.720 --> 00:35:55.950
the software. And you will have the same
issues on on the Raspberry Pi. So, yeah, I
00:35:55.950 --> 00:36:00.880
guess you could buy these devices, which
are like industrial grade shockproof and
00:36:00.880 --> 00:36:07.460
whatnot, and put some Linux on it and
do it better. But I don't think that
00:36:07.460 --> 00:36:11.650
the hardware or platform will
change anything at the moment.
00:36:11.650 --> 00:36:16.319
Herald: There is another question from
microphone number 4.
00:36:16.319 --> 00:36:21.749
Mic 4: Hi, more a social question, did you
get in contact with any development team,
00:36:21.749 --> 00:36:25.849
software development team in any of these
companies, or might it be that there is no
00:36:25.849 --> 00:36:33.080
one behind the emails and everything?
T: So I guess some of these companies are
00:36:33.080 --> 00:36:37.349
really so big, that they don't reply to
you if you don't have a support contract
00:36:37.349 --> 00:36:45.049
with them. But, for example, the support
of the ones that are not on my Twitter is
00:36:45.049 --> 00:36:49.730
kind of decent when it comes to two
security reports. And so my next steps
00:36:49.730 --> 00:36:57.220
will be to go via the ICS Cert, but, you
know, to report them. So, yes, there are
00:36:57.220 --> 00:37:03.737
development teams that will get in contact
with you, just not from all vendors.
00:37:03.737 --> 00:37:06.670
Herald: Thank you. We have another
question from the Internet.
00:37:06.670 --> 00:37:13.960
Signal Angel: Hello? OK. The Internet
wants to know what to do about, because
00:37:13.960 --> 00:37:18.259
there are a lot of old devices in the
field, how do you propose a vendor should
00:37:18.259 --> 00:37:24.200
deal with legacy devices and updates?
T: Yeah, so keeping legacy devices
00:37:24.200 --> 00:37:29.680
supported is very expensive because, for
example, if you buy a Qualcomm chip, they
00:37:29.680 --> 00:37:35.089
will eventually drop support for the Linux
kernel for it and so on. But if you buy
00:37:35.089 --> 00:37:39.619
like a Freescale automotive chip, they
guarantee you a certain time of support.
00:37:39.619 --> 00:37:43.490
But then you actually have to invest the
money to regularly provide the updates and
00:37:43.490 --> 00:37:48.859
ensure that your devices are secure. The
problem is that the lifetime of industrial
00:37:48.859 --> 00:37:55.470
installations currently is much larger
than the lifetime of this processors' supports
00:37:55.470 --> 00:38:00.819
and so on. So I guess we'll have to get
used to upgrading our hardware regularly
00:38:00.819 --> 00:38:07.400
or switch to, or figure out a different
way of deploying secure software onto
00:38:07.400 --> 00:38:11.259
them. But I really think the underlying
problem is, that we are still using
00:38:11.259 --> 00:38:16.229
memory unsafe languages. And I guess the
fact that there's cross site scripting
00:38:16.229 --> 00:38:20.150
just shows that there's no security
awareness really at those vendors
00:38:20.150 --> 00:38:29.395
whatsoever. At some of the vendors.
Herald: So, microphone number 2, please.
00:38:29.395 --> 00:38:34.349
Mic 2: I was wondering, you mentioned that
some of these facilities use GPRS.
00:38:34.349 --> 00:38:36.390
T: Yeah.
Mic 2: Do you know if they have mostly
00:38:36.390 --> 00:38:40.749
their own closed infrastructure, or if
they're using general consumer telecom
00:38:40.749 --> 00:38:44.849
stuff?
T: So they will use commercial
00:38:44.849 --> 00:38:50.479
networks mostly, and then they have custom
EPNs which have an IPSec tunnel or
00:38:50.479 --> 00:38:55.700
something similar to their premises. But
there's also there's also a company that
00:38:55.700 --> 00:39:02.589
sells industrial control SIM cards
which give you a public IP and you don't
00:39:02.589 --> 00:39:08.100
want to search on Shodan for that vendor.
Mic 2: Yeah. Thank you.
00:39:08.100 --> 00:39:11.050
Herald: There is a question from
microphone number 3.
00:39:11.050 --> 00:39:14.999
Mic 3: Hi there, isn't economics meant to
solve some of these problems? We're not
00:39:14.999 --> 00:39:20.359
talking about dirt cheap devices. How
surely at 300 bucks you should better have
00:39:20.359 --> 00:39:24.539
someone who's read security one and one.
How long before a large organization gets
00:39:24.539 --> 00:39:28.200
the result of their security audit and
goes to the aforementioned vendors and
00:39:28.200 --> 00:39:32.960
says, provide us something that's not
trivially hackable, otherwise we stop
00:39:32.960 --> 00:39:37.839
buying your rubbish?
T: Well, I mean, it's the same in all of
00:39:37.839 --> 00:39:45.329
IT, right? So everything has
vulnerabilities. And yes, there should be
00:39:45.329 --> 00:39:50.400
market pressure. But that's why I'm trying
to raise awareness for the issues that
00:39:50.400 --> 00:39:53.270
these devices have.
Mic 3: Thanks.
00:39:53.270 --> 00:39:55.729
Herald: There's another question from the
Internet.
00:39:55.729 --> 00:40:01.339
Signal Angel: Yep. The Internet wants to
know how and if it's a good idea to raise
00:40:01.339 --> 00:40:06.549
the level of awareness in public, because
they think it's a good approach to make
00:40:06.549 --> 00:40:11.869
people, the public know that, well,
infrastructure in the cities is at risk.
00:40:11.869 --> 00:40:16.000
T: Uh, sorry. Could you repeat the first
part of the question?
00:40:16.000 --> 00:40:21.339
Signal Angel: Yeah. They want to know how
to raise awareness for this in the public?
00:40:21.339 --> 00:40:27.789
T: Good question. I guess we need some
news articles or something about this in
00:40:27.789 --> 00:40:32.800
regular paper, but I personally think it's
just an accident waiting to happen. So
00:40:32.800 --> 00:40:37.999
eventually someone will turn off the
lights in a city or wherever, will open a
00:40:37.999 --> 00:40:44.773
flood valve or something. And that's when
the awareness will start.
00:40:44.773 --> 00:40:47.813
Herald: There's another question from
microphone number 4.
00:40:47.813 --> 00:40:51.680
Mic 4: OK, for what kind of industrial
processes are these devices you just
00:40:51.680 --> 00:40:57.108
demoed used?
T: So I've seen them in power utility. I
00:40:57.108 --> 00:41:02.350
know they're used in water dam
control systems. They are used and in
00:41:02.350 --> 00:41:07.039
serial connecting a CNC machine to the
network, they are used in connecting all
00:41:07.039 --> 00:41:10.690
kinds of stuff. Because if you have a big
plant, you have a ton of different
00:41:10.690 --> 00:41:15.719
sensors. So you might, you might need the
water level sensor. And for whatever
00:41:15.719 --> 00:41:20.680
reason, you only can get it with a modbus
and then you need to convert the modbus to
00:41:20.680 --> 00:41:25.119
TCP and then you need one of these
gateways. And so, I've seen in one
00:41:25.119 --> 00:41:28.529
cabinet, 20 of them. So they're
really used a lot I guess.
00:41:28.529 --> 00:41:31.869
Mic 4: OK, thank you. I just retweeted
your tweet to Star Alliance.
00:41:31.869 --> 00:41:37.979
T: Huh. laughs Thank you. laughs
Herald: So there's another question from
00:41:37.979 --> 00:41:41.260
the Internet.
Signal Angel: Yeah, the Internet wants to
00:41:41.260 --> 00:41:50.749
know if you did any research on MQTT
for example from like Beckhoff uses?
00:41:50.749 --> 00:41:54.489
T: I actually talked to someone who
recommended me to look at Beckhoff
00:41:54.489 --> 00:41:58.249
yesterday, but I've not looked at them
whatsoever yet.
00:41:58.249 --> 00:42:01.900
Herald: And there's another question from
microphone 3.
00:42:01.900 --> 00:42:07.450
Mic 3: OK, could you show the Moxa web
panel, because I would like to double
00:42:07.450 --> 00:42:16.619
check, which proves that they and they
would like you to see their Web page. And
00:42:16.619 --> 00:42:24.050
I think this browser isn't very secure.
T: OK, let's take a look.
00:42:24.050 --> 00:42:29.160
Mic 3: Yeah, and under gohead the
webserver small print.
00:42:29.160 --> 00:42:41.527
laughter
Herald: Nice finding.
00:42:41.527 --> 00:42:47.859
T: That's probably the issue here.
laughs
00:42:47.859 --> 00:42:55.658
Herald: Are there any more questions? Any
questions from the Internet?
00:42:55.658 --> 00:43:02.009
Signal Angel: The internet wants to know
how a memory safe language would prevent
00:43:02.009 --> 00:43:08.750
the authentication bypasses you showed?
T: Not one would not be protected against
00:43:08.750 --> 00:43:13.130
but it protects against a ton of other
stuff. It's just one example of where the
00:43:13.130 --> 00:43:18.420
industry needs to change. We need to stop
using memory unsafe languages. We need to
00:43:18.420 --> 00:43:23.910
start really thinking about security
design from the start, and we must not in
00:43:23.910 --> 00:43:28.319
2017, there's no excuse for having cross
site scripting or anything on the web
00:43:28.319 --> 00:43:35.720
page. That's also if we in the
Lantronics website, if you click logout,
00:43:35.720 --> 00:43:39.479
it tells you logout is not supported in
your browser.
00:43:39.479 --> 00:43:43.290
laughter
T: Probably because I'm not using Internet
00:43:43.290 --> 00:43:48.130
Explorer five.
Herald: So there's another question from
00:43:48.130 --> 00:43:53.239
microphone number 3.
Mic 3: Any remote part of the exploit
00:43:53.239 --> 00:43:57.750
where you did a buffer
overflow - I think.
00:43:57.750 --> 00:44:01.490
T: Yeah?
Mic 3: What I'm wondering is, are
00:44:01.490 --> 00:44:07.180
there.. isn't it like very standard to
have ALSR on these devices?
00:44:07.180 --> 00:44:10.239
T: No! laughts It should be, but it
isn't.
00:44:10.239 --> 00:44:16.199
Mic 3: Okay. Thank you though. That was
pretty much my question.
00:44:16.199 --> 00:44:23.428
Herald: Is there another question from the
Internet? It doesn't seem like it?
00:44:23.428 --> 00:44:36.052
Signal Angel: So, one just came in, OK, if
you want to hear it. Ok, nope.
00:44:36.052 --> 00:44:41.329
laughter
Herald: So, all right, give a very warm
00:44:41.329 --> 00:44:43.329
applause to Thomas Roth again!
00:44:43.329 --> 00:44:46.779
applause
00:44:46.779 --> 00:44:59.882
postroll music
00:44:59.882 --> 00:45:08.000
Subtitles created by c3subtitles.de
in the year 2021. Join, and help us!