< Return to Video

34C3 - SCADA - Gateway to (s)hell

  • 0:00 - 0:14
    prerol music
  • 0:14 - 0:24
    Herald: So a very warm welcome to Thomas
    Roth. He is a security researcher and his
  • 0:24 - 0:29
    specialty is exploiting techniques and
    reverse engineering and industrial
  • 0:29 - 0:38
    security. And the talk today will be
    about out SCADA the gateway to shell.
  • 0:38 - 0:45
    applause
  • 0:45 - 0:50
    And just one little notice: this talk
    will be in English and will be translated
  • 0:50 - 0:54
    in German as well.
    Thomas Roth: Thank you.
  • 0:54 - 0:55
    Herald: Yes.
  • 0:55 - 0:59
    Thomas Roth: Awesome, thank you. OK, yeah.
    Welcome to my talk gateway to shell. Who
  • 0:59 - 1:04
    am I? He already introduced me, but still
    my name is Thomas Roth. I'm a security
  • 1:04 - 1:09
    researcher. I do a lot of low level
    security, so a lot of ARM reverse
  • 1:09 - 1:13
    engineering, Coldfire and so on. And
    yeah, you can find me on Twitter or if you
  • 1:13 - 1:21
    want to write me an email. Feel free to
    send me one to thomas@stacksmashing.net.
  • 1:21 - 1:26
    Before we start a short introduction to
    the background of this talk, so, this year
  • 1:26 - 1:31
    I did some SCADA penetration tests and I
    found that while the PLC sensors
  • 1:31 - 1:35
    are pretty well covered in the security
    research area, I found that all the small
  • 1:35 - 1:40
    devices that surround SCADA environments
    are not really well covered. So basically
  • 1:40 - 1:44
    we have the big Siemens PLCs and so on,
    and there's a lot of research going on
  • 1:44 - 1:49
    about them. But there are also a ton of
    other small Ethernet devices involved in
  • 1:49 - 1:57
    industrial networks that are not really
    researched very well yet. And all devices
  • 1:57 - 2:01
    that we're going to talk about are running
    their latest respective firmware.
  • 2:01 - 2:07
    Unfortunately, there will be zero days and
    these are not theoretical attacks. Like if
  • 2:07 - 2:12
    you go to Shodan or similar search engine,
    you can find tens of thousands of these
  • 2:12 - 2:18
    devices vulnerable and open in the
    Internet. So let me give you a quick
  • 2:18 - 2:25
    introduction into the terminology in
    SCADA, because I know in the title I say
  • 2:25 - 2:29
    SCADA, but actually it should be ICS,
    which stands for industrial control
  • 2:29 - 2:37
    systems, because basically ICS describes
    the whole system from your supervision,
  • 2:37 - 2:42
    the big room with all the big screens up
    to your PLCs the sensors, the actors and
  • 2:42 - 2:47
    so on that you will find in your
    installation. And the term SCADA just
  • 2:47 - 2:51
    describes the supervision and control
    centers. So the big screens that you might
  • 2:51 - 2:55
    know from movies and so on, where when the
    bad guy comes, suddenly all the lights
  • 2:55 - 3:02
    turn red. Then there's something called a
    PLC, which is programable logic
  • 3:02 - 3:07
    controller. It's basically like an
    Arduino, just for industrial applications
  • 3:07 - 3:12
    and they are really easy to program and
    you can get them from Siemens or Schneider
  • 3:12 - 3:18
    and so on and so forth. Then there is
    something called an RTU, a remote terminal
  • 3:18 - 3:22
    unit, which is a small device that
    generally are, well, back in the day, was
  • 3:22 - 3:27
    only used for monitoring. But today you
    can actually program a lot of RTUs. So
  • 3:27 - 3:33
    it's kind of a mix between a PLC and an
    RTU. So it's basically a PLC in a remote
  • 3:33 - 3:41
    location. Alrighty, to the actual topic,
    industrial control gateways. So when you
  • 3:41 - 3:46
    look at industrial control network, you'll
    find that there are a lot of different
  • 3:46 - 3:50
    sensors and actors and a lot of them speak
    different protocols. So, for example, some
  • 3:50 - 3:56
    might be serial, some might be IP, some
    might be Modbus and so on. And so you can
  • 3:56 - 4:01
    buy these small gateways that connect all
    these different protocols to an IP
  • 4:01 - 4:07
    network. So, for example, via Ethernet or
    even via GPRS or Wi-Fi and so on. And I've
  • 4:07 - 4:12
    seen them in almost any industrial
    installation that I've seen. So, for
  • 4:12 - 4:16
    example, they're used in power plants.
    They are used in water dam control
  • 4:16 - 4:23
    systems. They are used to control the
    power grid and so on. And the security
  • 4:23 - 4:27
    concept is, "Hey, but these devices are
    airgapped!", so it doesn't matter really
  • 4:27 - 4:32
    if they are vulnerable or not fully up to
    date and so on, but that's not really true
  • 4:32 - 4:35
    because a lot of these devices, while they
    might be airgapped, they also have
  • 4:35 - 4:43
    antennas and they are interconnected by a
    ton of different wireless protocols such
  • 4:43 - 4:51
    as Wi-Fi, LoRa or GSM or even proprietary
    radio links. So, yeah, and even the
  • 4:51 - 4:55
    case studies show that basically in this
    case, you would have a monitoring network
  • 4:55 - 5:00
    that's connected via the cellular network
    to control the water mains and so on and
  • 5:00 - 5:05
    check the pressure. Or even worse, they
    even recommend that you connect the actors
  • 5:05 - 5:10
    like valves and water level gotchas and so
    on over GPS, which we know is not a secure
  • 5:10 - 5:18
    protocol to do anything that could
    be critical. Or you have stuff like
  • 5:18 - 5:24
    water storage tanks that are controlled
    via Wi-Fi and so on or even public in the
  • 5:24 - 5:33
    Internet. So, yeah, these devices are
    airgapped? Nope. So attacking in the field
  • 5:33 - 5:38
    I already mentioned, if you go to
    Shodan, you will find a ton of different
  • 5:38 - 5:43
    devices reachable via the Internet
    and even via GPS. So if you live
  • 5:43 - 5:49
    close to, for example, a dam or something,
    it's kind of interesting to look at an SDR
  • 5:49 - 5:52
    or similar radio equipment to see what's
    going over the airwaves, because you will
  • 5:52 - 5:59
    find a ton of interesting stuff and
    sometimes, you can even very trivially get
  • 5:59 - 6:04
    a physical access to the in field devices
    because they might just be in a white box
  • 6:04 - 6:08
    somewhere hidden. And if you break into
    it, you can pull out the SIM card and it
  • 6:08 - 6:12
    will put you directly into the SCADA
    network, if you're lucky. Don't do that,
  • 6:12 - 6:14
    by the way.
  • 6:14 - 6:17
    laughter
  • 6:17 - 6:25
    So, yeah, let's let's hack some gateways.
    So the equipment you will need to and
  • 6:25 - 6:29
    everything in this talk was done on this
    desk, just using these devices here, you
  • 6:29 - 6:33
    really just need a laptop, you need an
    oscilloscope or similar measurement
  • 6:33 - 6:37
    equipment just to ensure that you don't
    burn out your logic analyzer. You need a
  • 6:37 - 6:43
    logic analyzer, a soldering iron, a
    multimeter and a power supply. And that's
  • 6:43 - 6:48
    really basically it, because you can hack
    almost any embedded device that's using
  • 6:48 - 6:57
    these devices and to find potential
    targets. I have this kind of map where
  • 6:57 - 7:02
    first try to understand, can I get the
    firmware of the device or do I have to
  • 7:02 - 7:07
    somehow, for example, use J-Tech to get it
    out of the device? Can I actually buy the
  • 7:07 - 7:12
    devices at a sensible price? Because some
    of these devices cost like 600 € or so,
  • 7:12 - 7:18
    and if you buy ten of them, that gets
    expensive very quickly. And so, uh, I need
  • 7:18 - 7:24
    to check eBay and see what devices can I
    actually buy. And they should be half what
  • 7:24 - 7:29
    current, because if you look at all the
    devices, like 10 years old or so, they are
  • 7:29 - 7:34
    completely broken. You don't even have to
    look to start to look at their security.
  • 7:34 - 7:41
    So, yeah, the first device that I that I
    choose to really look at was the moxa
  • 7:41 - 7:51
    W2150A, which is this small device, which
    is also mounted on the board right here,
  • 7:51 - 7:54
    mainly because I found the phone
    was available and it looked like an
  • 7:54 - 7:59
    interesting device because it has Wi-Fi
    and so if I managed to break into it, I
  • 7:59 - 8:08
    can jump an airgap potentially. And the
    W2150A is just a simple device server. So
  • 8:08 - 8:15
    you can connect any serial device, any
    RS485 device simply to it and it will be
  • 8:15 - 8:21
    exposed via Ethernet or even via Wi-Fi.
    And you can download the firmware publicly
  • 8:21 - 8:29
    and it's available on eBay relatively
    cheap. So like 150 bucks or something. So
  • 8:29 - 8:33
    I downloaded the firmware and I
    looked at the entropy of the firmware and
  • 8:33 - 8:37
    I immediately saw that the entropy is very
    high, which means either it's very
  • 8:37 - 8:42
    compressed or it's encrypted,
    unfortunately, using a tool called
  • 8:42 - 8:47
    binwalk, which is really useful for
    looking into firmwares I saw that there's
  • 8:47 - 8:52
    no compression detected. And so it was
    very likely that this firmware image is
  • 8:52 - 9:00
    encrypted. But I noticed on the Web page
    that before you upgrade to version 2.0 or
  • 9:00 - 9:09
    2.1 of the firmware, you must upgrade to
    the firmware version 1.11. And I thought,
  • 9:09 - 9:14
    that's interesting. Let's look at the
    release notes for version 1.11. And it
  • 9:14 - 9:22
    turns out that 1.11 adds the support for
    the encrypted firmware. So I downloaded
  • 9:22 - 9:28
    the one point eleven firmware and sure
    enough, it's unencrypted. And if you've
  • 9:28 - 9:34
    ever done anything with ARM before, if you
    just look into a firmware hex dump, you
  • 9:34 - 9:40
    can immediately recognize whether it's ARM
    or not, because the first four bits of each
  • 9:40 - 9:46
    instructions are the conditional bits
    and those are almost always E. So if
  • 9:46 - 9:50
    you see a Hexdump and roughly every fourth
    byte is an E, you know, this is an ARM
  • 9:50 - 9:58
    firmware and it's not encrypted or
    anything else. And so, yeah, sure enough,
  • 9:58 - 10:03
    I ran binwalk on this image. This time we
    see there is a huge drop in entropy, which
  • 10:03 - 10:09
    is the bootloader and so on, and then a
    high entropy, which is basically the all
  • 10:09 - 10:15
    the compressed filesystems and so on. And
    binwalk was able to detect the SquashFS
  • 10:15 - 10:22
    filesystem and extract it for me very,
    very easy. And so my goal was to extract
  • 10:22 - 10:27
    the firmware, find the firmware upgrade
    code and somehow try to decipher the new
  • 10:27 - 10:34
    firmware. And so I was browsing through
    the files and sure enough, found the file
  • 10:34 - 10:41
    that was helpfully called
    libupgradeFirmware.so and if we look into
  • 10:41 - 10:45
    the symbols, which they luckily didn't
    remove or anything, there is a beautiful
  • 10:45 - 10:48
    symbol called firmware decrypt.
  • 10:48 - 10:51
    laughter
  • 10:51 - 10:56
    So we load the whole thing into
    disassembler and we see that
  • 10:56 - 11:04
    there's some fancy XORing going
    on in the bottom left corner. And I'm
  • 11:04 - 11:08
    going to walk you through what's, what
    exactly is happening in this code.
  • 11:08 - 11:13
    So basically, first, there's a variable
    called password loaded into the registar
  • 11:13 - 11:22
    R2 and then a second count variable is
    basically set and it starts looping and
  • 11:22 - 11:36
    increasing always by four and goes through
    this whole xor shebang and it turns out
  • 11:36 - 11:41
    that this is the obfuscation method for
    the AES Key. So, in password, in memory,
  • 11:41 - 11:46
    we have an obfuscated key and we can be
    obfusciated by just implementing the code
  • 11:46 - 11:54
    we see here in C or in the emulator.
    And sure enough, eventually this
  • 11:54 - 12:03
    will be used as the key into the ECB 128
    AES decryption. And so I implemented the
  • 12:03 - 12:09
    whole thing in C, it was almost a copy
    paste from the decompiler, so you can in
  • 12:09 - 12:14
    IAD Pro, you just hit F5, copy the C code
    at the bit, fix the memory offsets and so
  • 12:14 - 12:20
    on. And you have the whole key obfuscation
    method basically reverse engineered almost
  • 12:20 - 12:26
    automatically. And so I compile it. And
    sure enough, Moxa key extration, it turns
  • 12:26 - 12:31
    out that the key is two eight eight seven
    Conn seven five six four. I build a short
  • 12:31 - 12:39
    script to decrypt the 2.1 firmware and
    this time Binwalk finds all the files and
  • 12:39 - 12:42
    we can start reverse engineering the
    actual firmware.
  • 12:42 - 12:49
    applause
  • 12:49 - 12:54
    The scripts for this are available on my
    github. I'll push the actual decrypts stuff
  • 12:54 - 13:00
    after the talk because this is the first
    time this has been released. And so after
  • 13:00 - 13:03
    I was at this point, I knew that the
    firmware is.. I can decrypted the firmware
  • 13:03 - 13:08
    I can look into it. By the way, it's not
    signed or anything. The only verification
  • 13:08 - 13:14
    method is CRC32. And so at this point I
    knew, OK, I can buy this device and
  • 13:14 - 13:20
    start playing with it. And so I went to
    eBay, I bought one. I got it. I screwed it
  • 13:20 - 13:24
    open. And sure enough, there's an ARM
    processor in there. It's an Freescale
  • 13:24 - 13:29
    i.MX25, which is just a regular ARM
    processor. It's like 400 MHz or something,
  • 13:29 - 13:35
    I don't know. And I started probing all
    the all the small pins inside of the
  • 13:35 - 13:43
    device to try to find JTAG or serial
    or anything. And so I actually hooked up
  • 13:43 - 13:47
    my power supply to foot pedal so that I
    can probe and just press with my foot to
  • 13:47 - 13:54
    reset the device. And sure enough, I found
    that there's a full serial console
  • 13:54 - 14:01
    available inside of the device on these
    pins. And if you boot the device, it even
  • 14:01 - 14:05
    tells you, please press enter to activate
    this console, and so you do that and you
  • 14:05 - 14:07
    are root on the device.
  • 14:07 - 14:15
    applause
  • 14:15 - 14:19
    So that's kind of cool, but that means
    that you require physical access, so
  • 14:19 - 14:24
    that's not really a vulnerability, but
    it's very nice to have when doing security
  • 14:24 - 14:29
    research because it means you can suddenly
    debug all the code on there. And so if you
  • 14:29 - 14:35
    write an exploit, you can just touch GDB
    to the binary and start very, very simply,
  • 14:35 - 14:40
    writing the exploit. So at this point,
    I was trying to look at the available
  • 14:40 - 14:46
    services on the device. So for example,
    there is a web interface, there's a
  • 14:46 - 14:53
    proprietary configuration protocol,
    there's telnet, there's snmp, there is a
  • 14:53 - 14:59
    serial driver protocol and so on. And I
    started looking at the web interface and
  • 14:59 - 15:04
    there was cross site scripting that was
    Cross site request forgery, there was
  • 15:04 - 15:07
    insecure authentication where they
    basically hash on the client. So they have
  • 15:07 - 15:13
    some JavaScript that hashes your password
    and then locks you in. Then there's a
  • 15:13 - 15:18
    command injection which lets you execute
    code as root, there are stack overflows.
  • 15:18 - 15:24
    And just a week ago there was a zero day
    released for the web server. So yeah, demo
  • 15:24 - 15:37
    time. So just let me open up the Moxa
    Pitch right here. And so this one is
  • 15:37 - 15:41
    authenticated, so I'll just enter the
    default password, which, by the way, in
  • 15:41 - 15:46
    the field will 90 percent of the time
    these devices will be configured with
  • 15:46 - 15:55
    default credentials. But still, so, if we
    just start browsing through this thing and
  • 15:55 - 16:00
    go to the basic settings, we can start
    with a simple cross site scripting just in
  • 16:00 - 16:09
    the device name. One sec, so just for
    example we just paste in some JavaScript.
  • 16:09 - 16:15
    Submit the whole thing, and hello 34c3.
  • 16:15 - 16:20
    applause
  • 16:20 - 16:24
    I know what you're thinking, like cross
    site scripting, come on, that's not a
  • 16:24 - 16:29
    vulnerability, that's just nothing. So
    let's look at the ping test that's
  • 16:29 - 16:34
    integrated into this device. And funilly,
    a different device from Moxa that runs an
  • 16:34 - 16:40
    entirely different firmware had the same
    vulnerability in the past. But if I just
  • 16:40 - 16:46
    paste in my ping, so my IP address, a
    semicolon and then, for example, I cut
  • 16:46 - 16:52
    /etc/passwd and activate enter.
    Here we go.
  • 16:52 - 17:00
    applause
  • 17:00 - 17:08
    Kind of funny, but, yes, for sure not
    intended. All righty, but I know what
  • 17:08 - 17:13
    you're thinking, right, these are
    authenticated bugs in the web interface,
  • 17:13 - 17:17
    so we need something unauthenticated. We
    want something that's like cool and a real
  • 17:17 - 17:23
    exploit. Right? And so I decided to look
    at the.. this custom TCP protocol, which
  • 17:23 - 17:29
    runs on Port 4900. And my goal was to
    reverse engineer the whole protocol and
  • 17:29 - 17:34
    build a fuzzer for it, to find
    vulnerabilities, that turned out not to be
  • 17:34 - 17:41
    necessary. So during some testing, I just
    sent a lot of bytes onto this thing and
  • 17:41 - 17:49
    enabled crash debugging via the serial
    console. And sure enough, it crashed and
  • 17:49 - 17:59
    put my program countdown right to
    0x41414140. Wonderful. Thank you, Moxa.
  • 17:59 - 18:04
    applause
  • 18:04 - 18:22
    So, Demo time. So let's increase the size
    of this a bit. So I built a small script.
  • 18:22 - 18:34
    Just called moxa_pown and I'll just supply
    the IP address to it. Let's see. Opening a
  • 18:34 - 18:44
    second shell to connect to it via netcat.
    Here we go, we have a root shell on the
  • 18:44 - 18:45
    device.
  • 18:45 - 18:54
    applause
  • 18:54 - 19:02
    So, yeah, that was the Moxa w21508,
    basically rolls of the tongue. And so the
  • 19:02 - 19:09
    next device I decided to look at was the
    Advantech EKI-1522 which you can find
  • 19:09 - 19:17
    right here. And it's, again, just a simple
    serial device server this time without
  • 19:17 - 19:21
    Wi-Fi, even though they are available with
    Wi-Fi. It comes with two Ethernet ports
  • 19:21 - 19:26
    two serial ports and so on. And I
    basically followed the same steps again.
  • 19:26 - 19:31
    So I looked at the.. I downloaded the
    firmware. I looked at the edit using
  • 19:31 - 19:36
    binwalk. And this time we see almost no
    entropy. So there is.. this guy is
  • 19:36 - 19:40
    basically completely unencrypted. And
    again, we saw some ARM 32 bit it runs a
  • 19:40 - 19:51
    Linux kernel, 2.6.31 and a BOA Web server
    where the last update was in 2005. And the
  • 19:51 - 19:57
    firmware, I think, is from 2017. So these
    are kind of outdated. And I found
  • 19:57 - 20:01
    during the initial analysis just of the
    firmware that the main binary to look at
  • 20:01 - 20:07
    will be this edgserver binary. And so I
    loaded it into IDA pro and looked at the
  • 20:07 - 20:13
    different things that calls. And there
    are a lot of calls to functions like
  • 20:13 - 20:18
    string copy, to system, to sprintf and so
    on that are generally kind of considered
  • 20:18 - 20:26
    unsecure. And sure enough, I am doing
    static analysis. I found that there's some
  • 20:26 - 20:34
    code for sending an email as an alert, for
    example, when the system reboots. And
  • 20:34 - 20:39
    the full command invocation is mailx -s
    blah blah blah, and we have control over
  • 20:39 - 20:46
    some parts in the string because we can
    configure the two address in the UI. And if
  • 20:46 - 20:51
    we look at what's happening
    here, it basically just sets up this
  • 20:51 - 20:56
    format string. Then it goes to include the
    subject and then it gets some arguments
  • 20:56 - 21:04
    from the stack and basically calls
    into system. And so there's no filtering
  • 21:04 - 21:10
    going on at all. So we have an unfiltered
    part of the system, invocation, code
  • 21:10 - 21:15
    execution. And this was before I had the
    device in my hand. And this is kind of a
  • 21:15 - 21:19
    funny story because I first bought because
    it was just 40 bucks, I bought this
  • 21:19 - 21:25
    device, which in the firmware has the same
    bug, but the mail functionality is broken,
  • 21:25 - 21:34
    so I couldn't test it. So I had to go to
    eBay again, buy another one and buy the
  • 21:34 - 21:39
    bigger one. And so I ordered the bigger
    one on eBay. Looks like this. It comes
  • 21:39 - 21:46
    with a Cavium CNS C.P.U. It has JTAG
    exposed on the bottom there and serial
  • 21:46 - 21:51
    console is available again without any
    authentication. So beautiful. You just
  • 21:51 - 21:58
    connect your bus pirate or your UART
    adapter to it and you have full serial
  • 21:58 - 22:07
    console. So, again, we had to look at
    finding vulnerabilities for this device
  • 22:07 - 22:12
    and there, again, a ton of different
    services, there's like a Web interface
  • 22:12 - 22:16
    available. There is a proprietary
    configuration protocol that's based on
  • 22:16 - 22:23
    UDP. There is Telnet, there's snmp,
    there's a serial driver protocol and so
  • 22:23 - 22:28
    on. And again, looked at the website and
    again, cross site scripting cross side
  • 22:28 - 22:33
    request forgery, command injection, broken
    authentication, which basically if you log
  • 22:33 - 22:39
    in from one computer, it uses, I think
    http digest authentication, you can
  • 22:39 - 22:43
    connect from a completely different
    computer and it doesn't ask for a
  • 22:43 - 22:50
    password. I don't know why that is, but..
    Yeah. So I was thinking I was doing
  • 22:50 - 22:52
    something wrong, but it turned out it was
    just broken.
  • 22:52 - 22:55
    laughter
  • 22:55 - 23:03
    So, yeah, and there's, again, a stack
    overflow in another protocol. So I guess,
  • 23:03 - 23:14
    again, demo time. Let's first look at
    the device itself, so, you know the
  • 23:14 - 23:23
    password, firstly, we have a nice device
    description here. This is just a basic web
  • 23:23 - 23:29
    interface. Right. And we can, again, just
    copy in some basic JavaScript
  • 23:29 - 23:39
    hit the save button. Reload and there we
    go, cross site scripting yet again, OK,
  • 23:39 - 23:49
    again, not really interesting. Right. So,
    um, let's look at the stack overflow.
  • 23:49 - 24:04
    Again, I have a small script advantech_pown.
    For the IP there. And we have netcat
  • 24:04 - 24:12
    running on there. Sure enough, there we
    go, that's root on the Advantech device
  • 24:12 - 24:14
    again, via stack overflow.
  • 24:14 - 24:26
    applause
  • 24:26 - 24:32
    Yeah, so two of three devices have
    basically broken already. Let's look
  • 24:32 - 24:38
    at the next one. This one is a Lantronix
    EDS2100. And this one is kind of
  • 24:38 - 24:44
    interesting because it's not ARM. I
    normally I almost exclusively do ARMs. So
  • 24:44 - 24:48
    this one was kind of interesting. And this
    device, which is mounted somewhere right
  • 24:48 - 24:57
    here. Yeah. This device comes with a
    serial to ethernet secure device server.
  • 24:57 - 25:02
    It has two serial ports. It has
    Ethernet and you can buy it in two
  • 25:02 - 25:08
    variants. One comes with Linux and one is
    Evolution OS, which is I guess, a
  • 25:08 - 25:15
    proprietary operating system from
    Lantronics. And I'm using the EvolutionOS
  • 25:15 - 25:22
    variant in this talk. Looking at the
    firmware it turns out it's unencrypted and
  • 25:22 - 25:28
    it's coldfire architecture, which I've
    never done really anything with before,
  • 25:28 - 25:33
    and there are no obvious external software
    components. So if you go through this,
  • 25:33 - 25:37
    through the firmware, you'll find there's
    an SSH implementation, there's an SSL
  • 25:37 - 25:43
    implementation, but it's not openSSL and
    it's not anything very well known. And the
  • 25:43 - 25:47
    same is true for the web server and so on.
    It's not really anything that's well
  • 25:47 - 25:56
    known. And this time, while probing
    the device, I did not really find anything
  • 25:56 - 26:02
    interesting in terms of serial consoles or
    so, but it just found a potential debugger
  • 26:02 - 26:06
    port, but it didn't have a fitting
    debugger unfortunately. The CPU is from
  • 26:06 - 26:15
    NXP runs at 160MHz or something. Yeah.
    This time we actually have a web
  • 26:15 - 26:22
    interface, we have Telnet SSL and it even
    has a file system, so you have like FTP
  • 26:22 - 26:26
    and TFTP which allows you to download the
    configuration, upload the configuration
  • 26:26 - 26:31
    and so on. And it's kind of hard to secure
    it correctly because there are so many
  • 26:31 - 26:37
    protocols and it's not really clear what's
    set up by default. But yeah, you get
  • 26:37 - 26:44
    the idea. And this time the web interface
    was surprisingly secure. So there was no
  • 26:44 - 26:50
    cross site scripting. There was no command
    injection, because there's also not really
  • 26:50 - 26:55
    a shell that you could execute commands
    into. But I still found some stuff.
  • 26:55 - 27:02
    One is the configuration injection, which
    allows you basically to change the format
  • 27:02 - 27:07
    of the configuration using a different
    field. And I found an authentication
  • 27:07 - 27:12
    bypass, so I was able to write a small
    piece of code that takes a while and then
  • 27:12 - 27:24
    completely removes the password from the
    device. Demo time. So if we connect to the
  • 27:24 - 27:30
    Lantronics device, it will currently ask
    for a password, which in theory we don't
  • 27:30 - 27:45
    have. Let's clean up here a bit. I know
    it's just. And let's run Lantronix_pown,
  • 27:45 - 27:51
    oh, that was fast. That worked. Yeah, sure
    enough, the password is gone.
  • 27:51 - 28:00
    applause
  • 28:00 - 28:07
    Awesome. To be honest, I didn't expect the
    demos to go so smoothly, so I put in an
  • 28:07 - 28:14
    hour for the talk for this went very well
    so far, so that's good. So before we
  • 28:14 - 28:22
    finish already, some other devices are
    even worse. So, for example, as I
  • 28:22 - 28:27
    mentioned, I bought some other devices,
    for example, this Advantaech device and
  • 28:27 - 28:32
    this Moxa device and this Lantronix
    device, which are basically the
  • 28:32 - 28:39
    predecessors of the other devices. And
    those guys are really interesting to look
  • 28:39 - 28:46
    at, one could say. So, some of those are
    running eCos, which is an embedded Linux
  • 28:46 - 28:52
    platform, which was last released in 2009,
    and some devices run a Linux kernel with
  • 28:52 - 28:58
    the 2.4 version and you see Linux without
    any memory protection whatsoever. So even
  • 28:58 - 29:04
    if they, so even a small stack overflow in
    one of the userspace applications gives
  • 29:04 - 29:09
    you full root access to the device because
    you can directly exploit the kernel and
  • 29:09 - 29:13
    there are unfixed public vulnerabilities.
    So in the first penetration test that I
  • 29:13 - 29:19
    did, that included actually this device
    and Moxa and part of a small one. I found
  • 29:19 - 29:25
    that using SNMPWwalk, it gives you back
    the administration password via SNMP.
  • 29:25 - 29:27
    laughing
  • 29:27 - 29:32
    And so I went online. I tried to report
    it. And it turns out it's well known
  • 29:32 - 29:34
    there's a metasploit module for this
  • 29:34 - 29:37
    laughing
  • 29:37 - 29:42
    and it's unfixed, OK? And these devices
    are still in support. So I don't know why
  • 29:42 - 29:51
    the vendor is not patching this. Yeah. So
    the summary with trivial vulnerabilities
  • 29:51 - 29:57
    in most devices, or at least all that I've
    looked at, there are no security
  • 29:57 - 30:01
    mitigations whatsoever. So they don't even
    enable like the compiler flags that you
  • 30:01 - 30:06
    just set and then you have at least some
    kind of stack protection and some like
  • 30:06 - 30:11
    stack cookies and whatnot. And some
    vendors are really bad at responding to
  • 30:11 - 30:18
    vulnerability reports. So, yeah, I'm not
    going to name the vendor, but not even, on
  • 30:18 - 30:22
    Twitter I asked them to please give me a
    security contact and they responded,
  • 30:22 - 30:27
    please use our contact form. I said I did,
    three times. I send you emails, you're not
  • 30:27 - 30:31
    responding to me. And so they stopped
    responding to me on Twitter too.
  • 30:31 - 30:41
    laughing
    applause
  • 30:41 - 30:47
    So how to mitigate? Well, the only way
    that I would see to mitigate against this,
  • 30:47 - 30:53
    and I'm more on the deconstructive side of
    the story, is defense in depth. So never
  • 30:53 - 30:57
    directly expose any of these devices to
    the Internet, even if they say they
  • 30:57 - 31:02
    support VPN, even if they say they are a
    secure device of whatever, just don't do
  • 31:02 - 31:09
    it. Get a real VPN gateway and make sure
    that you never rely on a single level of,
  • 31:09 - 31:16
    for example, encryption. So, for example,
    WPA2 was broken by the crack attack and
  • 31:16 - 31:21
    they actually released a patch for it
    after two months. And these are these are
  • 31:21 - 31:26
    still two months where you are exposed to
    vulnerability on your potentially mission-
  • 31:26 - 31:34
    critical system. Also never use GPRS for
    these devices without VPN because it just,
  • 31:34 - 31:41
    it will go wrong. Okay. Yeah, thank you. I
    guess now we have time for Q&A. Thank you
  • 31:41 - 31:43
    all for coming.
  • 31:43 - 31:49
    applause
  • 31:49 - 31:58
    Herald: Thank you very much for the talk.
    So we have very much time for Q&A. So
  • 31:58 - 32:04
    please line up to the microphones and we
    have someone at microphone 4 already.
  • 32:04 - 32:09
    Mic 4: Yes, hello. Hello. Thanks for your
    talk. This is.. obviously this is a
  • 32:09 - 32:15
    problem. This is a part of the bigger
    problem of security in IT. Right. In
  • 32:15 - 32:19
    anything related to any kind of
    technology. And this is only going to go
  • 32:19 - 32:25
    worse with time, right. Internet of shit,
    internet of things and so and so on, so
  • 32:25 - 32:32
    forth. So my question is, you gave some
    ideas how to mitigate this in this very
  • 32:32 - 32:37
    specific area that use VPN, et cetera, et
    cetera. But my question is, so hacker
  • 32:37 - 32:42
    community is not very, let's say,
    interested in regulation. Right? And when
  • 32:42 - 32:47
    we see, when we see a government trying to
    do something with technology that usually
  • 32:47 - 32:52
    goes bad, we have this idea in our head
    that, OK, this can only go like this can
  • 32:52 - 32:57
    only go bad. Right. But so my question is:
    do you think that perhaps there is some
  • 32:57 - 33:01
    space for regulation here?
    T: There's definitely space for
  • 33:01 - 33:07
    regulation, but I think regulation does
    not solve the underlying technical issues.
  • 33:07 - 33:14
    So these devices, it's 2017 and these
    devices are using C-code. I think that's
  • 33:14 - 33:19
    just asking for trouble, basically. And so
    we really need to see this shift, even in
  • 33:19 - 33:23
    the embedded world, to switch to memory
    safe languages, for example Rust or
  • 33:23 - 33:28
    something similar, and really to stop
    using C in this kind of context. I don't
  • 33:28 - 33:36
    think there's anyone who can .. Thank you.
    applause
  • 33:36 - 33:39
    T: But there's definitely space for
    regulation.
  • 33:39 - 33:43
    Herald: Since there was a question from
    the Internet.
  • 33:43 - 33:48
    Signal Angel: OK, yeah, the Internet wants
    to know why you are not naming the bad
  • 33:48 - 33:52
    vendor, because it looks like it's the
    only option basically if they don't
  • 33:52 - 33:58
    respond to you. Let's say I asked them on
    Twitter and my Twitter is right there. And
  • 33:58 - 34:03
    if you click on Tweets and Replies..
    laughter
  • 34:03 - 34:06
    Signal Angel: Yeah, somebody just posted
    the link on IRC.
  • 34:06 - 34:11
    laughter
    T: I did not name them, just for the
  • 34:11 - 34:13
    record.
    laughter
  • 34:13 - 34:17
    applause
    Herald: So we have a question from
  • 34:17 - 34:23
    microphone number 2.
    Mic 2: So you shown an exploit for the
  • 34:23 - 34:30
    last device that disabled authentication.
    What did you use to achieve that?
  • 34:30 - 34:36
    T: So this one is unpatched and not yet
    fixed, so I would rather not disclose the
  • 34:36 - 34:39
    details yet.
    Mic 2: OK.
  • 34:39 - 34:43
    Herald: Microphone number 1, please.
    Mic 1: I wonder if you've also been
  • 34:43 - 34:48
    looking at a building automation system,
    control systems, or just industrial
  • 34:48 - 34:54
    automation control systems?
    T: So you can use these devices basically
  • 34:54 - 35:01
    wherever you want. And I think some of the
    Moxa ones are used in home automation. But
  • 35:01 - 35:06
    I've looked at I guess Crestron, it's
    called? But not in a lot of detail. So I'm
  • 35:06 - 35:10
    more on the industrial side at the moment.
    Mic 1: Thanks.
  • 35:10 - 35:15
    Herald: Microphone number 3.
    Mic 3: Any field experience or even just
  • 35:15 - 35:21
    opinions on using industrial strength
    Raspberry Pi hardware with community
  • 35:21 - 35:26
    supported Linux distributions or something
    like OpenBC whatever on them.
  • 35:26 - 35:31
    T: Yeah. So I guess the big trouble there
    is support, right? There are some, some
  • 35:31 - 35:35
    German companies and so on that provide
    support for industrial Raspberry Pis and
  • 35:35 - 35:41
    even like nice casing and so on. But I'm
    not sure if really Raspberry Pi is the way
  • 35:41 - 35:45
    to go here. I think there are
    boards that are.. the problem is not the
  • 35:45 - 35:50
    underlying stack, right? It's not the
    hardware. Really, that's the issue. It's
  • 35:50 - 35:56
    the software. And you will have the same
    issues on on the Raspberry Pi. So, yeah, I
  • 35:56 - 36:01
    guess you could buy these devices, which
    are like industrial grade shockproof and
  • 36:01 - 36:07
    whatnot, and put some Linux on it and
    do it better. But I don't think that
  • 36:07 - 36:12
    the hardware or platform will
    change anything at the moment.
  • 36:12 - 36:16
    Herald: There is another question from
    microphone number 4.
  • 36:16 - 36:22
    Mic 4: Hi, more a social question, did you
    get in contact with any development team,
  • 36:22 - 36:26
    software development team in any of these
    companies, or might it be that there is no
  • 36:26 - 36:33
    one behind the emails and everything?
    T: So I guess some of these companies are
  • 36:33 - 36:37
    really so big, that they don't reply to
    you if you don't have a support contract
  • 36:37 - 36:45
    with them. But, for example, the support
    of the ones that are not on my Twitter is
  • 36:45 - 36:50
    kind of decent when it comes to two
    security reports. And so my next steps
  • 36:50 - 36:57
    will be to go via the ICS Cert, but, you
    know, to report them. So, yes, there are
  • 36:57 - 37:04
    development teams that will get in contact
    with you, just not from all vendors.
  • 37:04 - 37:07
    Herald: Thank you. We have another
    question from the Internet.
  • 37:07 - 37:14
    Signal Angel: Hello? OK. The Internet
    wants to know what to do about, because
  • 37:14 - 37:18
    there are a lot of old devices in the
    field, how do you propose a vendor should
  • 37:18 - 37:24
    deal with legacy devices and updates?
    T: Yeah, so keeping legacy devices
  • 37:24 - 37:30
    supported is very expensive because, for
    example, if you buy a Qualcomm chip, they
  • 37:30 - 37:35
    will eventually drop support for the Linux
    kernel for it and so on. But if you buy
  • 37:35 - 37:40
    like a Freescale automotive chip, they
    guarantee you a certain time of support.
  • 37:40 - 37:43
    But then you actually have to invest the
    money to regularly provide the updates and
  • 37:43 - 37:49
    ensure that your devices are secure. The
    problem is that the lifetime of industrial
  • 37:49 - 37:55
    installations currently is much larger
    than the lifetime of this processors' supports
  • 37:55 - 38:01
    and so on. So I guess we'll have to get
    used to upgrading our hardware regularly
  • 38:01 - 38:07
    or switch to, or figure out a different
    way of deploying secure software onto
  • 38:07 - 38:11
    them. But I really think the underlying
    problem is, that we are still using
  • 38:11 - 38:16
    memory unsafe languages. And I guess the
    fact that there's cross site scripting
  • 38:16 - 38:20
    just shows that there's no security
    awareness really at those vendors
  • 38:20 - 38:29
    whatsoever. At some of the vendors.
    Herald: So, microphone number 2, please.
  • 38:29 - 38:34
    Mic 2: I was wondering, you mentioned that
    some of these facilities use GPRS.
  • 38:34 - 38:36
    T: Yeah.
    Mic 2: Do you know if they have mostly
  • 38:36 - 38:41
    their own closed infrastructure, or if
    they're using general consumer telecom
  • 38:41 - 38:45
    stuff?
    T: So they will use commercial
  • 38:45 - 38:50
    networks mostly, and then they have custom
    EPNs which have an IPSec tunnel or
  • 38:50 - 38:56
    something similar to their premises. But
    there's also there's also a company that
  • 38:56 - 39:03
    sells industrial control SIM cards
    which give you a public IP and you don't
  • 39:03 - 39:08
    want to search on Shodan for that vendor.
    Mic 2: Yeah. Thank you.
  • 39:08 - 39:11
    Herald: There is a question from
    microphone number 3.
  • 39:11 - 39:15
    Mic 3: Hi there, isn't economics meant to
    solve some of these problems? We're not
  • 39:15 - 39:20
    talking about dirt cheap devices. How
    surely at 300 bucks you should better have
  • 39:20 - 39:25
    someone who's read security one and one.
    How long before a large organization gets
  • 39:25 - 39:28
    the result of their security audit and
    goes to the aforementioned vendors and
  • 39:28 - 39:33
    says, provide us something that's not
    trivially hackable, otherwise we stop
  • 39:33 - 39:38
    buying your rubbish?
    T: Well, I mean, it's the same in all of
  • 39:38 - 39:45
    IT, right? So everything has
    vulnerabilities. And yes, there should be
  • 39:45 - 39:50
    market pressure. But that's why I'm trying
    to raise awareness for the issues that
  • 39:50 - 39:53
    these devices have.
    Mic 3: Thanks.
  • 39:53 - 39:56
    Herald: There's another question from the
    Internet.
  • 39:56 - 40:01
    Signal Angel: Yep. The Internet wants to
    know how and if it's a good idea to raise
  • 40:01 - 40:07
    the level of awareness in public, because
    they think it's a good approach to make
  • 40:07 - 40:12
    people, the public know that, well,
    infrastructure in the cities is at risk.
  • 40:12 - 40:16
    T: Uh, sorry. Could you repeat the first
    part of the question?
  • 40:16 - 40:21
    Signal Angel: Yeah. They want to know how
    to raise awareness for this in the public?
  • 40:21 - 40:28
    T: Good question. I guess we need some
    news articles or something about this in
  • 40:28 - 40:33
    regular paper, but I personally think it's
    just an accident waiting to happen. So
  • 40:33 - 40:38
    eventually someone will turn off the
    lights in a city or wherever, will open a
  • 40:38 - 40:45
    flood valve or something. And that's when
    the awareness will start.
  • 40:45 - 40:48
    Herald: There's another question from
    microphone number 4.
  • 40:48 - 40:52
    Mic 4: OK, for what kind of industrial
    processes are these devices you just
  • 40:52 - 40:57
    demoed used?
    T: So I've seen them in power utility. I
  • 40:57 - 41:02
    know they're used in water dam
    control systems. They are used and in
  • 41:02 - 41:07
    serial connecting a CNC machine to the
    network, they are used in connecting all
  • 41:07 - 41:11
    kinds of stuff. Because if you have a big
    plant, you have a ton of different
  • 41:11 - 41:16
    sensors. So you might, you might need the
    water level sensor. And for whatever
  • 41:16 - 41:21
    reason, you only can get it with a modbus
    and then you need to convert the modbus to
  • 41:21 - 41:25
    TCP and then you need one of these
    gateways. And so, I've seen in one
  • 41:25 - 41:29
    cabinet, 20 of them. So they're
    really used a lot I guess.
  • 41:29 - 41:32
    Mic 4: OK, thank you. I just retweeted
    your tweet to Star Alliance.
  • 41:32 - 41:38
    T: Huh. laughs Thank you. laughs
    Herald: So there's another question from
  • 41:38 - 41:41
    the Internet.
    Signal Angel: Yeah, the Internet wants to
  • 41:41 - 41:51
    know if you did any research on MQTT
    for example from like Beckhoff uses?
  • 41:51 - 41:54
    T: I actually talked to someone who
    recommended me to look at Beckhoff
  • 41:54 - 41:58
    yesterday, but I've not looked at them
    whatsoever yet.
  • 41:58 - 42:02
    Herald: And there's another question from
    microphone 3.
  • 42:02 - 42:07
    Mic 3: OK, could you show the Moxa web
    panel, because I would like to double
  • 42:07 - 42:17
    check, which proves that they and they
    would like you to see their Web page. And
  • 42:17 - 42:24
    I think this browser isn't very secure.
    T: OK, let's take a look.
  • 42:24 - 42:29
    Mic 3: Yeah, and under gohead the
    webserver small print.
  • 42:29 - 42:42
    laughter
    Herald: Nice finding.
  • 42:42 - 42:48
    T: That's probably the issue here.
    laughs
  • 42:48 - 42:56
    Herald: Are there any more questions? Any
    questions from the Internet?
  • 42:56 - 43:02
    Signal Angel: The internet wants to know
    how a memory safe language would prevent
  • 43:02 - 43:09
    the authentication bypasses you showed?
    T: Not one would not be protected against
  • 43:09 - 43:13
    but it protects against a ton of other
    stuff. It's just one example of where the
  • 43:13 - 43:18
    industry needs to change. We need to stop
    using memory unsafe languages. We need to
  • 43:18 - 43:24
    start really thinking about security
    design from the start, and we must not in
  • 43:24 - 43:28
    2017, there's no excuse for having cross
    site scripting or anything on the web
  • 43:28 - 43:36
    page. That's also if we in the
    Lantronics website, if you click logout,
  • 43:36 - 43:39
    it tells you logout is not supported in
    your browser.
  • 43:39 - 43:43
    laughter
    T: Probably because I'm not using Internet
  • 43:43 - 43:48
    Explorer five.
    Herald: So there's another question from
  • 43:48 - 43:53
    microphone number 3.
    Mic 3: Any remote part of the exploit
  • 43:53 - 43:58
    where you did a buffer
    overflow - I think.
  • 43:58 - 44:01
    T: Yeah?
    Mic 3: What I'm wondering is, are
  • 44:01 - 44:07
    there.. isn't it like very standard to
    have ALSR on these devices?
  • 44:07 - 44:10
    T: No! laughts It should be, but it
    isn't.
  • 44:10 - 44:16
    Mic 3: Okay. Thank you though. That was
    pretty much my question.
  • 44:16 - 44:23
    Herald: Is there another question from the
    Internet? It doesn't seem like it?
  • 44:23 - 44:36
    Signal Angel: So, one just came in, OK, if
    you want to hear it. Ok, nope.
  • 44:36 - 44:41
    laughter
    Herald: So, all right, give a very warm
  • 44:41 - 44:43
    applause to Thomas Roth again!
  • 44:43 - 44:47
    applause
  • 44:47 - 45:00
    postroll music
  • 45:00 - 45:08
    Subtitles created by c3subtitles.de
    in the year 2021. Join, and help us!
Title:
34C3 - SCADA - Gateway to (s)hell
Description:

more » « less
Video Language:
English
Duration:
45:08

English subtitles

Revisions