Eireann Leverett: Switches Get Stitches

Éireann: Things are blowing up, in
industrial systems, here in Germany, this

year! I had hoped that these things
wouldn't happen. This kind of future

wouldn't be one that we are living in. But
unfortunately it is. And I hope that we

can make that better, partly through the
course of this talk. But more, I think, in

the future with your help and your work.
So I'm sorry to begin this presentation

with such a dark thought but: This year's
theme is a new dawn. And it's always

darkest just before the dawn. So we're
going to go through some of that darkness

in industrial systems and SCADA-systems to
get to a better place, right? Now with

that said no hacker really gets to be
where they are without the help of other,

right? We stand on the shoulders of giants
and part of the key is not stepping on

their toes, on the way up. So I would like
to say thank you to a bunch of people who

are here and also some people who aren't
here. Particularly the Oslo hackerspace

where I hang out. And these people have
taught me a lot of things not just about

technology but about life and on
"aprendo", which is how Goya signed some

of his last paintings and sketches - which
basically means "I'm still learning". OK.

So with that said I hope that you will
enjoy this talk with its darkness and its

humor all at the same time. I used to be
in circus, as you may have guessed from

the mustache. So I encourage you not just
to view this as a technical vulnerability

presentation but also as kind of live
technical standup comedy. Instead of jokes

we have vulnerabilities. And I hope that
you will enjoy them. So these

vulnerabilities are in switches. I chose
to focus on switches and that will become

clear throughout the presentation, why I
chose to do that for industrial systems.

And we are looking primarily at three
different families of switches. Because I

don't want to pick on any one vendor. In
fact, the whole idea of this talk is to

continue giving it. I have two other
colleagues who couldn't be here with me

today, who have some vulnerabilities in
some other switches. And they look forward

to presenting those vulnerabilities as
part of this presentation in the future.

So every time we give this presentation
we'd like to give some new vulnerabilities

and show that this is systemic and endemic
risk. So the three switches we'll be

looking at today are the Siemens Scalance-
family, the GE Multilin-family and the

Garrettcom Magnum family. These switches
are usually not very big. They might be 8

ports, they might be 24 ports. And they're
used in a variety of different locations.

So this talk is for you, if you work in a
utility, if you test industrial Ethernet

switches, if you manage industrial
Ethernet networking, if you're comfortable

at a Linux commandline and you play with
web apps but you don't know as much about

reverse engineering. Don't worry, I'm
exactly the same. I suck at reverse

engineering. But I care about this stuff.
And so I'm learning. If you are a

developer of firmware then I think this
talk is for you as well. I hope you learn

something from it. If you like
vulnerabilities you'll enjoy this quite a

lot. I'm going to be sharing with you a
little collection I have, you know. Some

people collect stamps or stories or jokes.
I collect private keys. And I like to

share them with other enthusiasts such as
yourself. If you happen to work for one of

the switch manufacturers you know I've
spoken to before. Some of you I get on

with very well. We speak regularly. Some
of you not yet - but I hope you'll come

and have a chat with me later. Ok, most
SCADA or ICS presentations go a bit like

this: Pwn PLC, the RTU, the HMI - these
are terms, you know, that all of us in

SCADA know. Maybe most of you know them by
now, they're pretty popular. I hope you

do. But programmable logic controller,
remote terminal unit or human machine

interface. And the basic idea of the
presentation is if I pwn these things,

game over. Physical damage. I win. Isn't
the world a scary place? And I encourage

you to demand better content. I certainly
grew up with better content. I used to go

and see the presentations and the talks of
a guy called Jason Larson. And he has a

fantastic example of this. I want all of
you to try it, right now. Just think

about: If you had complete control over a
paint factory. What would you do to damage

it? No one is going to get hurt.
Everything's safe. It's a thought

experiment, right? What would you do to
damage it? Most people can't answer this

question. And on certain types of
processes I can't answer this question.

But other types I've worked with before
and I can answer this question. And I

encourage you to to ask it. But if you
like and you want to learn more go and see

Marmusha's talk - I think it's tomorrow.
Think of my talk as a frame for her talk.

She's going to be talking about how to
damage a chemical process. And what you

need to do as an engineer to do that. And
the reason she's doing that is to build a

better process in the future. You have to
break a few things to make them work a

little bit better. Okay. So what's the
point in industrial control systems

security? It's not credit card data. It's
not privacy. No disrespect to my privacy

friends in the room. I have the deepest
love and respect for the work that you do.

But confidentially ... confidentiality is
the lowest priority for us in industrial

systems. It would go: Availability,
integrity, confidentiality. And you might

even swap integrity and availability in
many cases. So, you have to protect the

sensor data or the control signals.
Everything else is maybe a vulnerability

on the path to getting this. But it's not
the most important thing that we're trying

to protect. So that's why I'm attacking
switches. That's where the process is,

right? Now these may not be core switches.
They're often a little bit further down in

the chain. They're field devices, right.
So you might find them in any of these

locations. And this last example is not
necessarily important be cause oil and gas

is important - but it's important because
it gives you the general format of all

industrial systems. You have sensor
network. And sensor data is traveling back

and forth. And you have control signal
data. That's it, basically. You might have

different control signals on different
protocols and you might have different

sensors on different protocols, giving you
different values like pressure or heat or

whatever. But most processes follow
basically this format. Okay. I don't do

SCADA 101. There are other people who do
this. I'm trying to do a little bit, to

set the reference for this talk, but
usually I avoid it. So basically there's

not much authentication or integrity in
industrial systems protocols. There's not

much cryptography. You would expect there
to be, maybe. I'm continually surprised

that I don't find any. And when I do find
it, it's badly implemented and barely

works. So once you have compromised a
switch or another part of the network you

can perform man-in-the-middle attacks on
the process. Or you can create malicious

firmwares on these different switches. And
that's what I'm trying to prevent. I'm

trying to find some of the different
methods that people can use to produce

these firmwares - and then get the vendors
to fix them, right. Okay. These are some

of the protocols. If you are new to this
space, if you want to do some more work in

this area, but you don't know what to work
on, take a picture of the slide or go and

find it later. And choose one of these
protocols and go and work on it. We need

people to go to these different
organizations. Some of them are

proprietary, some of them are open and
complain that there is not enough

cryptography going on in this space. And
yes you can use VPNs. But believe me, I

often don't find them. Okay. These are the
switches, the specific versions of the

firmware, in case you're here for
vulnerabilities instead of just me

waffling on about the basics. If you want
to go and look these up, if you're a

penetration tester working in this space,
you can go and find them all online. And

you can get a feeling for the kind of
coding practices that go into these

different devices. Now I've tried to
choose the vulnerabilities that I'm

presenting very carefully. To take you
gently from web app vulnerabilities into a

little bit deeper into the firmware. So
the first one we'll be looking at is

Siemens. And again, I'm not picking on any
particular vendor. In fact I'm very proud

of Siemens. They're probably here again.
They're here many years. And they fixed

these vulnerabilities within three months.
And I think that was awesome - especially

in the space that I work in. The average
patch-time in SCADA and ICS is 18 months.

So I think Siemens deserves a round of
applause for getting these fixed.

Applaus
So without further ado let's have some

fun, right. So MD5, you go to the web page
for this switch. This is the management

page of a switch, right. And you interact
with this webpage. And you have a look at

it. And on the client side they do MD5 of
the password. Okay. That's fascinating. I

don't think that's particularly secure.
But it's done in roughly the same format

as that Linux command. So I use the Linux
command instead of the JavaScript just to

make it easier for everyone. You have the
username at the beginning and the password

is in the middle. And then you have this
nonce that's at the end, a number you use

once, right. I was surprised to see the
nonce, and it's even called a nonce,

right. So somebody had done a little bit
of homework on their cryptography. And

they understood that they wanted to use,
you know, this number used once to prevent

replay of the hash every time. Okay,
that's some pretty good work.

Unfortunately this is MD5 and this is
protecting your electric utilities and

your water and your sewage systems. And
you can brute force this in a few seconds,

if the passwords are less than eight
characters. and if they're around 15 it

might take you 20 minutes or something.
You can do this from PCAPs, from network

traffic captures. And then you have the
cleartext password that you can use

forever after, with that switch. So, off
to a bad start, in my opinion. So these

are the nonces that we're looking at. I'm
glad to hear you laughing. It makes me, it

warms the heart, right. So you can see
that they are incrementing and that they

are hex. Yeah. What else can you say about
this? The last half is different than the

first half. Not only is it incrementing,
it is sequential. If you pull them quickly

enough. For those of you who also do a bit
of reverse engineering you might recognize

the first half as well. Anybody in the
room see any patterns in the first half of

the of the nonces? No? Hmm? Very good, IP
address. Mac address would have been a

good guess as well. I thought it was at
first. And I got very confused when I went

to look for the IP address. Because I went
to the switch itself. And the switches IP

address was not this in hex. It's the
clientside address. Which I just couldn't

believe, right? Like, it seems like it
makes a sort of sense if you're trying to

keep session IDs in state. And it's like
oh I want a different session for every IP

address. And then I'll just use time, I
use uptime in hex as the rest of my

session ID, right? You know, the entire IP
space and time that can't be brute force.

It has a kind of crazy logic to it, right.
Unfortunately it can be. And you can get

the uptime from the device using SNMP. And
of course if you don't want to use SNMP

you can get old-school and use the TCP-
sequence-ID numbers. So, not a lot of

entropy there, I guess, I would say. And I
think their lawyers agreed when they put

out the comments on this. All right. Not
only can you perform session hijacking.

And if you are attacking switches I'd like
to point out that session hijacking is not

necessarily a great attack in this
environment. Think about it like you would

at home, right. How often do you log into
your router? In fact even more importantly

how often do you upgrade the firmware on
your router? Everyone who has upgraded the

firmware on their router ever raise your
hand. Just for an experiment. Thank

goodness, right. But wait, keep them up
just for a minute. Everybody who's updated

it this year, keep your hand up. Everybody
else put them down. Everybody who has

updated in the last six months ... okay
... So that gives you a sense of how long

these vulnerabilities can be in play on an
industrial system's environment. If you

multiply that by about 10, right. Okay, so
you can simply upload a firmware image to

a Siemens Scalance device with this
version number without authentication. You

just need to know the URL. Cross-site
request forgery, right. I just say CSRF

all the time. I don't even remember what
it stands for. So you can upload or you

can download a logfile. Not that useful
but you get a sense of what's going on on

the switch. You know what usernames might
be present, whatever. Incidentally all of

these switches by default or at least this
one only have two usernames, right. So

it's "admin" and "operator" I think on
this switch. Or maybe it's not. But

anyway, there's two usernames "admin" and
"manager"? I know I get them mixed up now.

But the configuration includes password
hashes. I'm actually not even entirely

convinced they're hashes because when you
increase the length of your password it

increases. But I'll leave that for future
researchers to examine. You can download

the firmware image from the device, which
is nice. So you just make a request. You

just post an HTTP-request to this device.
And it gives you the firmware that it is

running back. That's not that big a deal,
right. Because you're just viewing data on

the switch. But you can upload firmware
and configuration to this device. Which is

an authentication bypass in and of itself.
But it's also interesting because I can

take a configuration file from one of the
devices that I have at home with a known

password. I can upload a new configuration
file with a password that I know. I can

use the device to do whatever I want to
do. And later I can re upload the old

configuration file that I got from the
device, so no one ever even realizes what's

been changed, right. So. I think that's a
disappointing state of affairs. And I

wrote a script to do this. So that you
wouldn't have to when you are doing

penetration tests of these device. And I
gave you a little ASCII menu because

sometimes I get bored. Cambridge is a
small town and there's not much to do in

the evening. So feel free to go and
examine my github-repository where I put

up some of this stuff. I'm Blackswanburst
on Github, and on Twitter. So like I say,

Siemens are some of my favorite people. So
I'm going to finish up with them. This is

old day, if you like all that you have
just seen. But I want you to keep in mind

that these vulnerabilities will still be
present in the wild for another two or

three years. And I encourage you to go and
have a look at your systems, if you have

any of these devices. And check them out.
And upgrade the firmware. I also hope this

encourages you that if you haven't done
much in industrial systems and SCADA you

don't have to be intimidated by all of the
engineering and the terminology, and the

verb beotch(?).. There is plenty for any
of you in this room to do in the

industrial systems space. You need to
spend a little time speaking to engineers

and translating your vulnerabilities into
something meaningful for them. But that's

just a matter of spending more time with
them and getting to know them. And I think

that's valuable too because they have a
lot of experience. They care very deeply

about safety. And I've learned quite a lot
of things from engineers. My general point

here is I'd like you to stop defending
banks and websites and other stuff. We

need your help in industrial systems, in
the utilities. We could really do with

living in a safer world rather than one
where you're just protecting other

people's money. So we're gonna move on to
the GE Multilin line. I worked on a GE

ML800 but these vulnerabilities affect
seven of the nine switches in this family.

Seven because one of the other switches is
an unmanaged switch. If you're a hardware

person maybe you want to go and play
around with those but not so much my thing

and the other one uses a different
firmware image but seven of the nine

switches use a similar firmware image GE
offers a worldwide 10 year warranty. So

let's see if that includes fixing
vulnerabilities. I think it should. What

do you think. No? Couple noes couple of
yeses, undecided. All right. CCC is

undecided on something that's novel. Let's
start with some new vulnerabilities. Cross

site scripting. Reflected, I grant you but
still cross site scripting and I want you

to pay attention to the details. I'm not
going to go slow for you and ask you to

think . I know it's morning, I know it's
tough but I am going to ask you to think.

See flash up there flash.php and the third
one. Yes, it runs flash in your browser.

So if you know something about Flash come
and have a look at the switch some time. I

didn't go for active script attacks. There are
so many attacks surface on this device. I

just I sometimes don't even know how I'm
going to finish looking at all of them. So

I just work with the web interface to
begin with. So you have this cross site

scripting times eight and I want you to
notice in the last section there

arbitrarily supplied URL parameters. I
don't know about you but I think that's

funny right. You can just make up
parameters to stick your cross site

scripting in. laughs It's unbelievable
right. Yeah. Anyways what does that look

like. It looks like that, they have an
error data page. OK maybe I'm using a

browser that they don't approve or
something but it deserves looking at. And

you can do quite a lot of things with
javascript on the client side these days.

Disturbing. Anyways I'm not a big fan of
XSS so I'm going to move on to things that

I think are worth my time. So if you fetch
the initial web page of this switch before

you've even logged in you get this config.
So this is pretty authentication. No

authentication, right. Now keep in mind that
these switches are designed for process

data, right. It's not carrying traffic to
images of cats. It's supposed to be for

engineering. So what happens if I add a
nocache parameter and I make it say 500000

digits long. I should just be able to
crash the web server. Right. Maybe maybe.

But you would not expect it to reboot the
switch. And it takes a minute or so for

the switch to reboot which is actually
really impressive comes up pretty quickly.

But you know obviously you can repeat
this. So I wanted to examine that a lot

further. I wanted to know more about that
that crash what was rebooting the switch.

But like I say I'm not a very good reverse
engineer. So you're going to go on a

little journey with me where I learned a
couple of things about reverse engineering

and I had to change my approach from
looking at the webapp style loans to

moving into this other stuff. So why is
why is it DoS even interesting. You'll

remember that I mentioned Misha's talk. So
the reason I mention her talk, this is it

right. Denial of Service on a Website. Who
cares it's tearing posters down as xkcd

once famously explained to us but in the
industrial system's environment it's very

different. It can be very serious right. A
simplistic example is you have an

application that has a heartbeat and if
you stop that heartbeat it might go into

some sort of safety state it might for
example scram a reactor. There is a famous

denial of service on PLCs that did scram a
reactor in real life. Does anybody know

what H2S is? Any oil and gas engineers in
the room? Okay so H2S alerts not reaching

their destinations is pretty serious
business right. For those of you who are

not aware of H2S it's a byproduct of
producing oil and gas and inhaled in very

very small amounts you can go unconscious
and in sort of larger amounts. Respiratory

failure. So if you take CA safety
seriously if you ever work on these rigs

in these environments you learn to care
about the wind sock. Right one of these

alerts goes out. An alarm goes off. There
are many different alarms you have to

memorize how they all sound on a rig and
then react to them and when you hear the

H2S alert you look up at the wind sock to
keep an eye on where the wind is and

trying to avoid being downwind of wherever
the leak is. So a simple denial of service

that we would not care about in a web
application environment in this

environment can be very serious. I'm not
saying it always is. It just can be

right. So denial of service goes up in our
list of problems especially when we're

looking at networking devices. Okay so
that's that's it for the denial of

service. But like I say we're going to
look at some other stuff. In fact the

story with the switch began with a
concerned citizen about three or four

years ago I found 10000 industrial systems
on the Internet as part of my master's

thesis and I was pretty uncomfortable with
that. So I sent that data to various

computer emergency response teams around
the world. I believe it was 52 of them

right. Not all of them were critical
infrastructure. A lot of them were small

stuff but maybe 1 in 100. I was told or in
one particular country when they got back

to me one in 20 were considered critical
infrastructure. And after that you have a

sort of reputation among the computer
emergency response teams of the world. So

people send you stuff you get anonymous
e-mails from someone called Concerned

Citizen. Thank you very much. They sent me
a firmware upgrade pcap of this particular

device. I suspect that they worked at one
of the utilities and they wanted me to see

how upgrading the firmware of this GE switch
was performed. So it all began with a pcap.

So I ran TCP trace to carve out all the
files and see what was going on and you

could see instantly that there was an FTP
session later looking at the switch I see

that you can also upgrade them over TFTP
so the management of the switch happens in

HTTPs and is encrypted but the firmware
upload goes across FTP right so you can

just carve the file out a little bit of
network forensics I guess. So instantly I

could see that this one is complete and
the ports on the end of the numbers give

me a clue of what's going on in the larger
stream. This one seems interesting. Let's

have a look at it. So. I tried running
file and binwalk I don't know about you

but I believe that hacking is a journey of
understanding and facts hacking is

understanding a system better than it
understands itself and nudging it to do

what you want right. And I also feel that
I should understand my tools. I don't

really understand my tools until I know
where they're going to fail me or they

have failed me in the past and in this
particular case I think binwalk is a

fantastic tool and file is a fantastic
tool. But they didn't tell me anything and

that was that was a journey of discovery
for me. So that was nice. It was like OK

binwalk doesn't always give me everything.
I think I was running an older version and

I think it would handle it now. But the
point is after been walked didn't give me

anything just resort to the old school
stuff right. Go strings and I found these

deflate and inflate copywrite strings and
I could tell that a certain portion of the

file was compressed. This is just from the
pcap. Remember this whole story. So I

tried to deflate the whole thing. That
didn't work again. I just did something

simple get a python script that checks
every byte to see which parts of the file

don't produce ZLIB errors when you try and
decompress them and you figure out what

sectors of this file are compressed. So
you go to your friend dd and you carve out

this section of the file right. So we have
this larger firmware image with this

little compressed section and we have now
cut this little compressed section out. I

suppose I could have loaded this up into
python and use ZLIB to decompress it. But

at the time I was still trying to use
command line tools and someone said I'll

just concatenate the gzip bytes on it.
Gzip inherits from inflate and deflate. So

if you just concatenate the bytes it
should still handle it. So I did that and

I got a decompressed binary. When you ran
strings on that it started to make a lot

more sense and you could find the opcodes
in it where previously it didn't make any

sense at all. So once you've got an image
like that what do you do. Well if you're

me you just grep for bugs. I think I
learned that from Ilija. If he's here in

the room thank you. Thank you very much. I
asked him like a year or two ago. How do

you how do you find so many bugs. And he
said: "Oh, I just, you know, I grep for

them, I use find." laughs And so I
started thinking about firmware images.

Like if I was going to grep for a bug in a
firmware image what would it be. And my

answer is hardcoded credentials and
default keys because you find them every

single time so I have this command aliased
on my machine and I just grep for it and I

find private keys and this is how you too
can end up with a private key collection.

So, there you go.

Applause

Yeah they're hardcoded keys,
but what are they for. It doesn't

stop there. You know you've got the keys,
but what do they do, right? That was the

next step of the journey for me. Two of
them you can see one sencrypted with a

password; we'll come back to that one
later. Let's start with the one on the

left. If you load this key up into
wireshark. and you use it to decrypt the

SSL you have a self decrypting pcap.
Remember at the beginning it was using

HTTPS to manage the device and upload this
firmware image. So if you happen to have

this firmware image you can decrypt all
the traffic. No forward secrecy, right?

Now you don't have to be lucky and have
concerned citizens send you an email. You

can download this image from the GE website
and you can carve the keys out of the

image in the same way that I did and
decrypt the SSL traffic of any pcap that

is sent to you. Now the passwords
underneath that are in clear text. You can

see them highlighted down here. Password
Manager and user manager. You can see them

up there as well and you can see that
we've decrypted the SSL with that key. So

default keys, right? Is it a big deal? I
believe the vendors in this case say you

can upload your own key to the device. For
those of you who aren't used to working in

embedded it sometimes is difficult to
generate a key on the device because you

don't have enough memory or you don't have
enough entropy or you don't have enough

processing power. That's the usual
excuses. And they're true I shouldn't say

excuses those those things are true. But
you could of course generate it on the

client side and upload it to the device
and that's what they allow you to do with

this switch which is great but where is
your encrypted channel in which to upload

this key? laughs So you can use the serial
device and make sure visually that there's no man

in the middle. But if you're doing this
remotely – and I'd like you to keep in

mind that most substations are remote –
if anyone here works in a utility are you

going to drive to every substation, plug
in a serial cable to change the keys on

all these devices? It's the sort of thing
you need to know in advance right? So the

problem with key management, particularly
with SSL and the industrial systems

environment, is that you have to manage
the keys. And these particular keys, well

the certificates are self signed so you
can't revoke them. And besides industrial

systems are never connected to the
Internet. So it wouldn't have made any

difference. So these are the kind of
problems we're dealing with in this space.

And that's why I'm trying to encourage
you. Whether you do crypto or privacy or

whatever spend a little time in the
embedded space, just for bit: there's

plenty of easy work. OK. So what about the
second key. It requires a password. I

didn't feel like brute forcing it. Maybe
you do. I don't know. I tried all the

strings in the image. A classic technique,
just in case someone had a hard coded the

password. I mean the hard coded
credentials were there but not the hard

coded password. So I guess I gotta start
reversing, and as I previously said I suck

at reversing. That's why I come to CCC, so
I can learn, right? But I did find this

PowerPC ROM image. and I think its running
eCos and redboot and I haven't even gotten

down to doing hardware stuff: taking it
apart, having look at, it but I probably

will in the future. So there's the image
I'm slowly starting to learn my way around

and figure out what's going on. So I had a
look at the image and I figured out that

this key is used for SSH, right? Well it
would be the other encrypted thing. But I

couldn't enable SSH on the device. I try
and enable SSH on the device and I'm

logged in as manager by the way. which is
highest level user on this particular

device, and I put it in the passwords that
I know and a bunch of other passwords and

they don't work. Like I said, I tried all
the strings in the image. So apparently to

enable ssh, I need a password for
something. Now maybe I'm just

misunderstanding or I'm not so clear on
what's going on but I don't know about

you. I kind of feel like if I buy a device
that's supposed to be used for a safety

critical process I should be allowed to
use SSH without having to call up the

vendor and get some special magic
password. So considering I don't like that

approach. What if I patched my own key
into the image right. I don't know the

password of their key but I know the
password of a key I can generate. So I

just need to make sure it's roughly the
right size and try and patch it in. Then

I've got some problems with compression
because I've got to reverse the whole

process that I just described to you patch
it into the larger binary. Will there be

any CRC or firmware signing? I don't know,
right. So the uploaded image is not a

valid image for this device. That's
correct: I messed with it. But I got this

error and it gave me a clue. It gave me a
clue that I did indeed have some of my

CRCs wrong so when I altered the image
again I got to this state. So you're

learning all the time by having a real
device. Now some of my friends they do

static analysis and they don't buy these
devices. I decided to buy this one. I

found one on eBay. It wasn't very
expensive. I mean it depends on your range

for expensive. But if you're helping
defend industrial systems I thought it was

worth the money. So I bought it and this
enables me to try firmware images out and

I can slowly start to figure out what I
need to patch on these firmware images to

do whatever I want. Luckily I just tried
to patch mine to have SSH because I

thought people deserve to have SSH. So
that's an Adler 32 up there on the left

and the other CRC is on the bottom so that
Adler 32 and some adjustment of file

length although zeros in that line just
above it eventually got me to the point

where it believes it's a corrupted binary.
And then we have this CRC on the end that

we need to have a look at. Now I'm a big
fan of suspense. I love suspense. I'm

going to leave that one is a cliffhanger
and an exercise for you watching. So I

said I was going to talk about GE ML800
but I'm also going to talk about

Garrettcom. Luckily it's not very
difficult. Garrettcom is the original

equipment manufacturer for the GE ML800
series. I noticed that because the

certificate I found attached to those
private keys said Garrettcom in it and I

went and looked at their firmware images
and they have similar CRC similar file

structures similar everything so I believe
that they are affected by the cross site

scripting, the denial of service, and
hardcoded keys. I understand from some

people that they have been in contact with
GE to try and fix some of this stuff but

their response to GE was mainly "Sorry,
this is the end of life on this device".

That's fine. I understand you're running a
business but you're selling equipment to

people who manage utilities that we all
depend on. If Sony goes bankrupt because

they get hacked that's one thing right.
But you can't just dissolve a utility and

start again. As my friend Klaus points out
regularly – fantastic insights into the

industrial system world, Klaus and Vanessa
– you can't just dissolve the utility and

start again. You still have the same
infrastructure you still have the same

workers. It doesn't work that way. You
can't bail out utilities that we depend

on. So sorry. End of Life... I don't even
understand why people buy these devices

and this code without code escrow. When
you buy the code make sure you have the

code in perpetuity for these systems so
that you can fix them when something like

this or something worse happens. If I'm
your worst nightmare, you have real

problems because there are very dark
people in the world actually damaging

furnaces in Germany. So me disclosing keys
on stage is scary for you. You need to get

a grip. So, garrettcom?
Here's your key too.

Applause

The strings come from the images.
Developers are funny people really. I like

this. I just put them up because they're
funny. Some people had some hard times, I

guess, writing some of this code. And my
respect to them! They do great work but

you know, there's a couple of things we
can improve on security in these devices.

So I once had the opportunity to stand in
front of six different vendors at the same

time their computer emergency response
teams at a conference and I said to them,

"Will any of you commit to an average
patch time for vulnerabilities of three

months?" An average patch time, because it
might take 8 months, as it so far has

taken in the case of GE and Garrettcom, to
work on these issues. It might take a long

time in some cases but as an average patch
time I think 3 months for things that we

all depend on is reasonable. So I asked
these six different teams in the same

room. If any of them would commit to this
and I heard silence for 30 seconds. So my

friend decided to call this the silence of
the vendors right. And I think that's that

sums it up. I'd like to see better patch
times. I'd like to see a computer

emergency response teams in each of these
vendors and I'd like to see someone

responsible for security in each of these
different utilities. I can dream, right? I

think that key management... the current
practice industrial systems is to take

some insecure protocol and wrap it in SSL
or TLS which is why we need the help of

you privacy people because TLS and SSL
are not the be all and end all. They often

sort of go the wrong way, right. For
example you can use TLS to do integrity

without encryption so you can verify that
every message has reached its destination

intact but it is not encrypted. And this
means that you can still do intrusion

detection analysis of the packets. That's
really good. But nobody uses that in SSL

in other ways right. I'm a big fan of
Shodan and use Shodan for a variety of

different things usually to get a sense of
the Internet as a whole, right? Let me

back up a little bit. When I was at
Cambridge I went to Darwin college and

because you're at Darwin college you read
up a bit on Darwin and you think about how

Darwin thought and I think the Internet is
kind of like that. When it was built by

the IETF and various people, who did
fantastic work, they imagined it one way

and then we inherited it and it grew and
it became an ecosystem and stuff happens

out there that you wouldn't expect. And so
that's why I like Shodan. It's kind of

like being a natural scientist: what's a
survey of the world, what kind of machines

are out there, what versions are they
running, when do people update their SSL..

err, you know, their certificates do they
do it before or after the certificate is

invalid. Do they always upgrade the
algorithm. Do they increase the key size.

You know how do things change right you
need to sort of study it as a whole and

that's my point when it comes to just
taking SSL and slapping it over a

protocol. It's not quite that simple. So
again we need your help. Where can we go

with these attacks. And you remember at
the beginning I pointed out the underpants

gnome. The emperor wears no clothes.
Altering switch configurations is a big

deal because you can exfiltrate process
data. That gives you a map of the process

because industrial systems are bespoke.
Each one of them is different. It does run

different traffic and we are lucky to work
on security in this space because our

users are numerate and literate and they
care about safety. They don't always

understand security but they do care about
safety. So if you can make it a safety

concern they care. There are also
engineers that many of these utilities who

look at the network 24/7. Not all of them
but some of them. Can you imagine a home

network or something else with that kind
of user base. We're lucky we should be

taking advantage of that user base. So
getting back to the point you know denial

of service attacks to disrupt the process
go and see Marmusha's talk. This will all

make a lot more sense when you go and see
her talk. Basically any man in the middle

attack can disrupt alter or drop traffic
at this point. If you can affect the

switches and the substation. And
exfiltrating in the data gives you a map

of the process which leads towards further
potential damage for the utilities. Now

it's not always that simple people will
get up on stage and they will tell you I

am awesome and this is how it's done and
it's easy to blow shit up. It's not true.

It takes a little bit of thought it takes
a little bit of work. I am certainly not

awesome. I am just a quality assurance
person from a former vendor. I just

decided to get into security and keep
going with it. So you can't always perform

these man in the middle attacks. People
will say you can. But the reason you can't

is real-time system constraints. Some
systems will stop receiving traffic five

milliseconds or microseconds later and
ignore anything. If a value doesn't arrive

in this time it doesn't care. So the idea
that you can route the traffic out to some

other country and then back in and disrupt
the process is bollocks. Sometimes you

have to alter the firmware to achieve
that. That depends on the process but I'm

just trying to give you a sense of how
performing actual attacks give you a sense

of what the limits are, what the
logistical burdens are for the attacker

and that's important stuff for us to know.
All right. Little bit of an overview.

Drunk session IDs. brute forcing
MD5+NONCE, cross site request forgery for

firmware upload (of all things),
reflected cross-site scripting (8 cases of

it) pre authentication denial of service,
hardcoded keys times 2 in a firmware

image, SSL without forward secrecy, self
signed certificates so there's no revoking

there's no managing of the keys on these
devices right. Not to mention utility

workers are busy already. They may not
have time to manage all of these devices

we might need to rethink that approach
right. Clear text passwords under SSL

because well no one can break SSL unless
you hard code the key in the firmware

that's downloadable from the internet.
Enable ssh with a password and three

quarter of a year waiting for fixes for
some of this stuff. I'm not happy with

that. I think that we could live in a much
better, much safer world. And to do so we

need to talk very seriously about some of
these issues. Don't take my opinion for

it. Listen to some other people. The best
thing about doing industrial systems work

is the diversity of approach. You know I
love that there are so many other people

doing SCADA and ICS. And I love that
they're going different directions. So in

the future I plan to be on another stage
with some friends and show you some more.

Thank you for listening mustache fans and
as a parting thought. More tax money is

spent on surveillance than on
defending common utilities.

Applaus

Herald: Thank you. It made me a scary
Sunday morning. They got a utility *<<

guess, mostly incomprehensable* down the
road. OK. We'll have some questions taken

please. As the session is recorded and
streamed anything you say, say it into a

mic. Any questions up? Wow, it is Sunday
morning.

Éireann: Number three, sure

Herald: everybody understood everything?
You're kidding me.

Éireann: I've got one right here
Herald: here is a question.

Question: Hey thanks I enjoyed your talk
and I think it's very important to raise

awareness. But I think it's not to raise
awareness. Not much in this community, but

within the engineering community and I see
it a lot of times and many engineers

having lots of problems doing that for
several reasons. There is maybe the

engineer who is thinking about this but
has its miniatures in the back has to deal

with service personnel which know how to
work a hammer and a screwdriver and on the

other side, engineers have to work with
customers which more those lazy people.

And so that's how these things happen. And
I think it's more important to raise

awareness of these kinds of things in the
engineering community.

Éireann: So just to repeat a little bit
for anybody else that couldn't hear it or

for the recording it's very important to
work with the engineers some of the

engineers understand the problem. But
typically management or lower level

service personnel don't always understand
the problem. And it's not important to

raise the awareness in the hacker
community. But more with the engineers is

what you were saying. Right. OK.
Absolutely true. Completely agree with

you. I don't just come to these
conferences and present to you guys. I go

and I present to the engineers too. And in
fact a couple of engineers have come to

this conference because we did work at
other conferences to see what the hacker

community is about and learn things from
the hacker community because this is a

place where you can learn if you're just
not afraid of getting pwned a couple of

times right. And it happens to me too
right. I learned a lot from getting

compromised on my machine and watching
someone do something. Anyways back to the

point I don't just work with engineers or
hackers. I also work with C-level

executives so I'm on a sabbatical from
IOActive at the moment. at the Cambridge

Center for Risk studies, and I'm working
with the insurance people which has its

challenges shall we say. But some of them
are very intelligent people and they want

to understand what's going on with hacking
attacks and they want to approach this

from a slightly different angle. My stake
in that is to be sure that when the

insurance people do get involved that they
actually ask for fixes and improve stuff.

So yes I do my best to raise awareness
wherever I can. And I'm not alone. You can

help me.
Questioner: Thank you

applause

Herald: OK, there's another question here.
Number two. Oh, and up there too, yes we

saw you. OK number two was first I think.
Go ahead

Question: incomprehensible. So you
mentioned a couple of things, err a couple

of vulnerabilities and I was wondering
what you would think an ideal system would

look like. You mentioned key provisioning
of course putting certificates. I assume

that they were different certificates for
different devices rather than the same

certificate for all devices. Okay that's a bad
thing. And and also sort of the way how

the software update management works. So
how would you if you could give them some

advice how to design a system
how would you do it?

Éireann: Okay. So first of all I wouldn't
hard code the keys as you as you discussed

to be in every device the same. It's one
thing to put in your documentation hey you

should update the keys but I mean if I can
patch binary file with a key then there's

no reason you couldn't do that on the
website where you download the firmware

image right. Just as an example as a
thought experiment sort of makes that

clear. The upgrade path for these devices
is download the firmware image from the

website to some machine and then carry it,
because all these systems are airgapped.

to some other location and then upload it
onto the switch right with hardcoded

credentials. So first off whenever you
provision a switch initially you provision

all of the credentials for that device.
That's standard practice of many routers

and other pieces of equipment today. And I
would think less about defending and

securing the device than on being
able to regularly check its integrity,

the integrity of the firmware that is
running and the integrity of the

configuration. So I'd focus on that and I'd
focus on being able to recover the switch

after it's been attacked. So you reverse
your thinking. You assume that one day

someone is going to crack your firmware
signing and crack this and crack that and

you focus on how can I quickly upload a
new firmware image that is known to be

good and verify that the one that is
uploaded is good to this device.

Questioner: Thank you.
Herald: There was a question up there on

the balcony.
Signal angel: Yes we have two questions

here on the net. So the first one is how
would you solve the end of life issue.

Sometimes incomprehensible clients just
gets really outdated.

Éireann: That's absolutely true and it is
slightly unfair of me to be a hard on the

vendors. But it's my job to take the
debate a little bit too far the other way.

So how would I solve the end of life issue
is the question from the internet. I don't

know. I think that's not a technical
problem it's a societal problem. Like when

we buy bridges they are bridges until they
fall down. When we buy roads they stay

there until they go away. I mean there is
probably some end of life issues in there

but it's almost more of a contractual
legal issue and someone should study that.

There are people studying that but it's
not my area of expertise but I'll try and

answer as best I can. I think code escrow
is a good way to go when you buy some of

these devices you say I want the code for
this device in the future. I want to have

access to it. If your company goes
bankrupt I need you to give up the source

code for these devices when you go
bankrupt or when you disappear or when

it's the end of life. There are a couple
of manufacturers out there doing open

source switches. There's a company called
Open gear who are awesome. They gave me a

switch to play with that I haven't had
time to look at yet. I think that's amazing

right. And their code is open source and
you can go and examine it. So you would

have the code anyway. Those are two
different approaches. I think there are

others you can solve this problem
technically or legally or socially but as

a society we depend on these utilities and
that code should not just vanish when it's

difficult or costly to keep it upgraded.
applause

Herald: There was a second
question from the Internet.

Signal angel: Yes, so the second one is:
what should a non-technical person in

the respect of incomprehensible set non-
technical person sent to manage small town

utility do as best practice?
Éireann: I think the first and most

important thing is to look for attacks.
I'm sorry I should probably repeat that

question just to be sure. What should
someone in a small town who manages

utility do to defend themselves and
protect himself. So the first thing is

look for attacks. Even if you spend a few
hours a week looking for something you

script something up or you hire some
college kid to come in and script

something and look for things on your
network and ask questions and yes they're

going to be a pain in the ass and is going
to be difficult. But you're going to learn

things about your network and you might
detect some attacks. The first problem in

utilities is no one is responsible for
security. It's not my job. It's kind of

the mantra so for a small utility find
someone whose job it is if you're a very

small utility there's probably some other
small utilities near you and you can hire

a resource together to come and visit your
different utilities and help you out. The

second one is watch your relationship with
your vendor when you purchase this

equipment you spend a lot of money on it.
Spend a little bit of time doing

penetration tests. Yes I like it when you
hire me but you don't have to hire me.

There are plenty of other people you can
hire who will have a look at the device

and find the simple vulnerabilities. So
when you purchase something make sure you

test it for security purposes and that's
very important because you can even put

into your contract if you fail the
security tests we will pay you less money.

And the vendors are not going to react
to security until you do that. So that's

the second answer. And I wish I had a
third to make it very neat but I don't.

Herald: OK. There was one more
question at mic 4 I think

Questioner: Yes hi thank you for
your time.

Herald: Talk into the mike please. Thank
you for your talk. Q Hi. I'm kind of a

newbie to the C3 community and I am not
sure about the question I want to ask you.

Probably many people understand in this
room but I don't know if I would like to

ask you what exactly do you
mean by arbitrary firmware.

Éireann: No problem. So the question was
What do you mean by arbitrary firmware. I

mean the firmware that I have altered that
was not manufactured by the vendor to do

whatever I want. How do you trust that
this switch sends all the packets that it

should send. What if it's, you know, my
handle is BSB right. What if it drops

every packet that has BSB in the packet.
Right. You can rewrite a firmware image to

do whatever the device can do and in some
cases more things than the device usually

does to damage itself for example. So an
arbitrary firmware is one in which anyone

writes the firmware and there is no
checking to be sure that this is the image

that you want on this device whether it's
provided by the vendor or the community

right. You still want checking that this
is the correct code or the code that you

wanted anyway. Right.
Herald: Okay thank you. Is that a question

here mic 1? OK go ahead.
Questioner: Yes please. In your

hypothetical question, you asked what
damage could I do in that paint factory.

But you can also reverse it. What kind of
company secrets can I obtain for example,

your favorite recipe for your hot
chocolate or the recipes of Coca-Cola.

They are vulnerable as well aren't they.
Éireann: Yes. So the question just again

for everyone else. You don't just have to
talk about damage in a paint factory or

any industrial system. You can also talk
about intellectual property and protecting

the recipes that we use to bake cookies or
make beer or whatever pharmaceuticals

whatever. And that's a fantastic question
and I'm glad you brought it up a couple of

years ago when I was doing... well, more
than a couple of years like eight years

ago, when I was doing industrial system
security I realized I wasn't getting a lot

of traction. It was before stuxnet, I was
a quality assurance guy. Everybody thought

I was fucking crazy right. Stuxnet,
career. It's wrong. It's really wrong. But

the point is I tried to take that
approach. I tried to say you have a

process in which you manufacture something
and you make money by the fact that that

process is relatively secret and if you
don't care about defending your workers

from being damaged then at least care
about the intellectual property because

I'll get security in by some sort of back
door right. I'm a little bit of a security

Machiavellian. I'll find a way to get
security into the system somehow. So I

tried to say intellectual property you
should be protected. And I found that they

didn't care so much. I mean maybe you'll
have more luck maybe post-stuxnet that

that's a better argument. I hope you do.
But it is an important question as well.

Right. It's not, it's not just potential
for damage. I think there's a lot more

espionage going on on these networks than
there is damage and sabotage. Herald: Okay

we'll take one more question on mike four.
Questioner: Thank you okay. My question

concerns the concepts of software defined
networking and open flow. So when I first

heard about software defined networking I
thought well this is a huge security issue

and there may be huge vulnerabilities.
After your joke I think this might

actually be a good idea to dumb down the
switches and put the intelligence

somewhere locked up in a safe place.
What's your opinion on that. Can they

actually improve security.
Éireann: Yes. So the question is what role

could software defined networking play in
these sorts of environments. And is it a

good idea from a security perspective.
Anytime someone has a revolution in

computing we also have to update our
security paradigm. So I think with

software defined networking it's not
whether it's good or bad it's that you

defend that network differently than you
defend one of these networks. So it's not

so much that as good as good or bad it's
neutral if you know how to defend your

network. I don't care what it is. As long
as someone is looking to defend it and

cares about how the flows are working. So
I think software defined networking in

these environments could be a very good
thing but the refresh rate on these

devices is not that high. So I don't think
we'll see it there for a little while even

though it might be a good thing
philosophically. It takes 5 10 15 20 years

to refresh these networks so it'll be a little
while. But it's not good or bad. It's just

learn to defend what you got is the
problem right.

Questioner: Okay thanks a lot.
Herald: Okay okay let's give a big hand

for Éireann and thank you.
Éireann: Thank you

applause

subtitles created by c3subtitles.de
Join, and help us!