Hacking Containers, Kubernetes and Clouds

Thomas Fricke: Thank you very much for the
invitation. So second talk tomorrow –

Thank you – ähm today. So this is my
background. More or less I do, Kubernetes

security and critical infrastructure,
founded several companies and are now my

main focus is on Kubernetes security. This
this rabbit hole of Kubernetes, if you

look deeper into it, then you should be a
little bit scared, and I want to explain

why. The first approach is the
application, and then the application

normally is run in containers. And the
containers, what is not really well known,

have access to service accounts in
Kubernetes, which is one of the major

flaws in Kubernetes at the moment. If you
take over the service account, it might be

that you can take over a cluster. and if
you can take over a cluster, you might

take over a node and then your entire
cloud service account, Which is the work

of somebody else, I will mention later on
these slides. So let's look what happens:

So, the target is I have an application
exposed to the internet, and I want to own

the entire cluster from outside.
Application might be vulnerable. Examples?

Yeah, lots of them. One example I want to
present is imagetragick, who normally

should not do eval or exec statements in
any framework – should be PHP, NodeJS, or

any other framework – and execute commands
in the context of your application,

because something can go wrong and
developers are responsible for this. Let's

see how it looks like: This is the attack
model based on an attack. I thought it was

old and has been fixed in 2016, but now
there was a new overview by Emil Lerner,

who again showed, yes, you can, in current
versions of ImageMagick, exploit this

attack. So, it works. ImageMagick is for
uploading images, so you convert the image

in a different format, scale the size, and
then if you do something wrong in this

image, you can own the entire container.
This also works for non containerized

applications if you have a server running
something with ImageMagick on it. Please

be careful. OK. If we have mastered this
step, the next step is, yes, we want

access to the service account. And this is
by default enabled in Kubernetes: So, you

have a Kubernetes design flaw because your
service account is exposed to the

container where the application runs it.
The next step of an attacker is

installation of additional software. So,
you want to take over. You need a curl or

kubectl or chmod, and then you are owner
of the service account and can actually do

commands by uploading pictures in
ImageTragick. So responsible for this flaw

is the image creator. Let's see what else
can happen. To get total control, you also

need role-binding to a cluster-admin role.
This is not enabled by default, but the

internet is always good for bad advice. So
if you copy the installation requirements

or recommendations from the internet,
somebody else might take over the entire

cluster. Let's look deeper into it: Worst
practice here is what you can see in the

elastic installation recommendation: They
just mentioned they have a newer version,

but they use the cluster admin permissions
here to install ElasticSearch in your

Kubernetes cluster. So they recommend it
and a lot of other applications also have

this – which is a little bit outdated, but
it's quite common – in the installation

requirements. Never, ever do this, please.
It also can come with Helm Charts, so you

have Helm Charts where the cluster-admin
role is included. Here you see it, it was

in Apache Heron, which is an Apache
project, and it uses the cluster-admin

role, so by a helm install you might be
affected by this flaw to. So with these

four steps, which effectively are three
steps, you have a cluster application

exposed, and through that path, you can
take over the entire cluster from the

outside, and do anything what the cluster-
admin world can do. Effectively, is this

cluster-admin role-binding is like a
doormat attack, so you have the best

cryptography, the most expensive locks on
one side and then you put the lock under

the doormat or under the flower at the
door or something like that. This is

something which is, not really, what you
want. I can do an example walkthrough

which shows how it goes. So, I've
published all my trainings notebooks on

GitHub. Here's the way you can build this
out-dated ImageTragick version in

OpenShift. So, I use CRC, which is the
code-ready container version. It's based

on the ImageTragick proof of concept by
Mike Williams. And here you run and create

a vulnerable image. A little bit lengthy.
It's compiled inside and so on. So, don't

get a full Version. Which is the reason
why I don't show it here, but effectively

at the end, you have a vulnerable
application in a container internal and in

OpenShift. And that's exactly what we need
to run the application. Here is the

exploit. And the exploit starts with the
deployment of this container, which is

standard Kubernetes. Here "oc" is like
kubesctl. So, you get an overview.

Additionally, in OpenShift, you have a
very simple version of creating a root,

which is connected to a hostname, and then
you can upload it by using that hostname.

You expose the deployment, you expose the
service which is created, you expose the

route finally, and then you have access.
The next step is you get this root and

then here you have a URL, which you can
use. And in a full demo, I would just

simply call this URL and then I can upload
images here. I've created these files,

which are valid postscript files, but you
see at the end there is a full command.

And here, because there's a curl in the
container, I can download a version of

kubectl. Effectively, the containers,
specially the RedHat containers are not so

vulnerable as others, but you have always
writable temp, which is enough to deploy

some software. So, we curl kubectl from
the internet, put it into temp, and then

we use a simple chmod command to activate
kubectl. So now we can call kubectl

commands from inside an image. It's a
death bells, more or less so. Exactly at

the right place. We have a working exploit
now and warning, it might also already

work in older versions of Kubernetes.
Because in newer versions will need some.

Pill of poison, additionally, and this is
exactly this cluster all binding to the

cluster admin, which needs to be done,
that we have full access from the outside

and if we do this, and expose our cluster
admin account to the same account, which

is already exposed inside the container,
we can execute commands with this kubectl

so we can create deployments by uploading
pictures. Which is exactly what you never

want, but an attacker now has full access
to your cluster by simply uploading

prepared malicious pictures. Can do this.
So this is an example here, just. Create

and delete. Containers and deployments
this way, you can effectively do

everything. And again, this is the problem
here from the application side. If you

have a vulnerable version of ImageMagick,
you can include commands, and you can

definitely install software on the
Kubernetes server side. There are several

trys to fix this. For example, you can use
better images like Red Hat does, so this

is a Red Hat health index, which is quite
good, but effectively these images have

the advantage only that you not run
anything as root. But you run the same as

another user I.D. and it's the same user
is allowed to write to the temp directory,

effectively, yeah, you don't need root for
installing software. So, the container

also was good practice, no root inside, it
has an immutable root file system, but the

curl which is completely unnecessary, was
also deployed, we had write access to

temp. We had a chmod. And the first thing
you would prevent. All the stuff I'm doing

here is and if you're going to and don't
learn anything from this talk, please go.

Look into your service account and try to
disable the automountServiceAccountToken

features, so all of the service accounts
which are not running operators don't need

this service account open. If you have an
operator, it might be broken now and it

can be, um, overwritten by the Pod
definition, but effectively this. entire

example would not work without this
service account token. So, we have fixed

that. We cannot fix the application
because this is something, uh, somebody

else is creating for us, and we might even
have a floor which is not affected, so

there might be a zero-day. The next thing
we must prevent is the installation of

software. Fix the images, so use really
immutable images. Temp only if you need

it. PID is 1, anyway. Uh, OK, you might
have some variable data, but you should

use containers from scratch, no curl, no
wget and this also affects Red Hat UBIs

And most of the standard images have this
flaw, so you have a full operating system

inside with all the tools you like. But
this is not your territory. It's just,

yeah, it's a tool for the attacker. So
please run only trusted images, build your

own images and build them from scratch.
This is my example I also have uploaded to

GitHub, how to harden the container, which
is based on nginx alpine. nginx alpine

normally is a very small container, but
you can do more. You can use the script,

which is in this repository, just to get
only the tools you need. So this is not

statically linked because the original
engine is not statically linked. But it's

very close. This means you only positively
install the software you need. This is

dynamically linked, therefor the -d, so we
use LVD. Extract all the dynamic link

libraries and then all the configuration
files which are necessary. It is the

password registry group. OK. Some licenses
and share. Need some directories for

logging and then you can install it from
scratch because this script installs it in

a directory \temp\harden and you can with
this. Multi-stage build you can install

all what you need from \temp\harden. And
then the next container is based on

scratch and you can use nginx the same way
you would use it more or less. An

application which is statically linked. So
now we have created a hardened image

without kubectl, curl inside. So, we are
much closer to a secure application. The

next thing is, yeah, role binding to
cluster admin role. Don't do this. If

something in your application goes wrong,
you have additional measures, which you

can take just to prevent the application
from break-out of the container. So, you

can separate the internet exposure of
services or ingresses in Kubernetes from

privilege operations. So you have node
settings. ElasticSearch is doing a lot of

these things, so a lot is really not true
so, doing a sysctl. Some applications have

hostPaths on or have connection to the
host inter-process communication, which is

not necessary if you have exposed it and
then separate the applications who need

this from the applications which don't
need it. So, cluster admin should be more

or less restricted to very privileged
operators. And by the way, Argo is also a

very privileged operator. Don't run an
Argo on a Kubernetes cluster in a security

critical environment because I've seen
Argo also is binding to cluster admin. It

doesn't mean that Argo by default is
unsafe, but it's a very complex

application and I would definitely run it
in a separate cluster, not in the critical

cluster. And what does an architecture fix
look like, here you have the lifecycle of

a Pod, so the time is going to from left
to right. Here you see if the container is

ready, it can be accessed from the
internet. And if you do something from the

init system, like a sysctl, please do it
inside a container which is not connected

to the internet, just to use the pause
container, as a pause container to limit

it and restrict it and that is not really
connected to the network. So, this is

something which covers the architecture.
Additionally, I already mentioned here the

network policy which will come later, so
this is our threat matrix. We have exposed

and not exposed services. You have
unprivileged and privileged things. The

dangerous ones are the privileged ones
which are exposed, but normally you only

have an exposed privileged application if
you have an IDE running in Kubernetes,

which is not what I would like to see in
critical infrastructure, something like

rstudio or have a web ui to a gitops
framework. And normally you only have a

web application. And what should not be
exposed under normal conditions is an

operator's sysctl, build systems, host
operators and so on. If you do this, it's

virtually not possible to own the cluster,
you should do all the three because if you

have security in depth, you can make a
mistake on one of these levels and the

other means other levels keep you from
being exploited. You can even do more

isolation on the network side, you have
network policies for egress on the node

side, you can activate seccomp, gvisor,
and the common Frameworks, SELinux,

Apparmor. You can use PodSecurity
policies, or in the future, the open

policy agent to prevent the node from
being hacked. For the identity and access

management, you should use individual
service accounts for all your tasks. So

you have enough of a lot of roles. You
should use role based access control to

check this. OK, but I promise, yes, we can
go even deeper, and this needs a little

help from your cloud administrator and
here, the example from Nico Meisenzahl,

who does a very similar example on hi-
jacking Kubernetes, and he's doing it,

obviously in one of the clouds. And what
he has found out is you can get access to

the azure.json file, which has user
assigned identities. This is not the

Kubernetes identities. This is the Azure
identity. You can get a token, you can get

a subscription, you can get a resource
group and then you can use a curl command,

with this token, to change things on the
API version of this resource group with

this subscription. So, you might be able
to hack your node with the privilege

container and then take over your cloud
account. And he told me that this is also

the truth for the other cloud, so it might
even work something similar in AWS and

GCP. So please, also protect your cloud
account. Understand your identity and

access management in the cloud. So, at
least, someone in the team should

understand it. And limit also the
underlying account to the bare minimum. It

might even be a good idea to block access
addresses like 169.254.something. And the

other clouds, as I already mentioned, also
might be affected. And my call to the

cloud providers, is don't deliver account
data in containers or nodes. This is not

necessary. It's yes, it's very
comfortable, as the service account

talking is very comfortable for running
operators, but it's a major security flaw

and it might be that you lose all your
accounts and data. Conclusion: We have a

full attach chain from the application to
the cloud account. And it's your task to

prevent it and fix it. This is called
shared responsibility, so the cloud

providers effectively only care for the
infrastructure, but not really for the

security in your clou d. This is your
task. OK. Thank you for your attention, I

hope it was interesting. Please ask your
questions. And now I'm open for he Q&A.

Applaus

Herald: Thank you for the talk. This is
working? Yeah. So do we have any questions

from the internet? I don't see any coming
in so far, but we, I think, usually a bit

ahead so I'll ask one:
Q: What do you think? So who's in in the

responsibility mainly to fix these
insecurities? Do you think this can be

fixed by better default in these
infrastructures and configuration files?

Is this to be fixed for better tutorials
and better education for the devop

engineers? What was the main point of
responsibility?

A: I definitely would prefer to have
secure default installations. But then you

have this shared responsibility in the
contracts: From a certain point, you are

responsible for the security of the
account, and we have seen this complexity

because this might be 20 steps. Every step
is very simple and every step is looking

very harmless, but all the steps together
might create a full exploit of a cloud. So

this must be overseen, and it's very hard
for developers who are cloud native and

are focusing on the application to have an
overview of the security. Developers now

have 10 or 100 times more code on the hard
disk than ten years before. And this means

developers are not able really to have a
full judgment about what is going on in

terms of security. This is something
developers talk about security, either

they are specialized on it or they have
not seen things like this. What I normally

notice: The developers are not aware of
these problems.

Q: OK. And what do you think, what can we
do about the complexity? So do you think

we need better education for people to
actually understand the systems? Or is

there a way in cloud infrastructures to
reduce the complexity?

A: Better education? And do all the simple
fixes. These are five steps, and the fixes

are also very simple. And you have to
check them and then you need a tool

because you might have 20 clusters like
this. Every cluster has 20 applications,

so this might be quite complicated. So you
need tools for an overview and in the

trainings material, you see examples how
you can check your Kubernetes clusters for

exploits like this.
Herald: OK, thank you very much. Thanks

for being here. We will continue in about
half an hour with the next talk, then

again in German. Thanks.
Thomas: Thank you very much. Applaus

Outro: Everything is licensed under CC BY
4.0. And it is all for the community, to

the unknown and for everyone.

Subtitles created by c3subtitles.de
in the year 2022. Join, and help us!