1
00:00:03,974 --> 00:00:08,515
Thomas Fricke: Thank you very much for the
invitation. So second talk tomorrow –
2
00:00:08,515 --> 00:00:13,731
Thank you – ähm today. So this is my
background. More or less I do, Kubernetes
3
00:00:13,731 --> 00:00:19,086
security and critical infrastructure,
founded several companies and are now my
4
00:00:19,086 --> 00:00:25,392
main focus is on Kubernetes security. This
this rabbit hole of Kubernetes, if you
5
00:00:25,392 --> 00:00:31,267
look deeper into it, then you should be a
little bit scared, and I want to explain
6
00:00:31,267 --> 00:00:36,764
why. The first approach is the
application, and then the application
7
00:00:36,764 --> 00:00:42,976
normally is run in containers. And the
containers, what is not really well known,
8
00:00:42,976 --> 00:00:48,458
have access to service accounts in
Kubernetes, which is one of the major
9
00:00:48,458 --> 00:00:54,474
flaws in Kubernetes at the moment. If you
take over the service account, it might be
10
00:00:54,474 --> 00:01:01,088
that you can take over a cluster. and if
you can take over a cluster, you might
11
00:01:01,088 --> 00:01:08,156
take over a node and then your entire
cloud service account, Which is the work
12
00:01:08,156 --> 00:01:15,121
of somebody else, I will mention later on
these slides. So let's look what happens:
13
00:01:15,121 --> 00:01:21,952
So, the target is I have an application
exposed to the internet, and I want to own
14
00:01:21,952 --> 00:01:28,565
the entire cluster from outside.
Application might be vulnerable. Examples?
15
00:01:28,565 --> 00:01:36,538
Yeah, lots of them. One example I want to
present is imagetragick, who normally
16
00:01:36,538 --> 00:01:43,876
should not do eval or exec statements in
any framework – should be PHP, NodeJS, or
17
00:01:43,876 --> 00:01:50,644
any other framework – and execute commands
in the context of your application,
18
00:01:50,644 --> 00:01:56,311
because something can go wrong and
developers are responsible for this. Let's
19
00:01:56,311 --> 00:02:04,604
see how it looks like: This is the attack
model based on an attack. I thought it was
20
00:02:04,604 --> 00:02:13,471
old and has been fixed in 2016, but now
there was a new overview by Emil Lerner,
21
00:02:13,471 --> 00:02:22,343
who again showed, yes, you can, in current
versions of ImageMagick, exploit this
22
00:02:22,343 --> 00:02:32,640
attack. So, it works. ImageMagick is for
uploading images, so you convert the image
23
00:02:32,640 --> 00:02:41,163
in a different format, scale the size, and
then if you do something wrong in this
24
00:02:41,163 --> 00:02:46,554
image, you can own the entire container.
This also works for non containerized
25
00:02:46,554 --> 00:02:52,520
applications if you have a server running
something with ImageMagick on it. Please
26
00:02:52,520 --> 00:03:02,160
be careful. OK. If we have mastered this
step, the next step is, yes, we want
27
00:03:02,160 --> 00:03:08,167
access to the service account. And this is
by default enabled in Kubernetes: So, you
28
00:03:08,167 --> 00:03:14,026
have a Kubernetes design flaw because your
service account is exposed to the
29
00:03:14,026 --> 00:03:20,040
container where the application runs it.
The next step of an attacker is
30
00:03:20,040 --> 00:03:27,229
installation of additional software. So,
you want to take over. You need a curl or
31
00:03:27,229 --> 00:03:35,386
kubectl or chmod, and then you are owner
of the service account and can actually do
32
00:03:35,386 --> 00:03:42,897
commands by uploading pictures in
ImageTragick. So responsible for this flaw
33
00:03:42,897 --> 00:03:50,920
is the image creator. Let's see what else
can happen. To get total control, you also
34
00:03:50,920 --> 00:03:57,514
need role-binding to a cluster-admin role.
This is not enabled by default, but the
35
00:03:57,514 --> 00:04:04,067
internet is always good for bad advice. So
if you copy the installation requirements
36
00:04:04,067 --> 00:04:11,224
or recommendations from the internet,
somebody else might take over the entire
37
00:04:11,224 --> 00:04:23,406
cluster. Let's look deeper into it: Worst
practice here is what you can see in the
38
00:04:23,406 --> 00:04:34,320
elastic installation recommendation: They
just mentioned they have a newer version,
39
00:04:34,320 --> 00:04:43,200
but they use the cluster admin permissions
here to install ElasticSearch in your
40
00:04:43,200 --> 00:04:53,680
Kubernetes cluster. So they recommend it
and a lot of other applications also have
41
00:04:53,680 --> 00:04:59,760
this – which is a little bit outdated, but
it's quite common – in the installation
42
00:05:00,480 --> 00:05:08,000
requirements. Never, ever do this, please.
It also can come with Helm Charts, so you
43
00:05:08,000 --> 00:05:16,560
have Helm Charts where the cluster-admin
role is included. Here you see it, it was
44
00:05:16,560 --> 00:05:23,200
in Apache Heron, which is an Apache
project, and it uses the cluster-admin
45
00:05:23,200 --> 00:05:40,000
role, so by a helm install you might be
affected by this flaw to. So with these
46
00:05:40,000 --> 00:05:46,800
four steps, which effectively are three
steps, you have a cluster application
47
00:05:46,800 --> 00:05:53,680
exposed, and through that path, you can
take over the entire cluster from the
48
00:05:53,680 --> 00:06:01,600
outside, and do anything what the cluster-
admin world can do. Effectively, is this
49
00:06:02,400 --> 00:06:09,680
cluster-admin role-binding is like a
doormat attack, so you have the best
50
00:06:09,680 --> 00:06:17,760
cryptography, the most expensive locks on
one side and then you put the lock under
51
00:06:17,760 --> 00:06:24,480
the doormat or under the flower at the
door or something like that. This is
52
00:06:24,480 --> 00:06:34,240
something which is, not really, what you
want. I can do an example walkthrough
53
00:06:34,240 --> 00:06:40,270
which shows how it goes. So, I've
published all my trainings notebooks on
54
00:06:40,270 --> 00:06:50,415
GitHub. Here's the way you can build this
out-dated ImageTragick version in
55
00:06:50,415 --> 00:06:57,834
OpenShift. So, I use CRC, which is the
code-ready container version. It's based
56
00:06:57,834 --> 00:07:05,134
on the ImageTragick proof of concept by
Mike Williams. And here you run and create
57
00:07:05,134 --> 00:07:13,582
a vulnerable image. A little bit lengthy.
It's compiled inside and so on. So, don't
58
00:07:13,582 --> 00:07:19,890
get a full Version. Which is the reason
why I don't show it here, but effectively
59
00:07:19,890 --> 00:07:29,800
at the end, you have a vulnerable
application in a container internal and in
60
00:07:29,800 --> 00:07:37,990
OpenShift. And that's exactly what we need
to run the application. Here is the
61
00:07:37,990 --> 00:07:48,560
exploit. And the exploit starts with the
deployment of this container, which is
62
00:07:48,560 --> 00:07:54,000
standard Kubernetes. Here "oc" is like
kubesctl. So, you get an overview.
63
00:07:54,720 --> 00:07:59,680
Additionally, in OpenShift, you have a
very simple version of creating a root,
64
00:08:00,320 --> 00:08:07,200
which is connected to a hostname, and then
you can upload it by using that hostname.
65
00:08:09,040 --> 00:08:16,230
You expose the deployment, you expose the
service which is created, you expose the
66
00:08:16,230 --> 00:08:22,920
route finally, and then you have access.
The next step is you get this root and
67
00:08:22,920 --> 00:08:30,814
then here you have a URL, which you can
use. And in a full demo, I would just
68
00:08:30,814 --> 00:08:38,063
simply call this URL and then I can upload
images here. I've created these files,
69
00:08:38,063 --> 00:08:45,000
which are valid postscript files, but you
see at the end there is a full command.
70
00:08:45,000 --> 00:08:50,957
And here, because there's a curl in the
container, I can download a version of
71
00:08:50,957 --> 00:08:58,374
kubectl. Effectively, the containers,
specially the RedHat containers are not so
72
00:08:58,374 --> 00:09:07,578
vulnerable as others, but you have always
writable temp, which is enough to deploy
73
00:09:07,578 --> 00:09:16,724
some software. So, we curl kubectl from
the internet, put it into temp, and then
74
00:09:16,724 --> 00:09:26,042
we use a simple chmod command to activate
kubectl. So now we can call kubectl
75
00:09:26,042 --> 00:09:39,680
commands from inside an image. It's a
death bells, more or less so. Exactly at
76
00:09:39,680 --> 00:09:46,400
the right place. We have a working exploit
now and warning, it might also already
77
00:09:46,400 --> 00:09:52,640
work in older versions of Kubernetes.
Because in newer versions will need some.
78
00:09:54,320 --> 00:10:01,600
Pill of poison, additionally, and this is
exactly this cluster all binding to the
79
00:10:01,600 --> 00:10:06,880
cluster admin, which needs to be done,
that we have full access from the outside
80
00:10:06,880 --> 00:10:15,840
and if we do this, and expose our cluster
admin account to the same account, which
81
00:10:15,840 --> 00:10:21,760
is already exposed inside the container,
we can execute commands with this kubectl
82
00:10:21,760 --> 00:10:29,200
so we can create deployments by uploading
pictures. Which is exactly what you never
83
00:10:29,200 --> 00:10:34,800
want, but an attacker now has full access
to your cluster by simply uploading
84
00:10:35,920 --> 00:10:46,178
prepared malicious pictures. Can do this.
So this is an example here, just. Create
85
00:10:46,178 --> 00:10:50,732
and delete. Containers and deployments
this way, you can effectively do
86
00:10:50,732 --> 00:11:10,560
everything. And again, this is the problem
here from the application side. If you
87
00:11:10,560 --> 00:11:20,560
have a vulnerable version of ImageMagick,
you can include commands, and you can
88
00:11:20,560 --> 00:11:27,440
definitely install software on the
Kubernetes server side. There are several
89
00:11:27,440 --> 00:11:34,800
trys to fix this. For example, you can use
better images like Red Hat does, so this
90
00:11:34,800 --> 00:11:39,760
is a Red Hat health index, which is quite
good, but effectively these images have
91
00:11:40,320 --> 00:11:48,480
the advantage only that you not run
anything as root. But you run the same as
92
00:11:49,440 --> 00:11:54,960
another user I.D. and it's the same user
is allowed to write to the temp directory,
93
00:11:54,960 --> 00:12:04,080
effectively, yeah, you don't need root for
installing software. So, the container
94
00:12:04,080 --> 00:12:12,000
also was good practice, no root inside, it
has an immutable root file system, but the
95
00:12:12,000 --> 00:12:16,960
curl which is completely unnecessary, was
also deployed, we had write access to
96
00:12:16,960 --> 00:12:23,200
temp. We had a chmod. And the first thing
you would prevent. All the stuff I'm doing
97
00:12:23,200 --> 00:12:28,800
here is and if you're going to and don't
learn anything from this talk, please go.
98
00:12:30,320 --> 00:12:36,000
Look into your service account and try to
disable the automountServiceAccountToken
99
00:12:36,720 --> 00:12:41,520
features, so all of the service accounts
which are not running operators don't need
100
00:12:42,160 --> 00:12:49,200
this service account open. If you have an
operator, it might be broken now and it
101
00:12:49,200 --> 00:12:58,480
can be, um, overwritten by the Pod
definition, but effectively this. entire
102
00:12:58,480 --> 00:13:04,560
example would not work without this
service account token. So, we have fixed
103
00:13:04,560 --> 00:13:10,240
that. We cannot fix the application
because this is something, uh, somebody
104
00:13:10,240 --> 00:13:14,960
else is creating for us, and we might even
have a floor which is not affected, so
105
00:13:14,960 --> 00:13:19,840
there might be a zero-day. The next thing
we must prevent is the installation of
106
00:13:19,840 --> 00:13:29,760
software. Fix the images, so use really
immutable images. Temp only if you need
107
00:13:29,760 --> 00:13:39,760
it. PID is 1, anyway. Uh, OK, you might
have some variable data, but you should
108
00:13:39,760 --> 00:13:47,920
use containers from scratch, no curl, no
wget and this also affects Red Hat UBIs
109
00:13:47,920 --> 00:13:52,880
And most of the standard images have this
flaw, so you have a full operating system
110
00:13:52,880 --> 00:14:02,320
inside with all the tools you like. But
this is not your territory. It's just,
111
00:14:02,320 --> 00:14:08,240
yeah, it's a tool for the attacker. So
please run only trusted images, build your
112
00:14:08,240 --> 00:14:16,560
own images and build them from scratch.
This is my example I also have uploaded to
113
00:14:16,560 --> 00:14:22,800
GitHub, how to harden the container, which
is based on nginx alpine. nginx alpine
114
00:14:22,800 --> 00:14:27,520
normally is a very small container, but
you can do more. You can use the script,
115
00:14:28,400 --> 00:14:33,920
which is in this repository, just to get
only the tools you need. So this is not
116
00:14:33,920 --> 00:14:39,557
statically linked because the original
engine is not statically linked. But it's
117
00:14:39,557 --> 00:14:55,905
very close. This means you only positively
install the software you need. This is
118
00:14:55,905 --> 00:15:02,048
dynamically linked, therefor the -d, so we
use LVD. Extract all the dynamic link
119
00:15:02,048 --> 00:15:09,735
libraries and then all the configuration
files which are necessary. It is the
120
00:15:09,765 --> 00:15:17,743
password registry group. OK. Some licenses
and share. Need some directories for
121
00:15:17,743 --> 00:15:23,154
logging and then you can install it from
scratch because this script installs it in
122
00:15:23,154 --> 00:15:32,397
a directory \temp\harden and you can with
this. Multi-stage build you can install
123
00:15:32,397 --> 00:15:41,578
all what you need from \temp\harden. And
then the next container is based on
124
00:15:41,578 --> 00:15:48,908
scratch and you can use nginx the same way
you would use it more or less. An
125
00:15:48,908 --> 00:15:57,423
application which is statically linked. So
now we have created a hardened image
126
00:15:57,423 --> 00:16:04,328
without kubectl, curl inside. So, we are
much closer to a secure application. The
127
00:16:04,328 --> 00:16:10,699
next thing is, yeah, role binding to
cluster admin role. Don't do this. If
128
00:16:10,699 --> 00:16:17,954
something in your application goes wrong,
you have additional measures, which you
129
00:16:17,954 --> 00:16:24,675
can take just to prevent the application
from break-out of the container. So, you
130
00:16:24,675 --> 00:16:30,432
can separate the internet exposure of
services or ingresses in Kubernetes from
131
00:16:30,432 --> 00:16:36,640
privilege operations. So you have node
settings. ElasticSearch is doing a lot of
132
00:16:36,640 --> 00:16:44,200
these things, so a lot is really not true
so, doing a sysctl. Some applications have
133
00:16:44,200 --> 00:16:51,851
hostPaths on or have connection to the
host inter-process communication, which is
134
00:16:51,851 --> 00:16:58,477
not necessary if you have exposed it and
then separate the applications who need
135
00:16:58,477 --> 00:17:04,135
this from the applications which don't
need it. So, cluster admin should be more
136
00:17:04,135 --> 00:17:09,463
or less restricted to very privileged
operators. And by the way, Argo is also a
137
00:17:09,463 --> 00:17:14,852
very privileged operator. Don't run an
Argo on a Kubernetes cluster in a security
138
00:17:14,852 --> 00:17:21,193
critical environment because I've seen
Argo also is binding to cluster admin. It
139
00:17:21,193 --> 00:17:26,798
doesn't mean that Argo by default is
unsafe, but it's a very complex
140
00:17:26,798 --> 00:17:33,800
application and I would definitely run it
in a separate cluster, not in the critical
141
00:17:33,800 --> 00:17:40,958
cluster. And what does an architecture fix
look like, here you have the lifecycle of
142
00:17:40,958 --> 00:17:47,536
a Pod, so the time is going to from left
to right. Here you see if the container is
143
00:17:47,536 --> 00:17:55,931
ready, it can be accessed from the
internet. And if you do something from the
144
00:17:55,931 --> 00:18:04,064
init system, like a sysctl, please do it
inside a container which is not connected
145
00:18:04,064 --> 00:18:09,722
to the internet, just to use the pause
container, as a pause container to limit
146
00:18:09,722 --> 00:18:15,721
it and restrict it and that is not really
connected to the network. So, this is
147
00:18:15,721 --> 00:18:22,786
something which covers the architecture.
Additionally, I already mentioned here the
148
00:18:22,786 --> 00:18:28,357
network policy which will come later, so
this is our threat matrix. We have exposed
149
00:18:28,357 --> 00:18:32,908
and not exposed services. You have
unprivileged and privileged things. The
150
00:18:32,908 --> 00:18:39,385
dangerous ones are the privileged ones
which are exposed, but normally you only
151
00:18:39,385 --> 00:18:46,295
have an exposed privileged application if
you have an IDE running in Kubernetes,
152
00:18:46,295 --> 00:18:50,502
which is not what I would like to see in
critical infrastructure, something like
153
00:18:50,502 --> 00:18:57,680
rstudio or have a web ui to a gitops
framework. And normally you only have a
154
00:18:57,680 --> 00:19:03,682
web application. And what should not be
exposed under normal conditions is an
155
00:19:03,682 --> 00:19:11,657
operator's sysctl, build systems, host
operators and so on. If you do this, it's
156
00:19:11,657 --> 00:19:20,702
virtually not possible to own the cluster,
you should do all the three because if you
157
00:19:20,702 --> 00:19:26,480
have security in depth, you can make a
mistake on one of these levels and the
158
00:19:26,480 --> 00:19:32,149
other means other levels keep you from
being exploited. You can even do more
159
00:19:32,149 --> 00:19:38,640
isolation on the network side, you have
network policies for egress on the node
160
00:19:38,640 --> 00:19:44,482
side, you can activate seccomp, gvisor,
and the common Frameworks, SELinux,
161
00:19:44,482 --> 00:19:49,646
Apparmor. You can use PodSecurity
policies, or in the future, the open
162
00:19:49,646 --> 00:19:56,070
policy agent to prevent the node from
being hacked. For the identity and access
163
00:19:56,070 --> 00:20:02,801
management, you should use individual
service accounts for all your tasks. So
164
00:20:02,801 --> 00:20:08,557
you have enough of a lot of roles. You
should use role based access control to
165
00:20:08,557 --> 00:20:18,300
check this. OK, but I promise, yes, we can
go even deeper, and this needs a little
166
00:20:18,300 --> 00:20:27,559
help from your cloud administrator and
here, the example from Nico Meisenzahl,
167
00:20:27,559 --> 00:20:34,867
who does a very similar example on hi-
jacking Kubernetes, and he's doing it,
168
00:20:34,867 --> 00:20:42,982
obviously in one of the clouds. And what
he has found out is you can get access to
169
00:20:42,982 --> 00:20:49,258
the azure.json file, which has user
assigned identities. This is not the
170
00:20:49,258 --> 00:20:55,186
Kubernetes identities. This is the Azure
identity. You can get a token, you can get
171
00:20:55,186 --> 00:21:01,670
a subscription, you can get a resource
group and then you can use a curl command,
172
00:21:01,670 --> 00:21:06,928
with this token, to change things on the
API version of this resource group with
173
00:21:06,928 --> 00:21:11,820
this subscription. So, you might be able
to hack your node with the privilege
174
00:21:11,820 --> 00:21:17,403
container and then take over your cloud
account. And he told me that this is also
175
00:21:17,403 --> 00:21:25,415
the truth for the other cloud, so it might
even work something similar in AWS and
176
00:21:25,415 --> 00:21:33,045
GCP. So please, also protect your cloud
account. Understand your identity and
177
00:21:33,045 --> 00:21:38,200
access management in the cloud. So, at
least, someone in the team should
178
00:21:38,200 --> 00:21:43,837
understand it. And limit also the
underlying account to the bare minimum. It
179
00:21:43,837 --> 00:21:50,515
might even be a good idea to block access
addresses like 169.254.something. And the
180
00:21:50,515 --> 00:21:57,200
other clouds, as I already mentioned, also
might be affected. And my call to the
181
00:21:57,200 --> 00:22:03,451
cloud providers, is don't deliver account
data in containers or nodes. This is not
182
00:22:03,451 --> 00:22:07,571
necessary. It's yes, it's very
comfortable, as the service account
183
00:22:07,571 --> 00:22:13,499
talking is very comfortable for running
operators, but it's a major security flaw
184
00:22:13,499 --> 00:22:22,640
and it might be that you lose all your
accounts and data. Conclusion: We have a
185
00:22:22,640 --> 00:22:29,310
full attach chain from the application to
the cloud account. And it's your task to
186
00:22:29,310 --> 00:22:36,005
prevent it and fix it. This is called
shared responsibility, so the cloud
187
00:22:36,005 --> 00:22:40,631
providers effectively only care for the
infrastructure, but not really for the
188
00:22:40,631 --> 00:22:46,037
security in your clou d. This is your
task. OK. Thank you for your attention, I
189
00:22:46,037 --> 00:22:52,109
hope it was interesting. Please ask your
questions. And now I'm open for he Q&A.
190
00:22:52,109 --> 00:22:58,369
Applaus
191
00:22:58,369 --> 00:23:04,720
Herald: Thank you for the talk. This is
working? Yeah. So do we have any questions
192
00:23:04,720 --> 00:23:12,480
from the internet? I don't see any coming
in so far, but we, I think, usually a bit
193
00:23:12,480 --> 00:23:17,200
ahead so I'll ask one:
Q: What do you think? So who's in in the
194
00:23:17,200 --> 00:23:21,280
responsibility mainly to fix these
insecurities? Do you think this can be
195
00:23:21,280 --> 00:23:26,880
fixed by better default in these
infrastructures and configuration files?
196
00:23:26,880 --> 00:23:31,840
Is this to be fixed for better tutorials
and better education for the devop
197
00:23:31,840 --> 00:23:35,200
engineers? What was the main point of
responsibility?
198
00:23:35,200 --> 00:23:44,800
A: I definitely would prefer to have
secure default installations. But then you
199
00:23:44,800 --> 00:23:51,360
have this shared responsibility in the
contracts: From a certain point, you are
200
00:23:51,360 --> 00:23:57,440
responsible for the security of the
account, and we have seen this complexity
201
00:23:58,400 --> 00:24:08,732
because this might be 20 steps. Every step
is very simple and every step is looking
202
00:24:08,732 --> 00:24:15,999
very harmless, but all the steps together
might create a full exploit of a cloud. So
203
00:24:15,999 --> 00:24:23,979
this must be overseen, and it's very hard
for developers who are cloud native and
204
00:24:23,979 --> 00:24:30,417
are focusing on the application to have an
overview of the security. Developers now
205
00:24:30,417 --> 00:24:36,727
have 10 or 100 times more code on the hard
disk than ten years before. And this means
206
00:24:36,727 --> 00:24:43,360
developers are not able really to have a
full judgment about what is going on in
207
00:24:43,360 --> 00:24:49,319
terms of security. This is something
developers talk about security, either
208
00:24:49,319 --> 00:24:55,694
they are specialized on it or they have
not seen things like this. What I normally
209
00:24:55,694 --> 00:24:59,960
notice: The developers are not aware of
these problems.
210
00:24:59,960 --> 00:25:06,685
Q: OK. And what do you think, what can we
do about the complexity? So do you think
211
00:25:06,685 --> 00:25:10,374
we need better education for people to
actually understand the systems? Or is
212
00:25:10,374 --> 00:25:14,274
there a way in cloud infrastructures to
reduce the complexity?
213
00:25:14,274 --> 00:25:24,160
A: Better education? And do all the simple
fixes. These are five steps, and the fixes
214
00:25:24,160 --> 00:25:29,840
are also very simple. And you have to
check them and then you need a tool
215
00:25:29,840 --> 00:25:35,520
because you might have 20 clusters like
this. Every cluster has 20 applications,
216
00:25:35,520 --> 00:25:40,800
so this might be quite complicated. So you
need tools for an overview and in the
217
00:25:40,800 --> 00:25:46,720
trainings material, you see examples how
you can check your Kubernetes clusters for
218
00:25:46,720 --> 00:25:51,920
exploits like this.
Herald: OK, thank you very much. Thanks
219
00:25:51,920 --> 00:25:57,840
for being here. We will continue in about
half an hour with the next talk, then
220
00:25:57,840 --> 00:26:04,123
again in German. Thanks.
Thomas: Thank you very much. Applaus
221
00:26:04,123 --> 00:26:12,552
Outro: Everything is licensed under CC BY
4.0. And it is all for the community, to
222
00:26:12,552 --> 00:26:13,791
the unknown and for everyone.
223
00:26:13,791 --> 00:26:15,000
Subtitles created by c3subtitles.de
in the year 2022. Join, and help us!