Herald: So could you give a warm welcome
of applause to Stephan Gronke who will be
talking to you in one minute.
Applause
Stephan: So hi everybody my name is
Stephan Gronke. I'm a software developer
since about 15 years, working in solo
projects and larger teams and smaller
teams. So mostly my development stack was
JavaScript and you will find some of the
tools that I mentioned coming from this
world, but I'm very sure you can also find
something for your project that applies
here. Here's my email address, my PGP key
and my favorite social network account.
Yeah, so a little spoiler what will happen
today: I will talk about development
process exploitation. So that means if you
are developing your software and somebody
joins your team and since you code for
review it could happen that it executes
code on your machine without your
knowledge. There are a few things that are
really hard to catch or I found hard to
catch and I want to share. Maybe you have
the same problems and you find that the
same mitigations apply for your project as
well. I will then continue and... let's
start with a software development process
that's a small cycle. So first of all it
starts with an operating system: you need
to have a computer to write a software and
that's something you need to trust first
off. If you... yes, so your operating
system contains keys and credentials, it
contains the source code you want to
develop and the tools that you have in
place. And the major risk is that tools
are vulnerable to some exploitation or
that your host is already compromised and
you write a software, commit it to your
coworkers and it isn't what you intended
to write. That's a larger problem here.
After you start writing code, the editor
is kind of the interface that I have to
write the files and edit the code. I find
it kind of complex to use an editor. On
the left you can see that many of the
editors come with a package manager
included, which is a good sign for the
complexity that these tools have. And I
don't know what tools you need but they
support you in development so it's very
good to, for example, have code linters
and auto-completion in place to write
better code. At the same time it can be a
problem because they can execute code
unattendedly. We will see in a moment. The
mitigation I came up with for the editor
part is that you have a virtualized
environment where you run your editor. So
when something happens and and it is
compromised, not your root system is also
compromised as well. You want to monitor
all your config files that you have in the
project and you want to get awareness what
exactly happens on my system when I run
and view this code.
The next part you will
probably use as a shell integration. So as
soon as you open your repository some of
the shells I saw just tell you what
branch you're working in and what files
were changed and so on. So that's
something that comes very neat if you're
developing but it can be a risk as well.
So my opinion on the shell integrations is
mostly that it's made for software
development on your own system. So when
you write the code and you can trust it
it's not a problem to use those tools at
all, but as soon as you get sources from
foreign developers it can be a problem. So
choose your tools wisely and don't execute
code from others if possible. The
versioning system that you commit your
code to is also a very good choice. For
example Git and yeah Git can execute hooks
on different occasions. For example when
you check out new code, if you commit and
so on. That means if you managed to clone
a repository and a ".git" folder is
included or a ".hg" folder is included, it
could mean that your operating system
decides to execute whatever is in the
hooks. It's not possible to store a ".git"
folder within a Git repository but it's
possible to store it in a Mercurial
repository or in a SVN or something, and
then your shell integration won't know
what the original source was and will
execute it anyway. One thing that was
introduced for example from Visual Studio
Code, this October, is that they now
support git hooks - which is a great
feature, right? chuckles
The mitigations
against this are pretty easy: you can
either set a different hooks path, which
is not within this project repository so
that you don't execute git hooks at all,
or you can use that little wrapper here
that you see, to for example check at
least that there is no file that is a git
hook within that folder before you execute
git. It's a very good choice if you want
to protect yourself from that
vulnerability.
So after you committed the
code and shared it to the versioning
server, you probably are going to build it
automatically. So some services like
Travis CI will run it, will run it for
you. So they will run tests, they will
compile the software and also they do the
package versioning and deployment to some
other places. It becomes a problem if you
can't reproduce the results from your
built runner, because it's an system you
don't control sometimes. And as soon as
you get the binary result from it - if you
compile the software that compiles to
binary - you need to check that result
somehow, because somebody could have
altered it without your knowledge and then
you will ship it to your users. Also a
problem on many of this build workers is:
you want to have this process very fast.
So that means you don't want to wait until
all the dependencies are installed and the
great service is that you have caching in
between these projects. This means that,
for example, if somebody managed to inject
the version to the cache of some CI
system, then it will eventually show up in
other projects as well and you can pivot
across the projects. Usually if you have a
build environment it has access to some
kind of development key. Mostly if you get
pull requests from externals, the keys are
stored encrypted and you don't have access
to them, but as soon as somebody has write
access to your repository also the keys
could be leaked. Let's make an example:
you have somebody offering you a software
and you don't give permission to edit the
master branch of the repository but as
soon as you open an open a branch anywhere
and make a pull request Travis CI or other
build runners will use that and decrypt
the passwords for you and give you access
to the credentials which you can then
print or do whatever you intend to.
Yeah,
and for me the best option here would be
to have reproducible builds because then
you can use different of the build workers
and compare results somehow so that you
see if one gets compromised the other two
will tell you: hey, there's a different
result, have a look please.
That would be great.
And also the build steps; I
mentioned building, testing and packaging
the software are totally different steps
so what you can do is you can have one
compartment per step so that you can have
a data at a finer level and see what
happens here. After you compiled the
software, you built the software, you need
to ship it to the user somehow so either
you store it in your own server or most
often you use a CDN. You just put it there
and it's the asset that's lying around.
Your users will come around, download it
from here and execute it, so what is the
problem here? The problem is, that if you
have an URL it's very hard to prove that
it's actually from the real maintainer. If
you call your software like if - if you
call your account like a different
project, then people won't be able to
notice the difference somehow. What you
can do to mitigate this, is to publish the
URLs that you legitly using and also sign
your assets so that users can check is
that something that the developer intended
to give me or is it something that is
really ... that is really not intended.
So, ...
Yes. And the next part is you need to
reach out to your users
so you make people aware,
that there is a project they can
check out, they can clone and usually you
have the package registries. A few slides
back you saw that the package managers are
also included in the editors, so that's
also something where you can ship the
software but the package manager I was
mostly looking at was for example NPM.
There was an interesting occasion where
somebody had a project called Kik.The
company Kik then tried to take it down and
the person just ignored it for the moment
but then Kik reached out to NPM directly
and they deleted the repository. In
consequence the developer removed all his
projects from the versioning server and a
few hours later malware showed up with the
same project names, so that means if you
have a software that uses that
dependencies and somebody freed up names
it would affect your repository as well
and compromise it. That's something that
needs mitigation. I think the best idea
here is to not only identify the
project by a unique identifier but also
have a GUID or an or a unique identifier
per project that does not change, so that
you can make a difference. That's
something that's up to the package
registries to implement. That's not
something we can do as a user but it's a
very common case - it's a very common case
that these packages fluctuate.
So for example, if somebody deletes it,
you don't have a backup of that.
A very good idea is also to store offline
backups of every package that you
check out and that you install to
your software because it's very bad
if you want to maintain your software and
you figure out there's something, there's
something missing and I can't recover
because it's deleted.
Yes, software developers have some
needs during their work.
I want my tooling to perform if my
code editor for example is in the VM and
the VM is slow, that's something that's
annoying all over the process. So then on
the other hand the velocity is something
that your manager will require from you if
you write commercial software or you try
to get something done and you can't spend
all day to work on chores and improve
the repository, the versioning and so on.
So that's something you need to deal with.
Another big factor for me is the
reliability. So as soon as your software
goes down and you are in holiday or
something, anybody else from the company
or from your team should be able to
recover what was there before, also known
as the bus-factor and, yeah, if you have
convenience like for example Ruby on Rails
gives you. It gives you a very good,
very easy start in the projects and that's
something you don't want to break by
making it too complicated with a
development environment. And also
something I've found to be more annoying
than helpful is, if you want to pair-
program and you have a very
compartmentalized environment, it's very
hard to share the resources that you need
to talk about with other developers,
expecting you're not in the same room but
working remotely, what is for me most
often the case.
A large problem that I saw is, if you
underhand somebody code,
if you go ahead and and check out
code from any online resources, it's
sometimes very hard to tell if the code
that you see in your, e.g. Git diff,
is what you really would expect to
see. I have some examples here, which can
show how this could work and how this
could look like if somebody tries
to inject code to your repository, that
you don't see. First of all, let's start
with something easy, that's phishing. What
you see here on the slide, on the left
side maybe you see the cursor. That's not
a full path, that is just a domain name.
The slashes in here are UTF-8 characters
so that thing here resolves to a hostname,
and if you control this host, you can get
a certificate for it and then the example
below you see, how it would look like, if
you install it. First I have a host that's
just running a web server on port 80, so
that you can see the result. Okay, I was
cheating a little bit. I was putting the
domain in the /etc/hosts so that I don't
have to buy it, for just showing it. It's
strange that dot zip is a domain actually,
but then if you install it, you would see
that you can send somebody a very
nice-looking link which looks like a
totally different project, but it's
pointing to your server instead; and I
found many of the package managers having
the nice feature of executing PostScript
hooks, so that means, if you have
installed it, it will run some commands
afterwards for you.
Then there is invisible code. If you go
online somewhere, find in a forum or in a
blog, you find an article and see, "hey,
that code is actually solving my problem",
you go ahead and copy/paste it. So, on the
left you see the source code how this
would look like in HTML for the blog. On
the right there's the result. So you can
go ahead, you can copy/paste from it and
if you paste it to a text area, you will
see, that the result is something that you
didn't expect. For example, if you copy a
large chunk of code, you won't go ahead
and review it on your local system again
and that could be the compromise for your
project.
Another example here is, you can
use ASCII characters,
the control characters to influence
the output in your terminal.
So if your terminal also
supports the legacy of ASCII control
characters, you can use that to just
revert the line and override it with
something you wouldn't expect. What you
see on top here, that harmless script is
the file. It's a little larger than you
would expect for just a echo foo, but not
something you would notice when you just
see it. Looking at it from a hex editor
you can see, that there is something more
going on than just the foo and if you
actually execute it, it will not print
something, it will create the pwned text,
which is a good example for you that your
host was compromised in this moment.
Another example I found online, so credit
to Ariel for this, so there is a byte
sequence you can use so that this even
works in a Git diff. So when you're
working exclusively in your terminal and
you're not doing reviews on GitHub or some
graphical tool, it could be the case that
you don't notice what was going on. What
you can see here on the left is, I created
an empty repository, I added a small
script and in the next step down here, I
added some improvement to the script,
which is actually the malicious commit
that's here in red. Afterwards, I just ran
a Git diff on the code and I see that
there is only no backdoor, oh sorry that
should be okay in the updated slides. So,
you don't see the evil.sh that it's
executed as well, if you run it. That's
something I consider very dangerous.
So, some mitigations: the best thing you
can do, is to make it expensive for your
attackers to compromise or try to. So
as soon as you have the chance to notice
what is going on, also retrospectively,
you can at least burn the capabilities and
tell others how your project was attempted
to compromise and that's something
that is, in my opinion, the best
mitigation against this complexity.
What you can also do, is you can test your
software from external services directly,
which will tell you, if some compromise
happened. For example, Git has it newly
introduced. They will check your packages,
the dependencies and will warn you about
some vulnerabilities that are commonly
known. The best thing you can do on your
local system is to build small
compartments, so that if some compromise
happens, it doesn't affect your full host.
Also not all your projects you have
access to. And it's very important, that
you have backups on a different system
than the hosts you're working on. So if
the compromise happens you still have
access to the original data and can
compare it and do some forensics on this.
So, the intrusion detection forensics;
there are some great tools available.
For example my favorites are DTrace and
Opensnoop. You can monitor changes and
access on the file system or on your
system at all. And you can e.g. set some
rules for your projects that are
specifically matching. So I am
not going to share some rules that match
for all projects, but you will figure out
what is e.g. important. Very
good start is e.g. to Opensnoop for
/etc/passwd if there was some access, then
you can e.g. say that's not something
what my software would do. And again it's
very important to have the backups of this
because in the moment where you execute
it, you can't trust your host at all.
The idea how to achieve this is,
if you have a VM per project for example,
you let it run for half a year,
you don't approve the situation. Instead
of having one system that you need to
update the software to, you need to update
afterwards all the projects that you're
working on frequently and that's something
that's easy to forget. So it's dangerous.
If you assume that every time you run some
command or every time you work in a
project, you spin up a new server entirely
from scratch, install the dependencies and
so on that's something that's not a risk
for you. Also, if you have for example a
virtualized server environment you can
have memory dumps at any time, you can
monitor the network and you can also diff
the filesystem. For example, if you stop
the server and just compare it to a
previous version and see, "hey, here is
something that was changed that I didn't
plan". It's great to know.
Very important is also to separate your
accounts. E.g., if you see large GitHub
accounts, people are making contributions
every day since years.
So it shows that the people
have access to very, to many projects from
the same machine. And the permission model
from GitHub for example, allows you to
store an SSH key for write access. But it
automatically has access to all the
repositories you control. So the best that
you can have here, is to make a
new GitHub account for, ... or to make a
new account on that versioning system
that only has exclusively write access
to that single repository.
So when you work in your
compartmented system and you want to
upload or pull changes, you can't
influence other repositories. That means
compromise doesn't spread across all your
projects and so on, which would be an
invitation for malware somehow or
ransomware. And you get a better
permission model if you create a GitHub
organization. In this case you can also
limit your own access in a better way. So
my recommendation is not to work on your
personal GitHub account but create an
organization for your project.
Something many projects are missing are to
find responsible persons for security
and to clearly communicate what is the
plan for incident response. Small example:
If you have a new project and you
find a vulnerability, you would like to
commit it, but you don't open an issue
publicly, because then everybody, every
user would be affected. You try to reach
out to some developers and if you don't
have any clue how to securely achieve
this, that can get you into trouble. And
there are quite a few projects which don't
communicate this and some of them don't
even respond to their security@ email
address, which is bad.
In this case I told you what I saw from
my experiences of working on the projects.
So that's basically my summary of
what can be harmful
and what can be good for your project.
Applause
Herald: Thank you and we now have time for
Q&A. In the room you can line up behind
the microphones and I can see we have a
question from the internet already.
Signal Angel: What about Git signed
commits? Any thoughts on that?
Answer: So as soon as you have signed
commits and I find that you also email
with the same PGP key, it's very
interesting that you have the PGP key on
the same host probably, then you have your
Git executable. So if somebody executes
Git hooks, they can steal your PGP keys
from this. I didn't find any tutorial
online which explains you, how to make it
manually, so that you don't use the Git
for signing the commits. But I think it
can be very good to sign the commits,
but it can be also dangerous, because your
email communication can be compromised.
Herald: Microphone number four.
Q: In the Git diff you showed us, there
were some control characters. I think
Git diff pipes to less by default, so
shouldn't they appear there somewhere?
A: No, they don't. I just checked with the
latest version today. So, that's something
that, well, we can also click on the blog
and see, if there is the video available.
Yeah, it's very hard to show from my
HTML slide how this works.
So this video animation, ...
maybe we can enlarge it a bit.
That's how it would work.
So most often, yes, if you pipe to less or
you use a hex editor to review,
then you would notice, yes.
Q: I somehow remember that, maybe it
only shows for longer diffs, but I think
when I type "git diff", I can scroll
around.
A: Ah, that's interesting. I need to try.
Herald: We have a question from microphone
number one.
Q: You mentioned Travis having access to
hidden variables and you being able to
leak those variables during pull requests.
What are your suggestions to mitigate
that?
A: Don't give people write access to your
repository, not even to branches that you
don't trust. So, as soon as they have
write access, they would also know the
secrets behind the variables in this case.
I like the security model, because if you
for example get contributions from
outside, nobody can trigger that and steal
your keys.
But as soon as you build it on your own
branch somewhere in the repository that
changes.
Q: Yes, but if you submit a pull request,
you don't necessarily have to have write
access to that repository.
A: Yes, that's what I mean. If you come
from outside and it's not within the same
repository, the secrets are not decrypted.
So you can't run the steps. For example,
you would not like to deploy directly from
a foreign branch, somewhere.
Herald: We have a question from microphone
number four.
Q: You mentioned the problem with
different compartments and how to exchange
those environments without people. I think
that problem has already been solved with
Vagrant and some kind of provisioning
software like Ansible.
Do you have any experience with checking
those results of those Vagrant boxes that
are automatically provisioned, like
having some server spec software to check
those environments afterwards, or having
some kind of hashing, how to find out, if
they have been reproduced the same way and
or if they have been any exploit used
in that process of setting up the Vagrant
environments.
A: Yes, so different levels you can look
at this. There was some, ..., I try to
find it, yes. You can for example memory
dump at any time, if you have the hosts
trying somewhere or was your question
exactly that you want to check, if your
environment that was spun up was not
compromised yet?
Q: Yeah, there has to be some kind of
process, how to verify that the produced
environments are the ones you expect them
to be, or if they have been compromised
and the problem is, I have used those
environments and tried, first I tried the
full disk encryption for the Vagrant boxes
but the problem is, it's always the same
key for the encryption, so that doesn't
work and even as you mentioned, you can
have a memory dump, so you can read out
that key so there's no real possibility to
set up a Vagrant box that can't be
tampered with afterwards. So there has to
be some kind of hash sum to compare those
produced results.
A: Yeah, so as soon as you have a
reproducible build and the result that
you, for example script languages are much
easier to achieve, because then you can
just diff the filesystem directory and
see, if there was some change. What I
would do in this case, is to run multiple
services and compare the results, if
that's possible. For example, you have
these reproducible builds, then run it on
a few servers which are independent and
compare what you have.
Herald: We have two more questions from
microphone number one and only a few
minutes left. Microphone number one.
Q: So, what's your recommendation for
handling credentials in application
configuration files? We need often some
database user and password or something
like this in, say Spring Boot Application
YML, or things like that?
And is there any best practice or any
framework which can handle such things or
we need to explicitly encrypt these
credentials in this application and then
decrypt for itself in the application, but
then you need symmetric keys, or?
A: Yes, so Ansible for example comes with
a mechanism that's called Ansible Vault,
which encrypts that with a passphrase that
you can enter in your command line as soon
as you touch the file. For example, if you
want to run Ansible then, it will ask you
for the password when starting up. So, if
you want to share that password with your
developers everybody has access to the
same keys, I would prefer to give
everybody, so every person in this team or
even every device a different key, if
that's possible somehow.
That's what I was trying to mention with
the GitHub accounts; that you don't use
one GitHub account but you use many of
them. If you, ... yeah.
Herald: We have one more question from
microphone number one and then a question
from the internet.
Q: Yeah, my question was more about, I
mean some of your recommendations are low-
hanging fruits, but some of them it's
like, it's just impossible. I mean it's
not sustainable, like it's very hard to
maintain and so I'm wondering, if you use
all of them every day or just part of them
or do you just leave like an ??? at the
end?
A: It depends on the project. So what I
try to do on my development system is to
have these compartment, so that one
compromised project would not affect
others. Because I'm not the only person
checking and merging the code, so and
that's something that gets quickly too
much for one person to review. So, I
can't review all the code that I'm running
currently on my computer, that's true. But
I can try to mitigate what the impact of
this will be.
Herald: And the question from the internet.
Signal: What tool would you recommend
for diffing a file system?
A: Diff. giggles Well, it worked for me
so far. Or what exactly is the question
about? Um, maybe you want to see, if did
the hash change in the files? So when you
have e.g. the script file one and
the script file B and they have a
different hash sum, that's something I
would consider something I would look up
manually. So as soon as I have an
indication that there was something wrong,
I would look it up manually and use any
tool that I have. Hex editor or whatever
is available.
Herald: Good. We have less than one minute
left. Are there any final remarks?
Stephan: Thank you.
Herald: Thank you very much.
Applause
postroll music
subtitles created by c3subtitles.de
in the year 2019. Join, and help us!