34C3 - Hardening Open Source Development

Herald: So could you give a warm welcome
of applause to Stephan Gronke who will be

talking to you in one minute.

Applause

Stephan: So hi everybody my name is

Stephan Gronke. I'm a software developer
since about 15 years, working in solo

projects and larger teams and smaller
teams. So mostly my development stack was

JavaScript and you will find some of the
tools that I mentioned coming from this

world, but I'm very sure you can also find
something for your project that applies

here. Here's my email address, my PGP key
and my favorite social network account.

Yeah, so a little spoiler what will happen
today: I will talk about development

process exploitation. So that means if you
are developing your software and somebody

joins your team and since you code for
review it could happen that it executes

code on your machine without your
knowledge. There are a few things that are

really hard to catch or I found hard to
catch and I want to share. Maybe you have

the same problems and you find that the
same mitigations apply for your project as

well. I will then continue and... let's
start with a software development process

that's a small cycle. So first of all it
starts with an operating system: you need

to have a computer to write a software and
that's something you need to trust first

off. If you... yes, so your operating
system contains keys and credentials, it

contains the source code you want to
develop and the tools that you have in

place. And the major risk is that tools
are vulnerable to some exploitation or

that your host is already compromised and
you write a software, commit it to your

coworkers and it isn't what you intended
to write. That's a larger problem here.

After you start writing code, the editor
is kind of the interface that I have to

write the files and edit the code. I find
it kind of complex to use an editor. On

the left you can see that many of the
editors come with a package manager

included, which is a good sign for the
complexity that these tools have. And I

don't know what tools you need but they
support you in development so it's very

good to, for example, have code linters
and auto-completion in place to write

better code. At the same time it can be a
problem because they can execute code

unattendedly. We will see in a moment. The
mitigation I came up with for the editor

part is that you have a virtualized
environment where you run your editor. So

when something happens and and it is
compromised, not your root system is also

compromised as well. You want to monitor
all your config files that you have in the

project and you want to get awareness what
exactly happens on my system when I run

and view this code.

The next part you will
probably use as a shell integration. So as

soon as you open your repository some of
the shells I saw just tell you what

branch you're working in and what files
were changed and so on. So that's

something that comes very neat if you're
developing but it can be a risk as well.

So my opinion on the shell integrations is
mostly that it's made for software

development on your own system. So when
you write the code and you can trust it

it's not a problem to use those tools at
all, but as soon as you get sources from

foreign developers it can be a problem. So
choose your tools wisely and don't execute

code from others if possible. The
versioning system that you commit your

code to is also a very good choice. For
example Git and yeah Git can execute hooks

on different occasions. For example when
you check out new code, if you commit and

so on. That means if you managed to clone
a repository and a ".git" folder is

included or a ".hg" folder is included, it
could mean that your operating system

decides to execute whatever is in the
hooks. It's not possible to store a ".git"

folder within a Git repository but it's
possible to store it in a Mercurial

repository or in a SVN or something, and
then your shell integration won't know

what the original source was and will
execute it anyway. One thing that was

introduced for example from Visual Studio
Code, this October, is that they now

support git hooks - which is a great
feature, right? chuckles

The mitigations

against this are pretty easy: you can
either set a different hooks path, which

is not within this project repository so
that you don't execute git hooks at all,

or you can use that little wrapper here
that you see, to for example check at

least that there is no file that is a git
hook within that folder before you execute

git. It's a very good choice if you want
to protect yourself from that

vulnerability.

So after you committed the
code and shared it to the versioning

server, you probably are going to build it
automatically. So some services like

Travis CI will run it, will run it for
you. So they will run tests, they will

compile the software and also they do the
package versioning and deployment to some

other places. It becomes a problem if you
can't reproduce the results from your

built runner, because it's an system you
don't control sometimes. And as soon as

you get the binary result from it - if you
compile the software that compiles to

binary - you need to check that result
somehow, because somebody could have

altered it without your knowledge and then
you will ship it to your users. Also a

problem on many of this build workers is:
you want to have this process very fast.

So that means you don't want to wait until
all the dependencies are installed and the

great service is that you have caching in
between these projects. This means that,

for example, if somebody managed to inject
the version to the cache of some CI

system, then it will eventually show up in
other projects as well and you can pivot

across the projects. Usually if you have a
build environment it has access to some

kind of development key. Mostly if you get
pull requests from externals, the keys are

stored encrypted and you don't have access
to them, but as soon as somebody has write

access to your repository also the keys
could be leaked. Let's make an example:

you have somebody offering you a software
and you don't give permission to edit the

master branch of the repository but as
soon as you open an open a branch anywhere

and make a pull request Travis CI or other
build runners will use that and decrypt

the passwords for you and give you access
to the credentials which you can then

print or do whatever you intend to.

Yeah,
and for me the best option here would be

to have reproducible builds because then
you can use different of the build workers

and compare results somehow so that you
see if one gets compromised the other two

will tell you: hey, there's a different
result, have a look please.

That would be great.

And also the build steps; I
mentioned building, testing and packaging

the software are totally different steps
so what you can do is you can have one

compartment per step so that you can have
a data at a finer level and see what

happens here. After you compiled the
software, you built the software, you need

to ship it to the user somehow so either
you store it in your own server or most

often you use a CDN. You just put it there
and it's the asset that's lying around.

Your users will come around, download it
from here and execute it, so what is the

problem here? The problem is, that if you
have an URL it's very hard to prove that

it's actually from the real maintainer. If
you call your software like if - if you

call your account like a different
project, then people won't be able to

notice the difference somehow. What you
can do to mitigate this, is to publish the

URLs that you legitly using and also sign
your assets so that users can check is

that something that the developer intended
to give me or is it something that is

really ... that is really not intended.

So, ...

Yes. And the next part is you need to
reach out to your users

so you make people aware,
that there is a project they can

check out, they can clone and usually you
have the package registries. A few slides

back you saw that the package managers are
also included in the editors, so that's

also something where you can ship the
software but the package manager I was

mostly looking at was for example NPM.
There was an interesting occasion where

somebody had a project called Kik.The
company Kik then tried to take it down and

the person just ignored it for the moment
but then Kik reached out to NPM directly

and they deleted the repository. In
consequence the developer removed all his

projects from the versioning server and a
few hours later malware showed up with the

same project names, so that means if you
have a software that uses that

dependencies and somebody freed up names
it would affect your repository as well

and compromise it. That's something that
needs mitigation. I think the best idea

here is to not only identify the
project by a unique identifier but also

have a GUID or an or a unique identifier
per project that does not change, so that

you can make a difference. That's
something that's up to the package

registries to implement. That's not
something we can do as a user but it's a

very common case - it's a very common case
that these packages fluctuate.

So for example, if somebody deletes it,
you don't have a backup of that.

A very good idea is also to store offline
backups of every package that you

check out and that you install to
your software because it's very bad

if you want to maintain your software and
you figure out there's something, there's

something missing and I can't recover
because it's deleted.

Yes, software developers have some
needs during their work.

I want my tooling to perform if my
code editor for example is in the VM and

the VM is slow, that's something that's
annoying all over the process. So then on

the other hand the velocity is something
that your manager will require from you if

you write commercial software or you try
to get something done and you can't spend

all day to work on chores and improve
the repository, the versioning and so on.

So that's something you need to deal with.
Another big factor for me is the

reliability. So as soon as your software
goes down and you are in holiday or

something, anybody else from the company
or from your team should be able to

recover what was there before, also known
as the bus-factor and, yeah, if you have

convenience like for example Ruby on Rails
gives you. It gives you a very good,

very easy start in the projects and that's
something you don't want to break by

making it too complicated with a
development environment. And also

something I've found to be more annoying
than helpful is, if you want to pair-

program and you have a very
compartmentalized environment, it's very

hard to share the resources that you need
to talk about with other developers,

expecting you're not in the same room but
working remotely, what is for me most

often the case.

A large problem that I saw is, if you
underhand somebody code,

if you go ahead and and check out
code from any online resources, it's

sometimes very hard to tell if the code
that you see in your, e.g. Git diff,

is what you really would expect to
see. I have some examples here, which can

show how this could work and how this
could look like if somebody tries

to inject code to your repository, that
you don't see. First of all, let's start

with something easy, that's phishing. What
you see here on the slide, on the left

side maybe you see the cursor. That's not
a full path, that is just a domain name.

The slashes in here are UTF-8 characters
so that thing here resolves to a hostname,

and if you control this host, you can get
a certificate for it and then the example

below you see, how it would look like, if
you install it. First I have a host that's

just running a web server on port 80, so
that you can see the result. Okay, I was

cheating a little bit. I was putting the
domain in the /etc/hosts so that I don't

have to buy it, for just showing it. It's
strange that dot zip is a domain actually,

but then if you install it, you would see
that you can send somebody a very

nice-looking link which looks like a
totally different project, but it's

pointing to your server instead; and I
found many of the package managers having

the nice feature of executing PostScript
hooks, so that means, if you have

installed it, it will run some commands
afterwards for you.

Then there is invisible code. If you go
online somewhere, find in a forum or in a

blog, you find an article and see, "hey,
that code is actually solving my problem",

you go ahead and copy/paste it. So, on the
left you see the source code how this

would look like in HTML for the blog. On
the right there's the result. So you can

go ahead, you can copy/paste from it and
if you paste it to a text area, you will

see, that the result is something that you
didn't expect. For example, if you copy a

large chunk of code, you won't go ahead
and review it on your local system again

and that could be the compromise for your
project.

Another example here is, you can
use ASCII characters,

the control characters to influence
the output in your terminal.

So if your terminal also
supports the legacy of ASCII control

characters, you can use that to just
revert the line and override it with

something you wouldn't expect. What you
see on top here, that harmless script is

the file. It's a little larger than you
would expect for just a echo foo, but not

something you would notice when you just
see it. Looking at it from a hex editor

you can see, that there is something more
going on than just the foo and if you

actually execute it, it will not print
something, it will create the pwned text,

which is a good example for you that your
host was compromised in this moment.

Another example I found online, so credit
to Ariel for this, so there is a byte

sequence you can use so that this even
works in a Git diff. So when you're

working exclusively in your terminal and
you're not doing reviews on GitHub or some

graphical tool, it could be the case that
you don't notice what was going on. What

you can see here on the left is, I created
an empty repository, I added a small

script and in the next step down here, I
added some improvement to the script,

which is actually the malicious commit
that's here in red. Afterwards, I just ran

a Git diff on the code and I see that
there is only no backdoor, oh sorry that

should be okay in the updated slides. So,
you don't see the evil.sh that it's

executed as well, if you run it. That's
something I consider very dangerous.

So, some mitigations: the best thing you
can do, is to make it expensive for your

attackers to compromise or try to. So
as soon as you have the chance to notice

what is going on, also retrospectively,
you can at least burn the capabilities and

tell others how your project was attempted
to compromise and that's something

that is, in my opinion, the best
mitigation against this complexity.

What you can also do, is you can test your
software from external services directly,

which will tell you, if some compromise
happened. For example, Git has it newly

introduced. They will check your packages,
the dependencies and will warn you about

some vulnerabilities that are commonly
known. The best thing you can do on your

local system is to build small
compartments, so that if some compromise

happens, it doesn't affect your full host.
Also not all your projects you have

access to. And it's very important, that
you have backups on a different system

than the hosts you're working on. So if
the compromise happens you still have

access to the original data and can
compare it and do some forensics on this.

So, the intrusion detection forensics;
there are some great tools available.

For example my favorites are DTrace and
Opensnoop. You can monitor changes and

access on the file system or on your
system at all. And you can e.g. set some

rules for your projects that are
specifically matching. So I am

not going to share some rules that match
for all projects, but you will figure out

what is e.g. important. Very
good start is e.g. to Opensnoop for

/etc/passwd if there was some access, then
you can e.g. say that's not something

what my software would do. And again it's
very important to have the backups of this

because in the moment where you execute
it, you can't trust your host at all.

The idea how to achieve this is,
if you have a VM per project for example,

you let it run for half a year,
you don't approve the situation. Instead

of having one system that you need to
update the software to, you need to update

afterwards all the projects that you're
working on frequently and that's something

that's easy to forget. So it's dangerous.
If you assume that every time you run some

command or every time you work in a
project, you spin up a new server entirely

from scratch, install the dependencies and
so on that's something that's not a risk

for you. Also, if you have for example a
virtualized server environment you can

have memory dumps at any time, you can
monitor the network and you can also diff

the filesystem. For example, if you stop
the server and just compare it to a

previous version and see, "hey, here is
something that was changed that I didn't

plan". It's great to know.

Very important is also to separate your
accounts. E.g., if you see large GitHub

accounts, people are making contributions
every day since years.

So it shows that the people
have access to very, to many projects from

the same machine. And the permission model
from GitHub for example, allows you to

store an SSH key for write access. But it
automatically has access to all the

repositories you control. So the best that
you can have here, is to make a

new GitHub account for, ... or to make a
new account on that versioning system

that only has exclusively write access
to that single repository.

So when you work in your
compartmented system and you want to

upload or pull changes, you can't
influence other repositories. That means

compromise doesn't spread across all your
projects and so on, which would be an

invitation for malware somehow or
ransomware. And you get a better

permission model if you create a GitHub
organization. In this case you can also

limit your own access in a better way. So
my recommendation is not to work on your

personal GitHub account but create an
organization for your project.

Something many projects are missing are to
find responsible persons for security

and to clearly communicate what is the
plan for incident response. Small example:

If you have a new project and you
find a vulnerability, you would like to

commit it, but you don't open an issue
publicly, because then everybody, every

user would be affected. You try to reach
out to some developers and if you don't

have any clue how to securely achieve
this, that can get you into trouble. And

there are quite a few projects which don't
communicate this and some of them don't

even respond to their security@ email
address, which is bad.

In this case I told you what I saw from
my experiences of working on the projects.

So that's basically my summary of
what can be harmful

and what can be good for your project.

Applause

Herald: Thank you and we now have time for
Q&A. In the room you can line up behind

the microphones and I can see we have a
question from the internet already.

Signal Angel: What about Git signed
commits? Any thoughts on that?

Answer: So as soon as you have signed
commits and I find that you also email

with the same PGP key, it's very
interesting that you have the PGP key on

the same host probably, then you have your
Git executable. So if somebody executes

Git hooks, they can steal your PGP keys
from this. I didn't find any tutorial

online which explains you, how to make it
manually, so that you don't use the Git

for signing the commits. But I think it
can be very good to sign the commits,

but it can be also dangerous, because your
email communication can be compromised.

Herald: Microphone number four.
Q: In the Git diff you showed us, there

were some control characters. I think
Git diff pipes to less by default, so

shouldn't they appear there somewhere?
A: No, they don't. I just checked with the

latest version today. So, that's something
that, well, we can also click on the blog

and see, if there is the video available.

Yeah, it's very hard to show from my
HTML slide how this works.

So this video animation, ...
maybe we can enlarge it a bit.

That's how it would work.

So most often, yes, if you pipe to less or
you use a hex editor to review,

then you would notice, yes.

Q: I somehow remember that, maybe it
only shows for longer diffs, but I think

when I type "git diff", I can scroll
around.

A: Ah, that's interesting. I need to try.
Herald: We have a question from microphone

number one.
Q: You mentioned Travis having access to

hidden variables and you being able to
leak those variables during pull requests.

What are your suggestions to mitigate
that?

A: Don't give people write access to your
repository, not even to branches that you

don't trust. So, as soon as they have
write access, they would also know the

secrets behind the variables in this case.
I like the security model, because if you

for example get contributions from
outside, nobody can trigger that and steal

your keys.
But as soon as you build it on your own

branch somewhere in the repository that
changes.

Q: Yes, but if you submit a pull request,
you don't necessarily have to have write

access to that repository.
A: Yes, that's what I mean. If you come

from outside and it's not within the same
repository, the secrets are not decrypted.

So you can't run the steps. For example,
you would not like to deploy directly from

a foreign branch, somewhere.
Herald: We have a question from microphone

number four.
Q: You mentioned the problem with

different compartments and how to exchange
those environments without people. I think

that problem has already been solved with
Vagrant and some kind of provisioning

software like Ansible.
Do you have any experience with checking

those results of those Vagrant boxes that
are automatically provisioned, like

having some server spec software to check
those environments afterwards, or having

some kind of hashing, how to find out, if
they have been reproduced the same way and

or if they have been any exploit used
in that process of setting up the Vagrant

environments.
A: Yes, so different levels you can look

at this. There was some, ..., I try to
find it, yes. You can for example memory

dump at any time, if you have the hosts
trying somewhere or was your question

exactly that you want to check, if your
environment that was spun up was not

compromised yet?
Q: Yeah, there has to be some kind of

process, how to verify that the produced
environments are the ones you expect them

to be, or if they have been compromised
and the problem is, I have used those

environments and tried, first I tried the
full disk encryption for the Vagrant boxes

but the problem is, it's always the same
key for the encryption, so that doesn't

work and even as you mentioned, you can
have a memory dump, so you can read out

that key so there's no real possibility to
set up a Vagrant box that can't be

tampered with afterwards. So there has to
be some kind of hash sum to compare those

produced results.
A: Yeah, so as soon as you have a

reproducible build and the result that
you, for example script languages are much

easier to achieve, because then you can
just diff the filesystem directory and

see, if there was some change. What I
would do in this case, is to run multiple

services and compare the results, if
that's possible. For example, you have

these reproducible builds, then run it on
a few servers which are independent and

compare what you have.
Herald: We have two more questions from

microphone number one and only a few
minutes left. Microphone number one.

Q: So, what's your recommendation for
handling credentials in application

configuration files? We need often some
database user and password or something

like this in, say Spring Boot Application
YML, or things like that?

And is there any best practice or any
framework which can handle such things or

we need to explicitly encrypt these
credentials in this application and then

decrypt for itself in the application, but
then you need symmetric keys, or?

A: Yes, so Ansible for example comes with
a mechanism that's called Ansible Vault,

which encrypts that with a passphrase that
you can enter in your command line as soon

as you touch the file. For example, if you
want to run Ansible then, it will ask you

for the password when starting up. So, if
you want to share that password with your

developers everybody has access to the
same keys, I would prefer to give

everybody, so every person in this team or
even every device a different key, if

that's possible somehow.
That's what I was trying to mention with

the GitHub accounts; that you don't use
one GitHub account but you use many of

them. If you, ... yeah.

Herald: We have one more question from
microphone number one and then a question

from the internet.
Q: Yeah, my question was more about, I

mean some of your recommendations are low-
hanging fruits, but some of them it's

like, it's just impossible. I mean it's
not sustainable, like it's very hard to

maintain and so I'm wondering, if you use
all of them every day or just part of them

or do you just leave like an ??? at the
end?

A: It depends on the project. So what I
try to do on my development system is to

have these compartment, so that one
compromised project would not affect

others. Because I'm not the only person
checking and merging the code, so and

that's something that gets quickly too
much for one person to review. So, I

can't review all the code that I'm running
currently on my computer, that's true. But

I can try to mitigate what the impact of
this will be.

Herald: And the question from the internet.
Signal: What tool would you recommend

for diffing a file system?
A: Diff. giggles Well, it worked for me

so far. Or what exactly is the question
about? Um, maybe you want to see, if did

the hash change in the files? So when you
have e.g. the script file one and

the script file B and they have a
different hash sum, that's something I

would consider something I would look up
manually. So as soon as I have an

indication that there was something wrong,
I would look it up manually and use any

tool that I have. Hex editor or whatever
is available.

Herald: Good. We have less than one minute
left. Are there any final remarks?

Stephan: Thank you.
Herald: Thank you very much.

Applause

postroll music

subtitles created by c3subtitles.de
in the year 2019. Join, and help us!