Hello and good evening on day two of the chaos communication Camp 2023 Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]
it's late in the evening this is meleeway stage in case you're wondering
and the next talk is going to be about incident report responses
so if you're curious about how to even get there to have an incident response how you could
prepare for an incident response and how you could support a new organization
uh, the incident response team in doing the job and trying to fix whatever broke
let's put it that way um we have the right talk for you
this is stories from the life of an incident from incident responders Harry and Chris
please a very warm Round of Applause [Applause]
so, good evening and thank you for joining us today um we will tell you a little bit of our
life as incident responders and I'm Chris I did my computer science
studies at the University of alang and Nuremberg I do this security stuff for
over 10 years now so my CV is a little bit longer at the moment I'm a detection
engineer before that I was a long time working in dfir so digital forensic incident
response in different organizations and
yeah I'm Harryr I studied electrical and computer engineering at RWTH
University and I played a lot of CTF and did some hacking stuff at chaos computer club RWTH
during my masters I worked at x41 dsac doing pen testing patch analysis
so I also have some kind of offensive security background on for around one year now I'm working at G data Advanced
analytics doing digital forensics and incident handling
first Christian will give you a short introduction and then he will tell you how a classical ransomware attack looks
like and in the second part of the talk I will tell you how the incident
responders work and what you can do in advance to make it go as smooth as possible and support the incident
response team so as Harryr told you I will probably
we'll talk about ransomware because the customers we usually have are small and
medium-sized businesses universities and hospitals and those are regularly
unfortunately regularly hit by um um
ransomware gangs the main reason for this and that's if you heard the last
talk um why they maybe not that responsive
and are not so interested in they just lack the resources so the manpower to do
uh proper security measurements to secure their systems especially in in erm
situations where you are for example in a hospital have medical devices
um which where you cannot simply install an AV on or even patch the system
because you lose the certification as a medical device then but also in in
companies manufacturing companies on the shop floor we're talking about systems
that have run times of 25 plus years so if you look back now 2023
we're talking about XP and older systems fun fact I was in a ransomware case and
Wannacry in 2017 when I got a call from from a person from the shop floor
asking me if we have a nt4 expert, um
that can tell us if WannaCry is affecting nt4 of course you don't need
to be a expert for NT-4 this one requires of course not affecting nt4
systems so due to the time uh slot we thought
memes are the best way to to tell you those stories and we have a lot of them
so in the first uh um section I tell you a little bit of how an attack Works
um there are a lot of different possibilities how you can describe and how to structure the how an attack works
there's the miter attack framework for example there was for example a talk Yesterday by Maker Salko
um here on the stage there's the original cyber kill chain from from Lockheed Martin you have
stuff from from companies like Mandy and their targeted the tech life cycle but
that's all in my opinion two two fine-grained it's that's the reason I
just take three simple steps yeah get a foothold in the door
look move play around and cash out those three uh I will just go over
so start with uh get a foot in the door so normally we
see three ways how attackers can can get into the environment in the ransomware
cases you have vulnerabilities in uh remote uh internet facing systems you
have the remote Services itself and you have malware
starting with the with the the vulnerabilities and um I just looked uh up the last four
years and maybe somebody remembers netscaler the the so-called Citrix
vulnerability in December 2019 um it was released mid of uh 2019 uh
December 2019 the first POC publicly available POC was in beginning of
January and the patch was available in middle of January so there was a round one week to one and a half weeks between
a public proof of concept for the vulnerability and uh patch for the vulnerability and what we saw
during 2020 a lot of companies patched but the patch didn't remove the the
compromise so they were already compromised and um yeah with it with the patch they
didn't remove the compromise so what we found what we could provable
see or proof evidence for uh was nine
month uh customer was breached after nine months using this this vulnerability
and we had other customers where we could see that the netscaler was affected after two years but we couldn't
prove that this this compromise was the reason for the actual ransomware case
and of course such vulnerabilities happen not that often
yeah so 2021 gave us uh hafnium exchange
vulnerability also a similar situation the patch
appeared as an out-of-band patch from Microsoft on a Tuesday evening 10 o'clock in German time
we saw during our uh incidents or the the assessments we did that
um the first exploit exploitation attempts were seen on Wednesday in the morning at
5:00 am so around seven eight hours later um I know one guy who could patch
because he was online when the patch was released otherwise Germany was unable to patch in
time and of course we can go on with 2021 proxy shell also
exchange vulnerability proxy nutshell also exchange vulnerability
we have uh in 2022 VMware Horizon the the virtual desktop infrastructure
from VMware just to name also open source stuff Zimbra a collaboration platform
including an email server uh has had a vulnerability actually the vulnerability
was in cpio from 2015 I think which led
to a compromise using via email so you send an email
with a cpio with a specially crafted archive file and you could drop a web
shell in one of the directories yeah you have of course 40 OS which is a
40 gate VPN and firewall operating system
and if you read the news we start at the beginning again
netscaler had some issues several weeks ago according to foxIT we have 1900
still unpatched net scalers worldwide how many patched
was netscale has exists that um have not been checked for compromise we
don't know of course so that will be a nice year probably
um so what can you can you do against this kind of of attack vector patch your systems is one thing as you
see this that doesn't lead to the the um or what you need to do afterwards in
such cases you need to check your systems for possible compromise
that is important to reduce this I highly suggest put your
uh Services behind some VPN so that only people who already have
connection to the VPN um can access your services or the services
they need and that would reduce the attack surface
at least to the VPN server so but I
of course we can also think about remote services without vulnerabilities
um there can be configuration mistakes so the admin does something wrong there can
be insecure default configurations like this um I don't know if you know it but the
local admins or the administrators on the Windows system are are
automatically in the remote desktop users group you know and so
we had several cases especially in the beginning of the pandemic when everybody moved from uh to the home offices and
they needed to put people fast in the position to to access their the assist
the internal systems again they just put a RDP server on the internet and hope for the best
um additionally if you put services on the internet of course brute forcing and
credential uh stuffing are attacks that are possible so brute forcing just trying the the
username and password combinations uh credential stuffing using already leaked
passwords or credentials from leaks you find on the internet
what you can do about this kind of of attack Vector is uh just as I said use
multi-factor Authentication and reduce the attack surface as in the
point with the vulnerabilities before by moving the services behind a VPN and
then use multi-factor authentication on VPN of course
the last Vector that we see normally that the attackers can get in the
network is malware we all know this about
those funny emails you get with the attachments
um include that have either Word documents
attached either zip files with with Visual Basic scripts javascripts and
what you can get isos you see a lot these days
um or what you can also have that you can have just a link inside the email and
you download the respective file from some some shady file sharing website
um what we saw over the last year was uh USB sticks again funnily
um I'm not sure if you have heard about raspberry Robin which is a malware that
warms via USB sticks um but I haven't seen it as a vector for
ransomware yet on my own but there are people who said that it's
an initial access broker for some of the ransomware gangs
so what can you do about this if you think the
you can of course ban simply some file extensions in your mail server or you
change the file Association types in your operating system meaning that you
don't open the JavaScript and Visual Basic script files using for example the
windows scripting host but open it with notepad and that will
of course some people will be
uh some people will think about what this this is then and ask the IT guys
but it's better than running the the script itself
one thing I I I don't like to to say it but keep your AV updated
um uh this is one thing keep it updated and read the logs
we see a lot of incidents where we see that the already
days or weeks before we you can could have seen that there's something going
on in your network yeah and if you see malware in your AV logs
then react to it just check it you don't know how long this AV this malware has
been on your system the thing is that
just because you're AV detected it now it might have been get received an
update for its signatures and the malware was active for days or weeks
before so when they are inside
then they usually look move and play around a little bit
so when they look around what they do
is they they enumerate AD they do Ports scan the you they search for
vulnerabilities they check uh what they how they can escalate
their privileges they try to find credentials
um Kerber roasting we heard in the talk before for example is this one thing
um they try to identify accounts you around
have running on your systems they can use they can get the credentials from and you reuse
and for that reason one of the most important things I think is that you
have a principle of least privileges in your environment so only what a account needs
you should be able to do you should use dedicated service
accounts for your services of course and just for your information
um service account is not an account that has a SVC underscore in front of it
and is otherwise a normal user account um there I exist educated service
accounts in Windows environments so use them use strong passwords I still
I today I can I know companies who still use eight
character long passwords um I think that's 20 minutes on a decent
graphics card today so use strong passwords length matters
12 plus characters is the minimum in my opinion
and don't reuse passwords especially on your
systems the local administrator especially in small and medium-sized businesses you see a lot that people use
the same password for all local administrators so if I have it on one
system I have the whole company don't reuse it
um yeah [Audience LoL]
so when they they move around in your network using either a password hashes they found valid credentials they they
discovered somewhere vulnerabilities are also used to to move around in your
environment they try to to establish persistence mechanisms here we heard
also in the talk before about the C2 channels command and control channels they use
um they install in some cases directly any desk team viewer and or other remote
control software or sometimes they also use tunneling softwares like ngrok or recently they
started using cloudflare G um and this is what you need to do is
prop have a proper Network segmentation and with the proper Network segmentation I don't talk about subnetting
subnetting means you just have different subnets you need to have a firewall between them
and you have need to have rules between them that
restrict access between your your subnets and one thing is especially important
please keep or use Network segmentation to
restrict the access to your backup and your Management Systems as far as
possible we see a lot especially as I said in the small and medium-sized businesses or in
the other organizations we we have as customers that
yeah they have they tell us in in in in in in workshops yeah we have
a network segmentation every building is one segment and on the question yeah you
can move between in a segment you can access everything yes you can and
between the the buildings you can have also a firewall and you cannot access
anything no you can access everything and also your your uh VMware Management
console oh yes yes we can so everybody can access it yes of course and that
doesn't work so
um when they play around they normally try
to to gain more privileges so privilege escalation is assisting local privilege escalations normally but also using
vulnerabilities um or misconfigurations insecure default
configurations um my personal favorites are Group
Policy preferences or passwords in group policy preferences
this is no longer possible since I think 2014 to put passwords in group policy
preferences however if you had your password stored in those preferences
before the patch in 2015 2014 then they're still there and yeah there
are AES encrypted but the encryption key is on the Microsoft website
so you can just download it and just take it and decrypt the keys
um then during that phase they also try to disable your security measures
the thing you can do is of course patch your system so you know try to get your
availabilities out of of this equation can try to configure your systems in a
secure way this is not always possible due to some shitty uh third-party software
and keep your AV um updated and please please as I said
already check the locks and act accordingly
so in the last phase they cash out that's when when they
start using a uh being your backup service so they copy
data from your your environment using um
file sharing platforms for example yeah Mega and set was was once the thing we
transfer we had already uh every every other file sharing platform you you can
think about is a possible way to exfiltrate data they also use their
their C2 communication channels so sometimes you they also they just use the the possibilities in any desk or in
in RDP clients or they use uh file transfer protocols
like um SS SFTP um
we saw for example in one case that they try to install filezilla on every machine they had access to
um because on the first one it didn't work on the second it didn't work on the third it didn't work yet because SFTP
was blocked uh outgoing and that is one of the things you can do to to prevent
exfiltration block at least
protocols you know that you don't need in your environment and proper Network segmentation of
course is a general thing so in the last step that's when they
start the encryption um they're running the ransomware or
normally they are have domain admins at that point so they can run it on all
domain connected systems they can also disable of course when they are domain admin they can disable
the AV before they they start to run somewhere ransomware's today disable services like
databases and such things so that they have the full power of the machine for
the uh for the encryption
um if you get lucky not how everything works perfectly because they use
group names and windows is especially picky when you have a non-english uh
windows installed for example in Germany the the group everybody is called yida
and we had cases where the ransomware didn't really work that well because they couldn't
change the permissions of the files first um
they use different encryption schemas normally they they come with the
asymmetric and the symmetric encryption type the asymmetrics or public key cryptography the public key comes with
the ransomware and is used to encrypt the symmetric keys they generate on in
your environment depending on the ransomware they they generate one key for each system or even one key for each
file it depends a little bit on the on the ransomware how it works but that's
the usual thing they use um I would never count on the the fact
that there are possibly maybe there could be
decryptable uh things um in in my opinion in my uh in my world
The ransomware Gangs have learned and used the standard Microsoft Windows or
some other publicly available libraries to to do the encryption
they executed by a remote tools like PSX Powershell or some use
gpos group policies to execute the ransomware on every
machines they they connected to the domain and what can you do about this no it's
it's hard but the the most important thing is have online backups offline
backup sorry thanks you you see you see off online backups
are not that are great but not that great offline backups is the most important this is the most important
thing so um don't have it connected to your environment
the the the USB disk on the system is not offline backup
um in my opinion if you see that something is is still encrypting
I I'm I'm always hesitant to say shut down the system because you can break
the encryption and maybe the file that is currently in under encryption or the files will never be decryptable if you
want to buy a decrypter um or gather the cryptos through some
discussions with the with the ransomware guys um if it's a VM just suspend it and
that's it and if everything is already encrypted
keep cool and call your incident responder
so now let's talk about incident response what happens when it's already too late and what can you do to support
your incident response team at first the things I'll say in this chapter are
for our company and how we work so other companies might work a little bit different than that
first for some reason incidents always come on Friday afternoon
so some customers think it is a good idea to try to solve a case by
themselves maybe until the end of the week and if they didn't solve it until the end of the week they call the
incident response team please don't do that it doesn't help your company and it
doesn't make your incident Response Team happy to have to work on the weekend and in addition the longer you wait with
calling the incident Response Team the longer the incident response will take and the more complicated forensics will
be because you have lock retention times while trying to do stuff by yourself
maybe you modify some of the systems and it becomes much more harder to do
precise forensics so what happens on our site when such a
new incident ticket arrives the first thing we do is team internal coordination
so we discussed do we have enough people do we have a person for each role in our
team we have three roads incident handling forensics analyst and Mayweather analyst
so first let's talk about incident handling incident Handler is responsible for all
the tasks that our customer facing and the first point is always get the customer out of that headless chicken
mode like we call it because when an incident comes at our customer site
everyone is like running around in so-called like headless chicken doing something but not doing anything helpful
so this is always the first task for the incident Handler Handler structure the customer do meetings and then do all the