Stories from the Life of an Incident Responder

Hello and good evening on day two of the chaos communication Camp 2023 Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]

it's late in the evening this is meleeway stage in case you're wondering

and the next talk is going to be about incident report responses

so if you're curious about how to even get there to have an incident response how you could
prepare for an incident response and how you could support a new organization

uh, the incident response team in doing the job and trying to fix whatever broke

let's put it that way um we have the right talk for you

this is stories from the life of an incident from incident responders Harry and Chris

please a very warm Round of Applause [Applause]

so, good evening and thank you for joining us today um we will tell you a little bit of our

life as incident responders and I'm Chris I did my computer science

studies at the University of alang and Nuremberg I do this security stuff for

over 10 years now so my CV is a little bit longer at the moment I'm a detection

engineer before that I was a long time working in dfir so digital forensic incident

response in different organizations and

yeah I'm Harryr I studied electrical and computer engineering at RWTH

University and I played a lot of CTF and did some hacking stuff at chaos computer club RWTH

during my masters I worked at x41 dsac doing pen testing patch analysis

so I also have some kind of offensive security background on for around one year now I'm working at G data Advanced

analytics doing digital forensics and incident handling

first Christian will give you a short introduction and then he will tell you how a classical ransomware attack looks

like and in the second part of the talk I will tell you how the incident

responders work and what you can do in advance to make it go as smooth as possible and support the incident

response team so as Harryr told you I will probably

we'll talk about ransomware because the customers we usually have are small and

medium-sized businesses universities and hospitals and those are regularly

unfortunately regularly hit by um um

ransomware gangs the main reason for this and that's if you heard the last

talk um why they maybe not that responsive

and are not so interested in they just lack the resources so the manpower to do

uh proper security measurements to secure their systems especially in in erm

situations where you are for example in a hospital have medical devices

um which where you cannot simply install an AV on or even patch the system

because you lose the certification as a medical device then but also in in

companies manufacturing companies on the shop floor we're talking about systems

that have run times of 25 plus years so if you look back now 2023

we're talking about XP and older systems fun fact I was in a ransomware case and

Wannacry in 2017 when I got a call from from a person from the shop floor

asking me if we have a nt4 expert, um

that can tell us if WannaCry is affecting nt4 of course you don't need

to be a expert for NT-4 this one requires of course not affecting nt4

systems so due to the time uh slot we thought

memes are the best way to to tell you those stories and we have a lot of them

so in the first uh um section I tell you a little bit of how an attack Works

um there are a lot of different possibilities how you can describe and how to structure the how an attack works

there's the miter attack framework for example there was for example a talk Yesterday by Maker Salko

um here on the stage there's the original cyber kill chain from from Lockheed Martin you have

stuff from from companies like Mandy and their targeted the tech life cycle but

that's all in my opinion two two fine-grained it's that's the reason I

just take three simple steps yeah get a foothold in the door

look move play around and cash out those three uh I will just go over

so start with uh get a foot in the door so normally we

see three ways how attackers can can get into the environment in the ransomware

cases you have vulnerabilities in uh remote uh internet facing systems you

have the remote Services itself and you have malware

starting with the with the the vulnerabilities and um I just looked uh up the last four

years and maybe somebody remembers netscaler the the so-called Citrix

vulnerability in December 2019 um it was released mid of uh 2019 uh

December 2019 the first POC publicly available POC was in beginning of

January and the patch was available in middle of January so there was a round one week to one and a half weeks between

a public proof of concept for the vulnerability and uh patch for the vulnerability and what we saw

during 2020 a lot of companies patched but the patch didn't remove the the

compromise so they were already compromised and um yeah with it with the patch they

didn't remove the compromise so what we found what we could provable

see or proof evidence for uh was nine

month uh customer was breached after nine months using this this vulnerability

and we had other customers where we could see that the netscaler was affected after two years but we couldn't

prove that this this compromise was the reason for the actual ransomware case

and of course such vulnerabilities happen not that often

yeah so 2021 gave us uh hafnium exchange

vulnerability also a similar situation the patch

appeared as an out-of-band patch from Microsoft on a Tuesday evening 10 o'clock in German time

we saw during our uh incidents or the the assessments we did that

um the first exploit exploitation attempts were seen on Wednesday in the morning at

5:00 am so around seven eight hours later um I know one guy who could patch

because he was online when the patch was released otherwise Germany was unable to patch in

time and of course we can go on with 2021 proxy shell also

exchange vulnerability proxy nutshell also exchange vulnerability

we have uh in 2022 VMware Horizon the the virtual desktop infrastructure

from VMware just to name also open source stuff Zimbra a collaboration platform

including an email server uh has had a vulnerability actually the vulnerability

was in cpio from 2015 I think which led

to a compromise using via email so you send an email

with a cpio with a specially crafted archive file and you could drop a web

shell in one of the directories yeah you have of course 40 OS which is a

40 gate VPN and firewall operating system

and if you read the news we start at the beginning again

netscaler had some issues several weeks ago according to foxIT we have 1900

still unpatched net scalers worldwide how many patched

was netscale has exists that um have not been checked for compromise we

don't know of course so that will be a nice year probably

um so what can you can you do against this kind of of attack vector patch your systems is one thing as you

see this that doesn't lead to the the um or what you need to do afterwards in

such cases you need to check your systems for possible compromise

that is important to reduce this I highly suggest put your

uh Services behind some VPN so that only people who already have

connection to the VPN um can access your services or the services

they need and that would reduce the attack surface

at least to the VPN server so but I

of course we can also think about remote services without vulnerabilities

um there can be configuration mistakes so the admin does something wrong there can

be insecure default configurations like this um I don't know if you know it but the

local admins or the administrators on the Windows system are are

automatically in the remote desktop users group you know and so

we had several cases especially in the beginning of the pandemic when everybody moved from uh to the home offices and

they needed to put people fast in the position to to access their the assist

the internal systems again they just put a RDP server on the internet and hope for the best

um additionally if you put services on the internet of course brute forcing and

credential uh stuffing are attacks that are possible so brute forcing just trying the the

username and password combinations uh credential stuffing using already leaked

passwords or credentials from leaks you find on the internet

what you can do about this kind of of attack Vector is uh just as I said use

multi-factor Authentication and reduce the attack surface as in the

point with the vulnerabilities before by moving the services behind a VPN and

then use multi-factor authentication on VPN of course

the last Vector that we see normally that the attackers can get in the

network is malware we all know this about

those funny emails you get with the attachments

um include that have either Word documents

attached either zip files with with Visual Basic scripts javascripts and

what you can get isos you see a lot these days

um or what you can also have that you can have just a link inside the email and

you download the respective file from some some shady file sharing website

um what we saw over the last year was uh USB sticks again funnily

um I'm not sure if you have heard about raspberry Robin which is a malware that

warms via USB sticks um but I haven't seen it as a vector for

ransomware yet on my own but there are people who said that it's

an initial access broker for some of the ransomware gangs

so what can you do about this if you think the

you can of course ban simply some file extensions in your mail server or you

change the file Association types in your operating system meaning that you

don't open the JavaScript and Visual Basic script files using for example the

windows scripting host but open it with notepad and that will

of course some people will be

uh some people will think about what this this is then and ask the IT guys

but it's better than running the the script itself

one thing I I I don't like to to say it but keep your AV updated

um uh this is one thing keep it updated and read the logs

we see a lot of incidents where we see that the already

days or weeks before we you can could have seen that there's something going

on in your network yeah and if you see malware in your AV logs

then react to it just check it you don't know how long this AV this malware has

been on your system the thing is that

just because you're AV detected it now it might have been get received an

update for its signatures and the malware was active for days or weeks

before so when they are inside

then they usually look move and play around a little bit

so when they look around what they do

is they they enumerate AD they do Ports scan the you they search for

vulnerabilities they check uh what they how they can escalate

their privileges they try to find credentials

um Kerber roasting we heard in the talk before for example is this one thing

um they try to identify accounts you around

have running on your systems they can use they can get the credentials from and you reuse

and for that reason one of the most important things I think is that you

have a principle of least privileges in your environment so only what a account needs

you should be able to do you should use dedicated service

accounts for your services of course and just for your information

um service account is not an account that has a SVC underscore in front of it

and is otherwise a normal user account um there I exist educated service

accounts in Windows environments so use them use strong passwords I still

I today I can I know companies who still use eight

character long passwords um I think that's 20 minutes on a decent

graphics card today so use strong passwords length matters

12 plus characters is the minimum in my opinion

and don't reuse passwords especially on your

systems the local administrator especially in small and medium-sized businesses you see a lot that people use

the same password for all local administrators so if I have it on one

system I have the whole company don't reuse it

um yeah [Audience LoL]

so when they they move around in your network using either a password hashes they found valid credentials they they

discovered somewhere vulnerabilities are also used to to move around in your

environment they try to to establish persistence mechanisms here we heard

also in the talk before about the C2 channels command and control channels they use

um they install in some cases directly any desk team viewer and or other remote

control software or sometimes they also use tunneling softwares like ngrok or recently they

started using cloudflare G um and this is what you need to do is

prop have a proper Network segmentation and with the proper Network segmentation I don't talk about subnetting

subnetting means you just have different subnets you need to have a firewall between them

and you have need to have rules between them that

restrict access between your your subnets and one thing is especially important

please keep or use Network segmentation to

restrict the access to your backup and your Management Systems as far as

possible we see a lot especially as I said in the small and medium-sized businesses or in

the other organizations we we have as customers that

yeah they have they tell us in in in in in in workshops yeah we have

a network segmentation every building is one segment and on the question yeah you

can move between in a segment you can access everything yes you can and

between the the buildings you can have also a firewall and you cannot access

anything no you can access everything and also your your uh VMware Management

console oh yes yes we can so everybody can access it yes of course and that

doesn't work so

um when they play around they normally try

to to gain more privileges so privilege escalation is assisting local privilege escalations normally but also using

vulnerabilities um or misconfigurations insecure default

configurations um my personal favorites are Group

Policy preferences or passwords in group policy preferences

this is no longer possible since I think 2014 to put passwords in group policy

preferences however if you had your password stored in those preferences

before the patch in 2015 2014 then they're still there and yeah there

are AES encrypted but the encryption key is on the Microsoft website

so you can just download it and just take it and decrypt the keys

um then during that phase they also try to disable your security measures

the thing you can do is of course patch your system so you know try to get your

availabilities out of of this equation can try to configure your systems in a

secure way this is not always possible due to some shitty uh third-party software

and keep your AV um updated and please please as I said

already check the locks and act accordingly

so in the last phase they cash out that's when when they

start using a uh being your backup service so they copy

data from your your environment using um

file sharing platforms for example yeah Mega and set was was once the thing we

transfer we had already uh every every other file sharing platform you you can

think about is a possible way to exfiltrate data they also use their

their C2 communication channels so sometimes you they also they just use the the possibilities in any desk or in

in RDP clients or they use uh file transfer protocols

like um SS SFTP um

we saw for example in one case that they try to install filezilla on every machine they had access to

um because on the first one it didn't work on the second it didn't work on the third it didn't work yet because SFTP

was blocked uh outgoing and that is one of the things you can do to to prevent

exfiltration block at least

protocols you know that you don't need in your environment and proper Network segmentation of

course is a general thing so in the last step that's when they

start the encryption um they're running the ransomware or

normally they are have domain admins at that point so they can run it on all

domain connected systems they can also disable of course when they are domain admin they can disable

the AV before they they start to run somewhere ransomware's today disable services like

databases and such things so that they have the full power of the machine for

the uh for the encryption

um if you get lucky not how everything works perfectly because they use

group names and windows is especially picky when you have a non-english uh

windows installed for example in Germany the the group everybody is called yida

and we had cases where the ransomware didn't really work that well because they couldn't

change the permissions of the files first um

they use different encryption schemas normally they they come with the

asymmetric and the symmetric encryption type the asymmetrics or public key cryptography the public key comes with

the ransomware and is used to encrypt the symmetric keys they generate on in

your environment depending on the ransomware they they generate one key for each system or even one key for each

file it depends a little bit on the on the ransomware how it works but that's

the usual thing they use um I would never count on the the fact

that there are possibly maybe there could be

decryptable uh things um in in my opinion in my uh in my world

The ransomware Gangs have learned and used the standard Microsoft Windows or

some other publicly available libraries to to do the encryption

they executed by a remote tools like PSX Powershell or some use

gpos group policies to execute the ransomware on every

machines they they connected to the domain and what can you do about this no it's

it's hard but the the most important thing is have online backups offline

backup sorry thanks you you see you see off online backups

are not that are great but not that great offline backups is the most important this is the most important

thing so um don't have it connected to your environment

the the the USB disk on the system is not offline backup

um in my opinion if you see that something is is still encrypting

I I'm I'm always hesitant to say shut down the system because you can break

the encryption and maybe the file that is currently in under encryption or the files will never be decryptable if you

want to buy a decrypter um or gather the cryptos through some

discussions with the with the ransomware guys um if it's a VM just suspend it and

that's it and if everything is already encrypted

keep cool and call your incident responder

so now let's talk about incident response what happens when it's already too late and what can you do to support

your incident response team at first the things I'll say in this chapter are

for our company and how we work so other companies might work a little bit different than that

first for some reason incidents always come on Friday afternoon

so some customers think it is a good idea to try to solve a case by

themselves maybe until the end of the week and if they didn't solve it until the end of the week they call the

incident response team please don't do that it doesn't help your company and it

doesn't make your incident Response Team happy to have to work on the weekend and in addition the longer you wait with

calling the incident Response Team the longer the incident response will take and the more complicated forensics will

be because you have lock retention times while trying to do stuff by yourself

maybe you modify some of the systems and it becomes much more harder to do

precise forensics so what happens on our site when such a

new incident ticket arrives the first thing we do is team internal coordination

so we discussed do we have enough people do we have a person for each role in our

team we have three roads incident handling forensics analyst and Mayweather analyst

so first let's talk about incident handling incident Handler is responsible for all

the tasks that our customer facing and the first point is always get the customer out of that headless chicken

mode like we call it because when an incident comes at our customer site

everyone is like running around in so-called like headless chicken doing something but not doing anything helpful

so this is always the first task for the incident Handler Handler structure the customer do meetings and then do all the

relevant decisions leading to a secure emergency operation mode that means in

this case that you have working core infrastructure so working domain controller maybe a working email server

and whatever you need or whatever you define as very business critical systems

let's go a little bit more in detail probably the first measure will be to cut off the internet connection because

you just buy you a lot of time with doing that no matter how many back doors

the attackers placed in the network if you cut off the internet connection the attackers can't access their back doors

anymore and then you will start to rebuild your network you will Define everything in

your current infrastructure as red Network and then start building up a

green network with clean systems maybe you will start with some admin workstations so that the

administrators can work properly and then we go through a prioritized system list and build up the most

important systems this can be of course like I said domain controllers or email

servers but also whatever is important for your current company and for the

business this is of course in a hospital something very different from a small

business or maybe a university and please have such a prioritized list before your first incident because when

you start discussing in an incident which system is the most important then everyone in the company will tell you

something else and everyone will tell you that their system is the most important and has to be migrated and

checked and analyzed at first and this isn't really helpful

so during incident handling there might also be obstacles for example if you

have backups and you encrypted your backups this is great but you should think about where to

store that encryption key because when you store it in your password manager and the password

manager database is in a VM which then gets encrypted by the ransomware well

you have backups but you don't have your key to decrypt the backup so this

doesn't help you very much and this is not only two for password

this is also true for like everything else which you could need during an incident like contact lists or network

lists and so on so

working with third parties is also a task for the incident Handler this could be a data Protection Agency

or the customers customers because they also often yeah have some panic and have

questions which the customer maybe can't answer so this is also a talk a task for

the incident Handler and of course also working with law enforcement but there's

one thing to keep in mind Law Enforcement wants to do criminal investigation incident response wants to

bring you back to business as soon as possible this is not necessarily the same goal but it finally makes sense to

allow your incident responders to share all information with law enforcement because this is beneficial for everyone

um so why do I do forensics at all one could ask well the less precise the

forensic results are the more conservative the rebuild needs to be so if you don't know anything about the

attack you have to rebuild everything from scratch so let's say you have maybe two week or

backups and you know the initial access of the attackers was only one week ago

then you could use that two weeks old backups for rebuilding which is really

great but if you don't know how long the attackers were in the

network then it's really hard to to say

a second Point why you should do forensics is to estimate the impact

especially in an early um phase of an attack it is important to know that the attackers only compromise

some user devices or did they already gain domain administrative privileges

because then the incident handling is completely different and of course it is also relevant to

know which attack path the attack is used so you can Harden your network and

improve in the future how should incident how should forensics

be it should of course be correct it should be non-disruptive for the customer so they can concentrate on

rebuilding their infrastructure and it should be fast what means correct you want to have a

dedicated forensics analyst so you want to have incident response team which is

doing incident response in their day-to-day business and you want in your forensics teams one

person who can really concentrate on forensics and is not also doing the incident handling and the malware

analysis and so on because then you can't really concentrate on doing the technical work

it also should be non-disruptive as I said that means for our company we do remote triage collection we use the tool

videos adapter for it this is an official logo of velociraptor it's a really nice tool because we just send

our customers an exe file the customer just has to execute the exe file and

then it collects all the logs we need all the um yeah information we need

and absolute uploads it to our um to our infrastructure so we don't

need to drive to the customer and like start copying hard disks for a week for weeks because this doesn't help anyone

I should also said it should be fast so we have a lot of Automation and tooling

to make the work really fast automatically pushing our the uploaded files we want

to lose this infrastructure automatically doing reporting and so on I won't go in detail here because this

is enough for a single talk about it maybe maybe on Congress let's see

so what happens when we do manual forensic analysis

there are some questions you normally want to answer one is of course about the vectors you want to find all vectors

the attack is placed in the network of course you want to have General IUC

so indicators of compromise so maybe which IP addresses did the attackers use so you can block them in the firewall or

which tools did they use so you can scan over the whole it infrastructure and

find further compromised systems and if you have the iocs you can also cross-correlate between cases or show

them share them with law enforcement another important thing is of course

lateral movement because you can use lateral movement to build a timeline of the attack so the mythology here is to

look where the attackers came from and where the attackers went so if you have a system a and you see the attackers

came from system B then you analyze system B and go back in time and the same you could do forward in time until

you found yeah all relevant systems which were compromised and have a proper timeline of the case

and other common thing in the life of an incident responder is to fall in a rabbit hole

because it can be really really interesting to do forensics but yeah you need to

check what the customer really wants to know or what your incident Handler really wants to know

and not just analyze something and fall into the rabbit hole because this yeah

really doesn't help anyone here also the 80 20 rule applies so I'd say you get like 80 percent of

the relevant forensics results in 20 of the time and often this is enough for the incident Handler to do the right

decisions to rebuild the customer Network customer's networks as fast as possible

so let's go into reporting nobody really likes reporting

but producing fancy forensics results doing fancy technical stuff is worthless

if you can can't explain it to a manager nobody pays us to do fancy technical

stuff nobody understands but we are paid to help our customers and explain the

situation to them and to do the right decisions so yeah nobody likes reporting but it is a really really important

thing so after learning the report the

incident is over and all work is done right no not really because such a

report normally contains a lot of recommendations what to do

for the customers and as I already said incident response more or less stops

when the customer is in an emergency operation mode with a core infrastructure and the most critical

systems are working but there's still a lot to do and of course security is a process so

there's always something to improve to stay ahead of the attackers

so now let's go into a quick recap

how to protect against those kind of attacks we had this uh the topics just

as a recap patch your systems it's important keep them updated

vulnerabilities come and go that's one thing have a sane privilege

account management so use the correct user accounts for the correct for the

the right task use the second two-factor authentication

for remote services have a proper Network segmentation so

put firewalls and access rules between your networks check your AV logs regularly and act

accordingly and not really protecting your you against an attack

but keeping the the symptoms a little bit um not that that devastating have

offline backups so how to make incident response easier

for you for yourself and for the incident Response Team first thing is don't try to solve this stuff by

yourself just call the incident response team of your trust as soon as you know

you have an incident the second thing is logging policy because if you have a very short lock

retention for example and you only have locks for the last like two days then it

might be hard to find out what happened and if you don't lock stuff at all

especially in the cloud environments this is a thing where you need to look at then it might also be hard to get

proper forensic results then as I said have offline contact

lists because your main server is down your accounts are compromised so you can't use like your Microsoft teams or

whatever you use in your company you need an alternative way to communicate to your colleagues and or

employees then as I said prioritized asset list and network plans

and a disaster recovery plan in best case and please all have all that in a

secure place in best case on paper well nobody likes doing stuff with their

trees I know but that trees don't get ransomware right

so that's it from our site more or less normally this is a point where

you can ask us questions but we want to spin that around and give you some

questions maybe someone can guess the right answer

what do you think is the shortest time to domain admin we saw an incident between initial compromise and the

attackers gaining domain admin

yeah it's not it somebody said 17 seconds two minutes well it's not that

bad six minutes is is what we have seen in incident hmm

what do you think think is the shortest lock retention on a domain controller so how long

zero 10 hours a year well it is 2.5 minutes

what what do you think is the highest number of domain administrator accounts

we saw in an incident 200 nah 50.

64 with 120 people working in IT

what do you think is the longest dwell time so the longest the longest time between being initially compromised and

realizing that you are compromised 60 now it's it's not that bad I hit here

a lot of years well it is like around two years

so well that's really it from our side no time for questions thank you

if you

if you have questions we will be there outside waiting come and join us

thank you thank you thank you Harryr and Kris warmer applause thank you

End of subtitles:[Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]