Stories from the Life of an Incident Responder

Hello and good evening on day two of the chaos communication Camp 2023 Translated by {Yang}{Li} (ITKST56 course assignment at JYU.FI)]

it's late in the evening this is meleeway stage in case you're wondering

and the next talk is going to be about incident report responses

so if you're curious about how to even get there to have an incident response how you could
prepare for an incident response and how you could support a new organization

uh, the incident response team in doing the job and trying to fix whatever broke

let's put it that way um we have the right talk for you

this is stories from the life of an incident from incident responders Harry and Chris

please a very warm Round of Applause [Applause]

so, good evening and thank you for joining us today um we will tell you a little bit of our

life as incident responders and I'm Chris I did my computer science

studies at the University of alang and Nuremberg I do this security stuff for

over 10 years now so my CV is a little bit longer at the moment I'm a detection

engineer before that I was a long time working in dfir so digital forensic incident

response in different organizations and

yeah I'm Harryr I studied electrical and computer engineering at RWTH

University and I played a lot of CTF and did some hacking stuff at chaos computer club RWTH

during my masters I worked at x41 dsac doing pen testing patch analysis

so I also have some kind of offensive security background on for around one year now I'm working at G data Advanced

analytics doing digital forensics and incident handling

first Christian will give you a short introduction and then he will tell you how a classical ransomware attack looks

like and in the second part of the talk I will tell you how the incident

responders work and what you can do in advance to make it go as smooth as possible and support the incident

response team so as Harryr told you I will probably

we'll talk about ransomware because the customers we usually have are small and

medium-sized businesses universities and hospitals and those are regularly

unfortunately regularly hit by um um

ransomware gangs the main reason for this and that's if you heard the last

talk um why they maybe not that responsive

and are not so interested in they just lack the resources so the manpower to do

uh proper security measurements to secure their systems especially in in erm

situations where you are for example in a hospital have medical devices

um which where you cannot simply install an AV on or even patch the system

because you lose the certification as a medical device then but also in in

companies manufacturing companies on the shop floor we're talking about systems

that have run times of 25 plus years so if you look back now 2023

we're talking about XP and older systems fun fact I was in a ransomware case and

Wannacry in 2017 when I got a call from from a person from the shop floor

asking me if we have a nt4 expert, um

that can tell us if WannaCry is affecting nt4 of course you don't need

to be a expert for NT-4 this one requires of course not affecting nt4

systems so due to the time uh slot we thought

memes are the best way to to tell you those stories and we have a lot of them

so in the first uh um section I tell you a little bit of how an attack Works

um there are a lot of different possibilities how you can describe and how to structure the how an attack works

there's the miter attack framework for example there was for example a talk Yesterday by Maker Salko

um here on the stage there's the original cyber kill chain from from Lockheed Martin you have

stuff from from companies like Mandy and their targeted the tech life cycle but

that's all in my opinion two two fine-grained it's that's the reason I

just take three simple steps yeah get a foothold in the door

look move play around and cash out those three uh I will just go over

so start with uh get a foot in the door so normally we

see three ways how attackers can can get into the environment in the ransomware

cases you have vulnerabilities in uh remote uh internet facing systems you

have the remote Services itself and you have malware

starting with the with the the vulnerabilities and um I just looked uh up the last four

years and maybe somebody remembers netscaler the the so-called Citrix

vulnerability in December 2019 um it was released mid of uh 2019 uh

December 2019 the first POC publicly available POC was in beginning of

January and the patch was available in middle of January so there was a round one week to one and a half weeks between

a public proof of concept for the vulnerability and uh patch for the vulnerability and what we saw

during 2020 a lot of companies patched but the patch didn't remove the the

compromise so they were already compromised and um yeah with it with the patch they

didn't remove the compromise so what we found what we could provable

see or proof evidence for uh was nine

month uh customer was breached after nine months using this this vulnerability

and we had other customers where we could see that the netscaler was affected after two years but we couldn't

prove that this this compromise was the reason for the actual ransomware case

and of course such vulnerabilities happen not that often

yeah so 2021 gave us uh hafnium exchange

vulnerability also a similar situation the patch

appeared as an out-of-band patch from Microsoft on a Tuesday evening 10 o'clock in German time

we saw during our uh incidents or the the assessments we did that

um the first exploit exploitation attempts were seen on Wednesday in the morning at

5:00 am so around seven eight hours later um I know one guy who could patch

because he was online when the patch was released otherwise Germany was unable to patch in

time and of course we can go on with 2021 proxy shell also

exchange vulnerability proxy nutshell also exchange vulnerability

we have uh in 2022 VMware Horizon the the virtual desktop infrastructure

from VMware just to name also open source stuff Zimbra a collaboration platform

including an email server uh has had a vulnerability actually the vulnerability

was in cpio from 2015 I think which led

to a compromise using via email so you send an email

with a cpio with a specially crafted archive file and you could drop a web

shell in one of the directories yeah you have of course 40 OS which is a

40 gate VPN and firewall operating system

and if you read the news we start at the beginning again

netscaler had some issues several weeks ago according to foxIT we have 1900

still unpatched net scalers worldwide how many patched

was netscale has exists that um have not been checked for compromise we

don't know of course so that will be a nice year probably

um so what can you can you do against this kind of of attack vector patch your systems is one thing as you

see this that doesn't lead to the the um or what you need to do afterwards in

such cases you need to check your systems for possible compromise

that is important to reduce this I highly suggest put your

uh Services behind some VPN so that only people who already have

connection to the VPN um can access your services or the services

they need and that would reduce the attack surface

at least to the VPN server so but I

of course we can also think about remote services without vulnerabilities

um there can be configuration mistakes so the admin does something wrong there can

be insecure default configurations like this um I don't know if you know it but the

local admins or the administrators on the Windows system are are

automatically in the remote desktop users group you know and so

we had several cases especially in the beginning of the pandemic when everybody moved from uh to the home offices and

they needed to put people fast in the position to to access their the assist

the internal systems again they just put a RDP server on the internet and hope for the best

um additionally if you put services on the internet of course brute forcing and

credential uh stuffing are attacks that are possible so brute forcing just trying the the

username and password combinations uh credential stuffing using already leaked

passwords or credentials from leaks you find on the internet

what you can do about this kind of of attack Vector is uh just as I said use

multi-factor Authentication and reduce the attack surface as in the

point with the vulnerabilities before by moving the services behind a VPN and

then use multi-factor authentication on VPN of course

the last Vector that we see normally that the attackers can get in the

network is malware we all know this about

those funny emails you get with the attachments

um include that have either Word documents

attached either zip files with with Visual Basic scripts javascripts and

what you can get isos you see a lot these days

um or what you can also have that you can have just a link inside the email and

you download the respective file from some some shady file sharing website

um what we saw over the last year was uh USB sticks again funnily

um I'm not sure if you have heard about raspberry Robin which is a malware that

warms via USB sticks um but I haven't seen it as a vector for

ransomware yet on my own but there are people who said that it's

an initial access broker for some of the ransomware gangs

so what can you do about this if you think the

you can of course ban simply some file extensions in your mail server or you

change the file Association types in your operating system meaning that you

don't open the JavaScript and Visual Basic script files using for example the

windows scripting host but open it with notepad and that will

of course some people will be

uh some people will think about what this this is then and ask the IT guys

but it's better than running the the script itself

one thing I I I don't like to to say it but keep your AV updated

um uh this is one thing keep it updated and read the logs

we see a lot of incidents where we see that the already

days or weeks before we you can could have seen that there's something going

on in your network yeah and if you see malware in your AV logs

then react to it just check it you don't know how long this AV this malware has

been on your system the thing is that

just because you're AV detected it now it might have been get received an

update for its signatures and the malware was active for days or weeks

before so when they are inside

then they usually look move and play around a little bit

so when they look around what they do

is they they enumerate AD they do Ports scan the you they search for

vulnerabilities they check uh what they how they can escalate

their privileges they try to find credentials

um Kerber roasting we heard in the talk before for example is this one thing

um they try to identify accounts you around

have running on your systems they can use they can get the credentials from and you reuse

and for that reason one of the most important things I think is that you

have a principle of least privileges in your environment so only what a account needs

you should be able to do you should use dedicated service

accounts for your services of course and just for your information

um service account is not an account that has a SVC underscore in front of it

and is otherwise a normal user account um there I exist educated service

accounts in Windows environments so use them use strong passwords I still

I today I can I know companies who still use eight

character long passwords um I think that's 20 minutes on a decent

graphics card today so use strong passwords length matters

12 plus characters is the minimum in my opinion

and don't reuse passwords especially on your

systems the local administrator especially in small and medium-sized businesses you see a lot that people use

the same password for all local administrators so if I have it on one

system I have the whole company don't reuse it

um yeah [Audience LoL]

so when they they move around in your network using either a password hashes they found valid credentials they they

discovered somewhere vulnerabilities are also used to to move around in your

environment they try to to establish persistence mechanisms here we heard

also in the talk before about the C2 channels command and control channels they use

um they install in some cases directly any desk team viewer and or other remote

control software or sometimes they also use tunneling softwares like ngrok or recently they

started using cloudflare G um and this is what you need to do is

prop have a proper Network segmentation and with the proper Network segmentation I don't talk about subnetting

subnetting means you just have different subnets you need to have a firewall between them

and you have need to have rules between them that

restrict access between your your subnets and one thing is especially important

please keep or use Network segmentation to

restrict the access to your backup and your Management Systems as far as

possible we see a lot especially as I said in the small and medium-sized businesses or in

the other organizations we we have as customers that

yeah they have they tell us in in in in in in workshops yeah we have

a network segmentation every building is one segment and on the question yeah you

can move between in a segment you can access everything yes you can and

between the the buildings you can have also a firewall and you cannot access

anything no you can access everything and also your your uh VMware Management

console oh yes yes we can so everybody can access it yes of course and that

doesn't work so

um when they play around they normally try

to to gain more privileges so privilege escalation is assisting local privilege escalations normally but also using

vulnerabilities um or misconfigurations insecure default

configurations um my personal favorites are Group

Policy preferences or passwords in group policy preferences

this is no longer possible since I think 2014 to put passwords in group policy

preferences however if you had your password stored in those preferences

before the patch in 2015 2014 then they're still there and yeah there

are AES encrypted but the encryption key is on the Microsoft website

so you can just download it and just take it and decrypt the keys

um then during that phase they also try to disable your security measures

the thing you can do is of course patch your system so you know try to get your

availabilities out of of this equation can try to configure your systems in a

secure way this is not always possible due to some shitty uh third-party software

and keep your AV um updated and please please as I said

already check the locks and act accordingly

so in the last phase they cash out that's when when they

start using a uh being your backup service so they copy

data from your your environment using um

file sharing platforms for example yeah Mega and set was was once the thing we

transfer we had already uh every every other file sharing platform you you can

think about is a possible way to exfiltrate data they also use their

their C2 communication channels so sometimes you they also they just use the the possibilities in any desk or in

in RDP clients or they use uh file transfer protocols

like um SS SFTP um

we saw for example in one case that they try to install filezilla on every machine they had access to

um because on the first one it didn't work on the second it didn't work on the third it didn't work yet because SFTP

was blocked uh outgoing and that is one of the things you can do to to prevent

exfiltration block at least

protocols you know that you don't need in your environment and proper Network segmentation of

course is a general thing so in the last step that's when they

start the encryption um they're running the ransomware or

normally they are have domain admins at that point so they can run it on all

domain connected systems they can also disable of course when they are domain admin they can disable

the AV before they they start to run somewhere ransomware's today disable services like

databases and such things so that they have the full power of the machine for

the uh for the encryption

um if you get lucky not how everything works perfectly because they use

group names and windows is especially picky when you have a non-english uh

windows installed for example in Germany the the group everybody is called yida

and we had cases where the ransomware didn't really work that well because they couldn't

change the permissions of the files first um

they use different encryption schemas normally they they come with the

asymmetric and the symmetric encryption type the asymmetrics or public key cryptography the public key comes with

the ransomware and is used to encrypt the symmetric keys they generate on in

your environment depending on the ransomware they they generate one key for each system or even one key for each

file it depends a little bit on the on the ransomware how it works but that's

the usual thing they use um I would never count on the the fact

that there are possibly maybe there could be

decryptable uh things um in in my opinion in my uh in my world

The ransomware Gangs have learned and used the standard Microsoft Windows or

some other publicly available libraries to to do the encryption

they executed by a remote tools like PSX Powershell or some use

gpos group policies to execute the ransomware on every

machines they they connected to the domain and what can you do about this no it's

it's hard but the the most important thing is have online backups offline

backup sorry thanks you you see you see off online backups

are not that are great but not that great offline backups is the most important this is the most important

thing so um don't have it connected to your environment

the the the USB disk on the system is not offline backup

um in my opinion if you see that something is is still encrypting

I I'm I'm always hesitant to say shut down the system because you can break

the encryption and maybe the file that is currently in under encryption or the files will never be decryptable if you

want to buy a decrypter um or gather the cryptos through some

discussions with the with the ransomware guys um if it's a VM just suspend it and

that's it and if everything is already encrypted

keep cool and call your incident responder

so now let's talk about incident response what happens when it's already too late and what can you do to support

your incident response team at first the things I'll say in this chapter are

for our company and how we work so other companies might work a little bit different than that

first for some reason incidents always come on Friday afternoon

so some customers think it is a good idea to try to solve a case by

themselves maybe until the end of the week and if they didn't solve it until the end of the week they call the

incident response team please don't do that it doesn't help your company and it

doesn't make your incident Response Team happy to have to work on the weekend and in addition the longer you wait with

calling the incident Response Team the longer the incident response will take and the more complicated forensics will

be because you have lock retention times while trying to do stuff by yourself

maybe you modify some of the systems and it becomes much more harder to do

precise forensics so what happens on our site when such a

new incident ticket arrives the first thing we do is team internal coordination

so we discussed do we have enough people do we have a person for each role in our

team we have three roads incident handling forensics analyst and Mayweather analyst

so first let's talk about incident handling incident Handler is responsible for all

the tasks that our customer facing and the first point is always get the customer out of that headless chicken

mode like we call it because when an incident comes at our customer site

everyone is like running around in so-called like headless chicken doing something but not doing anything helpful

so this is always the first task for the incident Handler Handler structure the customer do meetings and then do all the

relevant decisions leading to a secure emergency operation mode that means in

this case that you have working core infrastructure so working domain controller maybe a working email server

and whatever you need or whatever you define as very business critical systems

let's go a little bit more in detail probably the first measure will be to cut off the internet connection because

you just buy you a lot of time with doing that no matter how many back doors

the attackers placed in the network if you cut off the internet connection the attackers can't access their back doors

anymore and then you will start to rebuild your network you will Define everything in

your current infrastructure as red Network and then start building up a

green network with clean systems maybe you will start with some admin workstations so that the

administrators can work properly and then we go through a prioritized system list and build up the most

mportant systems this can be of course like I said domain controllers or email