When Disaster Hits

[deanwebb]

So, I'm involved in a major outage here... I'm a vendor at a customer site and it's a BIG outage.

What would happen if your network or a major customer's network were hit with a massive malware outbreak that cooked their AD, knocked out their management tools, and left big question marks about whether or not the network would be safe if clients started to connect again?

How fast would the people be able to work to deal not only with the basic issues, but all the knock-ons of having bottlenecks due to this or that being offline, software not behaving as expected, and human resources being pulled in multiple directions?

This is a very big question to consider, and I thought I'd kick off the discussion about it.

[icecream-guy]

I would think this would be one place for those in charge to declare a disaster and start implementing the DR procedures.

[deanwebb]

I would think this would be one place for those in charge to declare a disaster and start implementing the DR procedures.

Absolutely.

So the next question is, what systems exist that can act as replacements for the ones taken out in the disaster? Do you have other means of managing, assigning rights, and things like that? Do you have a plan to spin up backup capacity in the cloud?

[Otanx]

So assuming a full ownage, can't trust any system that was online. Here is a basic DR.

1. Cold spare, and offline project hardware inventory by hand. Find out what I have to work with.
2. Unrack all equipment in two racks at the DC. Tag and move it out of the way. Give me a space to work in.
3. Figure out what I can bring online. From a network perspective I always have a couple spare routers, and a few switches. May not have 10G, but can get some basic services.
4. Use laptops that were offline, or buy a couple laptops to configure network gear.
5. Once I have basic network connectivity to the internet the systems guys can use a couple laptops to start pulling down isos from known good websites. (VMWare, MS, Redhat, etc).
6. Bring online basic services (AD, DNS, NTP, DHCP).
7. At this point you have the basics. Now you need to figure out how far back the compromise happened, and start recovering from backups older than that. You do have offline backups right?
8. Realize I never requested the off-site backups to be delivered in step 1. Request that now, and get some food and/or sleep while we wait.
9. Recover systems from most critical to least till I run out of hardware.
10. At some point above here you need to figure out how to clean the compromised hardware, or begin the multi-million purchase request to buy all new.
11. Start cleaning, and re-installing the old hardware. Restoring services as you have equipment.

Rough draft. We don't have any cloud stuff so I didn't include that. At minimum you are resetting everyone's password for SaaS stuff.

-Otanx

[deanwebb]

^ Good list, now add to it checking all the IoT devices for possible compromise due to their being on the dirty network with a default vendor admin credential.

Prior to restoring vendor VPN connections or network connections to ancillary sites, verify that those sites are clean. Your pentesters can only check the access points you identify. If attackers can gain access via another route, they will gain access. It is quite possible to be hacked from multiple sources simultaneously.

[DesertFox]

Probably before starting to clean the old hardware some analysis task should take place? I can imagine that Chain of custody should be implemented so you have more info what and how happened prior destruction of all evidence (and helpfully not to have the same breach in 2 weeks).

[deanwebb]

Very true. If law enforcement is involved, they can be very good for chain of custody and forensics. Collect enough boxes to study to learn what happened. Once that's determined, you are looking at a very long process of re-imaging everything.

EVERYTHING.

[Otanx]

IoT etc would be under 10, and need to be cleaned (if they can be) or replaced. Assuming a full compromise I am not connecting anything from the old network into the new. Depending on the size and age of the IoT stuff this may be hard, or impossible. At that point someone needs to accept the risk of reconnecting this.

I didn't include any of the analysis, or cyber stuff in my list. This check list is to get the company up and running again. Another checklist would be needed for that. Assuming you have the skill set in house a quick stab would be;

1. Secure systems per current forensics best practices. Do you pull power, graceful shutdown, or leave systems running? Do you pull network cords from the system causing a link down, or do you leave them alone, and log traffic? It seems to change depending on the phase of the moon.
2. report to company leadership, and legal.
3. report to law enforcement
4. report to any compliance requirements (i.e. PII breach laws etc)
5. Create copies to use for investigation, more than one.
6. Secure originals.
7. Investigate compromise to determine. What, When, and How specifically. The who is not as important.
8. Report those findings to the COOP/DR team, and other stakeholders.
9. Continue investigation.

-Otanx

[deanwebb]

Assuming a full compromise I am not connecting anything from the old network into the new.

... and then the guys that run Payroll ask how they're going to get their jobs done if,

a) half their machines are dirty and the server is clean

or

b) half their machines are clean and the server is dirty.

This is where having a segmented data center is helpful...

[deanwebb]

My thoughts are now turning to information and access. There are things that will be available and absolutely great to have. They are the same things that, if you don't have them, you wish real hard that you did.

First on that list is an external hard drive with current copies of network diagrams. Even relatively recent copies will do the job. If a ransomware worm gets into the network share where these are kept, it's game over as far as sharing intel quickly with first responders.

Likewise, information on what SNMP communities exist and what devices they work with; SNMP v3 information and what devices accept those credentials; TACACS accounts that are not connected to AD that work; where network devices still have local accounts and those credentials; which devices do SSH with keys of length 1024 or greater; which devices are still stuck on telnet. Knowing this can do two things: help with getting access to determine if the network devices are compromised, and being able to make an educated guess about which devices and credentials are most likely compromised.

Keep that information on the emergency external drive, as well. I specify an external drive because a PC off the network is too easily left unpatched and could accidentally be connected to a hot network, whereupon all its information gets compromised by the ransomware.

What else... how about a client installed on each PC that is able to monitor the activity on the PC and also run scripts with local admin or system privileges? This client should be able to access the system independently of AD, which could be compromised in such a situation. Enterprise software distribution tools can be damaged in a major outage, so having the scripting ability can invoke a hardware install from a known clean network share.

Monitoring activity on endpoints is critical. Anything and everything that provides information for reporting is excellent. If it can provide spreadsheets that can be further analyzed, even better. If the client or AD account can reach most machines, but is cut off on a segment of the population, then it's a good bet that the ones where it has been cut off are compromised.

What else would be good to have on that external hard drive? What would be good to have in a sealed envelope?