NIST CSF: Recover: System Administration AND Cyber Security

The Recover phase of a NIST CSF-based Cyber Security program is about the longer-term clean up after an incident, as opposed to the Respond phase which is more focused on the short-term. The Incident Response Plan (IRP) has kicked in, you have taken steps to stop the problem and now it is time to get back to normal.

The Recover phase is a classic example of why Cyber Security is not just an IT function and why we say management has to be involved. Much of what we do in the Recover phase is done by System Administration people or Operations people, because recovering from hardware failure and recovering from a ransomware attack are damn near the same activity. Your Cyber Security program, in effect, outsources this work to their colleagues. But that outsourcing does not exempt this process from the usual Cyber Security rigor and methodology. Trust, but verify: yes, trust your colleagues to their job but also apply the usual oversight and standards of proof.

Let's take a real-life example, which was presented at a HIMSS conference a few years ago. A mid-sized health care system in the Pacific Northwest was using popular EMR software which was renowned for the large amount of downtime required to install updates and upgrades. That was a System Administration issue and they came up with a System Administration solution: two identical computers and a shared, duplicated storage system. This allowed the Sysadmin to run the software on Computer A, apply an update to the software on Computer B, then cut over to Computer A when the update was ready. Since the two computers shared the same data across their storage systems, the disruption to the users was minimal.

Then this organization experienced a ransomware attack when Computer A was hit with a virus. In other words, Computer A was infected and Computer A's copy of the data was encrypted. However, Computer B was uninfected and Computer B's data (although somewhere between 1 and 15 minutes out-of-date) was unencrypted. Given their IT infrastructure, their IRP was gloriously simple:

1. Shutdown Computer A (the Respond phase: stop the problem)
2. Start up Computer B (the Recover phase: fix the problem)

Total downtime: about 15 minutes. Total lost productivity: that 15 minutes plus about 15 minutes (backup lag), so call it 30 minutes. Temptation to pay the ransom: none. PR exposure: none; in fact, they got positive press.

Was this all System Administration or Operations? Yes. Was this their Respond phase and Recover phase of their Cyber Security IRP? Also yes.

This is an example of why we say that the CISO role usually has limited authority, but to do the job properly requires wide influence. The CISO relies on parts of the IT department for many crucial services, but is ultimately responsible when these services are provided in the context of an IRP. This means that the CISO (or whoever is in charge of the IRP) has to get proof from their colleagues that these services will be there when needed. The Recover phase therefore often requires a light touch and good people skills in addition to the technical chops needed to ensure that the job gets done.