Is there anything positive we can takeaway from the CrowdStrike Incident

CrowdStrike "Blue Screen of Death"

Considering all of the negative impacts, costs and disruption of the recent CrowdStrike incident is there anything positive that we can take way from this situation that is helpful going forward?

One thing that the CrowdStrike incident showed most organisations, even those that were not CrowdStrike users, is that there were gaps in tech infrastructure recovery procedures. Most organisations design their back-up procedures around the concept of restoring networks, configurations and data. All of this is predicated on the assumption that ALL of the organisations endpoints will be attached to a network so that the recovery actions can be performed. This was not the case with the CrowdStrike incident because network endpoints (i.e. desktop, laptops etc.) could not “fire-up” in order to connect to a network and thus preventing automated, scalable endpoint recovery.

The larger the organisation, the larger the problem became. Organisations with several hundred or even thousands of network endpoints were faced with a massive chellenge – recovering affected devices where automated, network dependent remote fixes were not possible. An additional complication for enterprises with thousands of systems, each with individual BitLocker recovery keys, was the one-at-a-time acquisition and entering of the key. CrowdStrike did provide guidance but it required a manual “Safe Mode” operation for each affected endpoint – which created significantly protracted recovery times.

So, the positive takeaway was that it potentially highlighted a “Rumsfeld principle” (something we don’t know we don’t know) or at least a DR scenario that was not previously considered – a situation where every endpoint in the organisation was unable to connect to a network. This is a thorny problem: how do you recover thousands of network endpoints that can’t get onto the network?

Microsoft has recently announced some solutions to this problem using either external boot media (i.e. a USB) or a safe mode option. Both, however, require that each endpoint is manually recovered, and the BitLocker issue remains a consideration.

So, what should organisations do? Is there a better way to be prepared? In my opinion it reaffirms the case that offsite recovery partners still remain an option. Being able to relocate to a site that can restore your IT infrastructure – including laptops, desktops etc onto a trusted environment can preserve an organisations operational capability during an extended period of disruption.

RiskCentric provide a range of of business continuity consulting services to organisations of all sizes