How a software update from cyber firm CrowdStrike caused one of the world’s biggest IT blackouts

Banks and health-care providers saw their services disrupted and TV broadcasters went offline as businesses worldwide grappled with the ongoing outage. Air travel has been hit hard, too, with planes grounded and services delayed.

So what happened, exactly? CNBC takes a look.

CrowdStrike is a cybersecurity vendor that develops software to help companies detect and block hacks. It is used by many of the world’s Fortune 500 companies, including major global banks, health-care and energy companies.

CrowdStrike is what’s known as an “endpoint security” firm as it uses cloud technology to apply cyber protections to devices that are connected to the internet.

This differs from alternative approaches used by other cyber firms, which involve applying protection directly to back-end server systems.

“Many companies use [CrowdStrike software] and install it on all of their machines across their organization,” Nick France, chief technology officer at IT security firm Sectigo, told CNBC’s “Squawk Box Europe” on Friday.

“So when an update happens that maybe has problems with it, it causes this problem where the machines reboot, and people can’t get back into their computers.”

On Friday, people around the world began encountering an error screen known as the “blue screen of death.”

This issue — a common problem among PCs, for example if a machine overheats — was the result of an update from CrowdStrike concerning its Falcon product. 

Falcon is a platform developed by the company that’s designed to stop cyber breaches using cloud technology — it is at the heart of the firm’s focus on endpoints. CrowdStrike said Friday it is in the process of rolling back the update globally.

CrowdStrike’s software requires deep access to a computer’s operating system to scan for threats. In the case of Friday’s outage, machines running Microsoft’s Windows operating system crashed due to a fault in the way a software update issued by CrowdStrike interacted with Windows.

“We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD [blue screen of death]) and get stuck in a restarting state. We approximate impact started around 19:00 UTC on the 18th of July,” Microsoft said in an update at 5:40 a.m. ET.

“We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance,” the company added.

Satnam Narang, senior staff researcher at Tenable, told CNBC on Friday that the outage was “very unprecedented.”

“The challenge here is that security software — because it’s doing its job to protect organizations — it has to have more privileged access to these machines,” he said.

So, while people may be seeing their IT issues as a problem with Windows, “it’s not actually a Windows issue, it’s related to a faulty or bad update from those security software,” Narang added.

Earlier, Microsoft said its cloud services had been restored after an outage that affected its Azure services and Microsoft 365 suite of apps in the central U.S. region. A company spokesperson said these are two different and nonrelated issues — one issue relates to Azure, the other is linked to CrowdStrike.

They added that they “anticipate a resolution is forthcoming,” in respect to the CrowdStrike problem.

CrowdStrike is “actively working with customers impacted by a defect found in a single content update for Windows hosts,” CEO George Kurtz said Friday in a update on social media platform X. He added that Mac and Linux hosts are not affected.

“This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed,” Kurtz said.

That fix could be hard to implement, though. Andy Grayland, chief information and security officer at threat intelligence firm Silobreaker, said that in order to implement a fix, engineers would have to go into each individual data center running windows.

They’d then have to log in, navigate to a certain CrowdStrike file, delete it and then reboot the entire system, he said.

“Where machines are encrypted, complex encryption keys also need to be entered manually. Unless Microsoft and CrowdStrike (if they are involved) pull something miraculous out of the bag, this could be painful to recover from.”