A routine software update caused cascading chaos Friday that has engulfed global businesses from airports and banks to retail and law enforcement.
It wasn’t a cyberattack, but was caused by one of the world’s largest cybersecurity companies deploying a flawed update while trying to keep their customers safe from hackers.
CrowdStrike, a cybersecurity company based in Austin, Texas, that’s widely used by businesses and government agencies that run on Microsoft computers, said that a defect in one of its updates for computers running the Windows operating system had caused the issue. The result was arguably the largest global information technology outage in history.
Here’s what we know:
What has been affected?
The outage appeared to affect almost every major business sector in one way or another.
Airports and flights across the world were severely delayed or canceled altogether, as the computers on which these services rely were disrupted by the issue. Three of the largest airlines in the United States — American Airlines, Delta Air Lines and United Airlines — all issued ground stops, citing communications problems.
Long lines snaked through airports from Berlin to Hong Kong, as electronic systems failed and staff members resorted to checking in passengers manually. One flier in the Indian city of Hyderabad posted a picture of a curiosity usually unseen in the modern digital age: a handwritten boarding pass.
The issue also affected banks, leaving some customers unable to access their money. People across Australia, New Zealand and elsewhere reported problems logging into their accounts at major retail banks. Meanwhile the London Stock Exchange, the largest of its kind in Europe, said some of its services had been disrupted although trading had not been impeded.
In retail, McDonald’s closed some of its stores in Japan because of what it said in an online statement was a “cash register malfunction.” And the British grocery chain Waitrose was forced to put up handwritten notes informing customers that it was only accepting cash. Some Starbucks locations temporarily closed after the company's mobile ordering system went down, while other locations stayed open and left their baristas scrambling to come up with new workflows.
Baltic Hub, a global shipping terminal in Poland, said in a statement it had been battling issues related to the outage. The Alaska State Troopers agency was among the law enforcement agencies to report issues, warning people that 911 was temporarily not working. And the British broadcaster Sky News, which is owned by NBCUniversal's parent company, Comcast, was briefly knocked off the air.
Some computers that were not on during the software update appear to have avoided the issue, according to initial reports.
What caused it?
Cybersecurity programs like CrowdStrike’s frequently and automatically update themselves to account for new tactics that malicious hackers have discovered. And there’s always a slight risk that any software update will be incompatible with other programs.
CrowdStrike’s update should have been routine, but an error in its code conflicted with Windows and proved catastrophic.
The company’s CEO, George Kurtz, said Friday morning on the "TODAY" show that while some customers would see their computers fixed automatically, others would require manual work to fix.
“The system was sent an update, and that update had a software bug in it and caused an issue with the Microsoft operating system,” Kurtz said. “Our systems are always looking for the latest attacks from these adversaries that are out there.”
How long until things go back to normal?
CrowdStrike identified the problem and issued a fix early Friday after the problem began to emerge. On Friday afternoon, Kurtz released a statement apologizing for the outage, adding that CrowdStrike was “operating normally.”
The fix required affected computers to download another software update, which some have been able to do automatically.
Many companies and services affected by the chaos, such as the New Hampshire Department of Safety, which said its 911 system was briefly down, reported that they were back online by Friday morning.
Others haven’t been able to load the update, which would potentially require IT workers at some companies to reboot and tinker with every single affected computer.
“Many of the customers are rebooting the system, and it’s coming up, and it’ll be operational because we fixed it on our end,” Kurtz said. “Some of the systems that aren’t recovering, we’re working with them. So it could be some time for some systems that just automatically won’t recover.”
