Is Cloudflare still acting up? Outage detailed in blog post
We've been hearing from you about Cloudflare's outage yesterday, and we checked to see what's going on
☁️ Cloudflare has shared what went wrong yesterday
❌ A wide-spread outage across its services caused platforms like X, Spotify, and Substack to experience issues
💾 Cloudflare says this was due to a database change - not a cyber attack
🧑🔧 If you’re still experiencing issues, let us know
Cloudflare, a DDoS protection provider that powers a large number of popular websites, had a huge outage yesterday that caused sites like X, Spotify, and Substack to experience a number of issues. We reported on the news, and it caught the attention of many of our followers on Substack, all while we awaited an answer from Cloudflare as to why the outage occurred in the first place. The company has since published a blog post about it - and it’s not what we thought.
What was originally reported as a spike in traffic on one of Cloudflare’s services has turned out to be a database change on the company’s end. It wasn’t a cyber attack, as some have speculated, but one of Cloudflare’s own robots that manages database output. Here’s how the company explains it:
The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems’ permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.
After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file. Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.
On Cloudflare’s status page, it says that the company is performing a slew of scheduled maintenance across a lot of its servers. It also resolved an issue with Magic WAN routing. In other words, it seems like the issue - at least on Cloudflare’s end - is resolved.
I’ve dug through a few comment sections across the web to see if anyone is still experiencing problems, and it seems like the general consensus is that the problem is over with. However, if you’re still having problems with certain sites, let us know by leaving a comment. Many of you chimed in yesterday to confirm that the outage was affecting you, so feel free to drop a line if you still have issues or if everything’s peachy.
Max Buondonno is an editor at The Shortcut. He’s been reporting on the latest consumer technology since 2015, with his work featured on CNN Underscored, ZDNET, How-To Geek, XDA, TheStreet, and more. Follow him on X @LegendaryScoop and Instagram @LegendaryScoop.