One bad regex took down Cloudflare

Last Tuesday, sites behind Cloudflare returned errors for about 27 minutes. Cloudflare explained it quickly: a new rule in its web application firewall included a regular expression that, on certain inputs, used huge amounts of CPU. It was deployed to every server at once, and CPUs everywhere hit 100%.

Two lessons any team can use:

• Regexes can be slow in surprising ways. Patterns with nested repetition can take exponential time. Test regexes against long, hostile inputs, or use engines that guarantee linear time.
• Roll out gradually, even for configuration and rules, not only code. A change that reaches 1% of servers first fails small.

And one more: have a fast way to switch a feature off globally. Cloudflare’s recovery started when they disabled the rule set.

Their public write-up is a good model for how to explain an outage.