WAF regular expression exhausts CPU worldwide for 27 minutes
Jul 2, 13:42 UTCJul 2, 14:09 UTC
Duration
27m
Impact
Critical
Root cause
Deployment
Cloudflare, 90 days
56 incidents
Affected
CDNWAFGlobal
Lesson: Rules and regexes are code; stage them, and use an engine with guaranteed linear time for untrusted input.
What happened
A new WAF rule for XSS detection contained a regular expression with catastrophic backtracking. Deployed globally through an automatic process, it pushed CPU to 100% on every HTTP server and the network dropped traffic for 27 minutes.
More from Cloudflare
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2500:03 UTC | Issues with Durable Objects | minor | Ongoing |
| Sep 2420:40 UTC | Intermittent 500 errors for backend services on Replicate | minor | 1h 54m |
| Sep 2417:33 UTC | Delays Starting Cloudflare Workers Builds | minor | 3h 19m |
| Sep 2417:26 UTC | Browser isolation session data sync issues | minor | 5h 11m |
| Sep 2415:20 UTC | Cloudflare Access updates delayed | minor | 3h 5m |
| Sep 2410:22 UTC | Customers using BYOIP can have issues updating their BGP prefixes, including advertising or withdrawing prefixes. | minor | 3m |
Also caused by deployment or rollout
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Aug 2014:43 UTC |  GitHub | Intermittent failures creating agent tasksGitHub | critical | 9h 54m |
| Aug 2000:31 UTC |  Box | [Medium] Issue with Microsoft Office Integration and Box EditBox | major | 1h 13m |
| Aug 622:22 UTC |  Slack | Trouble Using Search Bar For Some AdminsSlack | minor | 2h 44m |
| Jul 2220:20 UTC |  Amplitude | Some users may have experienced errors when accessing Amplitude.Amplitude | none | 0m |
| Jul 2110:23 UTC |  dbt Labs | Job runs failing at the git clone stepdbt Labs | critical | 2h 29m |
| Jul 1907:33 UTC | dbt Labs | Some EMEA customers experiencing Git clone failures (403 errors) on thier accountsdbt Labs | major | 6h 22m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.