Service Control crash loop returns 503s across Google Cloud
Jun 12, 17:49 UTCJun 12, 20:49 UTC
Duration
3h
Impact
Critical
Root cause
Config change
Google Cloud, 90 days
4 incidents
Affected
Service ControlCompute EngineBigQueryCloud StorageWorkspaceGlobal
Lesson: New code paths belong behind feature flags, and globally replicated policy data needs staged propagation like any binary.
What happened
A policy change inserted into the regional Spanner tables that Service Control reads hit an unguarded code path with a null pointer, crashing Service Control binaries worldwide. External API requests returned 503 errors; us-central1 recovered last.
More from Google Cloud
Full historyAlso caused by configuration change
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 2423:36 UTC |  Render | Key Value service restartsRender | minor | 0m |
| Sep 2414:08 UTC |  HubSpot | The Salesforce integration is unavailable for some customersHubSpot | none | 7h 18m |
| Sep 2218:20 UTC |  Twilio | Phone Number APIs and Console Were Returning Incorrect 404 ResponsesTwilio | none | 0m |
| Sep 1622:30 UTC |  Cloudflare | Hyperdrive Elevated Origin Connection Failure RatesCloudflare | none | 0m |
| Sep 1211:04 UTC |  Slack | Some customers experiencing blurry image previews and download issuesSlack | minor | 8h 56m |
| Sep 1107:18 UTC |  Snowflake | INC20000213Snowflake | critical | 6h 1m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.