Incidentsby GlyphService Control crash loop returns 503s across Google Cloud
Jun 12, 17:49 UTCJun 12, 20:49 UTC
Duration
3h
Impact
Critical
Root cause
Config change
Google Cloud, 90 days
4 incidents
Affected
Service ControlCompute EngineBigQueryCloud StorageWorkspaceGlobal
Lesson: New code paths belong behind feature flags, and globally replicated policy data needs staged propagation like any binary.
What happened
A policy change inserted into the regional Spanner tables that Service Control reads hit an unguarded code path with a null pointer, crashing Service Control binaries worldwide. External API requests returned 503 errors; us-central1 recovered last.
More from Google Cloud
Full historyMultiple services were experiencing issues in the europe-west2 region.On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 2026, 06:53 BST), a data center facility that hosts europe-west2-b, a zone within the europe-west2 cloud region, experienced an external electrical power disruption.lasted 6h 8mPostmortemGoogle Cloud Storage (GCS) customers experienced elevated latencies and error rates when performing upload or write operations.On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes.lasted 29h 46mPostmortemMultiple products in us-central1-b are experiencing network service degradation.Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected.lasted 4h 10mPostmortemWe are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.On Thursday, 20 August 2026, from 08:00 to 10:22 US/Pacific (15:00 to 17:22 UTC), multiple Google Cloud services in the us-west1 region experienced elevated latency, provisioning failures, increased error rates, and widespread service degradations for a total duration of 2 hours…lasted 3h 40mPostmortemGoogle Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes.lasted 12h 28mPostmortemGoogle Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions.On Tuesday, 14 July 2026 10:00 PT, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures.lasted 10h 40mPostmortem