Hosted Traces elevated write latency in prod-us-central-0 region.
Jan 21, 13:24 UTCJan 21, 15:21 UTC
Duration
1h 57m
Impact
Minor
Root cause
Not disclosed
Grafana Labs, 90 days
74 incidents
Affected
Not listed by the vendor.
Final update
We consider this incident as resolved since the latency hasn't been elevated since the fix was applied. The issue was caused by a latency spike in a downstream dependency, causing an increased backpressure on the Hosted Traces ingestion path, which degraded gateway performance and resulted in an elevated write latency. After clearing the affected gateway services the degraded state went away and normal operation was restored.
Timeline
- Resolved · Jan 21, 15:21 UTC
We consider this incident as resolved since the latency hasn't been elevated since the fix was applied. The issue was caused by a latency spike in a downstream dependency, causing an increased backpressure on the Hosted Traces ingestion path, which degraded gateway performance and resulted in an elevated write latency. After clearing the affected gateway services the degraded state went away and normal operation was restored.
More from Grafana Labs
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2221:27 UTC | Kubernetes Observability Billing & Usage Incorrect | minor | 16h 39m |
| Sep 2214:57 UTC | IRM Access Issues for a Small Group of Users | major | 20h 10m |
| Sep 2119:16 UTC | Mimir write request errors | none | 0m |
| Sep 1712:34 UTC | Elevated Latency Managing Cloud Provider Integrations in GCP US Central | minor | 0m |
| Sep 1510:00 UTC | Increased Execution Time for Browser Checks in Synthetic Monitoring | minor | 1h 17m |
| Sep 1507:49 UTC | Intermittent Metric Write Errors in GCP US Central (prod-us-central-0) | minor | 0m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.