Slowness in US region

Final update

On October 20, we experienced roughly three hours of degraded service in the US region. * From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed. * From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time. * From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC \(1% error rate\) and for approximately a minute at 14:15 UTC \(1.9% error rate\). Telemetry rejected during these periods was not ingested unless the sender successfully retried it. Customers may see gaps in their data for those periods, including in triggers and SLOs that cover it. Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed. Ou

1. Postmortem · Oct 21, 23:26 UTC
   On October 20, we experienced roughly three hours of degraded service in the US region. * From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. During this time, customers may have experienced slow or failed queries and a sluggish UI, and some trigger alerts may have been late or missed. * From 13:26 to approximately 16:10 UTC, SLO processing was delayed, so SLO data may have appeared out of date and burn alerts may have been delayed or missed during that time. * From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors, with lesser impact between 13:48 and 13:55 UTC \(1% error rate\) and for approximately a minute at 14:15 UTC \(1.9% error rate\). Telemetry rejected during these periods was not ingested unless the sender successfully retried it. Customers may see gaps in their data for those periods, including in triggers and SLOs that cover it. Our on-call engineers were first paged at 13:43 UTC by an alert about triggers not running on time, and we spun up our internal incident response as further alerts followed. Our investigation found that the primary database’s CPU was saturated by a large number of long-running read queries. We performed a database failover to relieve the load. This briefly eased the pressure, but the same queries soon reappeared on the new primary, so it did not resolve the problem. We th
2. Resolved · Oct 20, 16:26 UTC
   SLO processing has fully recovered. All services are now functional.
3. Investigating · Oct 20, 15:18 UTC
   We've identified and isolated the impacted service and are seeing recovery in query execution. SLO processing remains degraded.
4. Investigating · Oct 20, 14:04 UTC
   We are currently investigating the degradation of various services in our US region.