Elevated Latency and Errors for Grafana Cloud Logs in AWS US East (VA)

Final update

Between 12:09 and 15:39 UTC on September 30, Grafana Cloud Logs in AWS US East (VA) experienced intermittent elevated latency and a small number of failed requests. Impact has returned to normal levels and all services are operating as expected. Thank you for your patience.

1. Resolved · Sep 30, 15:39 UTC
   Between 12:09 and 15:39 UTC on September 30, Grafana Cloud Logs in AWS US East (VA) experienced intermittent elevated latency and a small number of failed requests. Impact has returned to normal levels and all services are operating as expected. Thank you for your patience.
2. Identified · Sep 30, 14:31 UTC
   An additional brief recurrence affected a small number of requests after our mitigation was deployed, though impact was smaller than earlier today. We continue to monitor and investigate a related, longer-running pattern. Next update by 15:40 UTC.
3. Identified · Sep 30, 13:43 UTC
   We have identified a contributing factor to the elevated latency and are rolling out a mitigation to increase write capacity in AWS US East (VA). We continue to investigate a related, longer-running pattern. Next update by 14:40 UTC.
4. Investigating · Sep 30, 13:40 UTC
   We are continuing to investigate this issue.
5. Investigating · Sep 30, 12:40 UTC
   We are investigating elevated latency and a small number of failed requests for Grafana Cloud Logs in AWS US East (VA). Some customers writing logs to this region may see delayed ingestion or occasional write errors. We are actively investigating and will share more as we learn it. Next update by 13:38 UTC.