Elevated Latency and Errors for Grafana Cloud Logs in AWS US East (VA)
Sep 30, 12:40 UTCSep 30, 15:39 UTC
Duration
2h 59m
Impact
Minor
Root cause
Not disclosed
Grafana Labs, 90 days
71 incidents
Affected
AWS US East - prod-us-east-3Logs - AWS US East - prod-us-east-3
Final update
Between 12:09 and 15:39 UTC on September 30, Grafana Cloud Logs in AWS US East (VA) experienced intermittent elevated latency and a small number of failed requests. Impact has returned to normal levels and all services are operating as expected. Thank you for your patience.
Timeline
- Resolved · Sep 30, 15:39 UTC
Between 12:09 and 15:39 UTC on September 30, Grafana Cloud Logs in AWS US East (VA) experienced intermittent elevated latency and a small number of failed requests. Impact has returned to normal levels and all services are operating as expected. Thank you for your patience.
- Identified · Sep 30, 14:31 UTC
An additional brief recurrence affected a small number of requests after our mitigation was deployed, though impact was smaller than earlier today. We continue to monitor and investigate a related, longer-running pattern. Next update by 15:40 UTC.
- Identified · Sep 30, 13:43 UTC
We have identified a contributing factor to the elevated latency and are rolling out a mitigation to increase write capacity in AWS US East (VA). We continue to investigate a related, longer-running pattern. Next update by 14:40 UTC.
- Investigating · Sep 30, 13:40 UTC
We are continuing to investigate this issue.
- Investigating · Sep 30, 12:40 UTC
We are investigating elevated latency and a small number of failed requests for Grafana Cloud Logs in AWS US East (VA). Some customers writing logs to this region may see delayed ingestion or occasional write errors. We are actively investigating and will share more as we learn it. Next update by 13:38 UTC.
More from Grafana Labs
Full historyPlugin installs and updates from the plugin catalog may faillasted 2h 41mIncreased Read Latency In prod-us-central-0 Regionlasted 1h 19mData Source Permission Errorslasted 0mDegraded performance for Grafana Cloud Metrics in prod-us-central-7lasted 1h 36mElevated query errors for Grafana Cloud Metrics in prod-us-east-3lasted 4h 14mCloud Logs writes - Elevated latency and error rates in prod-eu-central-0lasted 17m