Degraded Performance with Increased Latency for Grafana Cloud Metrics and Traces in GCP US Central

Final update

This incident has been resolved

1. Resolved · Oct 9, 00:39 UTC
   This incident has been resolved
2. Investigating · Oct 8, 21:42 UTC
   We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
3. Investigating · Oct 8, 20:41 UTC
   We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
4. Investigating · Oct 8, 19:13 UTC
   We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We have also received reports that some customers using Google Managed Prometheus in this region are experiencing issues when querying their GMP instances. We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
5. Investigating · Oct 8, 18:15 UTC
   We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We have also received reports that some customers using Google Managed Prometheus in this region are experiencing issues when querying their GMP instances. We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
6. Investigating · Oct 8, 17:30 UTC
   We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We continue to investigate this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 18:30 UTC
7. Investigating · Oct 8, 16:21 UTC
   We continue seeing elevated error rates and latency, and keep working with our cloud provider on this issue. Next update by 17:30 UTC.
8. Investigating · Oct 8, 15:19 UTC
   We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We are investigating this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 16:30 UTC.
9. Investigating · Oct 8, 14:40 UTC
   We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We are investigating this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 15:30 UTC.
10. Investigating · Oct 8, 13:52 UTC
    We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
11. Investigating · Oct 8, 13:39 UTC
    Update to scope: we have not confirmed any impact to Grafana Cloud Logs, and Logs has been removed from the affected components. We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Error rates have decreased, but intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
12. Investigating · Oct 8, 13:25 UTC
    We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics and Logs in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Error rates have decreased, but intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
13. Investigating · Oct 8, 13:06 UTC
    Beginning around 10:20 UTC, we have been seeing intermittent elevated error rates and latency affecting Grafana Cloud Metrics and Logs in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency and occasional errors affecting Grafana Cloud Traces in prod-us-central-0. Customers may see errors or delays when sending metrics, logs and traces. We are actively investigating. Next update by 13:40 UTC.