Delayed Metrics

Final update

This incident has been resolved.

1. Resolved · Jun 30, 22:19 UTC
   This incident has been resolved.
2. Identified · Jun 30, 19:58 UTC
   Almost all metrics have caught up aside from a small handful in `sin` and `syd`. We're continuing to monitor and expect these to complete in the next few hours.
3. Identified · Jun 30, 16:50 UTC
   Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.
4. Identified · Jun 30, 05:17 UTC
   Backlogged metrics are still being processed and ingestion delays persist for some customers
5. Identified · Jun 30, 00:58 UTC
   Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.
6. Identified · Jun 29, 20:02 UTC
   Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.
7. Identified · Jun 29, 15:56 UTC
   Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.
8. Identified · Jun 29, 08:43 UTC
   We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.
9. Identified · Jun 29, 05:34 UTC
   We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.
10. Identified · Jun 29, 02:58 UTC
    We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.
11. Identified · Jun 29, 01:46 UTC
    We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.
12. Investigating · Jun 29, 01:03 UTC
    We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.