activity log for US instance is delayed
May 8, 08:00 UTCMay 11, 16:37 UTC
Duration
3d 8h
Impact
Minor
Root cause
Database
Honeycomb, 90 days
4 incidents
Affected
ui.honeycomb.io - US1 Activity Log
Final update
This incident has been resolved.
Timeline
- Resolved · May 11, 16:37 UTC
This incident has been resolved.
- Monitoring · May 8, 20:46 UTC
Activity Log recovered at 07:00 UTC-07:00 today (about 6.5h ago). All streams are caught up.
- Monitoring · May 8, 10:40 UTC
All activity log streams are now caught up, except for the query runs table, which should have all data since the start of the incident backfilled by 1600 PDT.
- Monitoring · May 8, 10:22 UTC
We are continuing to monitor for any further issues.
- Monitoring · May 8, 10:21 UTC
We believe that the activity log should now be caught up.
- Monitoring · May 8, 10:13 UTC
Backfilling of the past 2h30m of data is now in progress and should complete shortly.
- Identified · May 8, 08:00 UTC
Activity log delay is rising, and is more than an hour behind due to a MySQL replica failover. Estimated recovery by 04:00 PDT.