Activity log outage in production US
Nov 25, 22:51 UTCNov 27, 00:14 UTC
Duration
25h 23m
Impact
Minor
Root cause
Database
Honeycomb, 90 days
4 incidents
Affected
ui.honeycomb.io - US1 Activity Log
Final update
As of 23:00 UTC, all systems are back to synchronized.
Timeline
- Resolved · Nov 27, 00:14 UTC
As of 23:00 UTC, all systems are back to synchronized.
- Monitoring · Nov 26, 17:24 UTC
We are continuing to work on catching up late data from yesterday night's replication issues and expect to complete this process in approximately 6 hours.
- Monitoring · Nov 26, 05:07 UTC
Due to issues with replication streams, the activity log will have a gap in its data. We estimate that a little over 6 hours of content will be missing, from 20:27 UTC until 2:43 UTC. We do not expect missing data to be backfilled. No other Honeycomb data or feature has been impacted. We are currently working on catching up on late data after this window.
- Identified · Nov 26, 05:02 UTC
We are continuing to work on a fix for this issue.
- Identified · Nov 25, 22:51 UTC
We have identified the cause and are working to resolve the issue.