Degraded querying and delayed records sent to Activity Log
Jan 6, 22:14 UTCJan 7, 19:54 UTC
Duration
21h 40m
Impact
Minor
Root cause
Not disclosed
Honeycomb, 90 days
4 incidents
Affected
ui.honeycomb.io - US1 Queryingui.honeycomb.io - US1 Activity Logui.eu1.honeycomb.io - EU1 Queryingui.eu1.honeycomb.io - EU1 Activity Log
Final update
This incident has been resolved.
Timeline
- Resolved · Jan 7, 19:54 UTC
This incident has been resolved.
- Identified · Jan 7, 19:53 UTC
We have stabilized activity log. We have confirmed activity log entries lost from 4:54pm-7:04pm ET in the US and 9:43pm-11:55pm GMT in the EU. Event ingest was unimpacted. All systems are currently operating normally
- Identified · Jan 7, 01:17 UTC
We have restored basic activity log functionality, though some delays remain, and we will analyze further tomorrow during business hours.
- Identified · Jan 6, 23:05 UTC
We are currently expecting to lose the last three hours of data for the beta activity log feature as part of remediation of this issue.
- Identified · Jan 6, 22:37 UTC
We have resolved the issue which impacted querying but are still working to get Activity Log up to date
- Identified · Jan 6, 22:14 UTC
We are seeing a spike in querying errors and the Activity Log has delayed ingest of new activity. We have identified the issue and are working to resolve