Querying and Ingest issues in EU

Final update

This incident started on December 5th, and is one of the longest in Honeycomb history, having been actively worked on and closed only on December 17th. Due to its impact and duration, we wanted to offer a partial and preliminary report to explain, at a high level, what happened. On December 5th, at 20:23 UTC, our Kafka cluster suffered a critical loss of redundancy. Our Kafka cluster contains multiple topics, including all telemetry events submitted by Honeycomb users, the rematerialization of state changes into the activity log, and multiple metadata topics used by Kafka to manage its own workloads. This led multiple partitions leaderless and by 20:35 UTC, we were getting alerts that roughly a quarter of our usual event topic partitions were unable to accept writes. For most Honeycomb customers, this does not result in ingest outages, because traffic gets redirected to other partitions environments can be assigned to. However, we noted some teams for whom all their assigned partitions fit within the impacted set, and for whom all ingest was down \(0.23% of all datasets were impacted\). By 21:30 UTC, we had identified impacted teams, were working on a traffic reassignment script, a

1. Postmortem · Dec 18, 19:09 UTC
   This incident started on December 5th, and is one of the longest in Honeycomb history, having been actively worked on and closed only on December 17th. Due to its impact and duration, we wanted to offer a partial and preliminary report to explain, at a high level, what happened. On December 5th, at 20:23 UTC, our Kafka cluster suffered a critical loss of redundancy. Our Kafka cluster contains multiple topics, including all telemetry events submitted by Honeycomb users, the rematerialization of state changes into the activity log, and multiple metadata topics used by Kafka to manage its own workloads. This led multiple partitions leaderless and by 20:35 UTC, we were getting alerts that roughly a quarter of our usual event topic partitions were unable to accept writes. For most Honeycomb customers, this does not result in ingest outages, because traffic gets redirected to other partitions environments can be assigned to. However, we noted some teams for whom all their assigned partitions fit within the impacted set, and for whom all ingest was down \(0.23% of all datasets were impacted\). By 21:30 UTC, we had identified impacted teams, were working on a traffic reassignment script, and thought other features \(such as SLO evaluation\) were unimpacted. However, at 22:08 UTC, we noticed, through the noise of all the alerts that had been going on, that our ingestion fleet had been in overload protection since 20:38 UTC, impacted everyone more broadly by returning errors on `api.eu
2. Resolved · Dec 17, 15:47 UTC
   As of 23:23UTC on December 16, the Activity Log has fully caught up and has remained up to date. All systems are now operational.
3. Monitoring · Dec 16, 20:17 UTC
   We successfully created and migrated over to a new kafka cluster in the EU. We have resumed consuming events for our SLO and Service Maps features. No further data loss is expected for service maps. SLOs are caught up and are fully synchronized. Activity logs are still delayed and expect to be caught up in about 5 hours. We consider this incident to be mitigated, but will continue to monitor until Activity logs are caught up.
4. Identified · Dec 16, 01:52 UTC
   As of today, our old EU cluster is still functional, yet in a bad state. We do not consider this outage over. However, after running a full evacuation in a pre-production cluster, we are now ready to run a production version of it. This will only be in the impacted EU cluster. We are planning to do it on 6pm UTC December 16th. We expect no ingestion downtime, although we will expect delays on processing events and alerts, will suffer minor data loss in our Service Maps feature, and lose 11 days worth of Activity log events starting the night of December 5th. We will provide updates throughout the migration.
5. Identified · Dec 12, 23:20 UTC
   We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is working through the next steps required to restore Activity Log processing. Next updates to come during Monday US business hours
6. Identified · Dec 12, 00:09 UTC
   We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is actively working through the next steps required to restore Activity Log processing.
7. Identified · Dec 11, 18:02 UTC
   We continue to work toward fully restoring the Activity Log. We’ve made meaningful progress on the underlying issues and continue to work through our remediation plan. Our engineering team is actively working through the next steps required to restore Activity Log processing.
8. Identified · Dec 11, 01:08 UTC
   We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is pausing for the evening and will resume working through the remaining steps required to resume Activity Log processing during US business hours tomorrow. We will provide another update tomorrow morning.
9. Identified · Dec 10, 17:34 UTC
   We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is actively working through the remaining steps required to resume Activity Log processing.
10. Identified · Dec 9, 22:40 UTC
    We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we have a clear path to recovery, and remediation work is underway. Our next steps involve coordinated infrastructure changes that will allow us to resume Activity Log processing. We’ll provide another update during US business hours Wednesday.
11. Identified · Dec 9, 02:05 UTC
    We are continuing to work on a fix for this issue.
12. Identified · Dec 9, 02:01 UTC
    We fixed the issue that was causing subsets of temporarily missing data in queries against certain partitions. We continue to work on restoring full functionality to Activity Log. We'll provide another update during US business hours Tuesday.
13. Identified · Dec 9, 01:02 UTC
    We continue to work on restoring full functionality to Activity Log and repair the partition causing certain customers to have subsets of unqueryable data. We will next provide an update during US business hours Tuesday.
14. Identified · Dec 8, 21:52 UTC
    The Activity Log is operational but is experiencing ingestion delays, which means recent activity may not be reflected immediately. We are working to restore full functionality. Currently, a small subset of data for certain customers may not be queryable due to a partition needing repair. We are working to return the data to queryability. After investigation of the impact starting on Friday to SLO and Trigger functionality, we're providing the update below with the details. Overview SLOs and Triggers were affected by Kafka and ingest outages beginning Friday, December 5 at 8:40 PM UTC. Trigger evaluations are now behaving normally except for some baseline triggers. SLOs with evaluation periods that include missing ingest or are on the affected partition will reflect that currently unqueryable data in their ongoing calculations. SLOs - No burn alerts were fired between December 5, 2025 at 8:40 PM UTC and December 6, 2025 at 7:15 PM UTC due to a processing delay. - Now that ingest is restored, burn alerts should behave normally again minus a subset that are also impacted by the small amount of currently unqueryable data. Triggers - When ingest was down on December 6, no events were received, so all EU Trigger evaluations were impacted. - Baseline Triggers with lookback windows that query data in the affected partition returning will also be affected by the small amount of currently unqueryable data. - Non-baseline triggers (which can only look back 24 hours) should behave norma
15. Identified · Dec 7, 20:23 UTC
    Our Kafka cluster is still operational and we are continuing to work towards restoring full reliability. We will resume work during business hours to restore full Activity Log functionality. We will continue to update as work progresses.
16. Identified · Dec 7, 02:18 UTC
    The Kafka cluster is still operational, but is not back to full capacity and resiliency. Work will continue tomorrow to ensure sufficient capacity, as well as during business hours to resume full Activity Log functionality. We will provide more updates tomorrow and as-needed if there are any changes to availability.