Incidentsby GlyphDelays in event ingestion
Jun 8, 20:46 UTCJun 13, 13:52 UTC
Duration
4d 17h
Impact
Minor
Root cause
Not disclosed
LaunchDarkly, 90 days
9 incidents
Affected
Data ExportMetricsRollout results processingFlag usage metricsExperiment results processingRollout results analysisExperiment results analysisAI insightsWarehouse native experimentationFeature management (non-core functionality) - Data ExportFeature management (non-core functionality) - Flag usage metricsExperimentation (core functionality) - Experiment results processingExperimentation (core functionality) - Experiment results analysisExperimentation (core functionality) - Warehouse native experimentation
Final update
This incident has been resolved.
Timeline
- Resolved · Jun 13, 13:52 UTC
This incident has been resolved.
- Monitoring · Jun 12, 22:15 UTC
We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
- Monitoring · Jun 11, 15:37 UTC
We have made major progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
- Monitoring · Jun 11, 05:14 UTC
We have made partial progress. We identified that a small percentage of event data was lost during two windows: June 9, 11:41 - 11:59 UTC and June 10, 23:55 UTC - June 11, 00:21 UTC. Some event-based features may show incomplete data while backfills and recovery continue. We are continuing to monitor and will provide another update when recovery is complete.
- Monitoring · Jun 11, 04:12 UTC
We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
- Monitoring · Jun 11, 04:07 UTC
We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
- Monitoring · Jun 11, 04:07 UTC
We are continuing to monitor for any further issues.
- Monitoring · Jun 11, 01:31 UTC
We have made partial progress. We have identified a brief period of event data loss, and some event-based features may show incomplete data while we continue recovery. We are continuing to monitor.
- Monitoring · Jun 10, 18:54 UTC
We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.
- Monitoring · Jun 9, 12:08 UTC
We have made partial progress. Some features may show data gaps while we complete recovery. We are continuing to monitor.
- Monitoring · Jun 9, 02:23 UTC
We are continuing to monitor for any further issues.
- Monitoring · Jun 9, 02:03 UTC
We are monitoring the status of our services.
- Monitoring · Jun 8, 21:24 UTC
A fix has been implemented and we are monitoring the results.
- Identified · Jun 8, 20:46 UTC
We are currently experiencing delays in event ingestion impacting multiple product areas starting at 12:40pm PT. We have identified the issue and are working on a fix. No data has been lost.
More from LaunchDarkly
Full historyEmail notifications are not being deliveredlasted 26mdelayed event processinglasted 0mElevated 5xx Errors (15:11–15:32 UTC)lasted 0mDelays in Guarded Rollouts, Progressive Rollouts, Release Pipelines, and Experimentationlasted 1h 37mElevated error rates in the LaunchDarkly applicationlasted 18h 55mFlag delivery errors in Streaming APIlasted 15m