Warehouse Native accounts may be experiencing export delays for data from Aug 6
Aug 7, 23:10 UTCAug 8, 08:25 UTC
Duration
9h 14m
Impact
Minor
Root cause
Not disclosed
Statsig, 90 days
3 incidents
Affected
Event Forwarding Pipeline (Warehouse Native)
Final update
This incident has been resolved
Timeline
- Resolved · Aug 8, 08:25 UTC
This incident has been resolved
- Monitoring · Aug 8, 04:00 UTC
Thank you for your patience. The main query-intensive jobs are through now, but it also took quite a bit longer than before. Catch up for subsequent jobs may therefore take up to 3am PT to complete. Our sincere apologies for the delay and we will continue to share updates as we monitor the job completion.
- Monitoring · Aug 8, 00:57 UTC
We are still making progress on re-running the exports. We should start to see some exports completing but the expected completion time for the full system to rerun has extended to 9pm PT. We will continue to keep you updated and aim to have the next update at 9pm PT! To clarify: 1. This only impacts customers who use Statsig warehouse native. If you are on Statsig cloud, you can ignore this status page. 2. For WHN customers, once we finish rerunning the final jobs, Statsig will automatically take care of refreshing the data for all experiments that are configured to incrementally refresh metrics.
- Monitoring · Aug 7, 23:10 UTC
This issue only impacts customers on Warehouse Native on the echidna pipeline. Around 11 am PT, we have identified that WHN accounts will have event and exposure export delays from August 6. No data is lost as the job rebuilds the full day from source so there will be no gaps once the post-revert run lands. Everything through August 5 is intact. Around 1pm PT, we have implemented a fix that should have mitigated this issue. We expect exports to complete around 7pm PT.
More from Statsig
Full historyAug 26 Incident Follow-up: Feature Gate Exposure Data Delayed for Some Warehouse-Native (WHN) Customerslasted 2d 17hDelayed Experiment & Feature Gate Results for Warehouse-Native (WHN) Customerslasted 0mFeature Flag Evaluation Errors (Statsig)lasted 1h 9mData ingestion delayedlasted 7h 38mWarehouse Native Pulse results delayed for customers with IP allowlists configuredlasted 0mStatsig Console temporarily downlasted 24m