Intermittent Impact from AWS Outage

Final update

We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.

1. Resolved · May 8, 02:05 UTC
   We’ve mitigated the impact from AWS’s Availability Zone issue, and service has remained stable. We have not observed further node failures or customer impact, so we’re marking this incident as resolved. We’ll continue to monitor AWS’s recovery separately.
2. Monitoring · May 8, 01:51 UTC
   We’ve moved affected infrastructure out of the impacted AWS Availability Zone, and traffic should now be restored. We’re continuing to monitor for any residual impact while AWS’s recovery progresses.
3. Identified · May 8, 00:54 UTC
   AWS’s Availability Zone-level issue is continuing to cause impact to our infrastructure. Some customers may experience extended downtime or elevated error rates while affected workloads recover on healthy infrastructure. We’re actively working to restore service for impacted customers and will continue to provide updates as recovery progresses.
4. Identified · May 8, 00:29 UTC
   AWS has identified an Availability Zone-level issue affecting our infrastructure. Some customer workloads may continue to experience intermittent errors while affected jobs are reallocated to healthy hardware and databases fail over. We’re continuing to monitor recovery and will provide updates as AWS mitigation progresses.
5. Investigating · May 8, 00:16 UTC
   We’re experiencing intermittent downtime from an upstream infrastructure provider. Some customer workloads may see brief error spikes while affected jobs are reallocated to healthy hardware and databases fail over. We’re monitoring recovery and will provide updates as mitigation progresses.