Elevated latency and errors in AWS us-east-1

Final update

This incident has been resolved. The AWS us-east-1 region is stable and all affected namespaces have been failed back.

1. Resolved · Oct 10, 00:51 UTC
   This incident has been resolved. The AWS us-east-1 region is stable and all affected namespaces have been failed back.
2. Monitoring · Oct 9, 23:11 UTC
   Latency and error rates in AWS us-east-1 have returned to normal levels and affected namespaces have recovered. We are continuing to monitor the region. We will provide another update once failback is complete.
3. Investigating · Oct 9, 22:50 UTC
   Mitigation remains in progress. We are continuing to fail over affected namespaces with Multi-Region Replication enabled to their replica regions. Some customers may continue to experience elevated latency while this work is underway. We will provide another update within 30 minutes.
4. Investigating · Oct 9, 22:38 UTC
   We have identified the cause of elevated latency affecting a subset of namespaces in AWS us-east-1. Namespaces with Multi-Region Replication enabled are being failed over to their replica region to restore service. We will provide another update within 30 minutes.
5. Investigating · Oct 9, 22:33 UTC
   We are investigating elevated latency and increased deadline exceeded errors affecting some namespaces in the AWS us-east-1 region. Workflows may progress more slowly than usual. Our team is actively working to identify the cause. We will provide another update within 30 minutes.