Incidentsby IbisCustomers in us-east-2 may experience some visibility related errors
Jul 15, 16:55 UTCJul 15, 21:15 UTC
Duration
4h 19m
Impact
Major
Root cause
Not disclosed
Temporal, 90 days
27 incidents
Affected
us-east-2Amazon Web Services (AWS) - us-east-2
Final update
All MRN-enabled namespaces failed over by Temporal have been failed back. This incident has been fully mitigated and is now resolved.
Timeline
- Resolved · Jul 15, 21:15 UTC
All MRN-enabled namespaces failed over by Temporal have been failed back. This incident has been fully mitigated and is now resolved.
- Monitoring · Jul 15, 20:29 UTC
We are seeing full recovery on our end, we are continuing to monitor.
- Monitoring · Jul 15, 19:58 UTC
Mitigation is in effect, we are seeing some recovery. Please submit a support ticket if you are seeing service degradation.
- Identified · Jul 15, 19:12 UTC
Mitigation is still in progress. We are scaling up service to help with mitigation efforts.
- Identified · Jul 15, 18:33 UTC
All MRN-enabled namespaces have been failed over as of 17:53 UTC. Mitigation continues to progress.
- Identified · Jul 15, 17:52 UTC
We are working on a fix, if your namespace are MRN enabled please fail over to your passive regions for faster recovery.
- Investigating · Jul 15, 16:55 UTC
Customers with namespaces in us-east-2 may experience errors with visibility queries. We are currently investigating.
More from Temporal
Full historyNew namespace creation restricted across all GCP regionsstill openElevated latency and errors in GCP us-west1lasted 17mElevated latency and errors in AWS us-east-1lasted 2h 17mElevated error rates for cross-region traffic between US AWS regionslasted 0mElevated GCP storage latency in us-central1 – precautionary namespace failoverlasted 4h 25mIncreased errors on Temporal Cloud control plane, Cloud Ops API, and Temporal UIlasted 29m