Customers in us-central1 and asia-southeast2 may experience a delay in workflow progress

Final update

The environment has been validated and all residual issues have been cleared. This incident is now resolved.

1. Resolved · Sep 11, 07:21 UTC
   The environment has been validated and all residual issues have been cleared. This incident is now resolved.
2. Monitoring · Sep 11, 00:17 UTC
   A configuration change affecting namespaces in the us-central1 and asia-southeast2 regions resulted in some workflow requests not being processed as expected. The change was rolled back and workflow processing resumed. We are continuing to validate the environment and clear any residual issues, and will provide further updates as that work completes.
3. Monitoring · Sep 10, 23:29 UTC
   We are continuing to monitor for any further issues.
4. Monitoring · Sep 10, 22:59 UTC
   While a small number of customer environments may still see some issues, mitigation has been in place and we continue to actively monitor.
5. Monitoring · Sep 10, 22:29 UTC
   We are continuing to monitor for any further issues.
6. Monitoring · Sep 10, 22:27 UTC
   The issue has been mitigated (as of 21:55 UTC) and we are actively monitoring the previously impacted environments.
7. Identified · Sep 10, 20:51 UTC
   Some customers with namespaces in the us-central1 and asia-southeast2 regions may experience errors and blocked progress on their workflows. We are currently rolling out a fix.