Delayed workflow execution for some tenants in the us-west-2 region

Final update

This incident has been resolved. Between approximately 22:35 and 00:21 UTC, our upstream cloud infrastructure provider experienced elevated failures launching new compute instances across the us-west-2 region, caused by a subsystem supporting storage volumes required during instance launch. Our provider completed remediation and has confirmed full recovery, and all of their services are operating normally. During this window, workflows for some tenants hosted in us-west-2 were delayed because new compute capacity could not start. All affected tenants recovered automatically once launches were restored, and delayed workflows have resumed and completed processing. The Atlan UI, search and APIs were not affected at any point. If any workflow run failed or did not complete during this window, please re-run it. No other action is required.

1. Resolved · Nov 7, 01:15 UTC
   This incident has been resolved. Between approximately 22:35 and 00:21 UTC, our upstream cloud infrastructure provider experienced elevated failures launching new compute instances across the us-west-2 region, caused by a subsystem supporting storage volumes required during instance launch. Our provider completed remediation and has confirmed full recovery, and all of their services are operating normally. During this window, workflows for some tenants hosted in us-west-2 were delayed because new compute capacity could not start. All affected tenants recovered automatically once launches were restored, and delayed workflows have resumed and completed processing. The Atlan UI, search and APIs were not affected at any point. If any workflow run failed or did not complete during this window, please re-run it. No other action is required.
2. Monitoring · Nov 7, 00:41 UTC
   All affected tenants in the us-west-2 region have recovered. New compute capacity is starting successfully, and delayed workflows have resumed and are processing normally. The Atlan UI, search and APIs were not affected at any point. Our upstream cloud infrastructure provider has identified the underlying cause as a subsystem supporting storage volumes required when launching new compute instances in the us-west-2 region. They are seeing early signs of recovery and continue to work on fully restoring launches; existing running infrastructure was not affected. Their next update is expected by 01:30 UTC. Until the provider confirms full resolution, we are pausing any non-essential infrastructure changes in us-west-2 and will continue to monitor closely. Further updates will follow.
3. Monitoring · Nov 7, 00:34 UTC
   All affected tenants in the us-west-2 region have recovered. New compute capacity is starting successfully, and delayed workflows have resumed and are processing normally. The Atlan UI, search and APIs were not affected at any point. Our upstream cloud infrastructure provider has not yet declared the underlying issue fully resolved, so we will continue to monitor closely until they confirm. Further updates will follow.
4. Identified · Nov 7, 00:05 UTC
   Our upstream cloud infrastructure provider has confirmed an ongoing operational issue in the us-west-2 region: elevated failures when launching new compute instances, caused by a problem with their block storage service. This is also affecting several other services that depend on it in that region. As a result, new compute capacity cannot start, and workflows for some tenants hosted in us-west-2 are delayed. The Atlan UI, search and APIs are not affected. Our provider is actively working on mitigation. Workflows will resume automatically once capacity is restored, and no action is required from customers. We are monitoring closely, and further updates will follow.
5. Identified · Nov 6, 23:49 UTC
   We have identified an issue with our upstream infrastructure provider in the us-west-2 region that is preventing new compute capacity from starting. As a result, some tenants hosted in us-west-2 may see workflows (including automations and background jobs) delayed or not starting. The Atlan UI, search and APIs are not affected. Our provider is working on mitigation, and we are monitoring closely. Workflows will resume automatically once capacity is restored; no action is needed from customers. We will post another update within 60 minutes or sooner as we learn more.