Some customers may be experiencing stuck workflows or activities

Final update

This incident has been resolved.

1. Resolved · Jun 16, 21:08 UTC
   This incident has been resolved.
2. Monitoring · Jun 16, 21:07 UTC
   Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
3. Monitoring · Jun 16, 21:00 UTC
   Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
4. Monitoring · Jun 16, 15:56 UTC
   The last few namespaces are still refreshing. We are starting to patch the code, and are monitoring the situation.
5. Monitoring · Jun 11, 20:25 UTC
   Refresh operations across both open and closed workflows are complete in most regions; we continue to make steady progress across the remaining few namespaces in the us-east-1, us-west-2,  us-east4, us-west1, us-central1, asia-south1, europe-west3. No additional impact has been observed since mitigation was implemented, and we continue to closely monitor the environment. The next update will be provided once the refresh process has completed.
6. Monitoring · Jun 8, 19:02 UTC
   Refresh across open and closed workflows continues to progress smoothly. Once this is done, the incident will be resolved.
7. Monitoring · Jun 5, 22:02 UTC
   Refresh across open and closed workflows continues to progress smoothly. We're monitoring the situation and will post updates once complete.
8. Monitoring · Jun 4, 22:35 UTC
   As refresh of closed workflows progresses, some users may see unblocked stuck parent workflows from closed child workflows; this is happening because the missed task for closed child workflows is refreshed and delivered to parent. It could also happen on other stuck workflows that are waiting for task updates which were missed due to this incident.
9. Monitoring · Jun 4, 17:40 UTC
   We are continuing to monitor the ongoing refresh process for all open workflows. We are also considering options for refreshing closed workflows. We will continue to share more updates as this progresses.
10. Monitoring · Jun 4, 07:08 UTC
    Workflows will continue to be refreshed overnight, and we are monitoring the situation.
11. Identified · Jun 4, 06:45 UTC
    Workflows will continue to be refreshed overnight, and we are monitoring the situation.
12. Identified · Jun 4, 06:14 UTC
    Refreshes are still in progress and we are monitoring the situation.
13. Identified · Jun 4, 04:27 UTC
    Rollback is complete; refresh of open workflows is underway to unblock impacted workflows.
14. Identified · Jun 4, 01:40 UTC
    The rollback is making progress across impacted cells, and we are continuing to monitor the situation.
15. Identified · Jun 3, 23:36 UTC
    Rollback is still in progress and we are continuing to monitor the situation.
16. Identified · Jun 3, 21:35 UTC
    A bug in the recent update in History pagination caused a rare edge case where an extra task could be added before the response was returned. This incorrectly advanced the read level, causing some tasks to be skipped. We’ve identified the issue, and are currently rolling back the update to stop this from happening again. We will be refreshing all executions in cells where the update was deployed and will monitor these closely. This will progress any stuck workflows, but will not impact any of the closed ones.