Project Actions Failing Across Multiple Regions

Final update

Project actions have been sustained at normal performance. This incident is resolved.

1. Resolved · Jul 18, 01:41 UTC
   Project actions have been sustained at normal performance. This incident is resolved.
2. Monitoring · Jul 17, 20:21 UTC
   The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.
3. Identified · Jul 17, 19:45 UTC
   We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.
4. Investigating · Jul 17, 19:20 UTC
   We are testing mitigation options to attempt to restore project action performance.
5. Investigating · Jul 17, 18:47 UTC
   We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.
6. Investigating · Jul 17, 18:16 UTC
   Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.
7. Investigating · Jul 17, 17:49 UTC
   Certain project actions are failing. We are actively investigating to determine the regions impacted and source of the issue.