Elevated error rates for sandbox operations

Final update

Resolved. Google Cloud fully resolved the underlying Compute Engine incident at 12:37 PDT and all sandbox operations in the US region are back to normal.

1. Resolved · Aug 20, 20:27 UTC
   Resolved. Google Cloud fully resolved the underlying Compute Engine incident at 12:37 PDT and all sandbox operations in the US region are back to normal.
2. Monitoring · Aug 20, 18:41 UTC
   We no longer see any impact on our customers. Sandbox create, pause, and resume in the US region are operating normally. This incident remains in monitoring while we wait for Google Cloud to fully resolve the underlying Compute Engine issue. We will post a final update once that is confirmed.
3. Monitoring · Aug 20, 18:02 UTC
   We are no longer seeing customer impact from the earlier Google Cloud Compute Engine incident. Sandbox create, pause, and resume in the US region have returned to normal. We are keeping this incident open and monitoring to confirm stability before marking it resolved.
4. Monitoring · Aug 20, 17:51 UTC
   We’ve identified the issue as related to an ongoing incident with one of our upstream providers. We’re monitoring their recovery and its impact on our service. Upstream incident: https://status.cloud.google.com/incidents/utF3FMFdQfwBzJcGG6vf
5. Monitoring · Aug 20, 17:23 UTC
   Service is recovering. Sandbox operations are stabilizing and error rates are returning to normal. We continue to monitor closely.
6. Investigating · Aug 20, 17:15 UTC
   We are investigating increased error rates for sandbox operations. Start times are slower than usual, and a small number of sandbox resumes (under 1%) are affected. Adding retries to your sandbox operations can help mitigate the impact while we work on a fix
7. Investigating · Aug 20, 16:25 UTC
   We are experiencing elevated error rates for sandbox operations. The team is investigating the root cause, and we will keep you updated as we learn more. We apologize for the inconvenience.