Delays starting jobs on Cloud Robots in the US region

Final update

## Customer impact From 15:00 to 16:38 UTC on 2026-10-02, Cloud Robots jobs in the US region took longer than usual to start, up to 22 minutes at the peak around 16:20 UTC. Orchestrator stayed available, no job was lost, no other region was affected. A small number of customers received a temporary limit error on new submissions while their earlier jobs were waiting. ## Detection Automated alerts fired at 15:07 UTC, seven minutes after impact began. Checks in the US region failed from about 15:32 UTC and the incident response started. ## Root cause At 14:55 UTC one customer's workflow began submitting jobs at about 20 times its normal rate. The service that admits jobs keeps a per-customer concurrency count in the database and updates it under a database lock on every job start and end. At that rate the lock queue for the customer's record grew until updates timed out. The waiting updates occupied the shared pool of workers that starts jobs for all customers in the region, so every customer's jobs waited behind them. The mechanism meant to isolate one customer's load became the bottleneck. ## Response The team identified the overloaded admission path and the workload behind it. A f

1. Postmortem · Oct 14, 14:20 UTC
   ## Customer impact From 15:00 to 16:38 UTC on 2026-10-02, Cloud Robots jobs in the US region took longer than usual to start, up to 22 minutes at the peak around 16:20 UTC. Orchestrator stayed available, no job was lost, no other region was affected. A small number of customers received a temporary limit error on new submissions while their earlier jobs were waiting. ## Detection Automated alerts fired at 15:07 UTC, seven minutes after impact began. Checks in the US region failed from about 15:32 UTC and the incident response started. ## Root cause At 14:55 UTC one customer's workflow began submitting jobs at about 20 times its normal rate. The service that admits jobs keeps a per-customer concurrency count in the database and updates it under a database lock on every job start and end. At that rate the lock queue for the customer's record grew until updates timed out. The waiting updates occupied the shared pool of workers that starts jobs for all customers in the region, so every customer's jobs waited behind them. The mechanism meant to isolate one customer's load became the bottleneck. ## Response The team identified the overloaded admission path and the workload behind it. A first configuration change at 15:46 UTC targeted the wrong record and had no effect. The correct change at 16:10 UTC exempted that customer's job type from the database lock and took effect at about 16:20 UTC. Start times dropped from about 13 seconds to under one second and the backlog of about 5200
2. Resolved · Oct 2, 17:03 UTC
   This issue has been resolved. Between 15:05 and 16:39 UTC, jobs on Cloud Robots in the US region were slow to start or stayed in the Pending state. Jobs have been starting normally since 16:39 UTC, and jobs that were delayed during this time have now been processed. We apologize for the disruption.
3. Monitoring · Oct 2, 16:51 UTC
   After further work on the issue, jobs on Cloud Robots in the US region are now starting normally. Jobs that were delayed earlier are being processed. We're continuing to monitor the service and will post another update once the issue is resolved.
4. Monitoring · Oct 2, 16:34 UTC
   We've applied a fix for the issue that delayed jobs on Cloud Robots in the US region, and jobs are starting normally again. Jobs that were delayed are now being processed. We're monitoring to make sure the service stays stable and will post another update within 60 minutes.
5. Investigating · Oct 2, 16:23 UTC
   Since about 15:05 UTC, some customers in the US region may see jobs that run on Cloud Robots take longer than usual to start, or stay in the Pending state. Jobs on your own robots are not affected, and neither are other regions. We've found the likely cause and are working to restore normal job processing. Jobs that are already queued are not lost and will be processed once the issue is resolved. We'll post another update within 60 minutes.