Who is down right now

26 down now · 292 this week

CircleCI · Developer toolsOct 1, 2026, 09:06 UTC 7 weeks ago

Delays starting Gen 2 Docker Jobs

MinorConfig changeUpdated 43d ago
Oct 1, 09:06 UTCOct 1, 10:45 UTC
Duration
1h 39m
Impact
Minor
Root cause
Config change
CircleCI, 90 days
23 incidents
Affected
Docker Jobs
Status page

Final update

## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances fr

Timeline

  1. Postmortem · Oct 9, 22:36 UTC
    ## **Summary** Between October 1 and October 8, 2026, CircleCI customers experienced five separate incidents in which jobs waited longer than normal to start, or failed to start. Three of the incidents were driven by limited instance availability from our cloud provider. One was caused by a configuration issue in our scheduling infrastructure, and one by memory limits on an internal service, which we raised during the incident. * **October 1, 07:57 to 10:24 UTC:** Docker Gen 2 jobs waited up to 30 minutes to start. A configuration issue prevented the Gen 2 scheduling servers from using their high-performance local disks. During a burst in traffic, the fallback disks saturated and job placement slowed. * **October 7, 13:27 to 14:06 UTC:** Machine jobs waited longer than normal because we could not acquire enough virtual machines from our cloud provider. * **October 7, 14:24 to 15:21 UTC:** Some jobs failed with an infrastructure error, and workflows were delayed. The service that gives each job its configuration at startup ran out of memory and restarted. * **October 7, 18:49 to 20:05 UTC:** Docker Gen 2 jobs waited longer than normal because we could not acquire enough instances from our cloud provider. * **October 8, 12:48 to 16:00 UTC:** Linux machine jobs and remote Docker jobs waited up to 50 minutes to start. Our cloud provider did not have enough of the instance type we request, and a defect in the logic that picks a different instance type kept us from using alternativ
  2. Resolved · Oct 1, 10:45 UTC
    This incident has been resolved.
  3. Monitoring · Oct 1, 10:24 UTC
    We're seeing queue times return to normal levels, and continue to monitor.
  4. Identified · Oct 1, 10:13 UTC
    A fix has been rolled out and we're starting to observe recovery.
  5. Identified · Oct 1, 09:47 UTC
    We have identified the root cause and we are actively working on a fix. Thank you for your patience.
  6. Investigating · Oct 1, 09:06 UTC
    Customers may experience delays in starting Docker Gen 2 Jobs. We are working to increase the throughput of the system to accomodate this.

More from CircleCI

Full history

Also caused by configuration change

All