Jobs using IP ranges failing with infrastructure errors
Nov 20, 21:22 UTCNov 20, 23:48 UTC
Duration
2h 26m
Impact
Minor
Root cause
Not disclosed
CircleCI, 90 days
23 incidents
Affected
Docker Jobs
Final update
We have seen recovery for IP range jobs for gen2 docker jobs. We appreciate your patience while we worked through the incident.
Timeline
- Resolved · Nov 20, 23:48 UTC
We have seen recovery for IP range jobs for gen2 docker jobs. We appreciate your patience while we worked through the incident.
- Monitoring · Nov 20, 23:11 UTC
We are continuing to monitor the rollout for recovery.
- Monitoring · Nov 20, 22:27 UTC
A fix has been fully deployed, we are now monitoring for recovery.
- Identified · Nov 20, 22:19 UTC
The fix for IP range docker jobs are rolling out across the fleet. We will monitor for recovery in infra fails.
- Identified · Nov 20, 21:58 UTC
We are continuing to rollout a fix for IP range docker jobs. Customers may continue to retry jobs that use IP ranges that experience infra fails. We will provide an update within 20 minutes on the fix.
- Identified · Nov 20, 21:22 UTC
Customers running Docker executor jobs with the IP ranges feature enabled are experiencing a higher rate of infrastructure failures. A fix is being rolled out. Customers who see an infrastructure failure on one of these jobs can rerun the failed job. We will provide another update as we have more information to share.
More from CircleCI
Full historyDelays and intermittent failures running Smarter Testinglasted 2h 48mIncreased wait time for Machine and Remote Docker jobsFrom 13:55 to 15:07 UTC on November 3, customers running Linux machine jobs and Remote Docker saw increased wait times for their jobs to start, and the delays were longest from 14:16 to 14:41.lasted 54mPostmortemDelays for gen2 dedicated IP range jobslasted 1h 3mIncreased Docker gen2 wait timeslasted 1h 24mIncreased wait times for macOS jobs on m4pro.largelasted 2h 16mDelays starting macOS jobs that use Xcode 26.6.0lasted 1h 13m