Jobs not starting
Final update
We have resolved the issues affecting job triggering, workflow starts, and API queries. Our systems have been stabilized and are operating normally. What was impacted: Job triggering, workflow starts, API queries, and pipeline page loading experienced disruptions for some customers. This affected all resource classes and executors. Resolution: We implemented mitigation measures to address high volume workflow queries impacting our internal systems and increased system capacity. All new jobs and workflows are now starting normally, pipeline pages are loading, and API queries are functioning as expected. What to expect: If you have jobs that became stuck during this incident, please rerun them. If you continue to experience issues after rerunning, please contact our support team. Some customers may still see jobs stuck in a cancelling state. Engineering is aware and addressing to mitigate risk. We will continue monitoring our systems and conducting a thorough review to identify additional preventive measures.
Timeline
- Resolved · Dec 3, 23:48 UTC
We have resolved the issues affecting job triggering, workflow starts, and API queries. Our systems have been stabilized and are operating normally. What was impacted: Job triggering, workflow starts, API queries, and pipeline page loading experienced disruptions for some customers. This affected all resource classes and executors. Resolution: We implemented mitigation measures to address high volume workflow queries impacting our internal systems and increased system capacity. All new jobs and workflows are now starting normally, pipeline pages are loading, and API queries are functioning as expected. What to expect: If you have jobs that became stuck during this incident, please rerun them. If you continue to experience issues after rerunning, please contact our support team. Some customers may still see jobs stuck in a cancelling state. Engineering is aware and addressing to mitigate risk. We will continue monitoring our systems and conducting a thorough review to identify additional preventive measures.
More from CircleCI
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2318:13 UTC | Delays in processing new signups and plan changes | none | 1h 53m |
| Sep 2116:22 UTC | Delays starting Linux Machine jobs | minor | 47m |
| Sep 1614:55 UTC | Customers may experience delays in UI updates | minor | 4h 35m |
| Sep 1514:42 UTC | Customers may experience delays in UI updates | minor | 2h 20m |
| Sep 1411:59 UTC | Increased wait times for macOS jobs | minor | 1h |
| Sep 1020:25 UTC | Issues loading pipelines for a small number of projects | minor | 1m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.