Who is down right now

26 down now · 292 this week

CircleCI · Developer toolsNov 19, 2026, 16:34 UTC 3 days ago

Delays and intermittent failures running Smarter Testing

MinorNot disclosedUpdated 21h ago
Nov 19, 16:34 UTCNov 19, 19:22 UTC
Duration
2h 48m
Impact
Minor
Root cause
Not disclosed
CircleCI, 90 days
23 incidents
Affected
Docker JobsMachine JobsmacOS JobsWindows Jobs
Status page

Final update

## Summary _\(All times UTC\)_ From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected. The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding. In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19. **Customers do not need to run a full analysis**. Impact data updates were delayed, and we deleted some queued updates during the incident, so some

Timeline

  1. Postmortem · Nov 20, 23:39 UTC
    ## Summary _\(All times UTC\)_ From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected. The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding. In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19. **Customers do not need to run a full analysis**. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data. The original status page can be found here: [Delays and intermittent failures running Smarter Testing](https://status.circleci.com/incidents/s501b363k4hs). ## Bac
  2. Resolved · Nov 19, 19:22 UTC
    Smarter Testing functionality is restored. We appreciate your patience as we worked through this.
  3. Monitoring · Nov 19, 19:08 UTC
    The maintenance was successful. Smarter Testing functionality has been restored. We are monitoring to ensure performance remains stable.
  4. Identified · Nov 19, 18:40 UTC
    At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail. Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
  5. Identified · Nov 19, 18:30 UTC
    Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again. Dynamic Test Splitting is still not running optimally. Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes. Thank you for your patience while our engineers work to resolve this. We will provide another update by 18:50 UTC
  6. Identified · Nov 19, 18:13 UTC
    Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again. Dynamic Test Splitting is still not running optimally. Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes. Thank you for your patience while our engineers work to resolve this. We will provide another update by 18:30 UTC.
  7. Identified · Nov 19, 17:45 UTC
    Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing. Thank you for your patience while our engineers work to resolve this.
  8. Identified · Nov 19, 17:30 UTC
    Since 17:09 UTC, we have been seeing delays in updating test impact analysis data. Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
  9. Identified · Nov 19, 17:04 UTC
    Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
  10. Investigating · Nov 19, 16:34 UTC
    Some Smarter Testing customers may not be able to run tests. We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.

More from CircleCI

Full history