Delays and intermittent failures running Smarter Testing
Final update
## Summary _\(All times UTC\)_ From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected. The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding. In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19. **Customers do not need to run a full analysis**. Impact data updates were delayed, and we deleted some queued updates during the incident, so some
Timeline
- Postmortem · Nov 20, 23:39 UTC
## Summary _\(All times UTC\)_ From 13:20 to 19:19 on November 19, customers using Smarter Testing saw degraded test selection. Between 15:50 and 18:00, some test jobs using Smarter Testing failed intermittently. In that window, 13% of jobs using Smarter Testing encountered an error, and the rest ran without errors. Most failures happened during two outages, from 16:20 to 16:38 and from 17:37 to 17:55. Customers not using Smarter Testing were not affected. The incident was caused by unusually large impact data that our system did not handle well. It retried the updates repeatedly, which kept CPU on the database behind Smarter Testing at 96-98% for over two hours. The database ran on an instance type that relies on burst capacity. When that capacity ran out at 16:20, the database stopped responding. In response, our engineering team stopped processing the oversized data and deleted the queued updates for it. At 18:50, we moved the database to a larger instance type that does not rely on burst capacity. We resolved the incident at 19:19. **Customers do not need to run a full analysis**. Impact data updates were delayed, and we deleted some queued updates during the incident, so some projects may select more tests than necessary until their next analysis. Running regular analysis reconverges on the correct impact data. The original status page can be found here: [Delays and intermittent failures running Smarter Testing](https://status.circleci.com/incidents/s501b363k4hs). ## Bac
- Resolved · Nov 19, 19:22 UTC
Smarter Testing functionality is restored. We appreciate your patience as we worked through this.
- Monitoring · Nov 19, 19:08 UTC
The maintenance was successful. Smarter Testing functionality has been restored. We are monitoring to ensure performance remains stable.
- Identified · Nov 19, 18:40 UTC
At 18:45 UTC, we will perform emergency maintenance, which is expected to take a few minutes. During this window, test jobs for customers using Smarter Testing may be delayed or fail. Thank you for your patience while our engineers work to resolve this. We will provide another update by 19:05 UTC.
- Identified · Nov 19, 18:30 UTC
Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again. Dynamic Test Splitting is still not running optimally. Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes. Thank you for your patience while our engineers work to resolve this. We will provide another update by 18:50 UTC
- Identified · Nov 19, 18:13 UTC
Since 17:55 UTC, test jobs for customers using Smarter Testing have been running again. Dynamic Test Splitting is still not running optimally. Dynamic Parallelism is also not running optimally, so jobs that use it should be expected to fall back to using all nodes. Thank you for your patience while our engineers work to resolve this. We will provide another update by 18:30 UTC.
- Identified · Nov 19, 17:45 UTC
Since 17:38 UTC, some test jobs for customers using Smarter Testing have been failing. Thank you for your patience while our engineers work to resolve this.
- Identified · Nov 19, 17:30 UTC
Since 17:09 UTC, we have been seeing delays in updating test impact analysis data. Test selection may be less precise, which may cause more tests than necessary to run. Thank you for your patience while our engineers work to resolve this.
- Identified · Nov 19, 17:04 UTC
Starting at 16:38 UTC, test jobs for customers using Smarter Testing should be back running. Dynamic Test Splitting may still not be working optimally for some customers.
- Investigating · Nov 19, 16:34 UTC
Some Smarter Testing customers may not be able to run tests. We are also seeing delays updating the impact analysis data map, which may cause more tests to be selected by Smarter Testing.