Incident with CodeQL, Webhooks, Notifications, and Slack Integration
Final update
On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied. We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.
Timeline
- Resolved · May 12, 17:43 UTC
On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied. We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.
More from GitHub
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2416:51 UTC | Disruption with billing information updates | minor | 3h 50m |
| Sep 2310:11 UTC | Incident across several services | minor | 18h 44m |
| Sep 2022:13 UTC | Incident with Pull Requests | minor | 1h 9m |
| Sep 1720:59 UTC | Elevated rate of errors for OpenAI models provided by Copilot | minor | 50m |
| Sep 1607:20 UTC | Degradation with Gemini 3.8 Flash | major | 10h 28m |
| Sep 1519:11 UTC | Disruption with some GitHub services | minor | 49m |
Also caused by deployment or rollout
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1106:22 UTC |  Cloudflare | Issues with Workers VPC hostname route resolution on 2026-09-11Cloudflare | none | 0m |
| Aug 2000:31 UTC |  Box | [Medium] Issue with Microsoft Office Integration and Box EditBox | major | 1h 13m |
| Aug 622:22 UTC |  Slack | Trouble Using Search Bar For Some AdminsSlack | minor | 2h 44m |
| Jul 2220:20 UTC |  Amplitude | Some users may have experienced errors when accessing Amplitude.Amplitude | none | 0m |
| Jul 2110:23 UTC |  dbt Labs | Job runs failing at the git clone stepdbt Labs | critical | 2h 29m |
| Jul 1907:33 UTC | dbt Labs | Some EMEA customers experiencing Git clone failures (403 errors) on thier accountsdbt Labs | major | 6h 22m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.