Incident with several GitHub Services

Lesson: Teams should maintain visibility across services to quickly detect and mitigate widespread platform incidents.

What happened

On September 13, 2026, between 08:43 and 10:44 UTC, GitHub experienced degraded availability across approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup. At peak, 8.8% of requests to create GitHub App installation access tokens failed. Token issuance for Actions workflows was also affected, impacting approximately 4% of workflows during the incident time frame. Creating issues through the web interface failed for about 96% of attempts, and signup failures were above 90%. The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal , how far the database replicas were lagging , and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit. When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on

1. Resolved · Sep 13, 10:44 UTC
   On September 13, 2026, between 08:43 and 10:44 UTC, GitHub experienced degraded availability across approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup. At peak, 8.8% of requests to create GitHub App installation access tokens failed. Token issuance for Actions workflows was also affected, impacting approximately 4% of workflows during the incident time frame. Creating issues through the web interface failed for about 96% of attempts, and signup failures were above 90%. The cause was an internal data-cleanup job that began writing to a shared database cluster at 07:33 UTC. That cluster stores permission data read on nearly every authenticated request. The safeguard that was pacing the background job watched only one health signal , how far the database replicas were lagging , and that signal stayed low the whole time. It did not account for the load building on the primary itself, so the job kept writing while the primary quietly ran toward its limit. When the primary ran out of available connections, requests that needed it could not complete. First, there was no quick timeout on these database calls, so request handlers waited on the stalled database instead of failing fast, and the shared request-handling capacity degraded into site-wide errors. Second, a retry loop around token creation kept re-sending the writes that were already failing, which held the database saturate
2. Investigating · Sep 13, 10:28 UTC
   Pull Requests is experiencing degraded performance. We are continuing to investigate.
3. Investigating · Sep 13, 10:26 UTC
   We have reduced load on this cluster with internal load-shedding and are seeing signs of recovery but continue to monitor
4. Investigating · Sep 13, 09:36 UTC
   We're seeing increased database replication delays on collab which is causing increased error rates in authorization endpoints and follow-on increased error rates across the system - we are investigating
5. Investigating · Sep 13, 09:25 UTC
   Actions is experiencing degraded performance. We are continuing to investigate.
6. Investigating · Sep 13, 09:16 UTC
   We are investigating reports of degraded availability for API Requests, Issues, Pages and Pull Requests