Incidentsby GlyphDegradation with Webhooks, Pull Requests and Actions
Jun 25, 17:50 UTCJun 25, 18:27 UTC
Duration
37m
Impact
Major
Root cause
Capacity
GitHub, 90 days
69 incidents
Affected
Not listed by the vendor.
Lesson: Service degradation can simultaneously impact multiple integrated platform features like webhooks, pull requests, and actions.
What happened
On June 25, 2026, between 17:33 UTC and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, Actions workflows, and Webhooks, with delays peaking at 7m. The issue was caused by underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances. The issue was mitigated by replacing the problem node at 17:49, after which all services saw recovery by 18:07.
Timeline
- Resolved · Jun 25, 18:27 UTC
On June 25, 2026, between 17:33 UTC and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, Actions workflows, and Webhooks, with delays peaking at 7m. The issue was caused by underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances. The issue was mitigated by replacing the problem node at 17:49, after which all services saw recovery by 18:07.