Who is down right now

20 down now · 248 this week

GitHub · Developer toolsDec 3, 2025, 22:30 UTC 11 months ago

Webhooks delivery degradation

InformationalCapacityUpdated 46d ago
Dec 3, 22:30 UTCDec 3, 22:30 UTC
Duration
0m
Impact
Informational
Root cause
Capacity
GitHub, 90 days
69 incidents
Affected
Not listed by the vendor.
Status page

Final update

On December 3, 2025, between 22:21 UTC and 23:44 UTC, the Webhooks service experienced a degradation that delayed writes of webhook delivery records to our database. During this period, many webhook deliveries were not visible in the webhook delivery UI or API for more than an hour after they were sent. As a result, customers were temporarily unable to request redeliveries for those delayed records. The underlying cause was throttling of database writes due to high replication lag. We mitigated the incident by temporarily disabling delivery history for a small number of very high‑volume webhook owners to reduce write pressure and stabilize the service. We are contacting the affected customers directly with more details. We are improving our webhook delivery storage architecture so it can scale with current and future webhook traffic, reducing the likelihood and impact of similar issues.

Timeline

  1. Resolved · Dec 3, 22:30 UTC
    On December 3, 2025, between 22:21 UTC and 23:44 UTC, the Webhooks service experienced a degradation that delayed writes of webhook delivery records to our database. During this period, many webhook deliveries were not visible in the webhook delivery UI or API for more than an hour after they were sent. As a result, customers were temporarily unable to request redeliveries for those delayed records. The underlying cause was throttling of database writes due to high replication lag. We mitigated the incident by temporarily disabling delivery history for a small number of very high‑volume webhook owners to reduce write pressure and stabilize the service. We are contacting the affected customers directly with more details. We are improving our webhook delivery storage architecture so it can scale with current and future webhook traffic, reducing the likelihood and impact of similar issues.

More from GitHub

Full history

Also caused by capacity and load

All