Increased timeouts due to surge in anonymous users

Final update

This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.

1. Resolved · Nov 9, 09:09 UTC
   This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
2. Monitoring · Nov 9, 09:09 UTC
   We are resolving this incident as error rates have remained low and we have prepared additional protections.
3. Monitoring · Nov 9, 09:05 UTC
   The degradation has been mitigated. We are monitoring to ensure stability.
4. Investigating · Nov 9, 08:47 UTC
   Error rates have declined again, and we’re preparing targeted traffic controls while monitoring for a recurrence. Logged-in users and authenticated requests have not been affected.
5. Investigating · Nov 9, 08:16 UTC
   We are continuing to observe high rates of 504 errors for anonymous requests. We are continuing to work on mitigation. Logged-in users and authenticated requests are unaffected.
6. Investigating · Nov 9, 07:43 UTC
   We have observed a repeated increase in 504 errors for anonymous and unauthenticated traffic, including users who are not logged in and release downloads. We are continuing to investigate and apply mitigations.
7. Monitoring · Nov 9, 07:03 UTC
   The problem with anonymous traffic receiving 504 status codes has been mitigated. We are continuing to observe for an abundance of caution.
8. Monitoring · Nov 9, 06:51 UTC
   The degradation has been mitigated. We are monitoring to ensure stability.
9. Investigating · Nov 9, 06:30 UTC
   We are debugging a spike in anonymous traffic. Some users who are not logged in may see errors on the site. CI jobs and release downloads may fail with a 504 Gateway Timeout status if they make anonymous calls.
10. Investigating · Nov 9, 05:59 UTC
    We are investigating reports of impacted performance for some GitHub services.