Authentication issues related to API requests

Lesson: Authentication failures can prevent successful API requests across the platform.

What happened

Between 15:05 UTC and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Customers experienced intermittent "logged out" behavior as erroneous 401 responses triggered repeated authentication flows in app integrations. Affected requests also experienced approximately 800ms of additional latency. A memcached proxy service rollout to our internal API infrastructure caused our authentication service to pick up an incorrect memcached host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached to use the correct host. To prevent similar issues in the future, we plan to migrate our authentication system to the new memcached infrastructure to improve resilience and strengthen overall reliability posture.

1. Resolved · Jun 10, 16:39 UTC
   Between 15:05 UTC and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Customers experienced intermittent "logged out" behavior as erroneous 401 responses triggered repeated authentication flows in app integrations. Affected requests also experienced approximately 800ms of additional latency. A memcached proxy service rollout to our internal API infrastructure caused our authentication service to pick up an incorrect memcached host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached to use the correct host. To prevent similar issues in the future, we plan to migrate our authentication system to the new memcached infrastructure to improve resilience and strengthen overall reliability posture.