Quay.io HTTP 502 on Pull/Push

Final update

On May 7, a production deployment introduced a database locking conflict that caused Quay.io to become unhealthy. Image pulls were intermittently degraded for approximately 30 minutes and image pushes were unavailable for approximately 3 hours. The team failed over to a secondary region to restore pulls, then reverted the deployment to fully restore service. We are implementing multi-phase deployment rollouts, stricter canary processes, and health probe improvements to prevent recurrence.

1. Postmortem · May 15, 17:12 UTC
   On May 7, a production deployment introduced a database locking conflict that caused Quay.io to become unhealthy. Image pulls were intermittently degraded for approximately 30 minutes and image pushes were unavailable for approximately 3 hours. The team failed over to a secondary region to restore pulls, then reverted the deployment to fully restore service. We are implementing multi-phase deployment rollouts, stricter canary processes, and health probe improvements to prevent recurrence.
2. Resolved · May 7, 21:10 UTC
   This incident has been resolved.
3. Monitoring · May 7, 20:44 UTC
   We are continuing to monitor for any further issues.
4. Monitoring · May 7, 20:34 UTC
   A fix has been implemented and we are monitoring the results.
5. Identified · May 7, 20:06 UTC
   We have identified the issue as caused by a recent deployment. Our team has reverted the deployment, and are gradually rolling back traffic to our primary region to re-enable pushes while monitoring.
6. Investigating · May 7, 18:20 UTC
   Pulls have been restored. Pushes are unavailable while we continue to investigate.
7. Investigating · May 7, 18:05 UTC
   We are currently investigating this issue.