Quay.io Intermittent HTTP 502 on Pull/Push

Final update

On May 9, a surge of concurrent automated image pushes caused database lock contention that degraded Quay.io for approximately 18 minutes. Both image pushes and pulls returned errors during this window. The team identified and disabled an internal feature that amplified the contention, restoring service. We are redesigning this feature to operate asynchronously and improving our database locking mechanisms to better handle high-concurrency traffic.

1. Postmortem · May 15, 17:11 UTC
   On May 9, a surge of concurrent automated image pushes caused database lock contention that degraded Quay.io for approximately 18 minutes. Both image pushes and pulls returned errors during this window. The team identified and disabled an internal feature that amplified the contention, restoring service. We are redesigning this feature to operate asynchronously and improving our database locking mechanisms to better handle high-concurrency traffic.
2. Resolved · May 9, 19:30 UTC
   For a period of 12 minutes (15:52–16:04 UTC), ~62% of manifest pulls and ~39% of manifest pushes failed with 502s. Quota insights has temporarily been disabled as part of remediation.