Quay.io Push/Pull Degraded

Final update

On April 21, a routine production deployment included a dependency upgrade that exposed a latent incompatibility, causing Quay.io application pods to crash on startup. Image pulls and pushes were fully unavailable for approximately 12 minutes before the team failed over to a secondary region, restoring pulls. Pushes remained unavailable for approximately 70 additional minutes while the deployment was reverted and traffic was incrementally shifted back to the primary region. We are enabling staging parity for all runtime features, improving automated rollback mechanisms, and isolating observability subsystems from the critical request path to prevent recurrence.

1. Postmortem · May 15, 17:18 UTC
   On April 21, a routine production deployment included a dependency upgrade that exposed a latent incompatibility, causing Quay.io application pods to crash on startup. Image pulls and pushes were fully unavailable for approximately 12 minutes before the team failed over to a secondary region, restoring pulls. Pushes remained unavailable for approximately 70 additional minutes while the deployment was reverted and traffic was incrementally shifted back to the primary region. We are enabling staging parity for all runtime features, improving automated rollback mechanisms, and isolating observability subsystems from the critical request path to prevent recurrence.
2. Resolved · Apr 22, 00:09 UTC
   Pull and Push APIs are fully up again. Service is working as expected.
3. Identified · Apr 21, 22:55 UTC
   Pull API is working again. We are actively working on bringing push API back online.
4. Investigating · Apr 21, 22:50 UTC
   We are experiencing degraded performance on quay.io push/pull API. We are actively investigating.