quay.io intermittent API failures

Final update

This incident was caused by a database change which was improperly rolled out. Quay engineers were deploying an update which dropped a no longer utilized table, however, this was deployed all at once rather than in phases. That meant that our existing deployment began crashing once the table was dropped, causing API failures since the table was being queried with every request to the API. The SRE team had to manually scale down the old deployment to allow the newer one to progress which restored API functionality. Going forward we are reviewing our process for database migrations and implementing additional checks to ensure that these migrations happen in two steps: application update first and database migrations second.

1. Postmortem · Feb 26, 18:02 UTC
   This incident was caused by a database change which was improperly rolled out. Quay engineers were deploying an update which dropped a no longer utilized table, however, this was deployed all at once rather than in phases. That meant that our existing deployment began crashing once the table was dropped, causing API failures since the table was being queried with every request to the API. The SRE team had to manually scale down the old deployment to allow the newer one to progress which restored API functionality. Going forward we are reviewing our process for database migrations and implementing additional checks to ensure that these migrations happen in two steps: application update first and database migrations second.
2. Resolved · Feb 18, 21:40 UTC
   Incident has been resolved and service is restored
3. Monitoring · Feb 18, 21:16 UTC
   A fix has been implemented and we are monitoring the results.
4. Identified · Feb 18, 20:51 UTC
   We are observing an increase in 500s on pulls and pushes, the team is current investigating.