elevated error rates on convex for pro instances

Final update

At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.

1. Postmortem · Mar 13, 03:36 UTC
   At approximately 6am Pacific we had a high rate of server errors for approximately 30 minutes for a subset of customer backends. This was caused by a 20x increase in traffic that saturated one of our databases and led to resource exhaustion on one of our function execution clusters, which is why the impact spilled beyond just that specific database. We’ve scaled up our infrastructure further and are deploying some changes to load shedding and retries to minimize the impact of such an event in future. While we aim to have no cascading impact on other customers from a load spike we’re also deploying dedicated isolated hardware for customers on our upcoming Business plan.
2. Resolved · Mar 12, 13:55 UTC
   This incident has been resolved.
3. Monitoring · Mar 12, 13:40 UTC
   A fix has been implemented and we are monitoring the results.
4. Investigating · Mar 12, 13:17 UTC
   We are currently investigating an issue leading to higher error rates