elevated latency/errors for a subset of customers
Oct 30, 23:43 UTCOct 31, 07:52 UTC
Duration
8h 8m
Impact
Major
Root cause
Not disclosed
Convex, 90 days
4 incidents
Affected
Free & StarterLive Traffic - Database ServicesLive Traffic - Function Runtime
Final update
This incident has been resolved.
Timeline
- Resolved · Oct 31, 07:52 UTC
This incident has been resolved.
- Monitoring · Oct 31, 06:00 UTC
We are continuing to monitor for any further issues.
- Monitoring · Oct 31, 05:59 UTC
There will be sporadic availability issues for a small subset of non-pro customers. We apologize for this impact. There was an issue with a downstream provider which resulted in a database cluster running with an inefficient table format. We're in the process of rewriting this data format but have had to shed load to provide headroom while reoptimizing the database.
- Monitoring · Oct 31, 04:24 UTC
All deployments should be restored. We are monitoring the database to ensure it remains stable.
- Identified · Oct 31, 03:08 UTC
We're continuing to work on a fix for the issue. Expect another update in ~90 minutes.
- Identified · Oct 31, 00:25 UTC
we've brought back a subset of free deployments. Continuing to remediate.
- Identified · Oct 30, 23:44 UTC
We've restored affected pro deployments. Working on bringing back affected free deployments.
- Identified · Oct 30, 23:43 UTC
We're investigating an issue with a database cluster leading to degraded performance and higher error rates for affected deployments.