elevated latency for a small subset of convex deployments
Mar 25, 16:27 UTCMar 25, 19:46 UTC
Duration
3h 20m
Impact
Informational
Root cause
Not disclosed
Convex, 90 days
4 incidents
Affected
Live Traffic - Function Runtime
Final update
We had elevated latencies on a single database that required us to temporarily shed load while resizing and reparenting the db. This lead to a period of elevated latencies for all instances on that database followed by downtime for a smaller subset of instances while we performed the maintenance operation. This issue should not occur under normal circumstances and was due an underlying database problem that required manual intervention to resolve. We apologize for the impact to those impacted!
Timeline
- Postmortem · Mar 25, 19:47 UTC
We had elevated latencies on a single database that required us to temporarily shed load while resizing and reparenting the db. This lead to a period of elevated latencies for all instances on that database followed by downtime for a smaller subset of instances while we performed the maintenance operation. This issue should not occur under normal circumstances and was due an underlying database problem that required manual intervention to resolve. We apologize for the impact to those impacted!
- Resolved · Mar 25, 19:46 UTC
This incident has been resolved.
- Identified · Mar 25, 17:54 UTC
A small subset of free instances are experiencing availability issues while we restore service.
- Investigating · Mar 25, 16:27 UTC
We're investigating elevated latency for functions on a subset of Convex deployments