elevated latency for a small subset of convex deployments
Mar 25, 16:27 UTCMar 25, 19:46 UTC
Duration
3h 20m
Impact
Informational
Root cause
Not disclosed
Convex, 90 days
6 incidents
Affected
Live Traffic - Function Runtime
Final update
We had elevated latencies on a single database that required us to temporarily shed load while resizing and reparenting the db. This lead to a period of elevated latencies for all instances on that database followed by downtime for a smaller subset of instances while we performed the maintenance operation. This issue should not occur under normal circumstances and was due an underlying database problem that required manual intervention to resolve. We apologize for the impact to those impacted!
Timeline
- Postmortem · Mar 25, 19:47 UTC
We had elevated latencies on a single database that required us to temporarily shed load while resizing and reparenting the db. This lead to a period of elevated latencies for all instances on that database followed by downtime for a smaller subset of instances while we performed the maintenance operation. This issue should not occur under normal circumstances and was due an underlying database problem that required manual intervention to resolve. We apologize for the impact to those impacted!
- Resolved · Mar 25, 19:46 UTC
This incident has been resolved.
- Identified · Mar 25, 17:54 UTC
A small subset of free instances are experiencing availability issues while we restore service.
- Investigating · Mar 25, 16:27 UTC
We're investigating elevated latency for functions on a subset of Convex deployments
More from Convex
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2821:09 UTC5 days ago | AI gateway unavailable | critical | 1h 4m |
| Sep 1823:01 UTC2 weeks ago | Dashboard logins degraded | none | 1h 7m |
| Aug 1623:33 UTC6 weeks ago | Account verification/team invite emails not working | major | 1h 53m |
| Aug 1203:41 UTC7 weeks ago | Some HTTP Actions returning 404 | minor | 20m |
| Aug 501:42 UTC8 weeks ago | Action failures for business deployments that use createFunctionHandle | major | 15m |
| Jul 1505:35 UTC2 months ago | New deployments unable to push Node Actions to Convex | minor | 57m |