Intermittent slowness in Vector US-EAST-1 region
May 29, 20:18 UTCMay 30, 07:51 UTC
Duration
11h 33m
Impact
Minor
Root cause
Not disclosed
Upstash, 90 days
4 incidents
Affected
AWS - US-EAST-1Vector - AWS - US-EAST-1
Final update
The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing memory utilization on the impacted nodes, rebalancing affected workloads where needed, and increasing available headroom capacity across the region. All affected indexes should now be operating normally, and based on the mitigations applied, we do not expect this issue to recur. We apologize for any inconvenience this may have caused.
Timeline
- Resolved · May 30, 07:51 UTC
The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing memory utilization on the impacted nodes, rebalancing affected workloads where needed, and increasing available headroom capacity across the region. All affected indexes should now be operating normally, and based on the mitigations applied, we do not expect this issue to recur. We apologize for any inconvenience this may have caused.
- Monitoring · May 29, 23:44 UTC
A fix has been implemented and we are monitoring the results.
- Investigating · May 29, 20:18 UTC
Some Upstash Vector indexes in the US-EAST-1 region may be experiencing intermittent slowness. Initial investigation indicates this may be related to memory pressure on the affected servers. Our team is actively investigating the root cause and working to restore normal performance. We will provide updates as soon as more information is available.
More from Upstash
Full historyBrief connection proxy degradation in us-east-1lasted 0mIntermittent DNS Resolution Errors for Upstash Vector in US East (us-east-1)lasted 0mUpstash Console login issuelasted 0mMessage Persistence Issue , QStash us-east-1lasted 0mFly.io infrastructure disruption affecting some Upstash Redis databases on Fly.io DFW Regionlasted 1h 15mQStash us-east-1 - URL Publish Errorslasted 0m