Intermittent latency in Eastern US
Sep 29, 16:26 UTCOct 6, 12:15 UTC
Duration
6d 19h
Impact
Minor
Root cause
Not disclosed
Supabase, 90 days
58 incidents
Affected
API Gateway
Final update
A fix for this issue has been implemented and our data shows previous impacted users have recovered.
Timeline
- Resolved · Oct 6, 12:15 UTC
A fix for this issue has been implemented and our data shows previous impacted users have recovered.
- Monitoring · Oct 5, 18:45 UTC
A fix for this issue has been implemented and our data shows recovery for users on projects that have previously been affected. We expect this situation to be resolved, but will keep monitoring for a while until fully resolving this incident.
- Identified · Oct 2, 21:06 UTC
While we have observed significant improvement in rates of high latency, we note that some users are still experiencing issues during peak Eastern US business hours. We continue to work with our networking partners to isolate and resolve any lingering sources of high latency. We recognize that this is a longstanding issue and appreciate the patience of affected customers. We will continue to share additional updates here as soon as they are available.
- Identified · Oct 1, 20:23 UTC
Our networking partners have implemented a mitigation to the latency issue at the problematic PoP in the Eastern US, and we have since observed significant improvement in rates of high latency across the region. We are continuing to work with those partners to isolate any additional root causes of latency and improve the reliability of our networking infrastructure across the region.
- Identified · Sep 30, 21:26 UTC
We are seeing signs of improvement and are continuing to work alongside our networking partners towards a resolution. We will provide another update as more information becomes available.
- Identified · Sep 30, 14:54 UTC
We're continuing to coordinate with our networking partners to deliver a permanent resolution. We will provide updates as soon as they're available.
- Identified · Sep 30, 02:14 UTC
We are working with our networking partners on a permanent fix. While we do not have an estimated resolution time, we will share updates here as they become available.
- Identified · Sep 29, 21:35 UTC
A mitigation for this issue was applied earlier today, but we do not have sufficient evidence to conclude that the mitigation resulted in improvement to this eastern US latency issue. We have tied the issue to a particular PoP, and we are continuing to work with our networking partners to bring this issue to a final resolution. We will provide another update within 4 hours, if not before. We appreciate your patience.
- Identified · Sep 29, 17:55 UTC
We are continuing our work to fix this latency issue, which has been observed to impact clients connecting to Supabase Projects from the eastern United States. These clients may include servers and serverless functions located in that geographical region, regardless of the region in which the Project is hosted.
- Identified · Sep 29, 16:26 UTC
We have identified an issue with increased latency for clients in the eastern US during spikes caused by bursty traffic. This is most noticeable during EDT working hours around the :00 and :30 hour marks. We are actively working on a fix. We will provide updates as the fix is implemented.
More from Supabase
Full historyGateway availability disruptionlasted 0mEdge Function Issues in ap-northeast-1lasted 3h 7m‘Project not specified’ errors reported for users in ap-northeast-2lasted 3h 12mProjects in sa-east-1 experiencing delayed event ingestionlasted 8h 44mElevated Error rates for All Supabase project APIslasted 1h 18mDatabase Connection Errorslasted 3h 4m