Degraded connectivity affecting us-east-2

Final update

We have resolved the incident.

1. Resolved · Apr 30, 02:12 UTC
   We have resolved the incident.
2. Monitoring · Apr 28, 22:32 UTC
   A subset of services in us-east-2 experienced intermittent connection timeouts earlier today caused by exhaustion of network resources on one of our regional load balancers. We have applied mitigations to restore connectivity and the service is currently operating normally. We are continuing to work with on rolling out additional capacity and safeguards to prevent recurrence. Customers who continue to see connection issues should reach out to support.
3. Investigating · Apr 28, 20:34 UTC
   We are seeing a recurrence of the connection issues affecting some customers in the us-east-2 region, resulting in connection timeouts. Our team is actively investigating and working to mitigate the issue as a priority. We will provide further updates as more information becomes available.
4. Resolved · Apr 28, 19:41 UTC
   The connection issues affecting some customers in the us-east-2 region have been resolved, and services are operating normally.
5. Monitoring · Apr 28, 19:18 UTC
   The connection issues affecting some customers in the us-east-2 region have been mitigated, and services are currently operating normally. We are continuing to monitor the situation closely to ensure stability and prevent recurrence. We will provide further updates if there are any changes.
6. Investigating · Apr 28, 18:44 UTC
   We are currently investigating connection issues affecting some customers, resulting in connection timeouts when connecting to ClickHouse. The issue has been isolated to infrastructure in the us-east-2 region. We have received multiple high-severity reports and are treating this with priority. Our engineering team is actively working to mitigate the issue and restore normal connectivity. We will provide further updates as more information becomes available.