IXP Communications Mining elevated error rate in US region

Final update

## Customer Impact On August 12, 2026, between approximately 15:10 and 18:15 UTC, Communications Mining (IXP) users in the United States region experienced intermittent request failures. The incident was confined to one of the region's deployment units, where users on it saw failures in bursts of five to ten minutes, with up to 5–7% of their requests failing with 5xx errors at peak. Between bursts the service operated normally, retried requests generally succeeded, and no data was lost. ## Root Cause A storm of requests to a rarely used API feature that computes machine-learning predictions on demand coincided with repeated retraining of the requested model. Each retraining invalidated cached predictions, turning each request into a multi-minute computation. The API placed no time limit on how long a request could wait for this computation, so these long-running requests progressively occupied all request-processing capacity, causing unrelated requests to fail. Automatic scaling reached its maximum capacity quickly and could not compensate. ## Detection Automated monitoring detected the failures at 15:15 UTC, about five minutes after impact began, and paged the on-call engineer. Th

1. Postmortem · Aug 18, 21:25 UTC
   ## Customer Impact On August 12, 2026, between approximately 15:10 and 18:15 UTC, Communications Mining (IXP) users in the United States region experienced intermittent request failures. The incident was confined to one of the region's deployment units, where users on it saw failures in bursts of five to ten minutes, with up to 5–7% of their requests failing with 5xx errors at peak. Between bursts the service operated normally, retried requests generally succeeded, and no data was lost. ## Root Cause A storm of requests to a rarely used API feature that computes machine-learning predictions on demand coincided with repeated retraining of the requested model. Each retraining invalidated cached predictions, turning each request into a multi-minute computation. The API placed no time limit on how long a request could wait for this computation, so these long-running requests progressively occupied all request-processing capacity, causing unrelated requests to fail. Automatic scaling reached its maximum capacity quickly and could not compensate. ## Detection Automated monitoring detected the failures at 15:15 UTC, about five minutes after impact began, and paged the on-call engineer. The incident was initially attributed only to the client generating the request storm, but client reports and further investigation showed a wider set of users was affected during the failure bursts. ## Response The on-call engineer traced the failures to the unbounded wait in the on-demand prediction
2. Resolved · Aug 12, 15:00 UTC
   A storm of requests on a a rarely used model feature in IXP Communications Mining caused a retry loop to lock down synchronous requests in the wider IXP API. This caused requests to fail with 500's as workers were busy with long-running requests. Automatic scaling up quickly reached it's maximum capacity and resolution was done only by applying a code fix that introduced strict timeouts to the contributing API request. The request storm started around 15:10 UTC and detected a 15:15 UTC by automatic alarms. Resolution was confirmed around 18:15 UTC. The incident was initially wrongly attributed to only the client making the storm of requests, but was later discovered to have impacted a wider range of users. Total impact limited to a handful of users in the US.