degraded-performance in MLS Observer thruough DU - US

Final update

## Customer impact Between August 31, 2026 at 3:40 pm UTC and September 1, 2026 at 12:00 am UTC, a subset of customers using Document Understanding custom-model extraction and classification in the US regions experienced intermittent request failures (HTTP 503 "Server overloaded") and slow processing. The impact was limited to document extraction requests served by custom models. It manifested in five waves of roughly twenty to forty minutes each, the heaviest between 5:10 pm and 5:50 pm UTC. During a wave the majority of requests to the affected models were rejected; between waves most requests completed normally. Document Understanding retried failed requests automatically, so most documents completed after a delay. A small share of documents submitted during the waves did not complete within the incident window and had to be resubmitted. No data was lost or exposed, and no incorrect results were returned. Extractions that appeared to return empty results, or remained in Running, were client-side timeouts inside the incident window ## Root cause Two independent defects combined under an ordinary volume of traffic. The first was in our request de-duplication mechanism. A small num

1. Postmortem · Sep 17, 21:29 UTC
   ## Customer impact Between August 31, 2026 at 3:40 pm UTC and September 1, 2026 at 12:00 am UTC, a subset of customers using Document Understanding custom-model extraction and classification in the US regions experienced intermittent request failures (HTTP 503 "Server overloaded") and slow processing. The impact was limited to document extraction requests served by custom models. It manifested in five waves of roughly twenty to forty minutes each, the heaviest between 5:10 pm and 5:50 pm UTC. During a wave the majority of requests to the affected models were rejected; between waves most requests completed normally. Document Understanding retried failed requests automatically, so most documents completed after a delay. A small share of documents submitted during the waves did not complete within the incident window and had to be resubmitted. No data was lost or exposed, and no incorrect results were returned. Extractions that appeared to return empty results, or remained in Running, were client-side timeouts inside the incident window ## Root cause Two independent defects combined under an ordinary volume of traffic. The first was in our request de-duplication mechanism. A small number of very large documents occupied shared GPU capacity and slowed all processing on the affected pool, so ordinary documents began exceeding their processing deadline. When a request's deadline expired, the de-duplication layer treated the work as abandoned even though the worker was still process
2. Resolved · Aug 31, 19:39 UTC
   The issue has been mitigated, and the service is currently operational. We will continue to closely monitor the service health and take further action if needed.
3. Monitoring · Aug 31, 18:24 UTC
   The issue has been mitigated, and the service is currently operational. We will continue to closely monitor the service health and take further action if needed.
4. Identified · Aug 31, 17:07 UTC
   We have identified the issue and implemented the necessary fix
5. Investigating · Aug 31, 16:55 UTC
   We are currently investigation the issue.