CML/CDW/CDF/CDE private deployments are failing in EU Control Plane region.

Final update

Private cluster deployments on our EU control plane were impacted by an issue stemming from our User Management Service \(UMS\). UMS is responsible for authenticating all calls to the Control Plane. The service did not autoscale as expected to meet the increased traffic on the EU control plane and was eventually overloaded. This caused delayed responses to authentication calls which in turn caused a breakdown in our network connectivity \(CCM\) from Control Plane to private workloads. Existing workloads were not impacted however any new private deployments or operations like update/upgrade/delete were impacted by this. We have implemented corrective measures within the automation to prevent similar occurrences in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform and appreciate your understanding.

1. Postmortem · Mar 11, 13:32 UTC
   Private cluster deployments on our EU control plane were impacted by an issue stemming from our User Management Service \(UMS\). UMS is responsible for authenticating all calls to the Control Plane. The service did not autoscale as expected to meet the increased traffic on the EU control plane and was eventually overloaded. This caused delayed responses to authentication calls which in turn caused a breakdown in our network connectivity \(CCM\) from Control Plane to private workloads. Existing workloads were not impacted however any new private deployments or operations like update/upgrade/delete were impacted by this. We have implemented corrective measures within the automation to prevent similar occurrences in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform and appreciate your understanding.
2. Resolved · Feb 28, 15:09 UTC
   Current Status: Our teams have confirmed that the solution implemented has addressed the issue. If you continue to experience issues please raise a support case with us. A root cause analysis (RCA) will be published within seven business days.
3. Investigating · Feb 27, 18:13 UTC
   We have made additional progress in isolating the issue today. Please expect further updates by tomorrow
4. Investigating · Feb 26, 23:00 UTC
   Current Status : We are continuing to work on isolating the issue. Please expect further updates by tomorrow.
5. Investigating · Feb 26, 17:50 UTC
   Current Status: Existing workloads should be working as expected. Only new deployments of private CML/CDW/CDF/CDE experiences are failing in EU Control Plane region. This is currently being investigated, we expect to have further updates towards the End of Business today.