Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityFeb 26, 2024, 17:50 UTC 2 years ago

CML/CDW/CDF/CDE private deployments are failing in EU Control Plane region.

MinorNot disclosedUpdated 49d ago
Feb 26, 17:50 UTCFeb 28, 15:09 UTC
Duration
45h 20m
Impact
Minor
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Cloudera Data FlowCloudera Data EngineeringCloudera Data WarehouseCloudera AICloudera Data Platform (EU) - DataFlowCloudera Data Platform (EU) - Data EngineeringCloudera Data Platform (EU) - Data WarehouseCloudera Data Platform (EU) - Machine Learning
Status page

Final update

Private cluster deployments on our EU control plane were impacted by an issue stemming from our User Management Service \(UMS\). UMS is responsible for authenticating all calls to the Control Plane. The service did not autoscale as expected to meet the increased traffic on the EU control plane and was eventually overloaded. This caused delayed responses to authentication calls which in turn caused a breakdown in our network connectivity \(CCM\) from Control Plane to private workloads. Existing workloads were not impacted however any new private deployments or operations like update/upgrade/delete were impacted by this. We have implemented corrective measures within the automation to prevent similar occurrences in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform and appreciate your understanding.

Timeline

  1. Postmortem · Mar 11, 13:32 UTC
    Private cluster deployments on our EU control plane were impacted by an issue stemming from our User Management Service \(UMS\). UMS is responsible for authenticating all calls to the Control Plane. The service did not autoscale as expected to meet the increased traffic on the EU control plane and was eventually overloaded. This caused delayed responses to authentication calls which in turn caused a breakdown in our network connectivity \(CCM\) from Control Plane to private workloads. Existing workloads were not impacted however any new private deployments or operations like update/upgrade/delete were impacted by this. We have implemented corrective measures within the automation to prevent similar occurrences in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform and appreciate your understanding.
  2. Resolved · Feb 28, 15:09 UTC
    Current Status: Our teams have confirmed that the solution implemented has addressed the issue. If you continue to experience issues please raise a support case with us. A root cause analysis (RCA) will be published within seven business days.
  3. Investigating · Feb 27, 18:13 UTC
    We have made additional progress in isolating the issue today. Please expect further updates by tomorrow
  4. Investigating · Feb 26, 23:00 UTC
    Current Status : We are continuing to work on isolating the issue. Please expect further updates by tomorrow.
  5. Investigating · Feb 26, 17:50 UTC
    Current Status: Existing workloads should be working as expected. Only new deployments of private CML/CDW/CDF/CDE experiences are failing in EU Control Plane region. This is currently being investigated, we expect to have further updates towards the End of Business today.

More from Cloudera

Full history