Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityFeb 20, 2024, 15:32 UTC 2 years ago

CML/CDE/CDF cluster upgrade

MinorBugUpdated 49d ago
Feb 20, 15:32 UTCMar 1, 19:57 UTC
Duration
10d 4h
Impact
Minor
Root cause
Bug
Cloudera, 90 days
1 incidents
Affected
Cloudera Data FlowCloudera Data EngineeringCloudera AICloudera Data Platform (US) - DataFlowCloudera Data Platform (US) - Data EngineeringCloudera Data Platform (US) - Machine LearningCloudera Data Platform (AP) - DataFlowCloudera Data Platform (AP) - Data EngineeringCloudera Data Platform (AP) - Machine LearningCloudera Data Platform (EU) - DataFlowCloudera Data Platform (EU) - Data EngineeringCloudera Data Platform (EU) - Machine Learning
Status page

Final update

On February 20, 2024, customers reported issues while performing a CML workspace upgrade. Post investigation, we identified a bug in a newly promoted build; that impacted CML/CDE/CDF upgrades; and as a result we temporarily disabled CML upgrades.  It is important to note that only the upgrade functionality was impacted, and this did not have any impact on existing workload operations.  A hotfix was deployed to production to address the bug post which CML upgrades were re-enabled.  We have implemented additional corrective measures within our automated test suite to proactively detect similar issues in the future. We sincerely apologize for any inconvenience this incident may have caused to our customers.

Timeline

  1. Postmortem · Apr 1, 18:55 UTC
    On February 20, 2024, customers reported issues while performing a CML workspace upgrade. Post investigation, we identified a bug in a newly promoted build; that impacted CML/CDE/CDF upgrades; and as a result we temporarily disabled CML upgrades.  It is important to note that only the upgrade functionality was impacted, and this did not have any impact on existing workload operations.  A hotfix was deployed to production to address the bug post which CML upgrades were re-enabled.  We have implemented additional corrective measures within our automated test suite to proactively detect similar issues in the future. We sincerely apologize for any inconvenience this incident may have caused to our customers.
  2. Resolved · Mar 1, 19:57 UTC
    Current Status: Our teams have successfully deployed a fix for the issue and confirmed that the issue has been resolved. If you are still experiencing issues or have any questions please raise a support case with us. A root cause analysis (RCA) will be published within seven business days. Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted.
  3. Monitoring · Feb 29, 20:51 UTC
    Current Status: Our teams have identified the source of the issue and have implemented a solution which is under monitoring. Please expect further updates tomorrow Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted
  4. Identified · Feb 29, 15:07 UTC
    Current Status: Our teams have identified the source of the issue. We are working on developing and implementing a solution to restore the service’s. We will have another update towards the end of business today. Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted
  5. Identified · Feb 29, 08:44 UTC
    The fix is currently being validated.
  6. Investigating · Feb 20, 15:32 UTC
    We are currently working on a fix for cluster upgrade failures that have been observed in the Control Plane regions. Please hold upgrading clusters for CML, CDE and CDF in any of the Control Plane regions till further update is made available.

More from Cloudera

Full history

Also caused by software bug

All