CML/CDE/CDF cluster upgrade

Final update

On February 20, 2024, customers reported issues while performing a CML workspace upgrade. Post investigation, we identified a bug in a newly promoted build; that impacted CML/CDE/CDF upgrades; and as a result we temporarily disabled CML upgrades.  It is important to note that only the upgrade functionality was impacted, and this did not have any impact on existing workload operations.  A hotfix was deployed to production to address the bug post which CML upgrades were re-enabled.  We have implemented additional corrective measures within our automated test suite to proactively detect similar issues in the future. We sincerely apologize for any inconvenience this incident may have caused to our customers.

1. Postmortem · Apr 1, 18:55 UTC
   On February 20, 2024, customers reported issues while performing a CML workspace upgrade. Post investigation, we identified a bug in a newly promoted build; that impacted CML/CDE/CDF upgrades; and as a result we temporarily disabled CML upgrades.  It is important to note that only the upgrade functionality was impacted, and this did not have any impact on existing workload operations.  A hotfix was deployed to production to address the bug post which CML upgrades were re-enabled.  We have implemented additional corrective measures within our automated test suite to proactively detect similar issues in the future. We sincerely apologize for any inconvenience this incident may have caused to our customers.
2. Resolved · Mar 1, 19:57 UTC
   Current Status: Our teams have successfully deployed a fix for the issue and confirmed that the issue has been resolved. If you are still experiencing issues or have any questions please raise a support case with us. A root cause analysis (RCA) will be published within seven business days. Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted.
3. Monitoring · Feb 29, 20:51 UTC
   Current Status: Our teams have identified the source of the issue and have implemented a solution which is under monitoring. Please expect further updates tomorrow Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted
4. Identified · Feb 29, 15:07 UTC
   Current Status: Our teams have identified the source of the issue. We are working on developing and implementing a solution to restore the service’s. We will have another update towards the end of business today. Customer Experience: During this time cluster upgrades for CML, CDE and CDF are impacted
5. Identified · Feb 29, 08:44 UTC
   The fix is currently being validated.
6. Investigating · Feb 20, 15:32 UTC
   We are currently working on a fix for cluster upgrade failures that have been observed in the Control Plane regions. Please hold upgrading clusters for CML, CDE and CDF in any of the Control Plane regions till further update is made available.