Clusters on the US Control Plane in an unreachable state
Nov 10, 21:13 UTCNov 10, 22:06 UTC
Duration
52m
Impact
Critical
Root cause
Network
Cloudera, 90 days
1 incidents
Affected
Cloudera Management ConsoleCloudera Data Platform (US) - CDP Management Console
Final update
The root cause of the issue was the expiration of certificates on our Cluster Connectivity Manager \(CCM\) instances. The certificates were renewed earlier this year however did not get fully installed across all services which caused the breakdown of communication between the services. To address this, we have enhanced our monitoring and devised a comprehensive remediation plan to detect and address similar incidents ahead of time. We regret the inconvenience this may have caused and strive to ensure every measure is taken to avoid similar issues in the future.
Timeline
- Postmortem · Nov 17, 21:08 UTC
The root cause of the issue was the expiration of certificates on our Cluster Connectivity Manager \(CCM\) instances. The certificates were renewed earlier this year however did not get fully installed across all services which caused the breakdown of communication between the services. To address this, we have enhanced our monitoring and devised a comprehensive remediation plan to detect and address similar incidents ahead of time. We regret the inconvenience this may have caused and strive to ensure every measure is taken to avoid similar issues in the future.
- Resolved · Nov 10, 22:06 UTC
This incident has been resolved.
- Investigating · Nov 10, 22:04 UTC
Current Status: Our teams have successfully deployed a fix for the issue and confirmed that the issue has been resolved. If you are still experiencing issues or have any questions please raise a support case with us. A root cause analysis (RCA) will be published within 7 Business days. Customer Experience: Customers on the US control plan may observe DataLake & DataHub cluster displaying an unreachable state Incident Start time: ~20:00 UTC November 10th, 2023 Incident End time: ~21:40 UTC November 10th, 2023
- Investigating · Nov 10, 21:13 UTC
Current Status: We are currently investigating an issue with our US Control Plane. We will have another update within 60 mins. Customer Experience: Customers on the US control plan may observe DataLake & DataHub cluster displaying an unreachable state Incident Start time: ~20:00 UTC November 10th, 2023
More from Cloudera
Full historyCloudera Management Console not accessible in US Control Planelasted 3h 29mIntermittent Performance Issues - US Control Planelasted 0mIntermittent Management Console Access Issues Across US, EU, and AP Regionslasted 0mFreeIPA connectivity issueslasted 6h 38mIntermittent Performance and Access Issues with the Cloudera Management Consolelasted 21h 19mDataHubs, DataLakes and FreeIPA are unreachable in US regionlasted 45m