CDP Control plane operations are impacted
Feb 13, 04:18 UTCFeb 13, 06:20 UTC
Duration
2h 2m
Impact
Critical
Root cause
Bug
Cloudera, 90 days
1 incidents
Affected
Cloudera Management ConsoleCloudera Data HubCloudera Data Platform (US) - CDP Management ConsoleCloudera Data Platform (US) - Data HubCloudera Data Platform (AP) - CDP Management ConsoleCloudera Data Platform (AP) - Data HubCloudera Data Platform (EU) - CDP Management ConsoleCloudera Data Platform (EU) - Data Hub
Final update
On Monday, February 13th at 04:18 UTC, Cloudera SRE detected a spike in errors related to the CDP Control plane management console. After investigation, it was determined that a recent production change had caused the outage. Although Cloudera follows strict software development lifecycle standards, an unforeseen bug due to complex dependencies was still encountered. The issue was resolved by rolling back to the previous version. To reduce the risk of similar issues in the future, we are improving our test suites, monitoring and dependency tracking to detect such scenarios as early in the development process as possible. Alongside also reviewing our rollback process to reduce mean time to recovery.
Timeline
- Postmortem · Feb 16, 09:33 UTC
On Monday, February 13th at 04:18 UTC, Cloudera SRE detected a spike in errors related to the CDP Control plane management console. After investigation, it was determined that a recent production change had caused the outage. Although Cloudera follows strict software development lifecycle standards, an unforeseen bug due to complex dependencies was still encountered. The issue was resolved by rolling back to the previous version. To reduce the risk of similar issues in the future, we are improving our test suites, monitoring and dependency tracking to detect such scenarios as early in the development process as possible. Alongside also reviewing our rollback process to reduce mean time to recovery.
- Resolved · Feb 13, 06:20 UTC
This incident has been resolved and the CDP Control plane is working as expected. we will publish the RCA as soon as possible.
- Monitoring · Feb 13, 05:50 UTC
A fix has been implemented and we are monitoring the results.
- Identified · Feb 13, 05:43 UTC
We are continuing to work on a fix for this issue.
- Identified · Feb 13, 05:42 UTC
The issue has been identified and a fix is being implemented.
- Investigating · Feb 13, 05:08 UTC
We are continuing to investigate the issue. Please note this impacts the management operations of the workload clusters.
- Investigating · Feb 13, 04:18 UTC
We're experiencing an elevated level of errors on CDP Control plane and are currently looking into the issue.
More from Cloudera
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 323:18 UTC4 weeks ago | Cloudera Management Console not accessible in US Control Plane | minor | 3h 29m |
| May 1118:12 UTC4 months ago | Intermittent Performance Issues - US Control Plane | none | 0m |
| Mar 1822:03 UTC6 months ago | Intermittent Management Console Access Issues Across US, EU, and AP Regions | minor | 0m |
| Sep 2510:23 UTC1 year ago | FreeIPA connectivity issues | minor | 6h 38m |
| Sep 2419:42 UTC1 year ago | Intermittent Performance and Access Issues with the Cloudera Management Console | minor | 21h 19m |
| Aug 1314:33 UTC1 year ago | DataHubs, DataLakes and FreeIPA are unreachable in US region | minor | 45m |
Also caused by software bug
All| Started | Vendor | Incident | Impact | Duration |
|---|---|---|---|---|
| Sep 1518:57 UTC2 weeks ago | ClickPipes failing on Kinesis in AWS us-east-1 | minor | 29h 46m | |
| Sep 318:20 UTC4 weeks ago | Retroactive Incident: Twilio Personalized Support Phone Line Affected | none | 0m | |
| Aug 819:48 UTC8 weeks ago | Alerting expressions pipeline failing when recovery settings | minor | 0m | |
| Aug 615:22 UTC8 weeks ago | Incident with Actions | critical | 10h 42m | |
| Jul 2314:14 UTC2 months ago | [Medium] Issues with Box Hubs | major | 16m | |
| Jun 1719:00 UTC3 months ago | Incident With Webhooks | none | 0m |