Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityFeb 13, 2023, 04:18 UTC 3 years ago

CDP Control plane operations are impacted

CriticalBugUpdated 49d ago
Feb 13, 04:18 UTCFeb 13, 06:20 UTC
Duration
2h 2m
Impact
Critical
Root cause
Bug
Cloudera, 90 days
1 incidents
Affected
Cloudera Management ConsoleCloudera Data HubCloudera Data Platform (US) - CDP Management ConsoleCloudera Data Platform (US) - Data HubCloudera Data Platform (AP) - CDP Management ConsoleCloudera Data Platform (AP) - Data HubCloudera Data Platform (EU) - CDP Management ConsoleCloudera Data Platform (EU) - Data Hub
Status page

Final update

On Monday, February 13th at 04:18 UTC, Cloudera SRE detected a spike in errors related to the CDP Control plane management console. After investigation, it was determined that a recent production change had caused the outage. Although Cloudera follows strict software development lifecycle standards, an unforeseen bug due to complex dependencies was still encountered. The issue was resolved by rolling back to the previous version.  To reduce the risk of similar issues in the future, we are improving our test suites, monitoring  and dependency tracking to detect such scenarios as early in the development process as possible. Alongside also reviewing our rollback process to reduce mean time to recovery.

Timeline

  1. Postmortem · Feb 16, 09:33 UTC
    On Monday, February 13th at 04:18 UTC, Cloudera SRE detected a spike in errors related to the CDP Control plane management console. After investigation, it was determined that a recent production change had caused the outage. Although Cloudera follows strict software development lifecycle standards, an unforeseen bug due to complex dependencies was still encountered. The issue was resolved by rolling back to the previous version.  To reduce the risk of similar issues in the future, we are improving our test suites, monitoring  and dependency tracking to detect such scenarios as early in the development process as possible. Alongside also reviewing our rollback process to reduce mean time to recovery.
  2. Resolved · Feb 13, 06:20 UTC
    This incident has been resolved and the CDP Control plane is working as expected. we will publish the RCA as soon as possible.
  3. Monitoring · Feb 13, 05:50 UTC
    A fix has been implemented and we are monitoring the results.
  4. Identified · Feb 13, 05:43 UTC
    We are continuing to work on a fix for this issue.
  5. Identified · Feb 13, 05:42 UTC
    The issue has been identified and a fix is being implemented.
  6. Investigating · Feb 13, 05:08 UTC
    We are continuing to investigate the issue. Please note this impacts the management operations of the workload clusters.
  7. Investigating · Feb 13, 04:18 UTC
    We're experiencing an elevated level of errors on CDP Control plane and are currently looking into the issue.

More from Cloudera

Full history

Also caused by software bug

All