Customer workloads in Azure cloud experiencing degraded performance
Apr 18, 08:32 UTCApr 18, 12:29 UTC
Duration
3h 57m
Impact
Major
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Cloudera Data HubCloudera Data Platform (US) - Data Hub
Final update
On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites
Timeline
- Postmortem · Apr 19, 05:40 UTC
On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites
- Resolved · Apr 18, 12:29 UTC
Datahub and Datalake services are fully operational now. Incident is now resolved
- Monitoring · Apr 18, 12:12 UTC
Rollback complete. Services are currently being monitored.
- Identified · Apr 18, 11:45 UTC
Rollback complete. Services are currently being monitored.
- Identified · Apr 18, 10:33 UTC
We are continuing to work on a fix for this issue.
- Identified · Apr 18, 09:47 UTC
Issue has been identified and rollback is in progress
- Investigating · Apr 18, 08:32 UTC
Datalake and DataHub workload management is in degraded state for few customers in Azure cloud
More from Cloudera
Full historyCloudera Management Console not accessible in US Control Planelasted 3h 29mIntermittent Performance Issues - US Control Planelasted 0mIntermittent Management Console Access Issues Across US, EU, and AP Regionslasted 0mFreeIPA connectivity issueslasted 6h 38mIntermittent Performance and Access Issues with the Cloudera Management Consolelasted 21h 19mDataHubs, DataLakes and FreeIPA are unreachable in US regionlasted 45m