Customer workloads in Azure cloud experiencing degraded performance
Apr 18, 08:32 UTCApr 18, 12:29 UTC
Duration
3h 57m
Impact
Major
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Cloudera Data HubCloudera Data Platform (US) - Data Hub
Final update
On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites
Timeline
- Postmortem · Apr 19, 05:40 UTC
On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites
- Resolved · Apr 18, 12:29 UTC
Datahub and Datalake services are fully operational now. Incident is now resolved
- Monitoring · Apr 18, 12:12 UTC
Rollback complete. Services are currently being monitored.
- Identified · Apr 18, 11:45 UTC
Rollback complete. Services are currently being monitored.
- Identified · Apr 18, 10:33 UTC
We are continuing to work on a fix for this issue.
- Identified · Apr 18, 09:47 UTC
Issue has been identified and rollback is in progress
- Investigating · Apr 18, 08:32 UTC
Datalake and DataHub workload management is in degraded state for few customers in Azure cloud
More from Cloudera
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 323:18 UTC4 weeks ago | Cloudera Management Console not accessible in US Control Plane | minor | 3h 29m |
| May 1118:12 UTC4 months ago | Intermittent Performance Issues - US Control Plane | none | 0m |
| Mar 1822:03 UTC6 months ago | Intermittent Management Console Access Issues Across US, EU, and AP Regions | minor | 0m |
| Sep 2510:23 UTC1 year ago | FreeIPA connectivity issues | minor | 6h 38m |
| Sep 2419:42 UTC1 year ago | Intermittent Performance and Access Issues with the Cloudera Management Console | minor | 21h 19m |
| Aug 1314:33 UTC1 year ago | DataHubs, DataLakes and FreeIPA are unreachable in US region | minor | 45m |