Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityApr 18, 2023, 08:32 UTC 3 years ago

Customer workloads in Azure cloud experiencing degraded performance

MajorNot disclosedUpdated 49d ago
Apr 18, 08:32 UTCApr 18, 12:29 UTC
Duration
3h 57m
Impact
Major
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Cloudera Data HubCloudera Data Platform (US) - Data Hub
Status page

Final update

On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites

Timeline

  1. Postmortem · Apr 19, 05:40 UTC
    On April/18/2023 between 8:30 UTC to 11:45 UTC Customers using DataHub and DataLake on Azure connecting to us-west CDP Control plane were experiencing timeouts causing environment creation failures. The team was notified about this incident immediately. On investigation it was found that latest release triggered a edge case bug which caused the metadata update failures with Azure. This incident was resolved by performing a rollback. The existing environments on AWS or GCP were not impacted due to this. As a mitigation item the team is working on adding additional test workloads across cloud environments to simulate these edge cases and also enhancing our test suites
  2. Resolved · Apr 18, 12:29 UTC
    Datahub and Datalake services are fully operational now. Incident is now resolved
  3. Monitoring · Apr 18, 12:12 UTC
    Rollback complete. Services are currently being monitored.
  4. Identified · Apr 18, 11:45 UTC
    Rollback complete. Services are currently being monitored.
  5. Identified · Apr 18, 10:33 UTC
    We are continuing to work on a fix for this issue.
  6. Identified · Apr 18, 09:47 UTC
    Issue has been identified and rollback is in progress
  7. Investigating · Apr 18, 08:32 UTC
    Datalake and DataHub workload management is in degraded state for few customers in Azure cloud

More from Cloudera

Full history