Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityOct 5, 2023, 19:27 UTC 3 years ago

Control Plane Issues

MajorNot disclosedUpdated 49d ago
Oct 5, 19:27 UTCOct 5, 23:07 UTC
Duration
3h 40m
Impact
Major
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Cloudera Management ConsoleCloudera IAMCloudera Data FlowCloudera Data EngineeringCloudera Data WarehouseCloudera Operational DatabaseCloudera AICloudera Data HubCloudera Data CatalogCloudera Replication ManagerCloudera ObservabilityCloudera Data Platform (US) - CDP Management ConsoleCloudera Data Platform (US) - Cloudera ObservabilityCloudera Data Platform (US) - CDP IAM
Status page

Final update

On Oct 5th, 2023 we had reports from customers that Cloudera Manager instances in their environments were reporting UNREACHABLE status for both Datalake’s and Datahub’s. Upon further investigation, it was identified that two independent production changes were attributed to the issue.  1. A new software release for connectivity between Control Plane and Workload; which hit an edge case during certain operations. This was addressed by rolling back to the previous software version.  2. The second independent production change was related to a decrease in resources allocated to one of our internal systems; responsible for storing key secrets. This was addressed by reverting to the previous configurations.  These issues did not manifest in our lower environments, where the changes were tested prior to rolling out to Production. As a mitigation action, our teams are working on adding monitoring and alerting around these corner cases and will introduce additional checks and balances before any resource changes are implemented in our production system.

Timeline

  1. Postmortem · Oct 16, 12:30 UTC
    On Oct 5th, 2023 we had reports from customers that Cloudera Manager instances in their environments were reporting UNREACHABLE status for both Datalake’s and Datahub’s. Upon further investigation, it was identified that two independent production changes were attributed to the issue.  1. A new software release for connectivity between Control Plane and Workload; which hit an edge case during certain operations. This was addressed by rolling back to the previous software version.  2. The second independent production change was related to a decrease in resources allocated to one of our internal systems; responsible for storing key secrets. This was addressed by reverting to the previous configurations.  These issues did not manifest in our lower environments, where the changes were tested prior to rolling out to Production. As a mitigation action, our teams are working on adding monitoring and alerting around these corner cases and will introduce additional checks and balances before any resource changes are implemented in our production system.
  2. Resolved · Oct 5, 23:07 UTC
    The fix implemented has seen positive results, resolving connectivity issues.
  3. Monitoring · Oct 5, 22:34 UTC
    The overall control plane has stabilized. We're investigating connectivity issues that are impacting some number of customers. A potential fix is being implemented to resolve the connectivity issues.
  4. Monitoring · Oct 5, 21:26 UTC
    We've implemented a fix to our control plane service. We're seeing services recover albeit with higher latency.
  5. Identified · Oct 5, 20:49 UTC
    Issues have been identified in an internal control plane service. Multiple services are currently impacted.
  6. Investigating · Oct 5, 19:27 UTC
    We are investigating problems affecting the US control plane.

More from Cloudera

Full history