Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityJan 31, 2024, 10:00 UTC 2 years ago

Degraded connectivity issues between Control plane and workload clusters

MinorNot disclosedUpdated 25d ago
Jan 31, 10:00 UTCJan 31, 10:00 UTC
Duration
0m
Impact
Minor
Root cause
Not disclosed
Cloudera, 90 days
1 incidents
Affected
Not listed by the vendor.
Status page

Final update

We recently implemented an upgrade to our internal proxy service, responsible for directing requests to workloads through the Cluster Connectivity Management \(CCM\) system. This update aimed to enhance the identification of healthy routes for traffic flow. However, an unforeseen issue impacted this functionality, causing routing through potentially unhealthy paths for selected configurations. The same was fixed by rolling back the release.  It's important to note that this problem did not appear during testing in our lower environments, where rigorous evaluation precedes production deployments. Since the incident, we have promptly implemented additional safeguards to effectively identify and prevent similar issues from affecting production systems in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform, and appreciate your understanding.

Timeline

  1. Postmortem · Feb 14, 15:20 UTC
    We recently implemented an upgrade to our internal proxy service, responsible for directing requests to workloads through the Cluster Connectivity Management \(CCM\) system. This update aimed to enhance the identification of healthy routes for traffic flow. However, an unforeseen issue impacted this functionality, causing routing through potentially unhealthy paths for selected configurations. The same was fixed by rolling back the release.  It's important to note that this problem did not appear during testing in our lower environments, where rigorous evaluation precedes production deployments. Since the incident, we have promptly implemented additional safeguards to effectively identify and prevent similar issues from affecting production systems in the future. We sincerely apologize for any inconvenience this service disruption may have caused. We remain committed to delivering a reliable and robust platform, and appreciate your understanding.
  2. Resolved · Jan 31, 10:00 UTC
    We observed degraded connectivity issues between control plane and workload for a subset of customers. This impacted operations like orchestration, auto-scaling and administration of workloads from control plane. The incident has been resolved now and no action is needed from customers. A root cause analysis (RCA) will be published within seven business days.

More from Cloudera

Full history