Who is down right now

26 down now · 292 this week

Cloudera · Data and observabilityFeb 14, 2024, 16:05 UTC 2 years ago

FreeIPA clusters are in an unreachable state

MajorBugUpdated 49d ago
Feb 14, 16:05 UTCFeb 14, 17:50 UTC
Duration
1h 44m
Impact
Major
Root cause
Bug
Cloudera, 90 days
1 incidents
Affected
Cloudera Data FlowCloudera Data EngineeringCloudera Data WarehouseCloudera Operational DatabaseCloudera AICloudera Data HubCloudera Data CatalogCloudera Data Platform (US) - DataFlowCloudera Data Platform (US) - Data EngineeringCloudera Data Platform (US) - Data WarehouseCloudera Data Platform (US) - Operational DatabaseCloudera Data Platform (US) - Machine LearningCloudera Data Platform (US) - Data HubCloudera Data Platform (US) - Data Catalog
Status page

Final update

On February 14, 2024, a scheduled monthly Kubernetes maintenance activity unintentionally caused a 30-minute outage for the FreeIPA service within the Control Plane. While the service itself remained functional on workloads, the issue stemmed from a bug introduced during the maintenance. This bug impacted the Inverting Proxy component, responsible for facilitating communication between the Control Plane and workloads, leading to the temporary disruption. The team promptly identified and rectified the bug, restoring full service approximately within 30 minutes. Additionally, we have implemented corrective measures within the maintenance automation to prevent similar occurrences in the future. We sincerely apologize for the inconvenience this incident may have caused.

Timeline

  1. Postmortem · Feb 24, 02:03 UTC
    On February 14, 2024, a scheduled monthly Kubernetes maintenance activity unintentionally caused a 30-minute outage for the FreeIPA service within the Control Plane. While the service itself remained functional on workloads, the issue stemmed from a bug introduced during the maintenance. This bug impacted the Inverting Proxy component, responsible for facilitating communication between the Control Plane and workloads, leading to the temporary disruption. The team promptly identified and rectified the bug, restoring full service approximately within 30 minutes. Additionally, we have implemented corrective measures within the maintenance automation to prevent similar occurrences in the future. We sincerely apologize for the inconvenience this incident may have caused.
  2. Resolved · Feb 14, 17:50 UTC
    Current Status: Our teams have confirmation that the solution implemented has addressed the issue. If you continue to experience issues please raise a support case with us. A root cause analysis (RCA) will be published within seven business days. Customer Experience: Customers may observe FreeIPA service in an unreachable status. Incident Start time: ~10:53 UTC Feburary 14th, 2024
  3. Monitoring · Feb 14, 16:54 UTC
    Current Status: Our teams have identified the source of the issue and have implemented a solution which is under monitoring. We will have another update within 60 mins. Customer Experience: Customer may observe FreeIPA service in an unreachable status. Incident Start time: ~10:53 UTC Feburary 14th, 2024
  4. Identified · Feb 14, 16:26 UTC
    Current Status: Our teams have identified the source of the issue. We are working on developing and implementing a solution to restore the service(s). We will have another update within 60 mins. Customer Experience: Customer may observe FreeIPA service in an unreachable status. Incident Start time: ~10:53 UTC Feburary 14th, 2024
  5. Investigating · Feb 14, 16:19 UTC
    We are continuing to investigate this issue.
  6. Investigating · Feb 14, 16:05 UTC
    Current Status: We are currently investigating a potential issue with the FreeIPA service. Customer Experience: Customer may observe FreeIPA service in an unreachable status.

More from Cloudera

Full history

Also caused by software bug

All