The intermittent performance and access issues with the Cloudera Management Console and FreeIPA were triggered by a scheduled service upgrade intended to improve platform stability.
Our investigation determined that a change in a core component introduced a latent configuration issue. This specific condition was not exposed in our testing environments, preventing dependent services from applying dynamic configuration updates in production and leading to the outage.
We've taken immediate action to prevent this issue from recurring:
System Fix: The problematic component change was rolled back, and a permanent patch was deployed to restore proper dynamic configuration functionality.
Process Overhaul: We've implemented a more rigorous upgrade process with mandatory, near-production scale validation steps that specifically test for these types of configuration failures.
Enhanced Monitoring: We significantly improved our monitoring and alerting capabilities to detect these abnormal service behaviours much earlier, ensuring a faster response time.
We are dedicated to providing a reliable platform and will continue to invest in our infrastructure and processes. Thank you again for your
Timeline
Postmortem · Oct 13, 20:04 UTC
The intermittent performance and access issues with the Cloudera Management Console and FreeIPA were triggered by a scheduled service upgrade intended to improve platform stability.
Our investigation determined that a change in a core component introduced a latent configuration issue. This specific condition was not exposed in our testing environments, preventing dependent services from applying dynamic configuration updates in production and leading to the outage.
We've taken immediate action to prevent this issue from recurring:
System Fix: The problematic component change was rolled back, and a permanent patch was deployed to restore proper dynamic configuration functionality.
Process Overhaul: We've implemented a more rigorous upgrade process with mandatory, near-production scale validation steps that specifically test for these types of configuration failures.
Enhanced Monitoring: We significantly improved our monitoring and alerting capabilities to detect these abnormal service behaviours much earlier, ensuring a faster response time.
We are dedicated to providing a reliable platform and will continue to invest in our infrastructure and processes. Thank you again for your patience and understanding
Resolved · Sep 25, 17:02 UTC
Our engineering teams have confirmed that the issues affecting the Management Console & FreeIPA services are now fully resolved, and all systems are confirmed to be operating normally. Should you continue to experience any trouble, please raise a support case with us.
We sincerely apologise for any inconvenience this may have caused and we will have the root cause analysis (RCA) published within 7 business days.
Monitoring · Sep 25, 10:23 UTC
Current Status: Our teams have identified the source of the issue and have implemented a solution which has fixed the issue. We are monitoring the situation.
Customer Experience: During this window customers may experience issues accessing FreeIPA service
