EU Control plane has issues accessing workload UIs

Final update

On Nov 3rd 2023 at ~09:40 UTC, few customers of the Europe Control plane reported issues about Workloads User Interface not being accessible via Management Console. Please do note the workload clusters were operational during this time. Upon further investigation it was identified that a recent production change hit an edge case exposing a new bug. The same was fixed immediately by rolling a new change. This issues did not manifest in our lower environments, where the changes were tested prior to rolling out to production. As a mitigation action, our teams are working on adding additional test cases to improve the coverage for this corner case.

1. Postmortem · Nov 10, 13:48 UTC
   On Nov 3rd 2023 at ~09:40 UTC, few customers of the Europe Control plane reported issues about Workloads User Interface not being accessible via Management Console. Please do note the workload clusters were operational during this time. Upon further investigation it was identified that a recent production change hit an edge case exposing a new bug. The same was fixed immediately by rolling a new change. This issues did not manifest in our lower environments, where the changes were tested prior to rolling out to production. As a mitigation action, our teams are working on adding additional test cases to improve the coverage for this corner case.
2. Resolved · Nov 3, 11:29 UTC
   Our teams have successfully deployed a fix for the issue and confirmed that the issue has been resolved. If you are still experiencing issues or have any questions please raise a support case with us. A root cause analysis (RCA) will be published within seven business days. Incident Start Time: 09:40 AM UTC November 3rd, 2023 Incident End Time: 11:00 AM UTC November 3rd, 2023
3. Investigating · Nov 3, 10:32 UTC
   We are continuing to investigate this issue.
4. Investigating · Nov 3, 10:28 UTC
   We are currently investigating a potential issue with EU Control Plane down(workload logins/UI are not working). We will have an update within 60 mins. Customer Experience: Customers may observe issues while logging in to the atlas URL, Ranger URL, and resource manager URL are not accessible.