DataHubs, DataLakes and FreeIPA are unreachable in US region

Final update

A recent system update, intended to enhance stability of our platform, inadvertently led to an unforeseen memory issue affecting a critical internal service responsible for platform access management. ‌ A subsequent incoming request spike caused the memory limits exhaustion, which resulted in the service to become intermittently unavailable. ‌ The memory allocation for the service was manually increased, which restored normal operations. This change was then made permanent to circumvent any recurrence. ‌ We have implemented more robust monitoring and alerting to detect similar issues in the future before they can impact our customers. This includes updating the existing alerts for resource exhaustion and adding new alerts based on the incident, to ensure that the right teams are notified immediately. ‌ We are dedicated to providing a reliable and performant platform. We will continue to invest in improving our infrastructure and processes to prevent future disruptions. We appreciate your patience and understanding as we worked to resolve this issue.

1. Postmortem · Aug 22, 18:15 UTC
   A recent system update, intended to enhance stability of our platform, inadvertently led to an unforeseen memory issue affecting a critical internal service responsible for platform access management. ‌ A subsequent incoming request spike caused the memory limits exhaustion, which resulted in the service to become intermittently unavailable. ‌ The memory allocation for the service was manually increased, which restored normal operations. This change was then made permanent to circumvent any recurrence. ‌ We have implemented more robust monitoring and alerting to detect similar issues in the future before they can impact our customers. This includes updating the existing alerts for resource exhaustion and adding new alerts based on the incident, to ensure that the right teams are notified immediately. ‌ We are dedicated to providing a reliable and performant platform. We will continue to invest in improving our infrastructure and processes to prevent future disruptions. We appreciate your patience and understanding as we worked to resolve this issue.
2. Resolved · Aug 13, 15:18 UTC
   Our teams have successfully deployed a fix, and this incident is now resolved. Full service was restored as of 15:05 UTC. Between 14:08 UTC and 15:05 UTC, customers may have experienced issues accessing DataHubs, DataLakes, and FreeIPA services. Service should now be operating normally. If you are still experiencing any issues, please contact our support team. A full root cause analysis (RCA) will be published within seven business days.
3. Investigating · Aug 13, 14:33 UTC
   Current Status: We are currently investigating a potential issue with DataHubs, DataLakes and FreeIPA service across US region. We will have an update within 60 mins. Customer Experience: Customer may observe issues when access DataHubs, DataLakes and FreeIPA service. Incident Start time : 14:08 UTC August 13th, 2025