Some users are experiencing issues accessing Hex projects

Final update

After fully analyzing the timeline of events, **we've confirmed that the root cause was a cascading failure triggered by the combination of a feature flag, deployment timing during peak traffic, and an inefficiency in our database connection management**. The issue began August 19 at 4:03 PM PDT, when a feature flag was updated to have clients query the server more frequently to eliminate a rare display issue in the client. It escalated August 20 at 10:00 AM PDT when we did a production deployment during peak traffic to add necessary support for a new feature release. This combination created a thundering herd problem where client reconnection attempts exponentially increased system pressure, saturating our database connections and triggering system-wide failures. Service was restored for most users at 10:52 PM PDT after manual intervention, and then fully restored at 11:26 AM EDT for all users after resetting backend pods. While this incident has been resolved, we are not satisfied by this sequence of events, from both the risk management perspective and the detection / response perspective. We are investing in the following mechanisms to improve: * We have increased baseline serv

1. Postmortem · Aug 26, 13:15 UTC
   After fully analyzing the timeline of events, **we've confirmed that the root cause was a cascading failure triggered by the combination of a feature flag, deployment timing during peak traffic, and an inefficiency in our database connection management**. The issue began August 19 at 4:03 PM PDT, when a feature flag was updated to have clients query the server more frequently to eliminate a rare display issue in the client. It escalated August 20 at 10:00 AM PDT when we did a production deployment during peak traffic to add necessary support for a new feature release. This combination created a thundering herd problem where client reconnection attempts exponentially increased system pressure, saturating our database connections and triggering system-wide failures. Service was restored for most users at 10:52 PM PDT after manual intervention, and then fully restored at 11:26 AM EDT for all users after resetting backend pods. While this incident has been resolved, we are not satisfied by this sequence of events, from both the risk management perspective and the detection / response perspective. We are investing in the following mechanisms to improve: * We have increased baseline server capacity and reverted the change to the feature flag that increased database load. We are also optimizing database connection usage to handle traffic spikes via better pooling and reuse. * Our monitoring around system load and client connection patterns will be made more comprehensive, including
2. Resolved · Aug 20, 18:26 UTC
   This incident has been resolved.
3. Monitoring · Aug 20, 17:52 UTC
   A fix has been implemented and we are monitoring the results.
4. Identified · Aug 20, 17:45 UTC
   The issue has been identified and a fix is being implemented.
5. Investigating · Aug 20, 17:30 UTC
   We are continuing to investigate this issue.
6. Investigating · Aug 20, 17:15 UTC
   We are continuing to investigate this issue.
7. Investigating · Aug 20, 17:00 UTC
   We are currently investigating this issue.