Issues with degraded platform performance

Final update

**What Happened:** Today, following a gradual rollout of a new feature, our platform experienced widespread cross-system performance degradation. While most features continued to work, a heavy load on our database impacted overall service availability, causing automations to run slowly and the automations store and agents to occasionally fail to load. **Please note that at no point was any data lost or compromised.** **Timeline:** * **US Region:** 15:30 – 17:10 UTC \(1 hour and 40 minutes of degradation\) ‌ **What caused it:** A new Dynamic Client Registration \(DCR\) feature for OAuth \(supporting MCP applications\) caused an unexpected chain reaction. Even with the limited rollout of this feature, frequent automated DCR requests triggered a continuous, global cache invalidation storm across our core caching infrastructure. With this shared cache repeatedly cleared, all traffic was directed straight to the database, causing a severe CPU spike. ‌ **How we responded:** Our engineering team identified the source and immediately performed a significant system scale-up to handle the extreme load. We then disabled the DCR endpoint in production. This stopped the cache invalidation loop

1. Postmortem · Jul 7, 21:28 UTC
   **What Happened:** Today, following a gradual rollout of a new feature, our platform experienced widespread cross-system performance degradation. While most features continued to work, a heavy load on our database impacted overall service availability, causing automations to run slowly and the automations store and agents to occasionally fail to load. **Please note that at no point was any data lost or compromised.** **Timeline:** * **US Region:** 15:30 – 17:10 UTC \(1 hour and 40 minutes of degradation\) ‌ **What caused it:** A new Dynamic Client Registration \(DCR\) feature for OAuth \(supporting MCP applications\) caused an unexpected chain reaction. Even with the limited rollout of this feature, frequent automated DCR requests triggered a continuous, global cache invalidation storm across our core caching infrastructure. With this shared cache repeatedly cleared, all traffic was directed straight to the database, causing a severe CPU spike. ‌ **How we responded:** Our engineering team identified the source and immediately performed a significant system scale-up to handle the extreme load. We then disabled the DCR endpoint in production. This stopped the cache invalidation loop, restoring database stability and returning system performance to normal. **What we're doing to prevent recurrence:** Moving forward, we plan to use a setup that will take the load off core caching systems. Because this was a limited gradual rollout, turning it off has zero impact on your current
2. Resolved · Jul 7, 17:30 UTC
   This incident has been resolved.
3. Monitoring · Jul 7, 17:14 UTC
   A fix has been implemented and we are monitoring the results.
4. Investigating · Jul 7, 16:40 UTC
   We are continuing investigating the issues related to platform not loading/working. This is also affecting automations, boards, board views, dashboards, AI features, my work, item creation, workspaces.
5. Investigating · Jul 7, 16:02 UTC
   We are continuing investigating the issues related to platform latency/slowness. This also might affect automations not triggering/loading. Thanks for your patience and understanding.
6. Investigating · Jul 7, 15:45 UTC
   We are currently experiencing issues related to platform latency/slowness. Our dedicated team is working to resolve this as quickly as possible