Asana down in US

Final update

Incident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

1. Postmortem · Feb 7, 22:46 UTC
   Incident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
2. Resolved · Feb 5, 21:54 UTC
   This incident has been resolved.
3. Monitoring · Feb 5, 21:35 UTC
   A fix has been implemented and we are monitoring the results.
4. Investigating · Feb 5, 21:34 UTC
   We have made configuration changes and are monitoring for recovery.
5. Investigating · Feb 5, 21:22 UTC
   We are continuing to investigate this issue.
6. Investigating · Feb 5, 21:19 UTC
   We are seeing a high rate of errors for the Asana application in the US region.