Investigating connectivity issues across accounts and devices

Final update

**What Happened:** On February 16th \(09:04 AM EST\), our US customers experienced a 28-minute system downtime due to an application-level bug. ‌ **What caused it:** A recent code change in the billing subsystem responsible for updating account plans introduced an edge-case bug that dramatically increased platform load, which caused system downtime. ‌ **How we responded:** Our monitoring system promptly identified the issue, triggering an immediate response from our engineering team. By activating failover mechanisms, reverting the problematic code change, and blocking the affected endpoint, we restored normal service by 09:32 AM EST. ‌ **What we're doing to prevent recurrence:** To enhance system resilience, we are implementing database optimizations, additional guardrails, and limits to prevent load spikes on the platform. It also includes strengthening our error detection and recovery processes. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [monday.com](http://monday.com)

1. Postmortem · Feb 16, 16:48 UTC
   **What Happened:** On February 16th \(09:04 AM EST\), our US customers experienced a 28-minute system downtime due to an application-level bug. ‌ **What caused it:** A recent code change in the billing subsystem responsible for updating account plans introduced an edge-case bug that dramatically increased platform load, which caused system downtime. ‌ **How we responded:** Our monitoring system promptly identified the issue, triggering an immediate response from our engineering team. By activating failover mechanisms, reverting the problematic code change, and blocking the affected endpoint, we restored normal service by 09:32 AM EST. ‌ **What we're doing to prevent recurrence:** To enhance system resilience, we are implementing database optimizations, additional guardrails, and limits to prevent load spikes on the platform. It also includes strengthening our error detection and recovery processes. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [monday.com](http://monday.com)
2. Resolved · Feb 16, 14:49 UTC
   The platform is now back to regular service. Please refresh your browser to access the platform. Thank you for your patience
3. Monitoring · Feb 16, 14:38 UTC
   A fix has been implemented, and we are monitoring results to restore full system stability. Automations may still be delayed.
4. Identified · Feb 16, 14:34 UTC
   Our team has identified the root cause of the issue and is working to resume regular service usage. We will continue to provide updates on their progress
5. Investigating · Feb 16, 14:10 UTC
   We are currently investigating reports of connectivity and loading issues across the platform. Our team is working to resolve this promptly.