Automations and Background Actions Delayed

Final update

On 2024-07-26, a deployment of new code introduced a bug which caused delays for automations and background actions for a subset of our customers. Jobs were delayed between 00:55 and 12:30 UTC. We reverted this deployment, which allowed our system to catch up. Impact: During the incident, a subset of customers experienced delayed background actions, such as rules, email notifications, imports, and exports. For a small fraction of customers, during a 7-hour period, automations may not have executed as intended. Moving forward: We have identified the root cause of this particular bug and will improve our systems and processes to prevent it from happening in the future.

1. Resolved · Jul 26, 21:10 UTC
   On 2024-07-26, a deployment of new code introduced a bug which caused delays for automations and background actions for a subset of our customers. Jobs were delayed between 00:55 and 12:30 UTC. We reverted this deployment, which allowed our system to catch up. Impact: During the incident, a subset of customers experienced delayed background actions, such as rules, email notifications, imports, and exports. For a small fraction of customers, during a 7-hour period, automations may not have executed as intended. Moving forward: We have identified the root cause of this particular bug and will improve our systems and processes to prevent it from happening in the future.