Intermittent engine failure in Dremio Cloud (US)

Final update

We have mitigated the issue impacting engine stability within Dremio Cloud (US) and have seen successful recovery of customer engines. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.

1. Resolved · Dec 17, 19:01 UTC
   We have mitigated the issue impacting engine stability within Dremio Cloud (US) and have seen successful recovery of customer engines. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.
2. Monitoring · Dec 17, 18:58 UTC
   We have mitigated the issue impacting engine stability within Dremio Cloud (US) and are continuing to monitor the recovery of customer engines. We apologize for any inconvenience caused.
3. Identified · Dec 17, 18:54 UTC
   We are continuing to work on a fix for this issue.
4. Identified · Dec 17, 18:52 UTC
   We have identified the issue impacting engine stability within Dremio Cloud (US). We have applied changes to our infrastructure and are working on recovering services. We will provide another update within the next 30 minutes.
5. Identified · Dec 17, 18:42 UTC
   We are in the process of mitigating the issue. More updates will be provided in 30 minutes.
6. Identified · Dec 17, 18:01 UTC
   We have identified the issue impacting engine stability within Dremio Cloud (US). We are continuing to work to mitigate this issue. We will provide another update within the next 30 minutes.
7. Identified · Dec 17, 17:27 UTC
   We have identified the issue impacting engine stability within Dremio Cloud (US). We are working to mitigate this issue. We will provide another update within the next 30 minutes.
8. Investigating · Dec 17, 16:56 UTC
   We are currently investigating an issue impacting engines within Dremio Cloud (US) that are causing them to shutdown while queries are in progress. We will provide another update within 30 minutes.