Queries stuck in metadata retrieval in Dremio Cloud (EU)

Final update

This incident has been resolved.

1. Resolved · Feb 10, 17:55 UTC
   This incident has been resolved.
2. Monitoring · Feb 10, 17:55 UTC
   One of our core indexing services that powers both discovery of sources and metadata refreshes starting queueing up. We have mitigated the issue by throttling the service. Our initial analysis suggests that the failure may have been triggered by a recent change in our underlying infrastructure on Google. We are collaborating with our partners at Google to conduct a thorough Root Cause Analysis and identify the exact cause of the issue.
3. Monitoring · Feb 10, 17:15 UTC
   We have identified the issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage and applied a mitigation. We are seeing recovery of customer environments and will continue to monitor. We apologize for any inconvenience caused and will publish a post-mortem for this incident in the coming days.
4. Investigating · Feb 10, 15:58 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We are experiencing issues with our upstream provider causing timeouts and are working to mitigate this issue. We will provide another update within 60 minutes.
5. Investigating · Feb 10, 14:57 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
6. Investigating · Feb 10, 13:51 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
7. Investigating · Feb 10, 12:52 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
8. Investigating · Feb 10, 11:53 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
9. Investigating · Feb 10, 10:55 UTC
   We are continuing to investigate an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.
10. Investigating · Feb 10, 10:02 UTC
    We are currently investigating an issue impacting queries within Dremio Cloud (EU) that causes queries to not progress past the metadata retrieval stage. We will provide another update within 60 minutes.