Intermittent BigQuery job failures using Workload Identity Federation in EMEA

Final update

The issue has been resolved, and all affected systems are now functioning normally. Please contact Support via email [support@getdbt.com](http://support@getdbt.com "support@getdbt.com") if you continue to experience delays and are unsure of the root cause.

1. Resolved · Sep 25, 01:50 UTC
   The issue has been resolved, and all affected systems are now functioning normally. Please contact Support via email [support@getdbt.com](http://support@getdbt.com "support@getdbt.com") if you continue to experience delays and are unsure of the root cause.
2. Monitoring · Sep 25, 00:43 UTC
   The impacted component appears to be recovering, we will continue to monitor the performance. Please contact us at support@getdbt.com with any questions or concerns.
3. Identified · Sep 24, 20:59 UTC
   We have identified an issue with intermittent failures for jobs that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
4. Monitoring · Sep 24, 20:21 UTC
   We have not observed any further occurrences and are monitoring to confirm the issue has fully cleared. dbt job runs have returned to their normal state and we are continuing to monitor.
5. Investigating · Sep 24, 18:41 UTC
   We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
6. Investigating · Sep 24, 17:32 UTC
   We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
7. Investigating · Sep 24, 15:19 UTC
   We're continuing to investigate intermittent failures for jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. Affected runs fail with "The service is currently unavailable" (HTTP 503) during the token exchange with Google Cloud. We haven't found any changes on our side, and the error appears to originate on Google Cloud's side. If you're affected, please open a case with Google Cloud support about the 503 errors from the Workload Identity Federation token exchange. Once you receive more information from Google, you can share it with us by opening a dbt Support ticket so we can investigate further. We will provide further updates as soon as new information becomes available.
8. Investigating · Sep 24, 13:27 UTC
   We're investigating an issue with BigQuery authentication that is causing some job runs to fail intermittently. This is impacting jobs in the EMEA multi-tenant region (AWS) that connect to BigQuery using Workload Identity Federation. Affected runs fail with "The service is currently unavailable" (HTTP 503) while authenticating with Google Cloud, before any queries run. The team is working on a resolution and we will provide updates as soon as new information becomes available. In the meantime, rerunning a failed job or using Rerun from failure will usually succeed. Jobs using other BigQuery authentication methods, jobs on other data platforms, and dbt Studio are not impacted and should be running as expected.