Flink job submission and job scale operations are affected in GCP asia-northeast1, europe-west8, europe-north1, southamerica-east1

Final update

No further issues have been observed. The incident is now resolved as of 2025-1203 02:03 UTC.

1. Resolved · Dec 3, 02:04 UTC
   No further issues have been observed. The incident is now resolved as of 2025-1203 02:03 UTC.
2. Monitoring · Dec 3, 01:32 UTC
   A fix has been applied to all regions and we've seen recovery in all affected regions. We'll continue to monitor for the next 30 minutes.
3. Identified · Dec 2, 23:14 UTC
   The team has successfully applied a fix to asia-northeast1 and we're seeing recovery there. Meanwhile the team is applying mitigations in the other affected regions.
4. Identified · Dec 2, 22:03 UTC
   The root cause for the issue has been identified and the team is working on applying a mitigation.
5. Investigating · Dec 2, 21:37 UTC
   This manifests itself in new Flink jobs remaining stuck in "Statement Status: Pending", existing Flink jobs being requested to stop remaining stuck in "Statement Status: Stopping". This issue started manifesting at 2025-12-02 19:53 UTC. We are currently investigating the issue and attempting to mitigate or provide a work-around.