Who is down right now

20 down now · 248 this week

Google Cloud · Cloud and hostingJul 14, 2026, 17:00 UTC 3 months ago

Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions.

MajorNetworkUpdated 45d ago
  • Root cause changed from Config change to Network on Sep 25, 2026
Jul 14, 17:00 UTCJul 15, 03:40 UTC
Duration
10h 40m
Impact
Major
Root cause
Network
Google Cloud, 90 days
4 incidents
Affected
VMWare engineaustralia-southeast2europe-west3

Lesson: Zonal outages can impact network connectivity across multiple regions for stretched cluster configurations.

What happened

Incident Report Summary On Tuesday, 14 July 2026 10:00 PT, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures. The disruption was traced to a network configuration update that introduced a conflict, causing inter-site communication failures, which triggered VMware Stretched Cluster failover events. Google engineers successfully mitigated the impact by rolling back the issue-causing configuration change and resetting the network state. We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Root Cause A network configuration update intended to prepare the cloud network infrastructure for new capabilities was deployed to the foundational network control plane. While the configuration payload itself was structurally valid, it exposed an implementation gap within the control plane's routing logic. This logical gap caused the underlying network hosts to misconfigure program routing

Timeline

  1. available · Jul 24, 07:10 UTC
    Incident Report Summary On Tuesday, 14 July 2026 10:00 PT, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures. The disruption was traced to a network configuration update that introduced a conflict, causing inter-site communication failures, which triggered VMware Stretched Cluster failover events. Google engineers successfully mitigated the impact by rolling back the issue-causing configuration change and resetting the network state. We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Root Cause A network configuration update intended to prepare the cloud network infrastructure for new capabilities was deployed to the foundational network control plane. While the configuration payload itself was structurally valid, it exposed an implementation gap within the control plane's routing logic. This logical gap caused the underlying network hosts to misconfigure program routing tables. As a result, traffic destined for the private IP address space used by GCVE Stretched Clusters was dropped. Because Stretched Clusters rely on this private address space to establish routing sessions for inter-zonal connectivity, the traffic drops severed communications between the active z
  2. available · Jul 20, 18:31 UTC
    Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using [**https://cloud.google.com/support**](https://cloud.google.com/support). Date/Time of the Issue (All time US/Pacific) Incident Start: 14 July 2026 10:00 Incident End: 14 July 2026 20:40 Duration: 10 hours, 40 minutes Summary On Tuesday, 14 July 2026, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures. The disruption was traced to a network configuration update that introduced a conflict, causing inter-site communication failures, which triggered VMware Stretch Cluster failover events. Google engineers successfully mitigated the disruption by rolling back the configuration change. Preliminary Root Cause A configuration change deployed within the Google Cloud network affected traffic between the VMware Engine stretched cluster zones. This resu
  3. available · Jul 15, 05:34 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers may have experienced zonal outages impacting network connectivity across multiple regions. **Description** We experienced an inter-site communication issue with Google Cloud VMware Engine (GCVE) Stretched Cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We identified the affected customers, and have worked with them to fully mitigate the issue by Tuesday, 2026-07-14 at 21:46 PDT. Preliminary analysis indicates that a network configuration update was the cause of the inter-zone network disruption. Our engineering team mitigated the issue by rolling back the faulty configuration to its last-known good value. If customers are still experiencing impact from this issue, please contact Support, and we will work with you to resolve any residual impact. We thank you for your patience while we worked to resolve this issue. **Customer Symptoms** Some GCVE customers using Stretched Cluster may have experienced inter-site communication failures to their GCVE environments within the affected zones. **Workaround** The issue is now mitigated.
  4. service_disruption · Jul 15, 03:27 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 22:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following to restore access to your workloads: * VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Cloud Support before proceedin
  5. service_disruption · Jul 15, 01:55 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 20:30 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before pro
  6. service_disruption · Jul 15, 01:15 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 19:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before pro
  7. service_disruption · Jul 15, 00:13 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. A remediation rollout is currently in progress to address the underlying network issue. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 18:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. Upon further investigation we identified that the northamerica-northeast2 region was not impacted. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaf
  8. service_disruption · Jul 14, 23:05 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our investigation has identified a recent configuration update that is the likely cause of the inter-zone network disruption. Teams are working on remediation. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 17:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch
  9. service_disruption · Jul 14, 22:16 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our investigation has identified underlying inter-zone communication failures and Border Gateway Protocol (BGP) session flapping between cluster zones. Specifically, network connectivity has been lost between the affected zones and the witness appliance. Because the witness appliance is currently unreachable, the cluster zones are unable to safely synchronize state. As a result, VMs on the affected sites are becoming isolated and may be left without writable data. **Failover / VM Migration:** Migrating affected VMs to the healthy, secondary zone of your stretch cluster remains the primary mitigation strategy. Because of the complexities surrounding failover risks and secondary zone health, we highly encourage you to open a ticket with Google Cloud Support if you are severely impacted. We do not currently have an ETA for resolution. We will provide another update by Tuesday, 2026-07-14 16:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to t
  10. service_disruption · Jul 14, 21:31 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an inter-site communication issue with Google Cloud VMware Engine (GCVE) stretched cluster customers beginning Tuesday, 2026-07-14 at 10:00 PDT. We have identified the customers that are affected by this issue and we are working with them on mitigation. Our preliminary investigation indicates this is stemming from an underlying network connectivity issue affecting the infrastructure that links the zones within a stretch cluster. This disruption is causing synchronization issues between the affected zones. We believe Storage and Compute services remain unaffected, and VMs are running as expected, though connectivity to them may be degraded. We will provide another update by Tuesday, 2026-07-14 15:00 PDT with current details. **Customer Symptoms** Some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** While we work on restoring full connectivity, we recommend the following workarounds to restore access to your workloads: - VM Migration (recommended): Where possible, migrate your affected VMs to the healthy and unaffected side of the stretch cluster. We strongly recommend consulting with Google Support before proceeding.
  11. service_disruption · Jul 14, 20:24 UTC
    **Summary** Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions. **Description** We are experiencing an issue with Google Cloud VMware Engine (GCVE) beginning Tuesday, 2026-07-14 at 10:00 PDT. We are continuing to investigate and mitigate the issue. We believe this disruption is isolated to network connectivity issues for stretch clusters, while Storage and Compute services appear to be unaffected. GCVE VMs are running as expected but customers may experience connectivity issues to the VMs. We will provide another update by Tuesday, 2026-07-14 14:00 PDT with current details. **Customer Symptoms** Some GCVE customers may experience inter-site communication failures to their GCVE environments within the affected zones. **Workaround** None at this time.

More from Google Cloud

Full history
MajorOct 20, 06:00 UTC
Multiple services were experiencing issues in the europe-west2 region.On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 2026, 06:53 BST), a data center facility that hosts europe-west2-b, a zone within the europe-west2 cloud region, experienced an external electrical power disruption.lasted 6h 8mPostmortem
MajorOct 8, 11:00 UTC
Google Cloud Storage (GCS) customers experienced elevated latencies and error rates when performing upload or write operations.On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes.lasted 29h 46mPostmortem
MajorSep 1, 14:42 UTC
Multiple products in us-central1-b are experiencing network service degradation.Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected.lasted 4h 10mPostmortem
CriticalAug 20, 15:40 UTC
We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.On Thursday, 20 August 2026, from 08:00 to 10:22 US/Pacific (15:00 to 17:22 UTC), multiple Google Cloud services in the us-west1 region experienced elevated latency, provisioning failures, increased error rates, and widespread service degradations for a total duration of 2 hours…lasted 3h 40mPostmortem
MajorJul 15, 23:57 UTC
Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes.lasted 12h 28mPostmortem
MajorJun 5, 07:00 UTC
Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.lasted 21d 12h

Also caused by network

All