Launches Failing in Multiple Regions

Final update

**Severity Level:** [High] Our Engineering team was able to identify and remediate the root cause of the issue. Users that were experiencing instance launch failures should no longer be impacted. In the event you’re unsure as to whether your service is still impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.

1. Resolved · Mar 5, 18:10 UTC
   **Severity Level:** [High] Our Engineering team was able to identify and remediate the root cause of the issue. Users that were experiencing instance launch failures should no longer be impacted. In the event you’re unsure as to whether your service is still impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
2. Identified · Mar 4, 21:25 UTC
   Severity Level: [High] We're currently investigating an issue with On-Demand Cloud instance launches across multiple regions. An issue with Firewall Rulesets is causing instance launches to fail or enter an Alert status for customers when creating new Firewall Rulesets in the affected regions. We have re-enabled launches, however customers without existing Firewall Rulesets may still experience failures in the following regions: • us-east-1 • us-midwest-1 • us-west-1 • us-west-2 • me-west-1 • europe-central-1 • ap-southeast-2 • asia-northeast-2 Please note that you will not be charged for an instance until it has reached a healthy boot state. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
3. Identified · Mar 4, 18:53 UTC
   Severity Level: [High] We're currently investigating an issue with On-Demand Cloud instance launches across multiple regions. A disruption to the creation of new Firewall Rulesets is causing instance launches to fail or enter an Alert status for customers without existing Firewall Rulesets in the affected regions. We have re-enabled launches, however customers without existing Firewall Rulesets may still experience failures in the following regions: • us-east-1 • us-midwest-1 • us-west-1 • us-west-2 • me-west-1 • europe-central-1 • ap-southeast-2 • asia-northeast-2 In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
4. Investigating · Mar 4, 18:37 UTC
   **Severity Level:** [High] We’re investigating instance launch failures for On-Demand Cloud with our service provider. Please bear with us as we work to re-enable instance launches in the following regions: • us-east-1 • us-midwest-1 • us-west-1 • us-west-2 • me-west-1 • europe-central-1 • ap-southeast-2 • asia-northeast-2 • asia-south-1 • asia-northeast-1 In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
5. Investigating · Mar 3, 23:36 UTC
   **Severity Level:** [High] **UPDATE:** • The below is still accurate. We're continuing the investigation with our service provider. Please bear with us as we work to re-enable instance launches in the regions listed below. We’re currently investigating instance failure launches with our On-Demand Cloud. Our engineers are actively working to identify the root cause of the issue. Instance Launches have been disabled for the following regions: • us-east-1 • us-midwest-1 • us-west-1 • us-west-2 • me-west-1 • europe-central-1 • ap-southeast-2 • asia-northeast-2 • asia-south-1 • asia-northeast-1 In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
6. Investigating · Mar 3, 19:22 UTC
   **Severity Level:** High We’re currently investigating instance failure launches with our On-Demand Cloud. Our engineers are actively working to identify the root cause of the issue. Instance Launches have been disabled for the following regions: • us-east-1 • us-midwest-1 • us-west-1 • us-west-2 • me-west-1 • europe-central-1 • ap-southeast-2 • asia-northeast-2 • asia-south-1 • asia-northeast-1 In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • [https://support.lambdalabs.com/hc/en-us/requests/new](https://support.lambdalabs.com/hc/en-us/requests/new "https://support.lambdalabs.com/hc/en-us/requests/new") To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
7. Investigating · Mar 3, 18:40 UTC
   **Severity Level:** High We’re currently investigating instance failure launches with our On-Demand Cloud. Currently, Instance launches are disabled. Our engineers are actively working to identify the root cause of the issue. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
8. Monitoring · Mar 3, 13:02 UTC
   **Severity Level:** [Medium] A fix has been implemented and we’re monitoring the results. To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.
9. Investigating · Mar 2, 22:42 UTC
   **Severity Level:** Medium We’re currently investigating a potential issue with our us-east-1 instance launches. Users will currently experience instance launch failures in this region. Our engineers are actively working to identify the root cause of the issue. Other regions are not currently impacted. In the event you’re unsure as to whether your service is impacted by this incident, please open a support ticket. We’ll get back to you as soon as possible. • To be automatically notified when the status of this incident changes, please click on the “Subscribe to updates” button. Thank you for your patience and understanding during this time.