Elevated API issues and Launch Failures

Final update

Between 6:46 AM PST and 12:31 PM PST, we experienced periods of API error rates, as well as connectivity issues for newly launched EC2 instances, primarily affecting a single Availability Zone (apne1-az1) in the AP-NORTHEAST-1 Region. Engineers were automatically engaged and immediately began investigating. The first impact window was between 6:46 AM PST and 9:46 AM PST, and primarily manifested as API errors and failed EC2 instance launches. Other AWS Services that rely on the EC2 APIs for their workflows, such as Autoscaling, ECS, and RDS, were also affected by this issue. Existing EC2 instances were not affected. During this first impact period, customers may have experienced API throttling or 5XX Responses. Additionally, some EC2 RunInstance API calls may have succeeded, but instances did not successfully launch, instead transitioning to a TERMINATED state. EC2 and EBS health data was missing or delayed, which resulted in CloudWatch alarms transitioning to an INSUFFICENT_DATA state. We recommend customers retry failed API requests or workflows that failed for any of the affected services. At 9:46 AM, we mitigated the API errors, but had not yet mitigated all issues. While moni

1. Resolved · Nov 11, 21:11 UTC
   Between 6:46 AM PST and 12:31 PM PST, we experienced periods of API error rates, as well as connectivity issues for newly launched EC2 instances, primarily affecting a single Availability Zone (apne1-az1) in the AP-NORTHEAST-1 Region. Engineers were automatically engaged and immediately began investigating. The first impact window was between 6:46 AM PST and 9:46 AM PST, and primarily manifested as API errors and failed EC2 instance launches. Other AWS Services that rely on the EC2 APIs for their workflows, such as Autoscaling, ECS, and RDS, were also affected by this issue. Existing EC2 instances were not affected. During this first impact period, customers may have experienced API throttling or 5XX Responses. Additionally, some EC2 RunInstance API calls may have succeeded, but instances did not successfully launch, instead transitioning to a TERMINATED state. EC2 and EBS health data was missing or delayed, which resulted in CloudWatch alarms transitioning to an INSUFFICENT_DATA state. We recommend customers retry failed API requests or workflows that failed for any of the affected services. At 9:46 AM, we mitigated the API errors, but had not yet mitigated all issues. While monitoring recovery of the EC2 APIs, we observed that both newly launched EC2 instances and existing instances where the network configuration was modified were experiencing connectivity issues. These connectivity issues were the result of delays in an internal subsytem responsible for propagating EC2 i
2. Update · Nov 11, 19:54 UTC
   We continue to work towards full recovery. Customers can launch new instances in the other two healthy Availability Zones (apne1-az2, apne1-az3). We also recommend that customers do not make changes that would impact connectivity to existing instances in the affected Availability Zone. This includes actions such as attaching and detaching network interfaces or gateways as well as VPC Peerings changes. We are continuing to work on resolving the intermittent connectivity issues to newly created or modified instances in a single Availability Zone (apne1-az1). This affects actions such as new instance launches, new cluster creation, autoscaling groups, gateways, and vpc gateway endpoints. Connectivity to existing resources and Instance to Instance communication are not impacted. We will provide you with another update by 1:00 PM PST or sooner as new information becomes available.
3. Update · Nov 11, 18:54 UTC
   We continue to work towards full recovery. We are observing some connectivity issues to newly created or modified instances in a single Availability Zone (apne1-az1). This affects actions such as new instance launches, new cluster creation, autoscaling groups, gateways, and vpc gateway endpoints. Connectivity to existing resources and Instance to Instance communication are not impacted. We have identified the root cause of this new issue and are working to mitigate the issue. We will provide you with another update by 12:00 PM PST or sooner as new information becomes available.
4. Update · Nov 11, 17:59 UTC
   We continue to see broad signs of recovery in the AP-NORTHEAST-1 Region. Our mitigation efforts are still underway, and we will provide you with another update on our progress by 10:45 AM PST or sooner as new information becomes available.
5. Update · Nov 11, 17:38 UTC
   We are seeing early signs of recovery for the elevated API error rates and launch failures in the AP-NORTHEAST-1 Region. EC2 and EBS CloudWatch health data is now reporting correctly. We will continue to monitor the progress of our recovery efforts and will provide you with another update by 10:00 AM PST or sooner as new information becomes available.
6. Update · Nov 11, 16:40 UTC
   We are experiencing elevated API error rates and launch failures in the AP-NORTHEAST-1 Region. While launch failures still appear to be concentrated in a single Availability Zone (apne1-az1), we are now seeing low levels of launch failures in other availability zones as well. Cloudwatch alarms for EC2 and EBS resources in the affected Availability Zone are transitioning into Insufficient Data state and alarms configured to alert off of Insufficient Data may transition to Alert state. The DescribeInstanceStatus API in the affected Availability Zone is returning Insufficient Data. Running instances are not affected, and you should rely on your own health telemetry to determine the health of your instance. In addition, we are seeing impact to other services such as ECS and Fargate Task Launch failures, ELB provisioning delays and Nat Gateway failures for Create and Disassociate workflows. We have identified the root cause and are working to mitigate the issue. We will provide you with another update by 9:45 AM PST or sooner as new information becomes available.
7. Update · Nov 11, 15:40 UTC
   We are experiencing elevated API error rates and launch failures in the AP-NORTHEAST-1 Region. Launch failures appear to be concentrated in a single Availability Zone (apne1-az1). We are working to identify the root cause and mitigate the issue.