Incidentsby Glyph
Internal network congestion disrupts US-EAST-1
Dec 7, 15:30 UTCDec 7, 22:40 UTC
Duration
7h 10m
Impact
Critical
Root cause
Network
AWS, 90 days
8 incidents
Affected
EC2 APIsInternal DNSConnectDynamoDBus-east-1
Lesson: Retry storms turn a small change into congestion; clients need backoff, and monitoring must not share the network it monitors.
What happened
An automated activity to scale capacity of a service in the main AWS network triggered unexpected behavior from a large number of clients inside the internal network, congesting the devices between the internal and main networks. DNS resolution recovered at 17:28 UTC and network devices at 22:22 UTC.