Who is down right now

Down now20This week248

20 down now · 248 this week

AWS · Cloud and hostingOct 20, 2025, 06:48 UTC 1 year ago

DynamoDB DNS failure takes down US-EAST-1

CriticalDNSUpdated 46d ago
Oct 20, 06:48 UTCOct 20, 21:20 UTC
Duration
14h 32m
Impact
Critical
Root cause
DNS
AWS, 90 days
8 incidents
Affected
DynamoDBEC2Network Load BalancerLambdaus-east-1
Read the postmortemStatus page

Lesson: Automation that manages critical DNS needs its own guard against writing an empty record, and dependent services need to recover from a stale state on their own.

What happened

A latent race condition in the DynamoDB DNS management system left an empty DNS record for the regional endpoint dynamodb.us-east-1.amazonaws.com, and the automation failed to repair it. EC2 launches and Network Load Balancer health checks failed in turn, giving three distinct periods of customer impact.

More from AWS

Full history
MinorNov 6, 22:51 UTC
Elevated Launch Failureslasted 2h 21m
MinorOct 30, 01:28 UTC
Increased invoke error rateslasted 0m
MinorOct 4, 16:14 UTC
Elevated packet losslasted 2h 42m
MinorSep 21, 23:26 UTC
Increased Error Rateslasted 58m
MinorSep 3, 21:49 UTC
Increased API Error Rateslasted 2h 18m
MinorAug 21, 02:02 UTC
Increased Error Rateslasted 38m

Also caused by dns

All
MajorOct 31, 04:35 UTC
Lambda: lambda.ai Unreachable Due to DNS Issuelasted 2h 32m
NoticeSep 10, 14:34 UTC
UiPath: Multiple services impacted in Korea and Japan due to cloud service provider issuelasted 0m
MinorJul 27, 20:34 UTC
Supabase: Delays in creation of DNS recordslasted 2h 2m
MinorJul 24, 23:03 UTC
Databento: Intermittent API outage due to an internal DNS issuelasted 17m
MajorJun 23, 01:05 UTC
Snowflake: INC20000046lasted 1h 30m
MinorJun 4, 10:12 UTC
DigitalOcean: DNS API Servicelasted 1h 35m