Who is down right now

Down now20This week248

20 down now · 248 this week

AWS · Cloud and hostingFeb 28, 2017, 17:37 UTC 9 years ago

Mistyped command removes S3 index servers in US-EAST-1

CriticalOperatorUpdated 46d ago
Feb 28, 17:37 UTCFeb 28, 21:54 UTC
Duration
4h 17m
Impact
Critical
Root cause
Operator
AWS, 90 days
8 incidents
Affected
S3Service Health Dashboardus-east-1
Read the postmortemStatus page

Lesson: Tools should refuse to remove capacity below a safe minimum, and the status page cannot depend on the system it reports on.

What happened

An S3 team member ran an established playbook command with one input entered incorrectly, removing far more servers than intended, including those behind the index and placement subsystems. Both needed full restarts; the status dashboard itself depended on S3.

More from AWS

Full history
MinorNov 6, 22:51 UTC
Elevated Launch Failureslasted 2h 21m
MinorOct 30, 01:28 UTC
Increased invoke error rateslasted 0m
MinorOct 4, 16:14 UTC
Elevated packet losslasted 2h 42m
MinorSep 21, 23:26 UTC
Increased Error Rateslasted 58m
MinorSep 3, 21:49 UTC
Increased API Error Rateslasted 2h 18m
MinorAug 21, 02:02 UTC
Increased Error Rateslasted 38m

Also caused by operator action

All
MajorFeb 6, 2025, 08:14 UTC
Cloudflare: R2 object storage disabled during a phishing report remediationWhile remediating a phishing URL report, an employee disabled the R2 Gateway service instead of a single bucket.lasted 1h 22mPostmortem
CriticalApr 5, 2022, 07:38 UTC
Atlassian: Maintenance script deletes 883 customer sitesA script meant to deactivate a legacy app was given the IDs of entire cloud sites and run in permanent delete mode, deleting 883 sites belonging to 775 customers between 07:38 and 08:01 UTC.lasted 12d 16hPostmortem
CriticalJan 31, 2017, 23:00 UTC
GitLab: Primary database data accidentally deleted, 18-hour restoreWhile fighting replication lag under spam load, an engineer accidentally removed data from the primary database server.lasted 19hPostmortem