Incidentsby GlyphMaintenance script deletes 883 customer sites
Apr 5, 07:38 UTCApr 18, 00:00 UTC
Duration
12d 16h
Impact
Critical
Root cause
Operator
Atlassian, 90 days
0 incidents
Affected
JiraConfluenceOpsgenieGlobal
Lesson: Deletion should be soft by default, and bulk restores need to be rehearsed at the scale of your largest possible mistake.
What happened
A script meant to deactivate a legacy app was given the IDs of entire cloud sites and run in permanent delete mode, deleting 883 sites belonging to 775 customers between 07:38 and 08:01 UTC. Some customers were down for up to 14 days while sites were restored one by one.
More from Atlassian
Full historyIssues with 403 user authentication errors across Atlassian productslasted 0mSlow loading times and extended response times for Cloud products in the EU-West region.lasted 2h 18mHigh RDS CPU on multiple environmentslasted 9h 16mSlowness in Jira.lasted 8d 3hData residency migrations halted.lasted 2d 23hSome products are hard downlasted 7h 3m
Also caused by operator action
All
AWS: Mistyped command removes S3 index servers in US-EAST-1An S3 team member ran an established playbook command with one input entered incorrectly, removing far more servers than intended, including those behind the index and placement subsystems.lasted 4h 17mPostmortem