GitLab · Developer toolsJan 31, 2017, 23:00 UTC

Primary database data accidentally deleted, 18-hour restore

CriticalOperatorUpdated 46d ago

Jan 31, 23:00 UTCFeb 1, 18:00 UTC

Duration

19h

Impact

Critical

Root cause

Operator

GitLab, 90 days

0 incidents

Affected

GitLab.comGlobal

Read the postmortem Status page

Lesson: A backup is only real once a restore has been tested; destructive commands on production hosts need an unmistakable prompt.

What happened

While fighting replication lag under spam load, an engineer accidentally removed data from the primary database server. Backups had not been working, so GitLab.com was restored from a six-hour-old staging snapshot on slower disks, losing projects, issues and comments made in that window.

Also caused by operator action

All

MajorFeb 6, 2025, 08:14 UTC

Cloudflare: R2 object storage disabled during a phishing report remediationWhile remediating a phishing URL report, an employee disabled the R2 Gateway service instead of a single bucket.lasted 1h 22mPostmortem

CriticalApr 5, 2022, 07:38 UTC

Atlassian: Maintenance script deletes 883 customer sitesA script meant to deactivate a legacy app was given the IDs of entire cloud sites and run in permanent delete mode, deleting 883 sites belonging to 775 customers between 07:38 and 08:01 UTC.lasted 12d 16hPostmortem

CriticalFeb 28, 2017, 17:37 UTC

AWS: Mistyped command removes S3 index servers in US-EAST-1An S3 team member ran an established playbook command with one input entered incorrectly, removing far more servers than intended, including those behind the index and placement subsystems.lasted 4h 17mPostmortem