Incidentsby GlyphOverloaded AWS Transit Gateway takes Slack down on the first workday of 2021
Jan 4, 14:57 UTCJan 4, 17:15 UTC
Duration
2h 18m
Impact
Critical
Root cause
Capacity
Slack, 90 days
7 incidents
Affected
MessagingWeb tierGlobal
Lesson: Managed network components have scaling limits too; pre-warm for known traffic spikes and make sure provisioning survives the incident it is meant to fix.
What happened
One of the AWS Transit Gateways linking Slack VPCs became overloaded as traffic returned after the holidays. Packet loss saturated the web tier, autoscaling tried to add 1,200 servers, and the provisioning service itself overloaded before the web tier recovered.
More from Slack
Full historySlack Real-Time Connection Disruptionlasted 0mSingle Sign On errorslasted 0mSlack for iOS closing unexpectedly when opening channelslasted 3d 1hFree Plan Customers Are Experiencing Message Failureslasted 6h 20mSome customers experiencing blurry image previews and download issueslasted 8h 56mTrouble loading Workspace-level Apps & Workflows settings page is returning blank pageslasted 2h 40m