Incidentsby GlyphOS update breaks networking across regions
Mar 8, 06:03 UTCMar 9, 08:58 UTC
Duration
26h 55m
Impact
Critical
Root cause
Deployment
Datadog, 90 days
9 incidents
Affected
Web appIngestionMonitorsAPMUS1US3US5EU1
Lesson: Automatic updates are deployments: stagger them, and never let the same change land on every region in the same hour.
What happened
An automatic operating system security update restarted a networking component on many hosts at once, removing network routes and disconnecting nodes across several regions and cloud providers. All services were operational on March 9 at 08:58 UTC, with historical data backfilled by March 10.
More from Datadog
Full historyMetrics query are disruptedlasted 2h 29mElevated Error Rates for Workflow Automation, App Builder, Bits Code, Bits Agent Builderlasted 57mDelayed RUM sessionslasted 2h 17mDelayed Eventslasted 48mUsers are unable to acknowledge, escalate, or resolve On-Call Pageslasted 27mDelayed Monitors Notificationslasted 46m