Postmortem library
66 postmortems from 13 vendors, back to 2017 · the vendors’ own write-ups, one lesson each
66 postmortems
| Date | Postmortem | Root cause | Duration | Link |
|---|---|---|---|---|
| Sep 12026 |  Google CloudMultiple products in us-central1-b are experiencing network service degradation.Teams should check service health dashboards when regional network service degradation affects cloud products.Network4h 8m | Network | Read | |
| Aug 272026 |  GitHubIncident with Copilot AI Model ProvidersEngineering teams should monitor provider health when using third party AI model services.Dependency2h 8m | Dependency | Read | |
| Aug 262026 | GitHubIncident with ActionsTeams should monitor service status when experiencing issues with automated workflow systems.Capacity2h 50m | Capacity | Read | |
| Aug 262026 | GitHubDisruption with some GitHub servicesTeams should check platform status dashboards during general service disruptions.Not disclosed59m | Not disclosed | Read | |
| Aug 202026 | Google CloudWe are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.Teams should monitor regional health alerts when experiencing elevated latencies and errors across cloud products.Capacity3h 40m | Capacity | Read | |
| Aug 202026 | GitHubIntermittent failures creating agent tasksTeams should track agent task creation pipelines to handle intermittent failures promptly.Deployment9h 54m | Deployment | Read | |
| Aug 172026 | GitHubIncident with GitHub.comEngineering teams should subscribe to platform status updates to remain informed during service incidents.Config change7h 36m | Config change | Read | |
| Aug 62026 | GitHubIncident with ActionsTeams should verify service health when experiencing disruptions with deployment actions.Bug10h 42m | Bug | Read | |
| Aug 62026 | GitHubIncident with Pages - Deployment LagTeams should account for deployment lag by monitoring static site hosting service health.Not disclosed1h 19m | Not disclosed | Read | |
| Jul 152026 | Google CloudGoogle Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.Teams should design multi region redundancy to withstand datacenter outages caused by cooling failures.Power12h 28m | Power | Read | |
| Jul 142026 | Google CloudGoogle Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions.Zonal outages can impact network connectivity across multiple regions for stretched cluster configurations.Network10h 40m | Network | Read | |
| Jun 252026 | GitHubDegradation with Webhooks, Pull Requests and ActionsService degradation can simultaneously impact multiple integrated platform features like webhooks, pull requests, and actions.Capacity37m | Capacity | Read | |
| Jun 172026 | GitHubIncident with Copilot AvailabilityAvailability issues can disrupt specific AI-powered developer tools within the broader platform ecosystem.Config change54m | Config change | Read | |
| Jun 162026 | GitHubDisruption with some GitHub servicesPlatform-wide disruptions can affect only a subset of services while others remain operational.Not disclosed30m | Not disclosed | Read | |
| Jun 102026 | GitHubAuthentication issues related to API requestsAuthentication failures can prevent successful API requests across the platform.Not disclosed1h 19m | Not disclosed | Read | |
| Jun 82026 | GitHubPull Requests and Issues unavailable for signed-out usersSpecific user segments like signed-out users may experience unique access issues to core features like pull requests and issues.Not disclosed1h 25m | Not disclosed | Read | |
| Jun 42026 | GitHubCopilot Code Review FailingAutomated code review features can fail independently of the main platform availability.Deployment1h 57m | Deployment | Read | |
| May 282026 | GitHubDisruption with OpenAI ModelsReliance on external AI models can lead to service disruptions if those models experience issues.Dependency1h 40m | Dependency | Read | |
| May 262026 | GitHubIncident with Actions and PagesMultiple related services like actions and pages can experience concurrent incidents.Not disclosed2h 21m | Not disclosed | Read | |
| May 152026 | GitHubActions is experiencing degraded availabilityAutomated workflow services can experience degraded availability independently of other platform components.Database35m | Database | Read | |
| May 72026 | GitHubCCR and CCA failing to start for PR commentsNot disclosed1h 54m | Not disclosed | Read | |
| May 62026 | GitHubIncident with Pull RequestsNot disclosed3h 39m | Not disclosed | Read | |
| May 62026 | GitHubDisruption with some GitHub servicesConfig change38m | Config change | Read | |
| May 62026 | GitHubIncident with Actions, we are investigating reports of degraded availabilityNot disclosed2h 25m | Not disclosed | Read | |
| May 52026 | GitHubIncident with ActionsNot disclosed3h 49m | Not disclosed | Read | |
| May 42026 | GitHubIncident with Issues and WebhooksNot disclosed55m | Not disclosed | Read | |
| Apr 272026 | GitHubDisruption with some GitHub servicesNot disclosed2h 14m | Not disclosed | Read | |
| Apr 272026 | GitHubGitHub search is degradedCapacity6h 15m | Capacity | Read | |
| Apr 232026 | GitHubIncident with multiple GitHub servicesDNS1h 18m | DNS | Read | |
| Apr 222026 | GitHubDisruption with Copilot chat and Copilot Coding AgentConfig change3h 43m | Config change | Read | |
| Apr 202026 | GitHubPartial degradation for code scanning default setup and for code qualityNot disclosed15h 36m | Not disclosed | Read | |
| Apr 162026 | GitHubIncident with CodespacesDependency3h 22m | Dependency | Read | |
| Apr 132026 | GitHubIncident with PagesDNS39m | DNS | Read | |
| Apr 92026 | GitHubDisruption with some GitHub servicesCapacity25m | Capacity | Read | |
| Apr 12026 | GitHubGitHub audit logs are unavailableNetwork4m | Network | Read | |
| Apr 12026 | GitHubDisruption with GitHub's code searchDeployment8h 43m | Deployment | Read | |
| Mar 242026 | GitHubTeams Github Notifications App is downNot disclosed2h 52m | Not disclosed | Read | |
| Mar 192026 | GitHubIssues with Copilot Coding AgentNot disclosed48m | Not disclosed | Read | |
| Mar 52026 | GitHubMultiple services are affected, service degradationConfig change2h 55m | Config change | Read | |
| Mar 32026 | GitHubIncident with all GitHub servicesNot disclosed1h 10m | Not disclosed | Read | |
| Feb 272026 | Google CloudVertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.Config change1h 58m | Config change | Read | |
| Feb 122026 | GitHubDisruption with some GitHub servicesNetwork34m | Network | Read | |
| Feb 122026 | GitHubIncident with CodespacesDeployment2h 3m | Deployment | Read | |
| Feb 92026 | GitHubIncident with Issues, Actions and Git OperationsNot disclosed1h 8m | Not disclosed | Read | |
| Feb 92026 | GitHubCopilot Policy Propagation DelaysNot disclosed17h 28m | Not disclosed | Read | |
| Feb 92026 | GitHubIncident with Pull RequestsNot disclosed1h 21m | Not disclosed | Read | |
| Feb 92026 | GitHubNotifications are delayedDatabase3h 35m | Database | Read | |
| Dec 52025 |  CloudflareBody parsing change for a React Server Components fix causes 25-minute outageUrgent security mitigations still need a staged rollout; speed of response is exactly when a global config push is most dangerous.Config change25m | Config change | Read | |
| Nov 182025 |  AsanaPartial downtime for Asana's MCP serverDependency0m | Dependency | Read | |
| Nov 182025 | CloudflareOversized Bot Management feature file breaks the core proxyTreat internally generated config files like user input: validate size and shape before they propagate, and keep global kill switches for each feature.Config change5h 46m | Config change | Read | |
| Oct 202025 |  AWSDynamoDB DNS failure takes down US-EAST-1Automation that manages critical DNS needs its own guard against writing an empty record, and dependent services need to recover from a stale state on their own.DNS14h 32m | DNS | Read | |
| Aug 52025 |  AnthropicThree infrastructure bugs intermittently degrade Claude responsesQuality regressions hide in normal variance; continuous evaluations on production traffic catch what user reports cannot.Bug44d | Bug | Read | |
| Jul 142025 | Cloudflare1.1.1.1 public DNS resolver unreachable for 62 minutesLatent config errors can sit for weeks; progressive deployment and legacy-system cleanup matter as much as the change that finally triggers them.Config change1h 2m | Config change | Read | |
| Jun 122025 | CloudflareWorkers KV storage failure cascades to Access, WARP and moreA shared internal primitive is a single point of failure for everything built on it; know which products have a hard dependency and give them a fallback.Dependency2h 28m | Dependency | Read | |
| Jun 122025 | Google CloudService Control crash loop returns 503s across Google CloudNew code paths belong behind feature flags, and globally replicated policy data needs staged propagation like any binary.Config change3h | Config change | Read | |
| Feb 62025 | CloudflareR2 object storage disabled during a phishing report remediationAbuse tooling needs the same guardrails as production changes: scope checks and a second pair of eyes before an action can disable a whole service.Operator1h 22m | Operator | Read | |
| Dec 112024 |  OpenAINew telemetry service overwhelms Kubernetes control planesTest changes at production cluster size, and keep break-glass access to the control plane that does not depend on the thing that is failing.Deployment4h 22m | Deployment | Read | |
| Jul 192024 |  CrowdStrikeFalcon Channel File 291 update crashes Windows hosts worldwideContent and configuration updates need the same staged rollout, validation and customer control as code releases.Config change1h 18m | Config change | Read | |
| Mar 82023 |  DatadogOS update breaks networking across regionsAutomatic updates are deployments: stagger them, and never let the same change land on every region in the same hour.Deployment26h 55m | Deployment | Read | |
| Apr 52022 |  AtlassianMaintenance script deletes 883 customer sitesDeletion should be soft by default, and bulk restores need to be rehearsed at the scale of your largest possible mistake.Operator12d 16h | Operator | Read | |
| Dec 72021 | AWSInternal network congestion disrupts US-EAST-1Retry storms turn a small change into congestion; clients need backoff, and monitoring must not share the network it monitors.Network7h 10m | Network | Read | |
| Jun 82021 |  FastlyCustomer configuration triggers latent bug, 85% of network errorsCustomer configuration is untrusted input to a shared fleet; isolate its blast radius and test the bug classes it can reach.Bug2h 48m | Bug | Read | |
| Jan 42021 |  SlackOverloaded AWS Transit Gateway takes Slack down on the first workday of 2021Managed network components have scaling limits too; pre-warm for known traffic spikes and make sure provisioning survives the incident it is meant to fix.Capacity2h 18m | Capacity | Read | |
| Jul 22019 | CloudflareWAF regular expression exhausts CPU worldwide for 27 minutesRules and regexes are code; stage them, and use an engine with guaranteed linear time for untrusted input.Deployment27m | Deployment | Read | |
| Feb 282017 | AWSMistyped command removes S3 index servers in US-EAST-1Tools should refuse to remove capacity below a safe minimum, and the status page cannot depend on the system it reports on.Operator4h 17m | Operator | Read | |
| Jan 312017 |  GitLabPrimary database data accidentally deleted, 18-hour restoreA backup is only real once a restore has been tested; destructive commands on production hosts need an unmistakable prompt.Operator19h | Operator | Read |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.