Postmortem library: the vendors' own write-ups, one lesson each

MinorNov 3

CircleCI: Increased wait time for Machine and Remote Docker jobsFrom 13:55 to 15:07 UTC on November 3, customers running Linux machine jobs and Remote Docker saw increased wait times for their jobs to start, and the delays were longest from 14:16 to 14:41.lasted 54mCapacityRead the postmortem

CriticalNov 2

Descript: Avatar OutageTeams should conduct decision first post mortems and pre mortems to turn failures into actionable learning and build trust.lasted 5h 37mRead the postmortem

MajorOct 20

Google Cloud: Multiple services were experiencing issues in the europe-west2 region.On Monday, 19 Oct 2026 at 22:53 US/Pacific (Tuesday, 20 Oct 2026, 06:53 BST), a data center facility that hosts europe-west2-b, a zone within the europe-west2 cloud region, experienced an external electrical power disruption.lasted 6h 8mPowerRead the postmortem

MajorOct 8

Google Cloud: Google Cloud Storage (GCS) customers experienced elevated latencies and error rates when performing upload or write operations.On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates while performing upload or write operations in the us-central1 region for a duration of 12 hours, 30 minutes.lasted 29h 46mCapacityRead the postmortem

CriticalSep 28

GitHub: Copilot Code Review is unable to complete reviewsOn September 28, 2026, between 19:23 UTC and 22:08 UTC the Copilot Code Review service was degraded and pull request reviews did not complete in all environments.lasted 52mRead the postmortem

CriticalSep 13

GitHub: Incident with several GitHub ServicesOn September 13, 2026, between 08:43 and 10:44 UTC, GitHub experienced degraded availability across approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup.lasted 1h 29mDatabaseRead the postmortem

MajorSep 1

Google Cloud: Multiple products in us-central1-b are experiencing network service degradation.Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected.lasted 4h 10mNetworkRead the postmortem

CriticalAug 27

GitHub: Incident with Copilot AI Model ProvidersOn August 27th, 2026, between approximately 09:20 and 12:14 UTC, the Copilot service experienced a degradation of the Kimi K3 model due to an issue with our upstream provider.lasted 2h 8mDependencyRead the postmortem

CriticalAug 26

GitHub: Incident with ActionsOn August 26, 2026 from 15:02 to 15:45 UTC, Actions jobs failed to start.lasted 2h 50mCapacityRead the postmortem

MinorAug 26

GitHub: Disruption with some GitHub servicesTeams should check platform status dashboards during general service disruptions.lasted 59mRead the postmortem

CriticalAug 20

Google Cloud: We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region.On Thursday, 20 August 2026, from 08:00 to 10:22 US/Pacific (15:00 to 17:22 UTC), multiple Google Cloud services in the us-west1 region experienced elevated latency, provisioning failures, increased error rates, and widespread service degradations for a total duration of 2 hours…lasted 3h 40mCapacityRead the postmortem

CriticalAug 20

GitHub: Intermittent failures creating agent tasksBetween 13:57 UTC on August 20 and 00:37 UTC on August 21, 2026, some users of the Copilot Cloud Agent experienced delays of up to 60 to 90 minutes in seeing the status and results of their agent tasks.lasted 9h 54mDeploymentRead the postmortem

MajorAug 19

Nebius: Region us-central1 partial degradationData center cooling failures can lead to rapid thermal shutdowns, requiring robust disaster recovery plans for whole-region power loss and manual recovery processes.lasted 21h 33mPowerRead the postmortem

CriticalAug 17

GitHub: Incident with GitHub.comOn August 17, 2026, from 13:28–21:15 UTC (7h 47m), GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot.lasted 7h 36mConfig changeRead the postmortem

CriticalAug 6

GitHub: Incident with ActionsOn August 6, 2026, between 15:05 UTC and 00:14 UTC on August 7, GitHub Actions experienced degraded availability.lasted 10h 42mBugRead the postmortem

MinorAug 6

GitHub: Incident with Pages - Deployment LagOn August 6, 2026, at 07:00 UTC, a configuration change inadvertently reduced the capacity of the service that processes GitHub Pages deployments.lasted 1h 19mRead the postmortem

MajorJul 15

Google Cloud: Google Cloud VMware Engine (GCVE), Google Cloud NetApp Volumes, and Bare Metal Solutions (BMS) services are experiencing a service outage in europe-west4-a due to a cooling failure.On Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes.lasted 12h 28mPowerRead the postmortem

MajorJul 14

Google Cloud: Google Cloud VMware Engine (GCVE) Stretched Cluster customers are experiencing zonal outages impacting network connectivity across multiple regions.On Tuesday, 14 July 2026 10:00 PT, Google Cloud VMware Engine (GCVE) Stretched Cluster customers in the australia-southeast2 and europe-west3 zones experienced inter-site communication failures.lasted 10h 40mNetworkRead the postmortem

MajorJun 25

GitHub: Degradation with Webhooks, Pull Requests and ActionsOn June 25, 2026, between 17:33 UTC and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, Actions workflows, and Webhooks, with delays peaking at 7m.lasted 37mCapacityRead the postmortem

CriticalJun 17

GitHub: Incident with Copilot AvailabilityOn June 17, 2026, between approximately 03:35 UTC and 04:44 UTC, GitHub Copilot was degraded and most of its frontier chat models were temporarily unavailable across all regions.lasted 54mConfig changeRead the postmortem

+27 more from GitHubAll GitHub postmortems

MinorFeb 27

Google Cloud: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.On Friday, 27 February 2026 at 04:37 US/Pacific, customers using Vertex AI Gemini API models experienced increased error rates.lasted 1h 58mConfig changeRead the postmortem

MajorFeb 12

GitHub: Disruption with some GitHub servicesFrom Feb 12, 2026 09:16:00 UTC to Feb 12, 2026 11:01 UTC, users attempting to download repository archives (tar.gz/zip) that include Git LFS objects received errors.lasted 34mNetworkRead the postmortem

MajorFeb 12

GitHub: Incident with CodespacesOn February 12, 2026, between 00:51 UTC and 09:35 UTC, users attempting to create or resume Codespaces experienced elevated failure rates across Europe, Asia and Australia, peaking at a 90% failure rate.lasted 2h 3mDeploymentRead the postmortem

+4 more from GitHubAll GitHub postmortems

MajorDec 5, 2025

Cloudflare: Body parsing change for a React Server Components fix causes 25-minute outageConfiguration changes to body parsing logic, made to detect and mitigate an industry-wide React Server Components vulnerability, caused a portion of the network to fail for about 25 minutes.lasted 25mConfig changeRead the postmortem

NoticeNov 18, 2025

Asana: Partial downtime for Asana's MCP serverCloudflare experienced an outage that impacted the Workers KV service, a storage infrastructure service our MCP server relies on.lasted 0mDependencyRead the postmortem

CriticalNov 18, 2025

Cloudflare: Oversized Bot Management feature file breaks the core proxyA change to database permissions made a ClickHouse query return duplicate rows, doubling the Bot Management feature file.lasted 5h 46mConfig changeRead the postmortem

CriticalOct 20, 2025

AWS: DynamoDB DNS failure takes down US-EAST-1A latent race condition in the DynamoDB DNS management system left an empty DNS record for the regional endpoint dynamodb.us-east-1.amazonaws.com, and the automation failed to repair it.lasted 14h 32mDNSRead the postmortem

MajorAug 5, 2025

Anthropic: Three infrastructure bugs intermittently degrade Claude responsesA context window routing error from August 5, an output corruption misconfiguration from August 25 and a TPU compiler bug from August 26 overlapped, degrading some responses.lasted 44dBugRead the postmortem

CriticalJul 14, 2025

Cloudflare: 1.1.1.1 public DNS resolver unreachable for 62 minutesA service topology configuration error, introduced weeks earlier with no impact, was triggered by a later change and withdrew the 1.1.1.1 prefixes from Cloudflare data centers.lasted 1h 2mConfig changeRead the postmortem

CriticalJun 12, 2025

Cloudflare: Workers KV storage failure cascades to Access, WARP and moreA failure in the third-party storage infrastructure behind Workers KV took down many products that rely on KV for configuration, authentication and asset delivery.lasted 2h 28mDependencyRead the postmortem

CriticalJun 12, 2025

Google Cloud: Service Control crash loop returns 503s across Google CloudA policy change inserted into the regional Spanner tables that Service Control reads hit an unguarded code path with a null pointer, crashing Service Control binaries worldwide.lasted 3hConfig changeRead the postmortem

MajorFeb 6, 2025

Cloudflare: R2 object storage disabled during a phishing report remediationWhile remediating a phishing URL report, an employee disabled the R2 Gateway service instead of a single bucket.lasted 1h 22mOperatorRead the postmortem

CriticalDec 11, 2024

OpenAI: New telemetry service overwhelms Kubernetes control planesA new telemetry service made every node run expensive Kubernetes API operations, overwhelming the API servers in the largest clusters.lasted 4h 22mDeploymentRead the postmortem

CriticalJul 19, 2024

CrowdStrike: Falcon Channel File 291 update crashes Windows hosts worldwideA Rapid Response Content update for the Falcon sensor defined 21 input fields while the sensor code supplied 20, causing an out-of-bounds read and a crash on Windows hosts that received it between 04:09 and 05:27 UTC.lasted 1h 18mConfig changeRead the postmortem

CriticalMar 8, 2023

Datadog: OS update breaks networking across regionsAn automatic operating system security update restarted a networking component on many hosts at once, removing network routes and disconnecting nodes across several regions and cloud providers.lasted 26h 55mDeploymentRead the postmortem

CriticalApr 5, 2022

Atlassian: Maintenance script deletes 883 customer sitesA script meant to deactivate a legacy app was given the IDs of entire cloud sites and run in permanent delete mode, deleting 883 sites belonging to 775 customers between 07:38 and 08:01 UTC.lasted 12d 16hOperatorRead the postmortem

CriticalDec 7, 2021

AWS: Internal network congestion disrupts US-EAST-1An automated activity to scale capacity of a service in the main AWS network triggered unexpected behavior from a large number of clients inside the internal network, congesting the devices between the internal and main networks.lasted 7h 10mNetworkRead the postmortem

CriticalJun 8, 2021

Fastly: Customer configuration triggers latent bug, 85% of network errorsA software deployment on May 12 introduced a bug that a specific customer configuration could trigger.lasted 2h 48mBugRead the postmortem

CriticalJan 4, 2021

Slack: Overloaded AWS Transit Gateway takes Slack down on the first workday of 2021One of the AWS Transit Gateways linking Slack VPCs became overloaded as traffic returned after the holidays.lasted 2h 18mCapacityRead the postmortem

CriticalJul 2, 2019

Cloudflare: WAF regular expression exhausts CPU worldwide for 27 minutesA new WAF rule for XSS detection contained a regular expression with catastrophic backtracking.lasted 27mDeploymentRead the postmortem

CriticalFeb 28, 2017

AWS: Mistyped command removes S3 index servers in US-EAST-1An S3 team member ran an established playbook command with one input entered incorrectly, removing far more servers than intended, including those behind the index and placement subsystems.lasted 4h 17mOperatorRead the postmortem

CriticalJan 31, 2017

GitLab: Primary database data accidentally deleted, 18-hour restoreWhile fighting replication lag under spam load, an engineer accidentally removed data from the primary database server.lasted 19hOperatorRead the postmortem