Incidentsby GlyphNew telemetry service overwhelms Kubernetes control planes
Dec 11, 23:16 UTCDec 12, 03:38 UTC
Duration
4h 22m
Impact
Critical
Root cause
Deployment
OpenAI, 90 days
80 incidents
Affected
ChatGPTAPISoraGlobal
Lesson: Test changes at production cluster size, and keep break-glass access to the control plane that does not depend on the thing that is failing.
What happened
A new telemetry service made every node run expensive Kubernetes API operations, overwhelming the API servers in the largest clusters. DNS-based service discovery depends on the control plane, so services failed as DNS caches expired, and engineers were locked out of the control plane needed to roll back.
More from OpenAI
Full historyElevated errors with image generationlasted 1h 49mElevated error rates with some U.S. business Mastercardsstill openChatGPT Work and Dots Errorslasted 13h 7mElevated error rates across paid ChatGPT planslasted 1h 54mFile upload errors in ChatGPTlasted 4h 53mElevated errors with search in Dots, ChatGPT Work, and certain ChatGPT thinking experienceslasted 1h 35m