Customer configuration triggers latent bug, 85% of network errors

Lesson: Customer configuration is untrusted input to a shared fleet; isolate its blast radius and test the bug classes it can reach.

What happened

A software deployment on May 12 introduced a bug that a specific customer configuration could trigger. On June 8 a customer pushed a valid configuration change that did, and 85% of the network returned errors. Most services recovered by 11:00 UTC; the incident was mitigated at 12:35 UTC.