Robot live stream issues in US region

Final update

## Customer impact Between 14:14 UTC and approximately 21:50 UTC on , customers in the US region were unable to start or view robot live streams. Robot job execution was not affected. ## Root cause On , we performed a planned rotation of the keys used to authorize live-stream connections. The US live streaming service loads these keys only at startup and had not been restarted after the new key was provisioned. As a result, the US live streaming service could not validate newly issued access tokens and rejected all connection attempts. ## Detection The issue was reported by customers as robot live streaming failing to connect in the US region. Our internal monitoring also detected the increase in errors. ## Response Our engineering team correlated the authorization failures with the key rotation and performed a rolling restart of the US live streaming service so it loaded the current keys. Errors stopped at approximately 21:50 UTC. After confirming live streams were connecting successfully, we resolved the incident at 22:50 UTC. ## Follow up To prevent recurrence of this issue, we are implementing several improvements: - Enable the live streaming service to reload keys without requ

1. Postmortem · Oct 7, 10:20 UTC
   ## Customer impact Between 14:14 UTC and approximately 21:50 UTC on , customers in the US region were unable to start or view robot live streams. Robot job execution was not affected. ## Root cause On , we performed a planned rotation of the keys used to authorize live-stream connections. The US live streaming service loads these keys only at startup and had not been restarted after the new key was provisioned. As a result, the US live streaming service could not validate newly issued access tokens and rejected all connection attempts. ## Detection The issue was reported by customers as robot live streaming failing to connect in the US region. Our internal monitoring also detected the increase in errors. ## Response Our engineering team correlated the authorization failures with the key rotation and performed a rolling restart of the US live streaming service so it loaded the current keys. Errors stopped at approximately 21:50 UTC. After confirming live streams were connecting successfully, we resolved the incident at 22:50 UTC. ## Follow up To prevent recurrence of this issue, we are implementing several improvements: - Enable the live streaming service to reload keys without requiring a restart. - Recalibrate alerting and monitoring mechanism.
2. Resolved · Sep 24, 22:50 UTC
   The issue affecting robot live streaming in the US region has been resolved. Live streaming is working as expected. We apologize for any inconvenience.
3. Monitoring · Sep 24, 22:24 UTC
   We've identified the cause of the robot live streaming issues in the US region and applied a fix. We're actively monitoring to confirm the issue is fully resolved.
4. Investigating · Sep 24, 21:45 UTC
   We're investigating issues with robot live streaming in the US region. We'll share an update as soon as we have more information.