[Europe, Japan, Australia] - [Orchestrator] - [Delays for ingestion of robot logs and search or display of the logs]

Final update

## Customer impact On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region: - Australia: approximately 15:45 UTC to 18:35 UTC - Europe: approximately 16:35 UTC to 18:00 UTC - Japan: approximately 16:45 UTC to 18:30 UTC Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated. ## Root cause We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over. In the affected regions, the traffic switch was

1. Postmortem · Oct 7, 11:08 UTC
   ## Customer impact On , between approximately 15:45 UTC and 18:45 UTC, customers hosted in the Europe, Australia and Japan regions were unable to see recent robot logs in Orchestrator. Logs written during the impact window did not appear in the Jobs and Logs views or through the Robot Logs API, and logs older than approximately 03:00 UTC that day were available only intermittently. The impact window differed per region: - Australia: approximately 15:45 UTC to 18:35 UTC - Europe: approximately 16:35 UTC to 18:00 UTC - Japan: approximately 16:45 UTC to 18:30 UTC Job execution, queues, triggers and all other Orchestrator functionality were not affected. No robot logs were lost. Logs produced during the impact window continued to be collected and became visible again once the issue was mitigated. ## Root cause We were performing a scheduled upgrade of the Elasticsearch service that stores Orchestrator robot logs. The upgrade uses a blue/green approach: a new cluster is created alongside the existing one, the data is restored into it from a backup, the pipeline that ingests new logs is started on it, and only then is traffic switched over. In the affected regions, the traffic switch was approved by the engineer running the upgrade before the data restore and the ingestion catch-up had completed, which is a deviation from our upgrade procedure. This was a human error. As a result, Orchestrator was reading robot logs from a cluster that did not yet contain recent data, while new log
2. Resolved · Oct 6, 18:44 UTC
   We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.
3. Monitoring · Oct 6, 18:43 UTC
   We have mitigated the issue impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.
4. Identified · Oct 6, 18:28 UTC
   We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.
5. Identified · Oct 6, 18:02 UTC
   We have identified the cause for an outage impacting ingestion of robot logs for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.
6. Investigating · Oct 6, 17:50 UTC
   We are investigating reports of an outage impacting [impacted functionality or user symptom] for Orchestrator in Europe. Impact: Users may expect delays for ingestion of robot logs and/or search/display of the logs. No logs will be lost. Next update: Our teams are working to understand the cause and scope and will share updates as available.