Replication bug causing slow recoveries

Final update

This issue was resolved in the 9.5.3 release of Elasticsearch.  Customers running versions 9.5.0, 9.5.1, or 9.5.2 of Elasticsearch are advised to update to a newer version as soon as possible.

1. Resolved · Sep 30, 20:44 UTC
   This issue was resolved in the 9.5.3 release of Elasticsearch.  Customers running versions 9.5.0, 9.5.1, or 9.5.2 of Elasticsearch are advised to update to a newer version as soon as possible.
2. Monitoring · Sep 3, 15:47 UTC
   Elasticsearch 9.5.3 has been released and all customers currently running 9.5.0, 9.5.1 or 9.5.2 are advised to update to this new version as soon as possible. Impacted customers are advised to follow the workaround steps from https://github.com/elastic/elasticsearch/issues/158212 after the upgrade.
3. Monitoring · Sep 3, 12:13 UTC
   Elasticsearch 9.5.3 has been released and all customers currently running 9.5.0, 9.5.1 or 9.5.2 are advised to update to this new version as soon as possible.
4. Identified · Sep 1, 15:46 UTC
   The fix has been identified and backported to the Elasticsearch 9.5 patch branch. The teams are performing the needed validations before scheduling and rolling out the 9.5.3 release.
5. Identified · Sep 1, 11:07 UTC
   We have identified the root cause and the team is working on a fix scheduled to be released in Elasticsearch version 9.5.3 What you can do: - If you have not yet upgraded to 9.5.*, you are not affected as upgrades to 9.5 have been disabled until 9.5.3 is released - If you are already running 9.5 and experience the symptoms of this issue, please contact Elastic Support. More information can be found on https://www.elastic.co/docs/release-notes/elasticsearch/known-issues#elasticsearch-9.5.2-known-issues
6. Investigating · Sep 1, 08:16 UTC
   We are aware of a bug affecting customers on Elasticsearch versions 9.5.0, 9.5.1 and 9.5.2 where bulk index operations processed on the primary shard can result in divergence on the replicas resulting in high disk usage and long-running recoveries. The team is working on a fix. More information can be found on https://www.elastic.co/docs/release-notes/elasticsearch/known-issues#elasticsearch-9.5.2-known-issues The next update will be in the next 2 hours or sooner if needed.