Retry on failure to acquire lock on remote metadata

[sachinpkale]

Description

• If we have 2 refresh flows executing concurrently (one for scheduled refresh, another as part of flush), there is a race condition where the scheduled refresh uploads the new segments, new segments_N is created with flush and the refresh that is part of flush becomes a no-op as it does not find any new segments that are not part of the reader.
• Due to this, the metadata file corresponding to commit generation on disk is not uploaded to remote store and results in NoSuchFileException.
• In this change, we retry on such exception. This makes sure the problem will not occur even if the same scenario explained above re-occurs. Following are the reasons.
  • if the ingestion is stopped, the first refresh would be a no-op.
  • if the ingestion is on-going, the second refresh would not be a no-op.

Related Issues

• [BUG][Remote Store] Snapshot failing with metadata not found for given commit generation and Primary Term #10217

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change a186d21

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26765/
• CommitID: fe38a4b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26815/
• CommitID: fe38a4b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26826/
• CommitID: a186d21
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26830/
• CommitID: a186d21
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/26832/
• CommitID: a186d21
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/26837/
• CommitID: a186d21

[codecov]

Codecov Report

Merging #10341 (a186d21) into main (d5a95b8) will increase coverage by 0.04%.
The diff coverage is 28.57%.

@@             Coverage Diff              @@
##               main   #10341      +/-   ##
============================================
+ Coverage     71.20%   71.24%   +0.04%     
- Complexity    58298    58355      +57     
============================================
  Files          4832     4832              
  Lines        274711   274721      +10     
  Branches      40031    40031              
============================================
+ Hits         195600   195724     +124     
+ Misses        62717    62636      -81     
+ Partials      16394    16361      -33     

Files	Coverage Δ
...rg/opensearch/snapshots/SnapshotShardsService.java	69.88% <28.57%> (-8.49%)	⬇️

... and 475 files with indirect coverage changes