[Remote Store] Fix stuck segments upload issue

[ashking94]

Description

There is a use case where when segments are getting uploaded to remote store and a flush/force merge happens, there is a possibility that the segment tracker's data can get erased while there is async refresh retry is happening. This leads to latch not getting counted down.

OpenSearch/server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

Lines 388 to 403 in a2febe9

	for (String src : filteredFiles) {
	// Initializing listener here to ensure that the stats increment operations are thread-safe
	UploadListener statsListener = createUploadListener();
	ActionListener<Void> aggregatedListener = ActionListener.wrap(resp -> {
	statsListener.onSuccess(src);
	batchUploadListener.onResponse(resp);
	}, ex -> {
	logger.warn(() -> new ParameterizedMessage("Exception: [{}] while uploading segment files", ex), ex);
	if (ex instanceof CorruptIndexException) {
	indexShard.failShard(ex.getMessage(), ex);
	}
	statsListener.onFailure(src);
	batchUploadListener.onFailure(ex);
	});
	statsListener.beforeUpload(src);
	remoteDirectory.copyFrom(storeDirectory, src, IOContext.DEFAULT, aggregatedListener);

Related Issues

Resolves #11020

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29347/
• CommitID: 6572fb0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change ee651e7

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29439/
• CommitID: ee651e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29439/
• CommitID: ee651e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Known flaky tests - #10193, #9499

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29465/
• CommitID: ee651e7
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[dblock]

@ashking94 Please see through gradle check failures above?

[dblock]

There should be a test that reproduces the bug as described and fails without the fix, please.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[ticheng-aws]

Hi @ashking94, the PR is stalled. Do we have any updates?

[ashking94]

Hi @ashking94, the PR is stalled. Do we have any updates?

I plan to take it up real soon.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[ashking94]

Part problem that this PR is solving has been fixed with #11896 already. Closing this PR.