[Remote Store Migration] Skip segrep lag computation for shard copies on docrep nodes

[shourya035]

Description

During the dual replication phase, we skip creating checkpoint timers for replication group members that are still left behind on docrep nodes and are yet to mover over to remote enabled nodes.

However, the segrep bytes behind calculation logic still considers those shard copies. Since those shards are still on docrep and doesn't understand the Segrep checkpoints, they add up to the bytes_behind computation logic, in turn exhibiting false sense of replication lag during the remote store migration process.

With this PR, if the primary shard:

• is allocated to a remote enabled node
  and
• does not have remote store based index settings (which means remote migration for that index is not yet complete)

we are skipping those allocationIds while calculating segment replication lag metrics

Added a new IT through which this issue was readily reproduced without the current code change

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 819ffae: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[ashking94]

lgtm overall

[github-actions]

❌ Gradle check result for 45bfacb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 23b6173: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 23b6173: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 71.65%. Comparing base (b15cb0c) to head (23b6173).
Report is 394 commits behind head on main.

❗ Current head 23b6173 differs from pull request most recent head ad5bede

Please upload reports for the commit ad5bede to get more accurate results.

Files	Patch %	Lines
...org/opensearch/index/seqno/ReplicationTracker.java	0.00%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #14119      +/-   ##
============================================
+ Coverage     71.42%   71.65%   +0.23%     
- Complexity    59978    61882    +1904     
============================================
  Files          4985     5113     +128     
  Lines        282275   290792    +8517     
  Branches      40946    41990    +1044     
============================================
+ Hits         201603   208374    +6771     
- Misses        63999    65217    +1218     
- Partials      16673    17201     +528     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for ad5bede: SUCCESS