[DOC] Add documentation for primary shard based balancing

[dreamer-89]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
Coming from opensearch-project/OpenSearch#5240 and opensearch-project/OpenSearch#6210, a new dynamic boolean type setting cluster.routing.allocation.balance.prefer_primary was introduced that defines if primary shards based balancing is desired i.e. whether primary shards should be distributed equally among your cluster nodes. Enabling this setting does not always guarantee equal number of primary shards on each node. This balance can't be hold specially post failover (node left/removed) which results in one node having more number of primary while other node contains the replica copy. In this situation, SameShardAllocationDecider prevents the shard relocations. This limitation can be addressed in future work as part of opensearch-project/OpenSearch#6642 and opensearch-project/OpenSearch#6481

Note: Please note this setting is intended only for segment replication feature which is still experimental.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
opensearch-project/OpenSearch#5240
opensearch-project/OpenSearch#6210

[Naarcha-AWS]

@dreamer-89: Is this issue on the roadmap for 2.6?

[dreamer-89]

@dreamer-89: Is this issue on the roadmap for 2.6?

Hi @Naarcha-AWS, thanks for the comment. I am working with @anasalkouz to get opensearch-project/OpenSearch#5240 added to 2.6.0 roadmap

[kolchfa-aws]

No PR yet but on track

[kolchfa-aws]

@dreamer-89 @anasalkouz Could you confirm that this is still going into 2.6?

[dreamer-89]

Hi @kolchfa-aws,

Thanks for checking on this. This actual feature change is under development and tracked in opensearch-project/OpenSearch#6210.
The original change which introduced the setting cluster.routing.allocation.balance.primary is already in 2.6 and may be removed as part of on-going work in 2.7. There are couple of options here:

1. Document this setting as under development feature and mark non-ready for production; with a note and that it can be removed in future.
2. Revert this change and thus remove the setting from 2.6. Checking with current core-oncall @kartg on this as we are past code freeze timeline.

CC @anasalkouz @mch2

[anasalkouz]

@dreamer-89 does keep the code will break any functionality for either users who is using SegRep or for users who don't use SegRep?
If it's not impacting them, I don't see any need to revert the changes, and I prefer to postpone the documentation till 2.7.

[dreamer-89]

@dreamer-89 does keep the code will break any functionality for either users who is using SegRep or for users who don't use SegRep? If it's not impacting them, I don't see any need to revert the changes, and I prefer to postpone the documentation till 2.7.

@anasalkouz : No the change is isolated. It doesn't break any functionality for both SegRep and DocRep. Thus, it wouldn't impact.

I think this should be reverted because once setting is exposed, removing it again is not a great idea even if it is marked experimental. Not exposing it altogether is better when it is identified that it may be removed later.

[dreamer-89]

@kolchfa-aws: Thanks for the heads up on this issue. We were able to timely remove the changes from going into 2.6.

Revert PR opensearch-project/OpenSearch#6444. Keeping this issue open for documentation for new setting introduced with opensearch-project/OpenSearch#6210.

[kolchfa-aws]

Thank you, @dreamer-89