Add a guardrail to limit maximum number of shard on the cluster

[RS146BIJAY]

Signed-off-by: Rishav Sagar [email protected]

Description

We have observed that as shard count on the cluster increases, it becomes more prone to instability and availability. We have observed multiple instances where user had issues in the past because of too many shards on their cluster.

As of now, OpenSearch allows us to restrict the number of shards per node using setting cluster.max_shards_per_node inside ShardLimitValidator. The maximum number of shards that can be present on the cluster is determined by maxShardsPerNode * nodeCount. Issue here is, we can increase this total shard limit on the cluster just by increasing the number of node. There is no way we can control the maximum number of shards on an entire cluster.

I would suggest to add a similar setting inside ShardLimitValidator which will control the maximum number of shards that can be present on a cluster. If the total number of shards exceeds this threshold, ShardLimitValidator will block any operation which creates new shards. The threshold value will be an optional configurable cluster setting, which user can set dynamically.

Issues Resolved

#6050

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10418/
• CommitID: 5fcf038
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10419/
• CommitID: 0ee3948
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10422/
• CommitID: 0d89de4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10423/
• CommitID: ad53c45

[Bukhtawar]

Overall LGTM, ensure we handle min case of node and cluster limit

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10466/
• CommitID: 9e5a63e

[codecov-commenter]

Codecov Report

Merging #6143 (b1397cf) into main (c5a1bdf) will decrease coverage by 0.01%.
The diff coverage is 87.09%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6143      +/-   ##
============================================
- Coverage     70.87%   70.86%   -0.01%     
+ Complexity    58830    58823       -7     
============================================
  Files          4776     4776              
  Lines        280993   281020      +27     
  Branches      40598    40599       +1     
============================================
  Hits         199141   199141              
- Misses        65488    65557      +69     
+ Partials      16364    16322      -42     

Impacted Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	91.89% <ø> (ø)
...va/org/opensearch/indices/ShardLimitValidator.java	92.68% <87.09%> (-3.69%)	⬇️
.../index/shard/IndexShardNotRecoveringException.java	0.00% <0.00%> (-100.00%)	⬇️
...adonly/AddIndexBlockClusterStateUpdateRequest.java	0.00% <0.00%> (-75.00%)	⬇️
...readonly/TransportVerifyShardIndexBlockAction.java	9.75% <0.00%> (-58.54%)	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	60.00% <0.00%> (-40.00%)	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	60.71% <0.00%> (-39.29%)	⬇️
...indices/readonly/TransportAddIndexBlockAction.java	20.68% <0.00%> (-37.94%)	⬇️
...luster/routing/allocation/RoutingExplanations.java	62.06% <0.00%> (-37.94%)	⬇️
...cluster/routing/allocation/RerouteExplanation.java	65.00% <0.00%> (-35.00%)	⬇️
... and 453 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[ashking94]

Curious on how would this behave since the node count is dynamic. Lets says initially max shard per node is 100 and there are 10 nodes with max shard per cluster as 1010. Now another node is added to the cluster in which case 1010 (max shard per cluster) is lesser than 1000*11. May be we can just put a condition as mentioned in this comment and remove this condition here? wdyt @sachinpkale @Bukhtawar @reta?

[ashking94]

and maxShardsInCluster = Math.min(maxShardsInCluster, maxShardsPerNode * nodeCount) can stay as a single else block.

[RS146BIJAY]

Addressed the comment.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10481/
• CommitID: 367840e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/10584/
• CommitID: dd5e23d
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/10586/
• CommitID: b1397cf

[Bukhtawar]

LGTM