Add a guardrail to limit maximum number of shard on the cluster

[RS146BIJAY]

Is your feature request related to a problem? Please describe.
We have observed that as shard count on the cluster increases, it becomes more prone to instability and availability. We have observed multiple instances where Cx had issues in the past because of too many shards on their cluster.

Describe alternatives you've considered
As of now, OpenSearch allows us to restrict the number of shards per node using setting cluster.max_shards_per_node inside ShardLimitValidator. The maximum number of shards that can be present on the cluster is determined by maxShardsPerNode * nodeCount. Issue here is, we can increase this total shard limit on the cluster just by increasing the number of node. There is no way we can control the maximum number of shards on an entire cluster.

Describe the solution you'd like
I would suggest to add a similar setting inside ShardLimitValidator which will control the maximum number of shards that can be present on a cluster. If the total number of shards exceeds this threshold, ShardLimitValidator will block any operation which creates new shards. The threshold value will be an optional configurable cluster setting, which Cx can set dynamically.

Considerations

• Should we have some upper cap on this threshold value.

[anasalkouz]

Closing this since PR already merged

[anasalkouz]

@RS146BIJAY Do you think we should to have some smart defaults for newly created clusters based on HW specs, index size ...etc.