Add search API param to specify read consistency level

[mch2]

Is your feature request related to a problem? Please describe.
Today to achieve strong read consistency the system exposes refresh policies on the write path to guarantee shards have refreshed on newly indexed documents before the requests are ack'd.

Segment replication does not support these policies and as such does not provide a mechanism for guaranteed read/write consistency on the write path. More on this below in additional context.

As a workaround, the current suggestion is to use the prefer primary preference option when performing reads. This works, but is not ideal for clients/plugins in that it requires prior knowledge of the replication strategy used on the index. Further it applies to both replication strategies and is not required for docrep.

Describe the solution you'd like
As an alternative, I propose we add a new expert level parameter that specifies the read consistency requirements.

I am thinking:

..setConsistencyPolicy(ConsistencyPolicy.STRONG);
or 
..setConsistencyPolicy(ConsistencyPolicy.EVENTUAL);

where eventual is the default. This would have the following applications:

For segrep indices, we internally route to the primary shard if it is determined that replicas are behind.
For docrep indices there is no change required. Newly indexed documents will already be refreshed on if refresh is pending.
Edit - There is a small nuance here and I think we would need to apply a manual refresh for docrep indices when STRONG is used. Today on the read path, DocRep indices will locally refresh only if search idle state is enabled, not if there are pending documents. If scheduled refresh has not yet run, it is possible that documents that have been written with no refresh policy (default) would not be returned. The impact here is that the reads using the STRONG policy would trigger a refresh before serving the request for it to truly be serving a strong read. Otherwise it will return only the latest refreshed on documents.

Describe alternatives you've considered
Do nothing.
Revisit supporting existing write level mechanisms (wait/immediate refresh policies) with segRep with tight constraints.
Blocking the reads until replicas have caught up, significantly impacting read latency.

Additional context
#6045
#8700

[andrross]

There is a small nuance here and I think we would need to apply a manual refresh for docrep indices when STRONG is used

This "strong" consistency proposal seems like a new feature for both seg rep and doc rep, because of this subtlety. My instinct is not to introduce something like this because OpenSearch isn't really the right tool for use cases that need strong consistency at scale.

Revisit supporting existing write level mechanisms (wait/immediate refresh policies) with segRep with tight constraints.

I'd be inclined to pursue this approach. Ideally we'd keep the semantics of all these options identical regardless of replication method, even if performance characteristics are different.

[dblock]

Is there a case to implement this as a parameter that specifies how many replicas need to ack the write instead of calling it "strong"? Thinking of something more similar to MongoDB write concern.