more flexible follower read dispatch strategy

[glorv]

Enhancement

In the cloud environment, the tidb clusters are commonly deployed across multiple "Access Zone"(AZ for short)s. Because the traffic cost across different AZ is much higher than within the same AZ. We can reduce the cost by dispatching read request to
the TiKV/TiFlash with a higher priority.

#28033 introduced a new tidb_replica_read option "closest-replica" that always dispatch all read request to the closest TiKV. But since the "follower read" feature need to propose an extra "readIndex" message through the raft state machine. This overhead may cause performance regression. For cop requests with small response, read from local AZ can not save the traffic because the extra "readIndex" messages are always cross AZ traffic. And the extra cpu usage to process these raft message can hurt the performance.

I propose to add a new option "closest-adaptive" for tidb_replica_read variable. This new option tries to dispatch request to the nearest store only when the response size is bigger than a certain threshold which is define by a new variable "tidb_adaptive_closest_read_threshold".

BTW, when the cop traffic from tidbs in different AZ are not evenly distributed, it may cause high load in some store while other are low loaded. This can cause high tail-lantency and low overall throughput. So when follow-read is enabled, the tikv client may need to track the load of each storage and force fallback to leader read when some stores' load are high.

[xhebox]

/cc @Yisaer

[nolouch]

So when follow-read is enabled, the tikv client may need to track the load of each storage and force fallback to leader read when some stores' load are high.

How to make PD's decision and client's decision not to affect each other?

[glorv]

So when follow-read is enabled, the tikv client may need to track the load of each storage and force fallback to leader read when some stores' load are high.

How to make PD's decision and client's decision not to affect each other?

In this design, we can't. by @BusyJay 's suggestion, we can move the decision making to PD, and the client only use the pd generated result. In the way, we can further make decision both for region schedule and traffic dispatch together.

BTW, because the client can adjust it decision very quickly. So even we implement the algorithm at the client side, it not a big problem. When PD do some schedule for some hot read, the client side can adjust the portion of each peer on the fly. In my benchmark, the traffic can be balance quickly within a few minutes.