kvclient: don't send follower reads to unhealthy nodes

[andrewbaptist]

Describe the problem

A draining node is likely to shut down soon. By moving the leaseholders off we stop all non-follower reads to that node, but we don't prevent follower reads from using it. SInce we already know that it is draining, we should either exclude or prioritize last the draining or suspect nodes. This could be enhanced to also exclude nodes with IO overload or generally take the health of the node into account when deciding which node to send a follower read to.

To Reproduce

1. Create a CRDB cluster with 6 nodes.
2. Run a workload of only follower reads at a low rate.
3. Drain and then stop one of the nodes.
4. Notice the P99.99 ~3s latency at the time the node is stopped due to read attempts waiting for an RPC timeout.

Expected behavior

From a client perspective, draining a node should allow clean shutdown without a performance impact. Follower reads break this model because while all the leases are moved, so normal reads and writes do not require this node, follower reads don't take this into account.

Additional context
This was noticed in a customer investigation where they had a 500ms client timeout on read requests and saw an elevated number of failures when they were cycling nodes.

Jira issue: CRDB-32365

[lyang24]

Excluding the draining node might be easier approach compare to tweak with replica sorting.

If we are adding to node health or io overload to the sorting equation. Should we make modifications to RoutingPolicy enum to reflect new semantic as well? I guess the sorting logic maybe be in OptimizeReplicaOrder function but I am not experienced enough to look for where to collect node health and IO overload info.

This is an interesting issue - I'm excited to read over the future pr that fixes this issue.

[amaddahian]

This issue was discussed on Roblox monthly call and has been flagged as a blocker for which we need to provide an ETA and/or target release in 23.x.