kvcoord: handle ReplicaUnavailableError

[tbg]

Related to #33007.

Once #71806 lands, replicas (regardless of Raft and Lease status) unable to access the replication layer will fail-fast requests. #74500 will allow DistSender to detect these errors and we could make use of that to improve the UX in cases in which DistSender has speculatively tried a Replica, received a fail-fast error, but has other options.
In particular, we may want to avoid returning fail-fast errors up the stack for follower reads, since we can always contact the leaseholder. On the other hand, one of the common reasons for fail-fast errors is loss of quorum, and this will affect all replicas in the same way and trying the leaseholder only adds latency, which may be undesirable. There is a trade-off to make here between hoping to get a result yet and letting the client handle the fail-fast error as soon as possible. Thinking this through a bit, it is almost as though we really want DistSender to have its own notion of circuit breaking on a per-Range basis, where an async probe periodically checks whether it can contact the Range. This is related

We may want to make sure to try other replicas when receiving a fail-fast error and only return when we've exhausted all potential options. But we must also avoid infinite redirect loops: if the leaseholder returns fail-fast errors, and the other replicas return a NotLeaseholderError pointing back at the leaseholder, we need to propagate the failure upwards. This is somewhat similar to #74503 (the case in which all replicas are gone and DistSender retries forever anyway).

Jira issue: CRDB-12122

[tbg]

The error will be returned as br.Error here:

cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go

Lines 2148 to 2157 in d54e0dd

	default:
	if ambiguousError != nil {
	return nil, roachpb.NewAmbiguousResultErrorf("error=%s [propagate]", ambiguousError)
	}
	// The error received is likely not specific to this
	// replica, so we should return it instead of trying other
	// replicas.
	return br, nil
	}

which would be the starting point for making behavioral changes.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!