release-21.1: kvserver: improve suspect replica GC heuristics

[erikgrinaker]

Backport 1/1 commits from #65062.
Backport 1/1 commits from #65609.

/cc @cockroachdb/release @cockroachdb/kv

---

kvserver: improve suspect replica GC heuristics

The replica GC queue will normally check a replica against the canonical
range descriptor every 12 hours. Under some circumstances the replica
may be considered suspect, which causes it to be checked against the
canonical descriptor every second instead. However, these heuristics
were fairly limited and missed a couple of cases that could cause stale
replicas to linger.

This patch adds two conditions to the suspect replica heuristics:
followers that have lost contact with their leader (which in particular
handles non-voting replicas), and quiescent replicas that lose contact
with any other voters (which could cause false underreplication alerts).

Since this change is expected to increase suspect replica matches, the
ReplicaGCQueueSuspectCheckInterval duration between checking suspect
replica descriptors was also increased from 1 to 5 seconds, and the
replicaGCQueueTimerDuration interval between replica GCs was increased
from 50 to 100 ms.

The previous logic would take into account replica activity such as
store startup and lease proposals as the offset for timeouts, but this
did not appear to have any significant benefit over simply using the
last check time, so these have been removed and the timeouts given more
appropriate names. The previous logic also failed to enforce the check
interval for suspect replicas, and would always check them in a tight
50ms loop, this has been fixed as well.

Resolves #62075, resolves #60259.

Release note (bug fix): Improved garbage collection of stale replicas by
proactively checking certain replicas that have lost contact with other
voting replicas.

kvserver: remove replica GC heuristic for quiesced followers

In #65062 we added a condition to the replica GC queue that quiescent
followers with a single unavailable voter would be considered suspect.
This was added to try to detect followers who were partitioned away from
the Raft group during its own removal from the range. However, this case
has a very high false positive rate, and on second thought it is probably
higher than it is worth.

There is already a secondary condition that considers followers who have
lost touch with their leader suspect, which would be somewhat sufficient
in this case, and with a far lower false positive rate. Even though this
heuristic is vulnerable to race conditions, it seems a better fit
considering that in the worst case the replica will always be GCed
within 12 hours anyway. We have also since moved range-level metrics
to the leaseholder, which reduces the impact of these stale replicas.

This patch therefore removes the quiescent replica condition, and
reduces ReplicaGCQueueSuspectCheckInterval from 5 to 3 seconds since
we now expect far fewer false positives.

Touches #65202.

Release note: None

[cockroach-teamcity]

This change is