kvserver: improve suspect replica heuristics in replica GC queue

[erikgrinaker]

The kvserver.replicaGCQueue will normally check replicas against the main replica descriptor every 10 days, to see if it should be GCed. However, some replicas may be considered suspect (e.g. if they are a Raft candidate), in which case they are checked every second. These heuristics are currently pretty limited, and probably need to cover additional scenarios -- one example is a non-voting replica which has lost its leader, which currently isn't considered suspect.

cockroach/pkg/kv/kvserver/replica_gc_queue.go

Lines 159 to 190 in 885eaec

	var isSuspect bool
	if raftStatus := repl.RaftStatus(); raftStatus == nil {
	// If a replica doesn't have an active raft group, we should check
	// whether or not it is active. If not, we should process the replica
	// because it has probably already been removed from its raft group but
	// doesn't know it. Without this, node decommissioning can stall on such
	// dormant ranges. Make sure NodeLiveness isn't nil because it can be in
	// tests/benchmarks.
	if repl.store.cfg.NodeLiveness != nil {
	if liveness, ok := repl.store.cfg.NodeLiveness.Self(); ok && !liveness.Membership.Active() {
	return true, replicaGCPriorityDefault
	}
	}
	} else if t := replDesc.GetType(); t != roachpb.VOTER_FULL && t != roachpb.NON_VOTER {
	isSuspect = true
	} else {
	switch raftStatus.SoftState.RaftState {
	case raft.StateCandidate, raft.StatePreCandidate:
	isSuspect = true
	case raft.StateLeader:
	// If the replica is the leader, we check whether it has a quorum.
	// Otherwise, it's possible that e.g. Node.ResetQuorum will be used
	// to recover the range elsewhere, and we should relinquish our
	// lease and GC the range.
	if repl.store.cfg.NodeLiveness != nil {
	livenessMap := repl.store.cfg.NodeLiveness.GetIsLiveMap()
	isSuspect = !repl.Desc().Replicas().CanMakeProgress(func(d roachpb.ReplicaDescriptor) bool {
	return livenessMap[d.NodeID].IsLive
	})
	}
	}
	}

Related to #61977.