loqrecovery,cli: add debug recover verify command

[aliher1911]

This commit adds debug recovery verify command which provides
the status of loss of quorum recovery plan application status.
The command is used after debug recover apply-plan was used to
stage a recovery plan on a cluster to check application progress.
It allows user to check which nodes still needs to be restarted,
outcome of recovery on restarted nodes and health of ranges on
the entire cluster.

Release note: None

Fixes #93043

[cockroach-teamcity]

This change is 

[aliher1911]

Might need to disable stress race on some integration tests that need to wait for meta queries to time out. Will do that once next round of CI runs is complete.

[aliher1911]

bors r=erikgrinaker

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[knz]

Reviewed 1 of 21 files at r1, 2 of 10 files at r2.
Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/testutils/listener.go line 38 at r2 (raw file):

}

func NewListenerRegistry() ListenerRegistry {

Please add explanatory comments on all the exported methods in this file in a follow-up PR. I am surprised this wasn't caught by our linter.

---

pkg/testutils/listener.go line 155 at r2 (raw file):

}

// Addr implements net.Listener interface

nit: period at end of comment.

---

pkg/util/grpcutil/grpc_util.go line 69 at r2 (raw file):

func IsConnectionUnavailable(err error) bool {
	if s, ok := status.FromError(errors.UnwrapAll(err)); ok {
		return s.Code() == codes.Unavailable || s.Code() == codes.FailedPrecondition

@aliher1911 what is this change about? I don't see why it is necessary in the rest of the PR. Also, the comment above the function has become stale. I would welcome a follow-up PR which clarifies both.

[aliher1911]

pkg/util/grpcutil/grpc_util.go line 69 at r2 (raw file):

Previously, knz (Raphael 'kena' Poss) wrote…

@aliher1911 what is this change about? I don't see why it is necessary in the rest of the PR. Also, the comment above the function has become stale. I would welcome a follow-up PR which clarifies both.

This method is used to check if remote servers are unavailable, but it was not working correctly when node itself blacklists remote end regardless if it is available or not. For the sake of check those servers are unavailable as well because cluster would never be able to connect. I'll add clarification comment in the follow up PR.