debug: The problemranges debug page hangs if nodes aren't responsive

[a-robinson]

On a cluster that's having some problems (e.g. the one from #15341), the problem ranges page is really tough to get to actually load. It just hangs for minutes on end, presumably because one or more of the nodes is being unresponsive. Assuming that this is the problem, the logic for rendering the page should just cut its losses at some point, return what it knows, and warn the user that the page is missing information from such-and-such nodes.

This doesn't appear to be deterministic though -- sometimes the page finishes loading in tens of seconds, and other times it continues hanging. It might just be a matter of the unhealthy nodes sometimes responding and sometimes not, but I'm not sure.

[BramGruneir]

One thing you can try as a temporary fix, is to look at just a specific node.
debug/problemranges?node_id=N

[a-robinson]

Yup, that did work much better.

[BramGruneir]

So I was looking into this. Right now, we only send requests to live nodes, and if there's an issue reading from the node liveness, we might not send any requests at all. This also means that nodes have a flaky liveness record, they may or may not be included in the request.

So I can forgo the liveness check and have it just try to hit every node with our standard timeout. And report the results of each attempted connection similar to how the new range page does it. See the last table in the pages #17433.

Not sure if that will solve the problem, but it should make it a bit easier to work with (and provide more info).

[BramGruneir]

So I've merged #17913 that should do a better job of dealing with down nodes on the problem range report.

Reopen this if there are new issues.

Bram