cli: allow node status to work in unavailable/broken clusters

[petermattis]

Expand the crdb_internal.gossip_liveness table to include columns
needed to satisfy the basic usage of node status. Specifically, added
address, build, started_at, updated_at and replicas
columns. Changed node status to use gossip_liveness instead of
kv_node_status. The latter table requires the range containing the
consistent node status descriptors to be available, while
gossip_liveness only retrieves info from gossip.

node status and node status --decommission will work on
unavailable/broken clusters as long as the node they are pointed to is
up. node status {--stats,--ranges,--all} continue to require a
reasonably healthy cluster.

Fixes #16489

Release note (cli change): Enhance node status to work on
unavailable/broken clusters.

[cockroach-teamcity]

This change is 

[petermattis]

This needs a test. @knz do any of the cli tests start up multi-node clusters? If yes, can you point me towards where that is done.

[petermattis]

Rather than extend gossip_liveness, perhaps I should be adding these new fields to gossip_nodes and then joining with that table.

[petermattis]

Rather than extend gossip_liveness, perhaps I should be adding these new fields to gossip_nodes and then joining with that table.

This is done. Still need to add a test.

[bdarnell]

LGTM aside from tests. None of the CLI tests start a multi-node cluster (but test_init_command.tcl comes closest). Or you could do it in a regular test using TestCluster (like I did for the remove-dead-replicas command). Not quite end-to-end, but close enough.

Reviewed 11 of 11 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained

[knz]

Yeah the hassle to do this using a TCL test is comparable to that of a Go test, so in this case I would recommend a Go test.

It can also become an acceptance test instead of a CLI test. In acceptance tests we have multi-node clusters and we're properly testing end-to-end.

Reviewed 11 of 11 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/sql/crdb_internal.go, line 1835 at r1 (raw file):

}

// crdbInternalGossipLivenessTable exposes local information about the nodes' liveness.

1. Extend this comment to explain the information can be stale / incomplete because gossip doesn't come with a guarantee of accuracy.
2. please check if the query against this table in workloadccl/fixture.go is still correct with your changes.

[petermattis]

(I still need to add a test).

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/sql/crdb_internal.go, line 1835 at r1 (raw file):
Extended the comment, though any time the term gossip is used, the reader should understand that consistency and freshness are not present.

please check if the query against this table in workloadccl/fixture.go is still correct with your changes.

Thanks for the pointer. That query is SELECT count(node_id) FROM "".crdb_internal.gossip_liveness. Do you have any reason to suspect it wouldn't continue to work?

[petermattis]

I've added a cli/node-status roachtest and also added that roachtest to the list that is run on every CI. PTAL.

Reviewable status: complete! 0 of 0 LGTMs obtained

[bdarnell]

Reviewed 2 of 6 files at r2, 9 of 9 files at r3.
Reviewable status: complete! 0 of 0 LGTMs obtained

[petermattis]

bors r=bdarnell

[craig]

Build failed

• GitHub CI (Cockroach)

[petermattis]

bors r=bdarnell

[craig]

Build succeeded

• GitHub CI (Cockroach)