webui: update to handle removal of decommissioned nodes

[erikgrinaker]

In #56529 we remove a node's status entry once it's decommissioned. This means that it no longer shows up in neither "Recently Decommissioned Nodes", nor the "Decommissioned Node History" in the UI. We may want to either remove these views, or change them to use ephemeral info from gossip liveness (e.g. crdb_internal.gossip_liveness) rather than the status entry (e.g. crdb_internal.kv_node_status).

Also requested a docs update in #61808, should coordinate any follow-up actions.

Epic CRDB-10792

[irfansharif]

Better yet, we should use the contents of node liveness directly (not the ephemeral gossip state): crdb_internal.kv_node_liveness. Probably we should expose a proper API for it if being depended on by the UI.

[irfansharif]

Related is #50707, which in hindsight is a very badly worded issue. Really we should purge all uses of these status entries in favor of liveness entries.

[tbg]

internal slack thread on the issue. I think the current presentation is actively dangerous. The UI will tell users that nodes that are actively undergoing the decommissioning process are "decommissioned" which users may interpret as them being removable from the cluster. This could lead to loss of quorum scenarios, as removing live nodes that houses replicas (which a decommissioning node is) in sufficient numbers does. As a quick fix, we can hide this section altogether.

[zachlite]

I ran into some confusion while going through the decommissioning lifecycle. @erikgrinaker can you advise?

I started to decommission a node, but when the node reached zero replicas, it disappeared from the Node List on the Overview page. I would have expected it to remain on the Node List with a DECOMMISSIONING label. The decommissioning node was also absent from the status report given by cockroach node status. Again, I would have expected the report to include all 4 nodes. Looking at my process manager, I could see all 4 nodes running.

Additionally, querying crdb_internal.kv_node_liveness yielded 4 nodes, with the decommissioning node's membership value set as decommissioned. Isn't a node supposed to be decommissioning until the node is stopped?

This was tested using a local 4 node cluster running 21.2.3 via roachprod.

[tbg]

No, the decommissioning process starts out marking the node as decommissioning, meaning replicas are moved off of it. Once zero replicas is reached, a transition of decommissioning -> decommissioned takes place. At this point, the node is hard-excluded from talking to the cluster, and the clusters considers this node to be no longer in existence. Whether the node is stopped or not plays no role in that. Recall that a node may be stopped throughout the entire decommissioning process (say it has a hardware failure).

[tbg]

The way to interpret

meaning replicas are moved off of it

in the latter case of a down node is that the rest of the cluster realizes that this node is not coming back, and will update their replica placement to make additional replicas elsewhere.

[zachlite]

Once zero replicas is reached, a transition of decommissioning -> decommissioned takes place.

Is the documentation using different semantics? It seems like a conflicting definition is given:

A node is considered to be decommissioned when it meets two criteria:

1. The node has completed the decommissioning process.
2. The node has been stopped and has not updated its liveness record for the duration configured via server.time_until_store_dead, which defaults to 5 minutes.

[erikgrinaker]

Yes, that is outdated. I opened a docs issue to fix it way back when we changed this, but it hasn't been done yet:

cockroachdb/docs#9968

I'll ping the docs team about it.