Update decommissioning process info

[erikgrinaker]

Erik Grinaker (erikgrinaker) commented:

Node decommissioning has seen some changes recently, e.g. in cockroachdb/cockroach#56529, that should be reflected in the 21.1 documentation:

https://www.cockroachlabs.com/docs/dev/remove-nodes.html

A node's status entry is now removed when the decommissioning operation completes (unless something goes wrong, in which case the operation can be run again). This means that the node will not show up under "Recently Decommissioned Nodes", since there is no longer an entry to show there. This needs to be updated a few places in the document (just search for "recently decommissioned"). I suppose the mentioned "historical list of decommissioned nodes" no longer applies either.

We may want to make some changes to DB Console following this as well.

Jira Issue: DOC-1022

[blathers-crl]

Hi @erikgrinaker, please add a C-ategory label to your issue. Check out the label system docs.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[taroface]

cc @thtruo @dhartunian on the need to update DB Console. Thank you for filing @erikgrinaker !

[erikgrinaker]

Thanks! I opened cockroachdb/cockroach#61812 for the DB Console work, but see that the backlink here didn't work since I moved the issue from the cockroach repo.

[erikgrinaker]

Hey @taroface, can we get this updated please? The docs are currently incorrrect.

[exalate-issue-sync]

Ryan Kuo (taroface) commented:
Erik Grinaker Yes, in the process as we speak. I’ve linked this to a larger overhaul of the drain/decommission docs. Thank you for bringing this back up!

Two questions:

• Is a node still considered to be successfully decommissioned after the {{server.time_until_store_dead}} duration; i.e., is the node status removed after this duration?
• Does this mean that the Recently Decommissioned Nodes and historical list of decommissioned nodes will never show nodes?

cc Cameron Nunez David Hartunian Tommy Truongchau in case you know the answer to the above

[exalate-issue-sync]

Erik Grinaker (erikgrinaker) commented:
Great, thanks!

• Is a node still considered to be successfully decommissioned after the {{server.time_until_store_dead}} duration; i.e., is the node status removed after this duration?

No, this is unrelated. At the end of the decommission command, it marks the node as decommissioned and removes its status entry. When the command completes, the decommissioning is complete.

Dead nodes has to do with nodes that are unavailable. After a certain time, we mark it as dead and will upreplicate its replicas to other nodes. A dead node can still come back online if it recovers, and if not it will need to be decommissioned via the decommission command as any other live node would.

• Does this mean that the Recently Decommissioned Nodes and historical list of decommissioned nodes will never show nodes?

It depends on where the DB console fetches their information from. Last time I checked it used the status entry, which is now removed during decommissioning, so they won’t show up. If they instead look at the liveness entry (which is not removed) they can show a list of decommissioned nodes if they want to. I’ll defer to the DB Console team on this (and note that the liveness entry is always the authoritative source for cluster membership, the status entry is secondary).