server: restart a large Cockroach cluster with no impact to foreground traffic

[lunevalex]

Is your feature request related to a problem? Please describe.

We recently observed a customer operating a cluster with 100 nodes. They needed to do a rolling restart of the cluster. To do that, they generated a shell script that went through all nodes, 1 at a time, then issued restart command, followed by (aggressive) 30 second sleep. This struck me as something that’s pretty risky, and not very user friendly. There are many issues here:

• Running such commands on large scale cluster will take very long time (100 node/ w 1 min sleep: 100 minutes at least)
  This is very error prone; you have to be connected all the time, you have to babysit this process, etc
• Restarting multiple nodes at the same time is not possible. Some form of cockroach supported command (either directly from the binary, as in ./cockroach restart or through k8 automation) would be able to do much better job (for example, cockroach could select multiple nodes that host disjoint sets of ranges).
• Restarting always results in a blip, particularly when restarting node hosting system ranges. This problem is captured here server,sql: complement query_wait with conn_wait to wait until clients/pool closes connections #66319.

Describe the solution you'd like

A single command is issued ./cockroach cluster restart, which cycles the whole cluster quickly and efficiently without any impact to foreground user traffic.

Ideally Cockroach would be smart enough to figure out how tor drain disjoint set of nodes quickly, and cycle through multiple nodes at a time, rather than doing it one at a time.

Describe alternatives you've considered

There is an argument to be made that this should live in an operator (i.e. k8s opeator) outside of CockroachDB.

Jira issue: CRDB-8257

[knz]

This needs an orchestration-level solution. CockroachDB nodes cannot "restart themselves".

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[knz]

xref crdb launcher proposal