server: proactively rebalance decommissioning nodes' replicas

[aayushshah15]

Previously, when a node was marked DECOMMISSIONING, other nodes in the
system would learn about it via gossip but wouldn't do much in the way
of reacting to it. They'd rely on their replicaScanner to gradually
run into the decommissioning node's ranges and rely on their
replicateQueue to then rebalance them.

This had a few issues:

1. It meant that even when decommissioning a mostly empty node, our
   worst case lower bound for marking that node fully decommissioned was
   one full scanner interval (which is 10 minutes by default).
2. If the replicateQueue ran into an error while rebalancing a
   decommissioning replica (see kvserver: nodes flapping on their liveness can stall cluster recovery operations #79266 for instance), it would only
   retry that replica after either one full scanner interval or after
   the purgatory interval. This meant that decommissioning could take
   excessively long towards the tail end of the process.

This patch improves this behavior by installing an idempotent callback
that is invoked every time a node is detected to be DECOMMISSIONING.
This callback spins up an async task that will first proactively enqueue
all of the decommissioning nodes ranges (that have a replica on the
local node) into the local node's replicateQueues. Then, this task will
periodically nudge the decommissioning node's straggling replicas in
order to requeue them (to alleviate (2) from above).

All this is managed by a lightweight decommissionMonitor, which is
responsible for managing the lifecycle of these async tasks.

Note: This PR is an alternative to, but subsumes, #80695. The main
difference between this patch and 80695 is that, here, every node that
learns about a decommissioning node will spin up a local nudger goroutine
for it, whereas in 80695, only the node that sent the DecommissionRequest
will spin up a nudger goroutine that then sends RPCs to other nodes in the
system to enqueue the decommissioning node's replicas.

Resolves #79453

Release note: None

[cockroach-teamcity]

This change is 

[aayushshah15]

Comparing master vs this patch on a 6 node cluster with the tpcc-200 warehouse dataset:

master

time rp ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self";
0.17s user 0.15s system 0% cpu 3:47.34 total

this patch

time rp ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self";
0.07s user 0.02s system 0% cpu 48.124 total

Repro steps

USER=aayushs
CLUSTER=${USER}-decommission-test;
rp destroy $CLUSTER
rp create $CLUSTER --nodes 6
./build/builder.sh mkrelease
rp put $CLUSTER ./cockroach-linux-2.6.32-gnu-amd64 cockroach
rp start $CLUSTER

rp sql $CLUSTER:1 -- -e "SET CLUSTER SETTING kv.snapshot_recovery.max_rate='1GiB'";
rp sql $CLUSTER:1 -- -e "SET CLUSTER SETTING kv.snapshot_rebalance.max_rate='1GiB'";
rp ssh $CLUSTER:1 -- './cockroach workload fixtures import tpcc --warehouses=200 {pgurl:1} --checks=false'

time rp ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self";

[aayushshah15]

Hey @AlexTalks, this is the PR that came up in today's retro. Since you're going to be leading a lot of the decommissioning work next release, I'd appreciate any thoughts you have on this. Please let me know if any of the motivations behind this patch are unclear.

[aayushshah15]

I ran a slightly larger test comparing decommissioning nodes from a cluster containing this patch vs master and it does look promising.

Running:

USER=aayushs
CLUSTER=${USER}-decommission-test;
rp destroy $CLUSTER
rp create $CLUSTER --nodes 7 --gce-machine-type=n1-standard-16
rp stage $CLUSTER cockroach
rp start $CLUSTER

rp sql $CLUSTER:1 -- -e "SET CLUSTER SETTING kv.snapshot_recovery.max_rate='1GiB'";
rp sql $CLUSTER:1 -- -e "SET CLUSTER SETTING kv.snapshot_rebalance.max_rate='1GiB'";
rp ssh $CLUSTER:7 -- './cockroach workload fixtures import tpcc --warehouses=4000 {pgurl:1} --checks=false'

time rp ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self";

On master:

roachprod ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self"  0.76s user 0.48s system 0% cpu 13:00.18 total

On this patch:

roachprod ssh $CLUSTER:1 -- "./cockroach node decommission --insecure --self"  0.31s user 0.45s system 0% cpu 6:34.53 total

[aayushshah15]

Closing in favor of #80993