kvserver: export remaining snapshot bytes

[kvoli]

Summary
Each snapshot may be a different size, it would be beneficial to track the total remaining snapshot bytes that are queued and in progress on a store's receiver snapshot semaphore. Additionally the remaining bytes that are queued on a store's sender snapshot semaphore.

Note we currently track the current reservations in bytes, which is the current size of the snapshot(s) being processed on a store capacity.reserved.

Solution

The solution is to add four additional exported metrics, with the last two optional and pending how useful they are:

1. range.snapshots.queued-rcvd: a gauge tracking the sum of all snapshot bytes that are currently queued on a store's receive queue, however have not gotten a reservation (begun processing).
2. range.snapshots.queued-send: a gauge tracking the sum of all snapshot bytes that are currently queued on a store's send queue, however have not begun gotten a reservation (begun processing).
3. range.snapshots.pending-rcvd: a gauge tracking the sum of all snapshot bytes that remain on a store's receiving side, for snapshots that have acquired a reservation. This could be updated more frequently, to track the "remaining bytes" i.e. reservation - processed.
4. range.snapshots.pending-send: a gauge tracking the sum of all snapshot bytes that remain on a store's sending side, for snapshots that have acquired a reservation. Similar to above, this is tracking the remaining bytes to be sent.

Context

(3) and (4) may not present much material benefit, as snapshots should in most cases be processed in under 16 (512mb/32mb/s) seconds. Whilst the default metric update interval is 10 seconds, In cases where the snapshot rate is set lower, it may provide utility - however the existing capacity.reserved metric, tracking the total (unprocessed + processed) in progress snapshot bytes may be more appropriate. This issue leaves them as optional.

related PR, for count rather than bytes: #84947

cc @AlexTalks

Jira issue: CRDB-18293

[miraradeva]

Added metrics (1) and (2) above as part of #100942. Austen helped me test the change using this roachprod script:

export cluster=$USER-snapqueue
# create the cluster and stage the binary (you will need to build the linux binary)
# use n1-4 as the initial cluster nodes. n5 to join later once there's data written and n6 as the workload runner
roachprod create $cluster -n 6 --gce-machine-type=n1-standard-8 
roachprod put $cluster cockroach cockroach # (assumes linux binary is named cockroach)
roachprod start $cluster:1-4  # start only the first 4 nodes
# load initial data size of 15gb (unreplicated), then run a 50% read workload at 300 target ops/s writing 16kb each op for 10 minutes
roachprod run $cluster:6 -- './cockroach workload run kv --drop --read-percent=50 --min-block-bytes=16000 --max-block-bytes=16000 --insert-count=1000000 --max-rate=300 --concurrency=128 --duration=10m {pgurl:1-4}'
roachprod start $cluster:5
# now watch the snapshot bytes queued metric, can also start the workload runner again and see what happens.

For metrics (3) and (4), I added another metric to keep track of the total reserved bytes sent/received of snapshots with reservations (range.snapshots.recv-reserved-bytes). We also have an existing metrics for total bytes sent/received (range.snapshots.rcvd-bytes). And the metrics we want for (3) and (4) are essentially the difference between these two. I ran the same workload and watched node 5 ramp up in terms of the two metrics above (I didn't do the actual difference because I haven't figured out grafana yet).

The lines seem very close, so the difference will likely be too small to tell us much.