kvserver: track snapshot bytes due to rebalancing

[kvoli]

We currently track the number of bytes received and bytes sent for snapshots, per store.

Snapshots occur for multiple reasons, we would like to gain insight into the portion attributable to rebalancing decisions, up-replication and "catch up snapshots" when a range is behind on it's log.

The solution is to track the cause of the snapshot and export the sent/recv bytes and count for each type of snapshot cause.

We update the bytes sent here:

cockroach/pkg/kv/kvserver/store_snapshot.go

Line 402 in 1c41988

bytesSentMetric.Inc(int64(b.Len()))

We update the bytes received here:

cockroach/pkg/kv/kvserver/store_snapshot.go

Line 291 in 1c41988

bytesRcvdCounter.Inc(int64(len(req.KVBatch)))

We track the snapshot type that is being sent, differentiating between up-replication/general raft and rebalancing:

cockroach/pkg/kv/kvserver/kvserverpb/raft.proto

Lines 125 to 133 in 1c41988

	enum Priority {
	UNKNOWN = 0;
	// RECOVERY is used for a Raft-initiated snapshots and for
	// up-replication snapshots (i.e. when a dead node has been
	// removed and the range needs to be up-replicated).
	RECOVERY = 1;
	// REBALANCE is used for snapshots involved in rebalancing.
	REBALANCE = 2;
	}

We should create two new counters:
range.snapshots.rebalancing.rcvd-bytes
range.snapshots.rebalancing.sent-bytes

Then increment these counters in a similar fashion to the currently tracked snapshot bytes:

cockroach/pkg/kv/kvserver/metrics.go

Lines 525 to 529 in 1c41988

	Name: "range.snapshots.rcvd-bytes",
	Help: "Number of snapshot bytes received",
	Measurement: "Bytes",
	Unit: metric.Unit_COUNT,
	}

The currently exported metrics for range snapshots are show in the screenshot below:

Additional Information:

Separate to this issue is to begin tracking the up-front range size when considering rebalancing. The snapshot bytes sent/recv due to rebalancing actions would be known after rebalancing has occurred. As such, it is separate from this issue and may be estimated using the range size up-front, with consideration for this metric as feedback.

Jira issue: CRDB-15374