kvserver: add rac2 v1 integration tests

Introduce several tests in `flow_control_integration_test.go`, mirroring
the existing tests but applied to the replication flow control v2
machinery.

The tests largely follow an identical pattern to the existing v1 tests,
swapping in rac2 metrics and vtables.

The following tests are added:

```
TestFlowControlBasicV2
TestFlowControlRangeSplitMergeV2
TestFlowControlBlockedAdmissionV2
TestFlowControlAdmissionPostSplitMergeV2
TestFlowControlCrashedNodeV2
TestFlowControlRaftSnapshotV2
TestFlowControlRaftMembershipV2
TestFlowControlRaftMembershipRemoveSelfV2
TestFlowControlClassPrioritizationV2
TestFlowControlQuiescedRangeV2
TestFlowControlUnquiescedRangeV2
TestFlowControlTransferLeaseV2
TestFlowControlLeaderNotLeaseholderV2
TestFlowControlGranterAdmitOneByOneV2
```

These tests all have at least two variants:

```
V2EnabledWhenLeaderV1Encoding
V2EnabledWhenLeaderV2Encoding
```

When `V2EnabledWhenLeaderV1Encoding` is run, the tests use a different
testdata file, which has a `_v1_encoding` suffix. A separate file is
necessary because when the protocol enablement level is
`V2EnabledWhenLeaderV1Encoding`, all entries which are subject to
admission control are encoded as `raftpb.LowPri`, regardless of their
original priority, as we don't want to pay the cost to deserialize the
raft admission meta.

The v1 encoding variants retain the same comments as the v2 encoding,
however any comments referring to regular tokens should be interpreted
as elastic tokens instead, due to the above.

Two v1 tests are not ported over to v2:

```
TestFlowControlRaftTransportBreak
TestFlowControlRaftTransportCulled
```

These omitted tests behave identically to `TestFlowControlCrashedNodeV2`
as rac2 is less tightly coupled to the raft transport, instead operating
on replication states (e.g., `StateProbe`, `StateReplicate`).

Resolves: cockroachdb#130187
Release note: None

pkg/kv/kvserver/testdata/flow_control_integration_v2/admission_post_split_merge

@@ -0,0 +1,146 @@
+echo
+----
+----
+-- Flow token metrics from n1 after issuing a regular 2*1MiB 3x replicated write
+-- that are yet to get admitted. We see 2*3*1MiB=6MiB deductions of
+-- {regular,elastic} tokens with no corresponding returns. The 2*1MiB writes
+-- happened on what is soon going to be the LHS and RHS of a range being split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.eval.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 42 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.send.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 42 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- (Splitting range.)
+
+
+-- Flow token metrics from n1 after further issuing 2MiB and 3MiB writes to
+-- post-split LHS and RHS ranges respectively. We should see 15MiB extra tokens
+-- deducted which comes from (2MiB+3MiB)*3=15MiB. So we stand at
+-- 6MiB+15MiB=21MiB now.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 27 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 21 MiB   
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 27 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 21 MiB   
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe the newly split off replica, with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+  71       | 3             
+
+
+-- (Merging ranges.)
+
+
+-- Flow token metrics from n1 after issuing 4MiB of regular replicated writes to
+-- the post-merged range. We should see 12MiB extra tokens deducted which comes
+-- from 4MiB*3=12MiB. So we stand at 21MiB+12MiB=33MiB tokens deducted now. The
+-- RHS of the range is gone now, and the previously 3*3MiB=9MiB of tokens
+-- deducted for it are released at the subsuming LHS leaseholder.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 0 B      
+  kvflowcontrol.tokens.eval.elastic.deducted    | 33 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 24 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 33 MiB   
+  kvflowcontrol.tokens.eval.regular.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 0 B      
+  kvflowcontrol.tokens.send.elastic.deducted    | 33 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 24 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 33 MiB   
+  kvflowcontrol.tokens.send.regular.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe only the merged replica with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+
+
+-- (Allow below-raft admission to proceed.)
+
+
+-- Flow token metrics from n1 after work gets admitted. We see all outstanding
+-- {regular,elastic} tokens returned, including those from:
+-- - the LHS before the merge, and
+-- - the LHS and RHS before the original split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 33 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 33 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 33 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 33 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+----
+----
+
+# vim:ft=sql

pkg/kv/kvserver/testdata/flow_control_integration_v2/admission_post_split_merge_v1_encoding

@@ -0,0 +1,153 @@
+# -- When using v1 encoding (V2EnabledWhenLeaderV1Encoding), all entries which
+# -- are subject to admission control are encoded as `raftpb.LowPri`,
+# -- regardless of their original priority; to avoid the overhead of
+# -- deserializing the raft admission metadata. Therefore, as the underlying
+# -- test is shared between the v1 and v2 encoding testdata files, the reader
+# -- should interpret any comments referring to regular tokens as referring to
+# -- elastic token.
+echo
+----
+----
+-- Flow token metrics from n1 after issuing a regular 2*1MiB 3x replicated write
+-- that are yet to get admitted. We see 2*3*1MiB=6MiB deductions of
+-- {regular,elastic} tokens with no corresponding returns. The 2*1MiB writes
+-- happened on what is soon going to be the LHS and RHS of a range being split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.eval.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.send.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- (Splitting range.)
+
+
+-- Flow token metrics from n1 after further issuing 2MiB and 3MiB writes to
+-- post-split LHS and RHS ranges respectively. We should see 15MiB extra tokens
+-- deducted which comes from (2MiB+3MiB)*3=15MiB. So we stand at
+-- 6MiB+15MiB=21MiB now.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe the newly split off replica, with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+  71       | 3             
+
+
+-- (Merging ranges.)
+
+
+-- Flow token metrics from n1 after issuing 4MiB of regular replicated writes to
+-- the post-merged range. We should see 12MiB extra tokens deducted which comes
+-- from 4MiB*3=12MiB. So we stand at 21MiB+12MiB=33MiB tokens deducted now. The
+-- RHS of the range is gone now, and the previously 3*3MiB=9MiB of tokens
+-- deducted for it are released at the subsuming LHS leaseholder.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 0 B      
+  kvflowcontrol.tokens.eval.elastic.deducted    | 33 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 0 B      
+  kvflowcontrol.tokens.send.elastic.deducted    | 33 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B      
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe only the merged replica with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+
+
+-- (Allow below-raft admission to proceed.)
+
+
+-- Flow token metrics from n1 after work gets admitted. We see all outstanding
+-- {regular,elastic} tokens returned, including those from:
+-- - the LHS before the merge, and
+-- - the LHS and RHS before the original split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B     
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.send.regular.returned    | 0 B     
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+----
+----
+
+# vim:ft=sql

pkg/kv/kvserver/testdata/flow_control_integration_v2/basic

@@ -0,0 +1,88 @@
+echo
+----
+----
+-- Flow token metrics, before issuing the regular 1MiB replicated write.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 0 B     
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B     
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B     
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 0 B     
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B     
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.send.regular.returned    | 0 B     
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+
+
+-- (Issuing + admitting a regular 1MiB, triply replicated write...)
+
+
+-- Stream counts as seen by n1 post-write. We should see three {regular,elastic}
+-- streams given there are three nodes and we're using a replication factor of
+-- three.
+SELECT name, value
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%stream%'
+ORDER BY name ASC;
+
+  kvflowcontrol.streams.eval.elastic.blocked_count | 0  
+  kvflowcontrol.streams.eval.elastic.total_count   | 3  
+  kvflowcontrol.streams.eval.regular.blocked_count | 0  
+  kvflowcontrol.streams.eval.regular.total_count   | 3  
+  kvflowcontrol.streams.send.elastic.blocked_count | 0  
+  kvflowcontrol.streams.send.elastic.total_count   | 3  
+  kvflowcontrol.streams.send.regular.blocked_count | 0  
+  kvflowcontrol.streams.send.regular.total_count   | 3  
+
+
+-- Another view of the stream count, using /inspectz-backed vtables.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+
+
+-- Flow token metrics from n1 after issuing the regular 1MiB replicated write,
+-- and it being admitted on n1, n2 and n3. We should see 3*1MiB = 3MiB of
+-- {regular,elastic} tokens deducted and returned, and {8*3=24MiB,16*3=48MiB} of
+-- {regular,elastic} tokens available. Everything should be accounted for.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB   
+  kvflowcontrol.tokens.eval.elastic.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB   
+  kvflowcontrol.tokens.send.elastic.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+----
+----
+
+# vim:ft=sql