kvserver: add rac2 v1 integration tests

Introduce several tests in `flow_control_integration_test.go`, mirroring
the existing tests but applied to the replication flow control v2
machinery.

The tests largely follow an identical pattern to the existing v1 tests,
swapping in rac2 metric and vtables.

The following tests are added:

```
TestFlowControlBasic
TestFlowControlRangeSplitMerge
TestFlowControlBlockedAdmission
TestFlowControlAdmissionPostSplitMerge
TestFlowControlCrashedNode
TestFlowControlRaftSnapshot
TestFlowControlRaftMembership
TestFlowControlRaftMembershipRemoveSelf
TestFlowControlClassPrioritization
TestFlowControlTransferLease
TestFlowControlGranterAdmitOneByOne
```

Another two tests are ommitted, `TestFlowControlRaftTransportBreak` and
`TestFlowControlRaftTransportCulled`, because they behave identically to
`TestFlowControlCrashedNode` as rac2 is less tightly coupled to the raft
transport.

One test, `TestFlowControlLeaderNotLeaseholder` has divering
pre-evaluation admit behavior and is therefore delayed to a subsequent
commit.

Lastly, two tests, `TestFlowControlQuiescedRange` and
`TestFlowControlUnquiescedRange` are dependent on cockroachdb#129581, and delayed
to a subsequent commit as well.

Part of: cockroachdb#130187
Release note: None

pkg/kv/kvserver/testdata/flow_control_integration_v2/admission_post_split_merge

@@ -0,0 +1,138 @@
+echo
+----
+----
+-- Flow token metrics from n1 after issuing a regular 2*1MiB 3x replicated write
+-- that are yet to get admitted. We see 2*3*1MiB=6MiB deductions of
+-- {regular,elastic} tokens with no corresponding returns. The 2*1MiB writes
+-- happened on what is soon going to be the LHS and RHS of a range being split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.eval.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 42 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 18 MiB   
+  kvflowcontrol.tokens.send.elastic.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 42 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 6.0 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- (Splitting range.)
+
+
+-- Flow token metrics from n1 after further issuing 2MiB and 3MiB writes to
+-- post-split LHS and RHS ranges respectively. We should see 15MiB extra tokens
+-- deducted which comes from (2MiB+3MiB)*3=15MiB. So we stand at
+-- 6MiB+15MiB=21MiB now.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 27 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 21 MiB   
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 21 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 27 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 21 MiB   
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe the newly split off replica, with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+  71       | 3             
+
+
+-- (Merging ranges.)
+
+
+-- Flow token metrics from n1 after issuing 4MiB of regular replicated writes to
+-- the post-merged range. We should see 12MiB extra tokens deducted which comes
+-- from 4MiB*3=12MiB. So we stand at 21MiB+12MiB=33MiB tokens deducted now. The
+-- RHS of the range is gone now, and the previously 3*3MiB=9MiB of tokens
+-- deducted for it are released at the subsuming LHS leaseholder.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%regular%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.regular.available   | 24 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 33 MiB   
+  kvflowcontrol.tokens.eval.regular.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 24 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 33 MiB   
+  kvflowcontrol.tokens.send.regular.returned    | 9.0 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe only the merged replica with its own three streams.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+
+
+-- (Allow below-raft admission to proceed.)
+
+
+-- Flow token metrics from n1 after work gets admitted. We see all outstanding
+-- {regular,elastic} tokens returned, including those from:
+-- - the LHS before the merge, and
+-- - the LHS and RHS before the original split.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 33 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 33 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 33 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 33 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 33 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+----
+----
+
+# vim:ft=sql

pkg/kv/kvserver/testdata/flow_control_integration_v2/basic

@@ -0,0 +1,89 @@
+echo
+----
+----
+-- Flow token metrics, before issuing the regular 1MiB replicated write.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 0 B     
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B     
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B     
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 0 B     
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B     
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 0 B     
+  kvflowcontrol.tokens.send.regular.returned    | 0 B     
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+
+
+-- (Issuing + admitting a regular 1MiB, triply replicated write...)
+
+
+-- Stream counts as seen by n1 post-write. We should see three {regular,elastic}
+-- streams given there are three nodes and we're using a replication factor of
+-- three.
+SELECT name, value
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%stream%'
+ORDER BY name ASC;
+
+  kvflowcontrol.streams.eval.elastic.blocked_count | 0  
+  kvflowcontrol.streams.eval.elastic.total_count   | 3  
+  kvflowcontrol.streams.eval.regular.blocked_count | 0  
+  kvflowcontrol.streams.eval.regular.total_count   | 3  
+  kvflowcontrol.streams.send.elastic.blocked_count | 0  
+  kvflowcontrol.streams.send.elastic.total_count   | 3  
+  kvflowcontrol.streams.send.regular.blocked_count | 0  
+  kvflowcontrol.streams.send.regular.total_count   | 3  
+
+
+-- Another view of the stream count, using /inspectz-backed vtables.
+SELECT range_id, count(*) AS streams
+    FROM crdb_internal.kv_flow_control_handles_v2
+GROUP BY (range_id)
+HAVING count(*) = 3 
+ORDER BY streams DESC;
+
+  range_id | stream_count  
+-----------+---------------
+  70       | 3             
+
+
+-- Flow token metrics from n1 after issuing the regular 1MiB replicated write,
+-- and it being admitted on n1, n2 and n3. We should see 3*1MiB = 3MiB of
+-- {regular,elastic} tokens deducted and returned, and {8*3=24MiB,16*3=48MiB} of
+-- {regular,elastic} tokens available. Everything should be accounted for.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB   
+  kvflowcontrol.tokens.eval.elastic.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB   
+  kvflowcontrol.tokens.send.elastic.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 3.0 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 3.0 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+----
+----
+
+# vim:ft=sql

pkg/kv/kvserver/testdata/flow_control_integration_v2/blocked_admission

@@ -0,0 +1,97 @@
+echo
+----
+----
+-- (Issuing 5 regular 1MiB, 3x replicated write that's not admitted.)
+
+
+-- Flow token metrics from n1 after issuing 5 regular 1MiB 3x replicated writes
+-- that are yet to get admitted. We see 5*1MiB*3=15MiB deductions of
+-- {regular,elastic} tokens with no corresponding returns.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 9.0 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 15 MiB   
+  kvflowcontrol.tokens.eval.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.eval.regular.available   | 33 MiB   
+  kvflowcontrol.tokens.eval.regular.deducted    | 15 MiB   
+  kvflowcontrol.tokens.eval.regular.returned    | 0 B      
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.elastic.available   | 9.0 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 15 MiB   
+  kvflowcontrol.tokens.send.elastic.returned    | 0 B      
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B      
+  kvflowcontrol.tokens.send.regular.available   | 33 MiB   
+  kvflowcontrol.tokens.send.regular.deducted    | 15 MiB   
+  kvflowcontrol.tokens.send.regular.returned    | 0 B      
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B      
+
+
+-- Observe the total tracked tokens per-stream on n1.
+SELECT range_id, store_id, crdb_internal.humanize_bytes(total_tracked_tokens::INT8)
+    FROM crdb_internal.kv_flow_control_handles_v2
+
+  range_id | store_id | total_tracked_tokens  
+-----------+----------+-----------------------
+  70       | 1        | 5.0 MiB               
+  70       | 2        | 5.0 MiB               
+  70       | 3        | 5.0 MiB               
+
+
+-- Observe the individual tracked tokens per-stream on the scratch range.
+SELECT range_id, store_id, priority, crdb_internal.humanize_bytes(tokens::INT8)
+    FROM crdb_internal.kv_flow_token_deductions_v2
+
+  range_id | store_id |  priority  | tokens   
+-----------+----------+------------+----------
+  70       | 1        | normal-pri | 1.0 MiB  
+  70       | 1        | normal-pri | 1.0 MiB  
+  70       | 1        | normal-pri | 1.0 MiB  
+  70       | 1        | normal-pri | 1.0 MiB  
+  70       | 1        | normal-pri | 1.0 MiB  
+  70       | 2        | normal-pri | 1.0 MiB  
+  70       | 2        | normal-pri | 1.0 MiB  
+  70       | 2        | normal-pri | 1.0 MiB  
+  70       | 2        | normal-pri | 1.0 MiB  
+  70       | 2        | normal-pri | 1.0 MiB  
+  70       | 3        | normal-pri | 1.0 MiB  
+  70       | 3        | normal-pri | 1.0 MiB  
+  70       | 3        | normal-pri | 1.0 MiB  
+  70       | 3        | normal-pri | 1.0 MiB  
+  70       | 3        | normal-pri | 1.0 MiB  
+
+
+-- (Allow below-raft admission to proceed.)
+
+
+-- Flow token metrics from n1 after work gets admitted. We see 15MiB returns of
+-- {regular,elastic} tokens, and the available capacities going back to what
+-- they were.
+SELECT name, crdb_internal.humanize_bytes(value::INT8)
+    FROM crdb_internal.node_metrics
+   WHERE name LIKE '%kvflowcontrol%tokens%'
+ORDER BY name ASC;
+
+  kvflowcontrol.tokens.eval.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.eval.elastic.deducted    | 15 MiB  
+  kvflowcontrol.tokens.eval.elastic.returned    | 15 MiB  
+  kvflowcontrol.tokens.eval.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.eval.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.eval.regular.deducted    | 15 MiB  
+  kvflowcontrol.tokens.eval.regular.returned    | 15 MiB  
+  kvflowcontrol.tokens.eval.regular.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.elastic.available   | 24 MiB  
+  kvflowcontrol.tokens.send.elastic.deducted    | 15 MiB  
+  kvflowcontrol.tokens.send.elastic.returned    | 15 MiB  
+  kvflowcontrol.tokens.send.elastic.unaccounted | 0 B     
+  kvflowcontrol.tokens.send.regular.available   | 48 MiB  
+  kvflowcontrol.tokens.send.regular.deducted    | 15 MiB  
+  kvflowcontrol.tokens.send.regular.returned    | 15 MiB  
+  kvflowcontrol.tokens.send.regular.unaccounted | 0 B     
+----
+----
+
+# vim:ft=sql