kv/kvserver: TestFlowControlUnquiescedRange failed

[cockroach-teamcity]

kv/kvserver.TestFlowControlUnquiescedRange failed with artifacts on master @ f295bd861a3a427652b19c2254d2401ebb4a3c8e:

=== RUN   TestFlowControlUnquiescedRange
    test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestFlowControlUnquiescedRange3338863982
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:98: 
        Test server was configured to route SQL queries to a secondary tenant (virtual cluster).
        If you are only seeing a test failure when this message appears, there may be a problem
        specific to cluster virtualization or multi-tenancy.
        
        To investigate, consider using "COCKROACH_TEST_TENANT=true" to force-enable just
        the secondary tenant in all runs (or, alternatively, "false" to force-disable), or use
        "COCKROACH_INTERNAL_DISABLE_METAMORPHIC_TESTING=false" to disable all random test variables altogether.
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* use of SQL servers requires an enterprise license. see https://cockroachlabs.com/pricing?cluster=0f23fe1c-018b-42cd-9531-6edc34b309f7 for details on how to enable enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* use of SQL servers requires an enterprise license. see https://cockroachlabs.com/pricing?cluster=0f23fe1c-018b-42cd-9531-6edc34b309f7 for details on how to enable enterprise features
*
*
* WARNING: test tenant requested by configuration, but code organization prevents start!
* use of SQL servers requires an enterprise license. see https://cockroachlabs.com/pricing?cluster=0f23fe1c-018b-42cd-9531-6edc34b309f7 for details on how to enable enterprise features
*
    flow_control_integration_test.go:2405: condition failed to evaluate within 3m45s: expected to track 2097152 tokens in aggregate, got 6291456
    panic.go:540: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/33e1d369c27b9c01b2b6009c561815a3/logTestFlowControlUnquiescedRange3338863982
--- FAIL: TestFlowControlUnquiescedRange (240.35s)

Parameters: TAGS=bazel,gss,race , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-30229

[pav-kv]

This is a fairly new test (April). Flaked a few times in CI recently too (so flakes both under --race and not):

flow_control_integration_test.go:2405: condition failed to evaluate within 45s: expected to track 2097152 tokens in aggregate, got 4194304

@irfansharif Looks like some kind of incorrect token counting / leak?

Reproes easily under dev test --stress pkg/kv/kvserver --filter=TestFlowControlUnquiescedRange.