sql: logictest: default test tenant unable to split index

[cockroach-teamcity]

pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test.TestLogic_crdb_internal failed with artifacts on release-22.2 @ a5951d589426b3dfd5ce406c25750fb4e56efc10:

=== RUN   TestLogic_crdb_internal
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/0205c25cd2c9009394cc81369c4dd2f5/logTestLogic_crdb_internal1663832886
    test_log_scope.go:79: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
[11:34:24] rng seed: -2154276009083321188

    logic.go:2762: let $testdb_id = 106
    logic.go:2762: let $testdb_foo_id = 108
    logic.go:2762: let $schema_bar_id = 111
=== CONT  TestLogic_crdb_internal
    logic.go:3927: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0205c25cd2c9009394cc81369c4dd2f5/logTestLogic_crdb_internal1663832886
--- FAIL: TestLogic_crdb_internal (48.47s)
=== RUN   TestLogic_crdb_internal/max_retry_counter
    logic.go:2705: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3059/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test_/fakedist-disk_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/crdb_internal:791: SELECT start_pretty, end_pretty FROM crdb_internal.ranges
        WHERE split_enforced_until IS NOT NULL
        AND (start_pretty LIKE '/Table/112/1%' OR start_pretty LIKE '/Table/112/2%')
        expected:
            /Table/112/1/1  /Table/112/1/2
            /Table/112/1/2  /Table/112/1/3
            /Table/112/1/3  /Table/112/2/1
            /Table/112/2/1  /Table/112/2/2
            /Table/112/2/2  /Table/112/2/3
            /Table/112/2/3  /Table/112/3/1
            
        but found (query options: "retry") :
[11:35:11] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3059/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test_/fakedist-disk_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/crdb_internal: 107 statements
    logic.go:2019: 
         /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3059/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test_/fakedist-disk_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/crdb_internal:803: too many errors encountered, skipping the rest of the input
[11:35:11] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3059/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test_/fakedist-disk_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/crdb_internal with config fakedist-disk: 107 tests, 2 failures
[11:35:11] --- total progress: 107 statements
--- total: 107 tests, 2 failures
    --- FAIL: TestLogic_crdb_internal/max_retry_counter (46.10s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-24375

[rharding6373]

Looks like more fallout from #96390, given that this test is doing splits and the failure is in the tenant.

[rharding6373]

I'm not 100% sure that tenant admin split capability is the culprit, since I would expect the split statements to fail if that were the case.

[yuzefovich]

I don't think it's #96390 given this failure is on 22.2 branch and that PR doesn't seem to have been backported. Someone's gotta do the bisect.

[mgartner]

Doing the bisect now.

[mgartner]

I've bisected to c31fdb0. @xinhaoz do you mind taking a look at this? The failure can be reproduced by stressing the crdb_internal logic test:

./dev test './pkg/sql/logictest/tests/local-vec-off' -f 'TestLogic_crdb_internal' --stress

I've observed the failure after ~85 runs.

[rytaft]

Looks like this was fixed by #97187