pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test: TestDataDriven failed

[cockroach-teamcity]

pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test.TestDataDriven failed with artifacts on release-22.2 @ 62c6828d227a6c0e05cc48f63c0fc881fb77c39a:

        <no input to command>
        ----
        delete {source=1,target=1}
        delete {source=1,target=2}
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/866/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test_/spanconfigreconcilerccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/protectedts:121:
        state [1 args]
        <no input to command>
        ----
        /{Min-System/NodeLiveness}                 ttl_seconds=3600 num_replicas=5
        /System/NodeLiveness{-Max}                 ttl_seconds=600 num_replicas=5
        ...
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/866/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test_/spanconfigreconcilerccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/protectedts:128:
        release [1 args]
        <no input to command>
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/866/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test_/spanconfigreconcilerccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/protectedts:131:
        release [1 args]
        <no input to command>
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/866/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/spanconfigreconcilerccl_test_/spanconfigreconcilerccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/spanconfigccl/spanconfigreconcilerccl/testdata/protectedts:134: 
        expected:
        delete /Table/10{6-7}
        upsert /Table/10{6-7}                      range default
        delete /Table/10{7-8}
        upsert /Table/10{7-8}                      range default
        
        found:
        delete /Table/10{6-7}
        upsert /Table/10{6-7}                      protection_policies=[{ts: 4}]
        delete /Table/10{7-8}
        upsert /Table/10{7-8}                      protection_policies=[{ts: 4}]
        delete /Table/10{6-7}
        upsert /Table/10{6-7}                      range default
        delete /Table/10{7-8}
        upsert /Table/10{7-8}                      range default
I230614 06:36:14.285341 4072637 testutils/testcluster/testcluster.go:130  [-] 1428  TestCluster quiescing nodes
W230614 06:36:14.285669 4560892 kv/kvserver/intentresolver/intent_resolver.go:799  [-] 1429  failed to gc transaction record: could not GC completed transaction anchored at /Table/31/1/1/0: node unavailable; try another peer
I230614 06:36:14.285944 4161802 2@rpc/context.go:1393  [n1] 1430  closing [{n1: 127.0.0.1:36519 (default)} {n0: 127.0.0.1:36519 (default)}]
W230614 06:36:14.287874 4083472 sql/sqlliveness/slinstance/slinstance.go:242  [n1] 1431  exiting heartbeat loop
W230614 06:36:14.288981 4079432 kv/txn.go:705  [n1] 1432  failure aborting transaction: node unavailable; try another peer; abort caused by: expire-sessions: failed to read query result: context canceled
E230614 06:36:14.289063 4079432 jobs/registry.go:749  [n1] 1433  error expiring job sessions: expire-sessions: failed to read query result: context canceled
W230614 06:36:14.289171 4079432 kv/txn.go:705  [n1] 1434  failure aborting transaction: node unavailable; try another peer; abort caused by: txn exec: context canceled
E230614 06:36:14.289222 4079432 jobs/registry.go:757  [n1] 1435  failed to serve pause and cancel requests: txn exec: context canceled
W230614 06:36:14.289258 4079432 jobs/registry.go:826  [n1] 1436  canceling all adopted jobs due to stopper quiescing
I230614 06:36:14.287989 4083855 sql/stats/automatic_stats.go:509  [n1] 1437  quiescing auto stats refresher
    --- FAIL: TestDataDriven/protectedts (5.91s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-28755

[pav-kv]

Same as #98038. Probably just need to backport the fix #98082 to 22.2.

[pav-kv]

@irfansharif The backport #105061 did not apply cleanly, could you PTAL?

[irfansharif]

#105061.