pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121401

cockroach-teamcity · 2024-03-30T06:25:00Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed with artifacts on master @ 2a5e231716c436781f12452d800651f51c6383b7:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/020c62f143b22628b86ce9738a03dba8/logTestTenantLogicCCL_cluster_locks_tenant1701138771
    test_log_scope.go:81: use -show-logs to present logs inline
    logic.go:3021: let $root_session = 17c17635ff3679170000000000000001
    logic.go:3021: let $testuser_session = 17c1763622ff33360000000000000001
    logic.go:3021: let $txn1 = e37b30d8-6b9b-4355-b2e3-f1242d0b0e2d
    logic.go:3021: let $txn2 = af298c7a-c371-44ca-bdf4-40074d3c0821
    logic.go:3021: let $txn3 = 11a97fb2-2348-4597-947e-6fafa3105ac2
    logic.go:3021: let $txn4 = ed672888-de7e-4a43-9b49-0c5d4e7f2482
[06:15:11] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
    logic.go:2963: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:230: SELECT count(*) FROM crdb_internal.cluster_locks WHERE table_name IN ('t','t2')
        expected:
            4
            
        but found (query options: "retry") :
            0
            
[06:15:57] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 36 statements
[06:15:57] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 36 tests, 1 failures
    logic.go:4145: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:235: error while processing
    logic.go:4145: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:235: too many errors encountered, skipping the rest of the input
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/020c62f143b22628b86ce9738a03dba8/logTestTenantLogicCCL_cluster_locks_tenant1701138771
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (55.03s)

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries _{This test on roachdash | Improve this report!

Jira issue: CRDB-37224}

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2024-03-31T06:25:24Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed with artifacts on master @ 7fc4c7bcbbf0c75a62d056da0bf79a5a32714650:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/020c62f143b22628b86ce9738a03dba8/logTestTenantLogicCCL_cluster_locks_tenant4275148946
    test_log_scope.go:81: use -show-logs to present logs inline
    logic.go:3021: let $root_session = 17c1c50f030a07cc0000000000000001
    logic.go:3021: let $testuser_session = 17c1c50f260e0a6c0000000000000001
    logic.go:3021: let $txn1 = 0b46de9d-c696-424b-b1f2-7b323709065a
    logic.go:3021: let $txn2 = ffea5e69-0a64-4234-9e5c-9b63a80a70f8
    logic.go:3021: let $txn3 = e5dcb3c7-ad20-44db-a97b-0ff6dd33a973
    logic.go:3021: let $txn4 = 1b59198f-e363-4836-b0c5-427b9cf128a3
[06:20:05] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5519/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
[06:20:05] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5519/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 30 tests, 0 failures
    logic.go:4145: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5519/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: error while processing
    logic.go:4145: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5519/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: 
        expected success, but found
        (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=1b59198f key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.00577965 epo=0 ts=1711866002.430824821,2 min=1711866002.385191052,0 seq=2} lock=true stat=PENDING rts=1711866002.430824821,2 wto=false gul=1711866002.885191052,0
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/020c62f143b22628b86ce9738a03dba8/logTestTenantLogicCCL_cluster_locks_tenant4275148946
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (8.79s)

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121410 pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed [C-test-failure O-robot T-sql-queries branch-release-24.1 release-blocker]

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-04-01T07:32:38Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed on master @ 7fc4c7bcbbf0c75a62d056da0bf79a5a32714650:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant259184025
    test_log_scope.go:81: use -show-logs to present logs inline
[04:15:38] setting distsql_workmem='23626B';
[04:15:38] setting distsql_workmem='23626B';
    logic.go:3021: let $root_session = 17c20cd9df595b370000000000000001
    logic.go:3021: let $testuser_session = 17c20cd9f6bfd5370000000000000001
    logic.go:3021: let $txn1 = a7649597-8f42-473a-ac94-60c37cc799b7
    logic.go:3021: let $txn2 = 6d40ae60-a18e-475d-8182-6d51d927ba64
    logic.go:3021: let $txn3 = e9976108-b378-4808-90d8-1b8398e9463d
    logic.go:3021: let $txn4 = 02314e22-9223-43a0-afa6-3bd2fc6b268d
[04:15:41] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
[04:15:41] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 30 tests, 0 failures
    logic.go:4145: 
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: error while processing
    logic.go:4145: 
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: 
        expected success, but found
        (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=02314e22 key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.00223224 epo=0 ts=1711944938.766138890,2 min=1711944938.734692448,0 seq=2} lock=true stat=PENDING rts=1711944938.766138890,2 wto=false gul=1711944939.234692448,0
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant259184025
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (8.27s)

Parameters:

attempt=1
run=7
shard=16

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121410 pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed [C-test-failure O-robot T-sql-queries branch-release-24.1 release-blocker]

_{This test on roachdash | Improve this report!}

yuzefovich · 2024-04-01T17:03:47Z

This is the same as #120821, so #121052 didn't fix this, cc @nvanbenschoten

yuzefovich · 2024-04-01T17:05:46Z

Actually, there are two failures modes: one is an unexpected output (also in #121410), and another is the "missing intent" retryable error. #120821 was about the former.

cockroach-teamcity · 2024-04-02T07:19:36Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed on master @ c43f54cdde5b7578f4a0ca61de41463f0d690993:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant1076918167
    test_log_scope.go:81: use -show-logs to present logs inline
[05:09:21] setting distsql_workmem='23468B';
[05:09:21] setting distsql_workmem='23468B';
    logic.go:3021: let $root_session = 17c25e5cec51c6c70000000000000001
    logic.go:3021: let $testuser_session = 17c25e5d0477e1bf0000000000000001
    logic.go:3021: let $txn1 = 970b50d4-b3df-4224-a308-7f92b546807f
    logic.go:3021: let $txn2 = e3a198fd-4b64-4c83-b114-fdeb2568317e
    logic.go:3021: let $txn3 = 435069cf-34fb-4e51-815a-a96e09bc4913
    logic.go:3021: let $txn4 = 5a4965c2-77f8-4270-a5f3-1dbaf29b808b
[05:09:25] --- progress: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
[05:09:25] --- done: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 30 tests, 0 failures
    logic.go:4145: 
        /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: error while processing
    logic.go:4145: 
        /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: 
        expected success, but found
        (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=5a4965c2 key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.00447482 epo=0 ts=1712034562.070819290,2 min=1712034562.039439716,0 seq=2} lock=true stat=PENDING rts=1712034562.070819290,2 wto=false gul=1712034562.539439716,0
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant1076918167
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (7.70s)

Parameters:

attempt=1
run=11
shard=16

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

kvoli · 2024-04-02T20:09:52Z

Could this potentially be the same issue as #120931 @arulajmani?

cockroach-teamcity · 2024-04-03T06:55:33Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed on master @ 7166724ff02b9b7f0d5787f74d919487ca347bd0:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant3785568505
    test_log_scope.go:81: use -show-logs to present logs inline
[06:24:06] setting distsql_workmem='66419B';
[06:24:06] setting distsql_workmem='66419B';
    logic.go:3021: let $root_session = 17c2b105b237b51e0000000000000001
    logic.go:3021: let $testuser_session = 17c2b105cab92c5d0000000000000001
    logic.go:3021: let $txn1 = c7fb482b-0155-4ffd-984f-5e7758297a3d
    logic.go:3021: let $txn2 = 9a72223b-4abd-4ed8-9aa9-7b7e65a7842b
    logic.go:3021: let $txn3 = 445a0884-bd85-40d9-aedb-f8f71f3a8bec
    logic.go:3021: let $txn4 = 6f6757ab-acd3-4771-90cc-ecf0521b670f
[06:24:10] --- progress: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
[06:24:10] --- done: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 30 tests, 0 failures
    logic.go:4145: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: error while processing
    logic.go:4145: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: 
        expected success, but found
        (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=6f6757ab key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.00299485 epo=0 ts=1712125446.904616363,2 min=1712125446.876344397,0 seq=2} lock=true stat=PENDING rts=1712125446.904616363,2 wto=false gul=1712125447.376344397,0
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant3785568505
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (7.85s)

Parameters:

attempt=1
run=6
shard=16

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2024-04-04T07:07:34Z

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test.TestTenantLogicCCL_cluster_locks_tenant failed on master @ 0d3c5b71415de867fedde97de01486d424209848:

=== RUN   TestTenantLogicCCL_cluster_locks_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant3050186744
    test_log_scope.go:81: use -show-logs to present logs inline
[06:12:32] setting distsql_workmem='26067B';
[06:12:32] setting distsql_workmem='26067B';
    logic.go:3021: let $root_session = 17c2fef89e2a50bf0000000000000001
    logic.go:3021: let $testuser_session = 17c2fef8b6afa8790000000000000001
    logic.go:3021: let $txn1 = 1feda179-d311-4b87-8565-b6509653544d
    logic.go:3021: let $txn2 = ad6a66e7-60f8-466b-ba6d-6b7c2f02607c
    logic.go:3021: let $txn3 = c5aa6d79-7520-45bd-a830-4e55dd84d1ff
    logic.go:3021: let $txn4 = f71bdc25-df23-4b8e-b5c2-12c6b888379e
[06:12:35] --- progress: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant: 26 statements
[06:12:35] --- done: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant with config 3node-tenant: 30 tests, 0 failures
    logic.go:4148: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: error while processing
    logic.go:4148: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test_/3node-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/cluster_locks_tenant:204: 
        expected success, but found
        (40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=f71bdc25 key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.01998693 epo=0 ts=1712211152.650190821,2 min=1712211152.619659444,0 seq=2} lock=true stat=PENDING rts=1712211152.650190821,2 wto=false gul=1712211153.119659444,0
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantLogicCCL_cluster_locks_tenant3050186744
--- FAIL: TestTenantLogicCCL_cluster_locks_tenant (8.06s)

Parameters:

attempt=1
run=7
shard=16

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

kvoli · 2024-04-04T19:24:44Z

cc @nvanbenschoten, this is the issue I mentioned during our meeting.

nvanbenschoten · 2024-04-04T23:16:24Z

I was able to bisect this failure back to ec14795. That commit is interesting, as it's a re-revert of a previous revert (3d87bec) which also found the change to be destabilizing to tests.

nvanbenschoten · 2024-04-04T23:27:08Z

I wonder if that change is playing poorly with the use of params.ReplicationMode = base.ReplicationAuto (in logic tests here), where we now have a desire to split apart a tenant into more ranges. This will be disruptive to logictests.

msbutler · 2024-04-05T14:54:56Z

a flake here: https://teamcity.cockroachdb.com/viewLog.html?buildId=14703790&buildTypeId=Cockroach_BazelEssentialCi

nvanbenschoten · 2024-04-05T17:17:06Z

I believe the change in behavior here from:

expected:
    4
    
but found (query options: "retry") :
    0

to

(40001) restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_ASYNC_WRITE_FAILURE - missing intent on: /Tenant/10/Table/106/1/"c"/0): "sql txn" meta={id=f71bdc25 key=/Tenant/10/Table/106/1/"b" iso=Serializable pri=0.01998693 epo=0 ts=1712211152.650190821,2 min=1712211152.619659444,0 seq=2} lock=true stat=PENDING rts=1712211152.650190821,2 wto=false gul=1712211153.119659444,0

can be traced back to enabling pipelining for DeleteRange requests. This trips up the DELETE statement here.

121857: sql: run tenant logic tests with manual replication mode r=nvanbenschoten a=nvanbenschoten Fixes #121401. This commit removes the use of automatic replication mode for secondary tenant SQL logic tests, so that the configuration more closely matches that for single tenant SQL logic tests. In doing so, it eliminates some of the recent flakiness which has come from 3d87bec, where ranges now split and rebalance much more often for secondary tenants. To accomplish this, the commit updates the `distsql_tenant_locality` tests to manually upreplicate. The commit also needed to fix up `cluster_locks_tenant`, which was broken by #99735. Now that we're running with manual replication mode, we can't rely on a range split or lease transfer to clear unreplicated locks, which was undermining the point of the test anyway. Release note: None Co-authored-by: Nathan VanBenschoten <[email protected]>

Fixes #121401. This commit removes the use of automatic replication mode for secondary tenant SQL logic tests, so that the configuration more closely matches that for single tenant SQL logic tests. In doing so, it eliminates some of the recent flakiness which has come from 3d87bec, where ranges now split and rebalance much more often for secondary tenants. To accomplish this, the commit updates the `distsql_tenant_locality` tests to manually upreplicate. The commit also needed to fix up `cluster_locks_tenant`, which was broken by #99735. Now that we're running with manual replication mode, we can't rely on a range split or lease transfer to clear unreplicated locks, which was undermining the point of the test anyway. Release note: None

cockroach-teamcity added this to the 24.1 milestone Mar 30, 2024

cockroach-teamcity added this to SQL Queries Mar 30, 2024

github-project-automation bot moved this to Triage in SQL Queries Mar 30, 2024

cockroach-teamcity mentioned this issue Mar 30, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121410

Closed

yuzefovich added T-kv KV Team and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Apr 1, 2024

yuzefovich removed this from SQL Queries Apr 1, 2024

cockroach-teamcity mentioned this issue Apr 3, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121600

Closed

cockroach-teamcity mentioned this issue Apr 4, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121753

Closed

kvoli assigned nvanbenschoten Apr 4, 2024

kvoli added P-1 Issues/test failures with a fix SLA of 1 month C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. GA-blocker labels Apr 4, 2024

cockroach-teamcity mentioned this issue Apr 5, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121825

Closed

nvanbenschoten mentioned this issue Apr 5, 2024

sql: run tenant logic tests with manual replication mode #121857

Merged

craig bot closed this as completed in 5f2a4f8 Apr 5, 2024

blathers-crl bot mentioned this issue Apr 5, 2024

release-24.1: sql: run tenant logic tests with manual replication mode #121870

Merged

This was referenced Apr 8, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121907

Closed

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121919

Closed

github-project-automation bot added this to KV Aug 28, 2024

github-project-automation bot moved this to Incoming in KV Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121401

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121401

cockroach-teamcity commented Mar 30, 2024 •

edited by cockroach-jira-scripts

Loading

cockroach-teamcity commented Mar 31, 2024

cockroach-teamcity commented Apr 1, 2024

yuzefovich commented Apr 1, 2024

yuzefovich commented Apr 1, 2024

cockroach-teamcity commented Apr 2, 2024

kvoli commented Apr 2, 2024

cockroach-teamcity commented Apr 3, 2024

cockroach-teamcity commented Apr 4, 2024

kvoli commented Apr 4, 2024

nvanbenschoten commented Apr 4, 2024

nvanbenschoten commented Apr 4, 2024

msbutler commented Apr 5, 2024

nvanbenschoten commented Apr 5, 2024

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121401

pkg/ccl/logictestccl/tests/3node-tenant/3node-tenant_test: TestTenantLogicCCL_cluster_locks_tenant failed #121401

Comments

cockroach-teamcity commented Mar 30, 2024 • edited by cockroach-jira-scripts Loading

cockroach-teamcity commented Mar 31, 2024

cockroach-teamcity commented Apr 1, 2024

yuzefovich commented Apr 1, 2024

yuzefovich commented Apr 1, 2024

cockroach-teamcity commented Apr 2, 2024

kvoli commented Apr 2, 2024

cockroach-teamcity commented Apr 3, 2024

cockroach-teamcity commented Apr 4, 2024

kvoli commented Apr 4, 2024

nvanbenschoten commented Apr 4, 2024

nvanbenschoten commented Apr 4, 2024

msbutler commented Apr 5, 2024

nvanbenschoten commented Apr 5, 2024

cockroach-teamcity commented Mar 30, 2024 •

edited by cockroach-jira-scripts

Loading