Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test: TestCCLLogic_regional_by_row_read_committed failed #111481

Closed
cockroach-teamcity opened this issue Sep 29, 2023 · 27 comments · Fixed by #113111
Closed

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test: TestCCLLogic_regional_by_row_read_committed failed #111481

cockroach-teamcity opened this issue Sep 29, 2023 · 27 comments · Fixed by #113111
Assignees
@yuzefovich
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 29, 2023 •
edited by cockroach-jira-scripts
Loading

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 27e7c166b5591b0e05bb2fd23780309b490d7af9:

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.
[07:01:16] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3188/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 11 statements
    logic.go:2576: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3188/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:01:17] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3188/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3993: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3188/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3993: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3188/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed239503891
--- FAIL: TestCCLLogic_regional_by_row_read_committed (29.31s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-31919
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Sep 29, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Sep 29, 2023
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Sep 29, 2023
@msirek
Copy link
Contributor

msirek commented Sep 29, 2023

A bisect of this points to d0c46fe
But I haven't fully verified this.

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ fad649d89721ddb3e9f3dcab1ad5d14f74c91bc9:

	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.
[05:30:21] setting distsql_workmem='92534B';
[05:30:22] setting distsql_workmem='92534B';
    logic.go:2576: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3980/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[05:30:24] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3980/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3993: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3980/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3993: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3980/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed1093372550
--- FAIL: TestCCLLogic_regional_by_row_read_committed (21.03s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@msirek
Copy link
Contributor

msirek commented Sep 30, 2023

This test also fails under stress at the commit prior to d0c46fe: 4b2badd, but with a different error:

dev test pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant --filter=TestCCLLogic_regional_by_row_read_committed --stress

FAIL: //pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test (run 828 of 1000) (see /private/var/tmp/_bazel_msirek/70437e77fb93b11771e5e34486b10eac/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/testlogs/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test/run_828_of_1000/test.log)
ERROR: /Users/msirek/go/src/github.com/cockroachdb/cockroach/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/BUILD.bazel:3:8: Testing //pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test (run 828 of 1000) failed: Test failed, aborting
==================== Test output for //pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test (run 828 of 1000):
I230929 23:57:42.423516 1 (gostd) rand.go:250  [-] 1  random seed: -2889133837769026593

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1558: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.
test logs left over in: /Users/msirek/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/6de158fee3dfcf3c1fb1427c6ae12320/logTestCCLLogic_regional_by_row_read_committed2329509628
E230930 00:04:58.895208 40531 2@rpc/peer.go:580  [T10,Vtenant-10,nsql5,rnode=1,raddr=127.0.0.1:58327,class=default,rpc] 2  disconnected (was healthy for 4.055s): grpc: connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)" [code 14/Unavailable]
E230930 00:04:58.990365 44281 2@rpc/peer.go:580  [T10,Vtenant-10,nsql8,rnode=1,raddr=127.0.0.1:58327,class=default,rpc] 3  disconnected (was healthy for 3.025s): grpc: connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)" [code 14/Unavailable]
I230930 00:04:59.162660 50820 sql/sqlliveness/slstorage/slstorage.go:394  [n3,intExec=expire-sessions] 4  deleted session 0101801b00ff1b6c7f422d9013d7738bf4f5d6 which expired at 1696031907.752038000,0
--- FAIL: TestCCLLogic_regional_by_row_read_committed (9.86s)
    test_log_scope.go:167: test logs captured to: /Users/msirek/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/6de158fee3dfcf3c1fb1427c6ae12320/logTestCCLLogic_regional_by_row_read_committed2329509628
    test_log_scope.go:81: use -show-logs to present logs inline
    logic.go:1656:
        :0: error while processing
    logic.go:1656: dial tcp 127.0.0.1:58325: connect: connection refused
    panic.go:522: -- test log scope end --
FAIL
I230930 00:04:59.222348 1 (gostd) testmain.go:118  [-] 1  Test //pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test exited with error code 1
================================================================================
Target //pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test up-to-date:
  _bazel/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test
ERROR: Test failed, aborting
INFO: Elapsed time: 2939.185s, Critical Path: 95.03s
INFO: 502 processes: 114 remote cache hit, 13 internal, 375 darwin-sandbox.
//pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant:multiregion-9node-3region-3azs-tenant_test NO STATUS
  /private/var/tmp/_bazel_msirek/70437e77fb93b11771e5e34486b10eac/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/testlogs/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test/run_828_of_1000/test.log

@michae2
Copy link
Collaborator

michae2 commented Oct 1, 2023

d0c46fe added the logic test that is failing, but I don't think the failure is related to read committed isolation. I'm able to reproduce it using this smaller logic test which runs entirely at serializable isolation:

# michae2@michae2-crl-mbp 1 (1) ~crdb % cat pkg/ccl/logictestccl/testdata/logic_test/unique

# tenant-cluster-setting-override-opt: sql.zone_configs.allow_for_secondary_tenant.enabled=true sql.multi_region.allow_abstractions_for_secondary_tenants.enabled=true
# LogicTest: multiregion-9node-3region-3azs multiregion-9node-3region-3azs-vec-off multiregion-9node-3region-3azs-tenant multiregion-9node-3region-3azs-no-los

statement ok
CREATE DATABASE multi_region_test_db PRIMARY REGION "ca-central-1" REGIONS "ap-southeast-2", "us-east-1" SURVIVE REGION FAILURE

statement ok
USE multi_region_test_db

statement ok
CREATE TABLE ab (
  region crdb_internal_region NOT NULL,
  a STRING NOT NULL,
  b STRING NOT NULL,
  PRIMARY KEY (region, a),
  UNIQUE INDEX (region, b),
  FAMILY (region, a, b)
) LOCALITY REGIONAL BY ROW AS region

statement ok
INSERT INTO ab VALUES ('ap-southeast-2', 'aaa', 'bbb'), ('ca-central-1', 'ccc', 'ddd')

statement error pgcode 23505 pq: duplicate key value violates unique constraint "ab_region_b_key"\nDETAIL: Key \(region, b\)=\('ca-central-1', 'ddd'\) already exists
UPDATE ab SET region = 'ca-central-1', b = 'ddd' WHERE a = 'aaa'

# michae2@michae2-crl-mbp 146 (1) ~crdb % ./dev testlogic ccl --config=multiregion-9node-3region-3azs-tenant --files=unique --ignore-cache --stress
...
--- FAIL: TestCCLLogic_unique (16.53s)
    test_log_scope.go:167: test logs captured to: /Users/michae2/go/src/github.com/cockroachdb/cockroach/tmp/_tmp/59032834933fdc70df3058d7ece5f2d7/logTestCCLLogic_unique2408354390
    test_log_scope.go:81: use -show-logs to present logs inline
    logic.go:2576:

        /private/var/tmp/_bazel_michae2/239414c2d9c0d0876d35ed2ff4c5fe74/sandbox/darwin-sandbox/372/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/unique:23:
        expected:
        pq: duplicate key value violates unique constraint "ab_region_b_key"\nDETAIL: Key \(region, b\)=\('ca-central-1', 'ddd'\) already exists
        got:
        pq: duplicate key value got decoding error, key=/Min val=raw_bytes:"\357\340\022\215\003\022ccc\000\001" timestamp:<wall_time:1696196042593730000 > : insufficient bytes to decode uvarint value
    logic.go:4172: -- test log scope end --
FAIL

Notice the slightly changed error message, it always seems to happen when key is /Min. And it doesn't seem to reproduce when using --config=multiregion-9node-3region-3azs, only with --config=multiregion-9node-3region-3azs-tenant so I suspect that both regional by row tables and multi-tenancy are required. I will keep investigating.

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ f6f355b50e0dbf28633e25ddd05f2775141af31e:

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.

pkg/sql/logictest/logic.go:1556: ((*logicTest).newCluster)
	NOTICE: .StartTenant() called via implicit interface TenantControlInterface;
HINT: consider using .TenantController().StartTenant() instead.
[06:59:21] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3189/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 7 statements
    logic.go:2576: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3189/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[06:59:23] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3189/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3993: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3189/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3993: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3189/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed131772037
--- FAIL: TestCCLLogic_regional_by_row_read_committed (34.44s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 4494e945a9070862cfae1b668b3f8b667c41be17:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed302651180
    test_log_scope.go:81: use -show-logs to present logs inline
[07:25:34] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3183/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 6 statements
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3183/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:25:35] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3183/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3183/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3183/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed302651180
--- FAIL: TestCCLLogic_regional_by_row_read_committed (37.49s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@michae2 michae2 self-assigned this Oct 4, 2023
@michae2 michae2 moved this from Triage to Active in SQL Queries Oct 4, 2023
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 1b83c04ed0106dd0d3380821707861008ec32b73:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed310769343
    test_log_scope.go:81: use -show-logs to present logs inline
[07:22:40] setting distsql_workmem='62209B';
[07:22:40] setting distsql_workmem='62209B';
[07:22:43] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2819/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 8 statements
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2819/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:22:44] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2819/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2819/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2819/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed310769343
--- FAIL: TestCCLLogic_regional_by_row_read_committed (27.30s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 3b438b4a59ad4759e5bf22ff0b6dd6e678c2be0d:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed827710860
    test_log_scope.go:81: use -show-logs to present logs inline
[15:39:37] setting distsql_workmem='82227B';
[15:39:37] setting distsql_workmem='82227B';
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4916/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[15:39:39] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4916/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 32 statements
[15:39:39] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4916/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4916/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4916/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed827710860
--- FAIL: TestCCLLogic_regional_by_row_read_committed (20.83s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 7e1f909e9f2ddcef2e418d6c2df11f3139a2b85a:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2490017701
    test_log_scope.go:81: use -show-logs to present logs inline
[06:54:24] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3031/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 8 statements
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3031/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[06:54:26] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3031/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3031/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3031/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2490017701
--- FAIL: TestCCLLogic_regional_by_row_read_committed (27.83s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 5af56c0ecd701d452c19fb89908012294b5336aa:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2266644065
    test_log_scope.go:81: use -show-logs to present logs inline
[07:15:30] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 12 statements
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:15:31] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2266644065
--- FAIL: TestCCLLogic_regional_by_row_read_committed (26.60s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 11, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 619aa6edad0269f75d12cb7d4b88b52f09825c7b:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2820547160
    test_log_scope.go:81: use -show-logs to present logs inline
[07:57:44] setting distsql_workmem='78741B';
[07:57:44] setting distsql_workmem='78741B';
[07:57:47] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 7 statements
[07:57:49] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 22 statements
    logic.go:2578: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:57:50] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3995: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3995: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3216/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2820547160
--- FAIL: TestCCLLogic_regional_by_row_read_committed (35.69s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 45f6344a2fc7d2d52f410e1ecad98f8c928cfeea:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed1879932226
    test_log_scope.go:81: use -show-logs to present logs inline
[05:32:45] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3978/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 12 statements
    logic.go:2579: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3978/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value violates unique constraint "river_pkey"
        DETAIL: Key (region, name)=('us-east-1', 'Skykomish') already exists.
[05:32:45] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3978/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3996: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3978/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3996: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3978/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed1879932226
--- FAIL: TestCCLLogic_regional_by_row_read_committed (24.09s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 45f6344a2fc7d2d52f410e1ecad98f8c928cfeea:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2813368806
    test_log_scope.go:81: use -show-logs to present logs inline
[07:32:58] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3351/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 11 statements
    logic.go:2579: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3351/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:32:59] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3351/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3996: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3351/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3996: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3351/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed2813368806
--- FAIL: TestCCLLogic_regional_by_row_read_committed (29.05s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@sumeerbhola sumeerbhola mentioned this issue Oct 18, 2023
Merged
@stevendanna stevendanna mentioned this issue Oct 19, 2023
Merged
@yuzefovich
Copy link
Member

yuzefovich commented Oct 19, 2023 •
edited
Loading

I'm making some incremental progress on understanding this issue. I found that the reproduction cycle is faster for me with the original TestCCLLogic_regional_by_row_read_committed failure, so I've been using that. I added a bunch of logging, here is the current output:

        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: j=1 b=&{0xc0229d0e60 [calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
         calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
         calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Rows
        0: /Tenant/10/Table/113/1/"\xc0"/"Skykomish"/0=nil
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
         calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Rows
        0: /Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0=nil
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
         calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        ] {0,0 <nil> 0,0 (n0,s0):? 0 normal <nil> CONSISTENT LEASEHOLDER Block 0s 0 0 0 false false false false false 0 {0 0 LAG_BY_CLUSTER_SETTING false} <nil> <nil> <nil> false [] false} {0 1697757719598956812 FROM_SQL REMOTE false} [{0xc01fc342c0} {0xc01fc342c8} {0xc01fc342d0} {0xc01fc342d8} {0xc0271a8b98}] 123 false <nil> 0xc0065b3c40 [{1 0xc000696be8 [] [] <nil> 0 0} {1 0xc000696bf0 [] [] <nil> 0 0} {1 0xc000696bf8 [{[254 146 246 113 137 18 192 0 1 18 83 107 121 107 111 109 105 115 104 0 1 136] <nil>}] [] <nil> 0 0} {1 0xc000696c00 [{[254 146 246 113 138 18 192 0 1 18 83 97 108 105 115 104 32 83 101 97 0 1 136] <nil>}] [] <nil> 0 0} {1 0xc000696c88 [] [] <nil> 0 0} {0 <nil> [] [] <nil> 0 0} {0 <nil> [] [] <nil> 0 0} {0 <nil> [] [] <nil> 0 0}] [] [{[254 146 246 113 137 18 192 0 1 18 83 107 121 107 111 109 105 115 104 0 1 136] <nil>} {[254 146 246 113 138 18 192 0 1 18 83 97 108 105 115 104 32 83 101 97 0 1 136] <nil>} {[] <nil>} {[] <nil>} {[] <nil>} {[] <nil>} {[] <nil>} {[] <nil>}] 2}
        reqs
        0: delete:<header:<key:"\376\222\366q\212\022@\000\001\022Snohomish\000\001\210" sequence:1 > > 
        1: delete:<header:<key:"\376\222\366q\211\022@\000\001\022Skykomish\000\001\210" sequence:2 > > 
        2: conditional_put:<header:<key:"\376\222\366q\211\022\300\000\001\022Skykomish\000\001\210" sequence:3 > value:<raw_bytes:"\227Q\362$\n6\nSalish Sea" timestamp:<> > > 
        3: init_put:<header:<key:"\376\222\366q\212\022\300\000\001\022Salish Sea\000\001\210" sequence:4 > value:<raw_bytes:"\235\355\243\361\003\022Skykomish\000\001" timestamp:<> > > 
        4: end_txn:<header:<key:"\376\222\366q\211\022@\000\001\022Skykomish\000\001\210" sequence:5 > commit:true deadline:<wall_time:1697758023981565505 > lock_spans:<key:"\376\222\366q\211\022@\000\001\022Skykomish\000\001\210" > lock_spans:<key:"\376\222\366q\211\022\200\000\001\022Skykomish\000\001\210" > lock_spans:<key:"\376\222\366q\211\022\300\000\001\022Skykomish\000\001\210" > in_flight_writes:<key:"\376\222\366q\212\022@\000\001\022Snohomish\000\001\210" sequence:1 > in_flight_writes:<key:"\376\222\366q\211\022@\000\001\022Skykomish\000\001\210" sequence:2 > in_flight_writes:<key:"\376\222\366q\211\022\300\000\001\022Skykomish\000\001\210" sequence:3 > in_flight_writes:<key:"\376\222\366q\212\022\300\000\001\022Salish Sea\000\001\210" sequence:4 > > 
        results
        0: calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        1: calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        2: calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Rows
        0: /Tenant/10/Table/113/1/"\xc0"/"Skykomish"/0=nil
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        3: calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Rows
        0: /Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0=nil
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        4: calls=1, Err=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > 
        Resume: span=<nil>, reason=RESUME_UNKNOWN, bytes=0
        origErr=unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697757719581895625 > : "sql txn" meta={id=cf9f7e80 key=/Tenant/10/Table/113/1/"@"/"Skykomish"/0 iso=ReadCommitted pri=0.00799033 epo=0 ts=1697757719.599003908,0 min=1697757719.598951171,0 seq=5} lock=true stat=PENDING rts=1697757719.599003908,0 wto=false gul=1697757720.098951171,0
        hypothetical error = duplicate key value violates unique constraint "river_region_outlet_key"
        : duplicate key value got decoding error: insufficient bytes to decode uvarint value

(Note that if at least one request in a kv.Batch results in an error, then all results are updated to have that error set, so all 5 above have it.)

The interesting bit is that j that is the position of the result in kv.Batch that we use to come up with an error is 1, meaning that it points to the result of the delete request. Unsurprisingly, that results in empty KeyValue{} which is treated as empty key for the decoding purposes.

I added a "hypothetical error" by additionally attempting to get an error using j = 3 (which corresponds to the init_put that we expect to fail), and the hypothetical error matches what we expect.

Thus, it seems like we're setting origPErr.Index.Index incorrectly. I'll keep looking.

@yuzefovich
Copy link
Member

I think it has something to do with range splits. I disabled spanconfig.reconciliation_job.enabled, and the problem didn't repro within an hour, while with that setting enabled usually I get a repro within a few minutes. This starts to smell more like a KV issue, but I'll spend a bit more time on it before handing off.

@yuzefovich
Copy link
Member

yuzefovich commented Oct 20, 2023 •
edited
Loading

I think I understand what's happening. The problem is in the DistSender when it has to retry a BatchRequest because the range cache had stale routing information.

Here are the logs I got 
I231020 03:34:30.526311 55909 kv/kvserver/replica_command.go:439  [T1,Vsystem,n2,split,s2,r108/1:/Tenant/1{0/Table/…-1}] 2367  initiating a split of this range at key /Tenant/10/Table/113/2/"\xc0" [r109] (span config)
E231020 03:34:30.527780 49080 kv/kvclient/kvcoord/dist_sender.go:1953  [T10,Vcluster-10,nsql1,client=127.0.0.1:54448,hostssl,user=root] 2368  reply error Delete [/Tenant/10/Table/113/2/"@"/"Snohomish"/0,/Min), InitPut [/Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0,/Min), [txn: ddef4553]: key range /Tenant/10/Table/113/2/"@"/"Snohomish"/0-/Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0/NULL outside of bounds of range /Tenant/10/Table/113/2/"@"-/Tenant/10/Table/113/2/"@"/PrefixEnd; suggested ranges: [desc: r107:/Tenant/10/Table/113/2/"@"{-/PrefixEnd} [(n2,s2):1, (n6,s6):2, (n8,s8):3, next=4, gen=51], lease: repl=(n2,s2):1 seq=7 start=1697772866.124097289,0 epo=1 pro=1697772866.125432862,0, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING desc: r108:/Tenant/1{0/Table/113/2/"@"/PrefixEnd-1} [(n2,s2):1, (n6,s6):2, (n8,s8):3, next=4, gen=51], lease: repl=(n2,s2):1 seq=7 start=1697772866.124097289,0 epo=1 pro=1697772866.125432862,0, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING]: "sql txn" meta={id=ddef4553 key=/Tenant/10/Table/113/1/"@"/"Skykomish"/0 iso=ReadCommitted pri=0.01635565 epo=0 ts=1697772870.520523662,0 min=1697772870.520456914,0 seq=0} lock=true stat=PENDING rts=1697772870.520523662,0 wto=false gul=1697772871.020456914,0 (positions=[0 3])
I231020 03:34:30.528232 49080 kv/kvclient/kvcoord/dist_sender.go:1987  [T10,Vcluster-10,nsql1,client=127.0.0.1:54448,hostssl,user=root] 2369  likely split; will resend. Got new descriptors: [desc: r107:/Tenant/10/Table/113/2/"@"{-/PrefixEnd} [(n2,s2):1, (n6,s6):2, (n8,s8):3, next=4, gen=51], lease: repl=(n2,s2):1 seq=7 start=1697772866.124097289,0 epo=1 pro=1697772866.125432862,0, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING desc: r108:/Tenant/1{0/Table/113/2/"@"/PrefixEnd-1} [(n2,s2):1, (n6,s6):2, (n8,s8):3, next=4, gen=51], lease: repl=(n2,s2):1 seq=7 start=1697772866.124097289,0 epo=1 pro=1697772866.125432862,0, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING] (positions=[0 3])
I231020 03:34:30.528898 55734 kv/kvserver/replica_evaluate.go:376  [T1,Vsystem,n2,tenant=10,s2,r108/1:/Tenant/1{0/Table/…-1}] 2370  index=0, ba=InitPut [/Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0,/Min), [txn: ddef4553]
goroutine 49080 [running]:
runtime/debug.Stack()
	GOROOT/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	GOROOT/src/runtime/debug/stack.go:16 +0x19
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0043ea580, {0x7b5abd0, 0xc01e4d0fc0}, 0xc01a953680?, {0xc010dbfd10, 0xc015617110, 0xc015617180, 0x0, 0x0}, 0x1)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2395 +0x12ad
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0043ea580, {0x7b5abd0?, 0xc01e4d0fc0}, 0xc01a953680, {{0xc00534c300, 0x17, 0x20}, {0xc00534c300, 0x18, 0x20}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1887 +0x7b4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0043ea580, {0x7b5abd0, 0xc01e4d0fc0}, 0xc00b4aca20, {{0xc00534c300, 0x17, 0x20}, {0xc00534c300, 0x18, 0x20}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1656 +0x13b6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0043ea580, {0x7b5abd0?, 0xc01e4d0fc0}, 0xc00b4aca20, {{0xc00534c240, 0x16, 0x20}, {0xc00534c300, 0x18, 0x20}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1988 +0x17ba
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0043ea580, {0x7b5abd0, 0xc01e4d0fc0}, 0xc00b4ac360, {{0xc00534c240, 0x16, 0x20}, {0xc00534c300, 0x18, 0x20}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1656 +0x13b6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendParallelCommit(0xc0043ea580, {0x7b5abd0, 0xc01e4d0fc0}, 0xc00b4ac360, {{0xc00534c2c0, 0x16, 0x20}, {0xc00534c300, 0x18, 0x20}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1183 +0xa5e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0043ea580, {0x7b5abd0, 0xc01e4d0f90}, 0xc00b4ac360)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1073 +0x750
...
E231020 03:34:30.530431 49080 kv/kvclient/kvcoord/dist_sender.go:1953  [T10,Vcluster-10,nsql1,client=127.0.0.1:54448,hostssl,user=root] 2371  reply error InitPut [/Tenant/10/Table/113/2/"\xc0"/"Salish Sea"/0,/Min), [txn: ddef4553]: unexpected value: raw_bytes:"d\214\322\370\003\022Fraser\000\001" timestamp:<wall_time:1697772870515220440 > : "sql txn" meta={id=ddef4553 key=/Tenant/10/Table/113/1/"@"/"Skykomish"/0 iso=ReadCommitted pri=0.01635565 epo=0 ts=1697772870.520523662,0 min=1697772870.520456914,0 seq=4} lock=true stat=PENDING rts=1697772870.520523662,0 wto=false gul=1697772871.020456914,0 (positions=[1])

Here is what I think is happening:

  • a range containing the unique secondary index is split
  • the DistSender has stale information that doesn't know about that split, so it attempts to execute a Batch with two requests (Delete and InitPut) which have positions [0, 3] (which refer to the ordinals within the original multi-range batch that had 5 request (Delete, Delete, CPut, InitPut, EndTxn))
  • sendToReplicas results in a kvpb.RangeKeyMismatchError, so the DistSender recursively calls divideAndSendBatchToRanges
  • crucially, it doesn't pass positions slice here
  • that BatchRequest with Delete and InitPut is split into two, and InitPut gets positions = [1]
  • that InitPut results in kvpb.ConditionFailedError with origPErr.Index.Index = 0 which is then adjusted via positions[0] = 1, so in the end we have origPErr.Index.Index = 1
  • that is incorrect - we should have set it to 3 to refer to the "original" batch with 5 requests correctly.

AFAICT this bug has been present for a long time, and it's not multitenant-specific - we just happened to hit it in this config.

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 20, 2023
Closed
@kvoli kvoli mentioned this issue Oct 20, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed on master @ 59a69ae0c5474a8b323afac269055bf6ce8afa34:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCCLLogic_regional_by_row_read_committed2596540546
    test_log_scope.go:81: use -show-logs to present logs inline
[17:32:51] setting distsql_workmem='98203B';
[17:32:51] setting distsql_workmem='98203B';
    logic.go:2581: 
         
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[17:32:52] --- done: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: outputs.zip/logTestCCLLogic_regional_by_row_read_committed2596540546
--- FAIL: TestCCLLogic_regional_by_row_read_committed (11.01s)

Parameters: attempt=1 , run=10 , shard=15

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed on master @ 59a69ae0c5474a8b323afac269055bf6ce8afa34:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCCLLogic_regional_by_row_read_committed3428247824
    test_log_scope.go:81: use -show-logs to present logs inline
[17:32:54] setting distsql_workmem='30036B';
[17:32:54] setting distsql_workmem='30036B';
    logic.go:2581: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[17:32:56] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: outputs.zip/logTestCCLLogic_regional_by_row_read_committed3428247824
--- FAIL: TestCCLLogic_regional_by_row_read_committed (13.42s)

Parameters: attempt=1 , run=16 , shard=15

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed on master @ 59a69ae0c5474a8b323afac269055bf6ce8afa34:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCCLLogic_regional_by_row_read_committed127580084
    test_log_scope.go:81: use -show-logs to present logs inline
[17:32:51] setting distsql_workmem='75373B';
[17:32:51] setting distsql_workmem='75373B';
[17:32:54] --- progress: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 26 statements
    logic.go:2581: 
         
        /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[17:32:54] --- done: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: outputs.zip/logTestCCLLogic_regional_by_row_read_committed127580084
--- FAIL: TestCCLLogic_regional_by_row_read_committed (13.17s)

Parameters: attempt=1 , run=9 , shard=15

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed on master @ 59a69ae0c5474a8b323afac269055bf6ce8afa34:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCCLLogic_regional_by_row_read_committed970791551
    test_log_scope.go:81: use -show-logs to present logs inline
    logic.go:2581: 
         
        /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[17:31:15] --- done: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: outputs.zip/logTestCCLLogic_regional_by_row_read_committed970791551
--- FAIL: TestCCLLogic_regional_by_row_read_committed (11.02s)

Parameters: attempt=1 , run=7 , shard=15

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rickystewart
Copy link
Collaborator

FYI: Previous 4 comments are failures on EngFlow. Failures seem the same as previous.

@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 22, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed on master @ bf9a22dc85683966ee65e84d3eeadf2b44185127:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestCCLLogic_regional_by_row_read_committed186029346
    test_log_scope.go:81: use -show-logs to present logs inline
[21:19:36] setting distsql_workmem='95462B';
[21:19:36] setting distsql_workmem='95462B';
    logic.go:2581: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[21:19:37] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:522: -- test log scope end --
test logs left over in: outputs.zip/logTestCCLLogic_regional_by_row_read_committed186029346
--- FAIL: TestCCLLogic_regional_by_row_read_committed (10.16s)

Parameters: attempt=1 , run=11 , shard=15

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@yuzefovich
Copy link
Member

I should get to it this week, so I'll take this over.

@yuzefovich yuzefovich assigned yuzefovich and unassigned michae2 Oct 24, 2023
@yuzefovich
Copy link
Member

I think this diff

--- a/pkg/kv/kvclient/kvcoord/dist_sender.go
+++ b/pkg/kv/kvclient/kvcoord/dist_sender.go
@@ -1879,6 +1879,9 @@ func (ds *DistSender) sendPartialBatch(
                        if !intersection.Equal(rs) {
                                log.Eventf(ctx, "range shrunk; sub-dividing the request")
                                reply, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, batchIdx)
+                               if pErr != nil && pErr.Index != nil && pErr.Index.Index != -1 && positions != nil {
+                                       pErr.Index.Index = int32(positions[pErr.Index.Index])
+                               }
                                return response{reply: reply, positions: positions, pErr: pErr}
                        }
                }
@@ -1986,6 +1989,9 @@ func (ds *DistSender) sendPartialBatch(
                        // with unknown mapping to our truncated reply).
                        log.Infof(ctx, "likely split; will resend. Got new descriptors: %s (positions=%v)", tErr.Ranges, positions)
                        reply, pErr = ds.divideAndSendBatchToRanges(ctx, ba, rs, isReverse, withCommit, batchIdx)
+                       if pErr != nil && pErr.Index != nil && pErr.Index.Index != -1 && positions != nil {
+                               pErr.Index.Index = int32(positions[pErr.Index.Index])
+                       }
                        return response{reply: reply, positions: positions, pErr: pErr}
                }
                break

should fix the problem. I'm currently thinking through (and asking for pointers) on how to write a targeted unit test.

@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 8315a4bc997fb8b8d679079e14d6d7ca94d53bc6:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed759714094
    test_log_scope.go:81: use -show-logs to present logs inline
[07:49:05] setting distsql_workmem='50442B';
[07:49:06] setting distsql_workmem='50442B';
[07:49:08] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3279/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 7 statements
    logic.go:2581: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3279/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:49:09] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3279/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3279/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3279/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:523: -- test log scope end --
--- FAIL: TestCCLLogic_regional_by_row_read_committed (44.51s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@yuzefovich yuzefovich mentioned this issue Oct 26, 2023
Merged
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ 2205334339583cea57db33e3c0d992f672e660e8:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed2888135195
    test_log_scope.go:81: use -show-logs to present logs inline
[06:17:01] setting distsql_workmem='42930B';
[06:17:01] setting distsql_workmem='42930B';
[06:17:03] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5286/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 11 statements
    logic.go:2581: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5286/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[06:17:03] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5286/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5286/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/5286/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/24b232bb2ddb5cbd23310506078b09b9/logTestCCLLogic_regional_by_row_read_committed2888135195
--- FAIL: TestCCLLogic_regional_by_row_read_committed (19.34s)
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

This was referenced Oct 26, 2023
Closed
Closed
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row_read_committed failed with artifacts on master @ bfc5c35b112fe612d837ef247b2c61e532e8184d:

=== RUN   TestCCLLogic_regional_by_row_read_committed
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row_read_committed3389879511
    test_log_scope.go:81: use -show-logs to present logs inline
[07:47:56] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 7 statements
[07:47:59] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed: 23 statements
    logic.go:2581: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:180: 
        expected:
        pq: duplicate key value violates unique constraint "river_region_outlet_key"\nDETAIL: Key \(region, outlet\)=\('us-east-1', 'Salish Sea'\) already exists\.
        got:
        pq: duplicate key value got decoding error: insufficient bytes to decode uvarint value
[07:48:00] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed with config multiregion-9node-3region-3azs-tenant: 32 tests, 1 failures
    logic.go:3998: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: error while processing
    logic.go:3998: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3282/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_read_committed:183: too many errors encountered, skipping the rest of the input
    panic.go:523: -- test log scope end --
--- FAIL: TestCCLLogic_regional_by_row_read_committed (45.40s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@craig craig bot closed this as completed in f515f70 Nov 1, 2023
@github-project-automation github-project-automation bot moved this from Active to Done in SQL Queries Nov 1, 2023
This was referenced Nov 1, 2023
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuzefovich yuzefovich

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

kvcoord: fix error index in case of range split yuzefovich/cockroach
5 participants
@rickystewart @msirek @yuzefovich @cockroach-teamcity @michae2