Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ccl/multiregionccl: TestMultiRegionDataDriven failed #92235

Closed
cockroach-teamcity opened this issue Nov 21, 2022 · 27 comments
Closed

ccl/multiregionccl: TestMultiRegionDataDriven failed #92235

cockroach-teamcity opened this issue Nov 21, 2022 · 27 comments
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 21, 2022

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ bf7c1cf83275be4645fc77581be2cb9d5e8502c4:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3645059334
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3645059334
--- FAIL: TestMultiRegionDataDriven (226.53s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2722/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (135.09s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/multiregion

This test on roachdash | Improve this report!

Jira issue: CRDB-21666

@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Nov 21, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Nov 21, 2022
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 2a71a5cca1f66313fd39956f1a3db070f67ade15:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3832146062
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3832146062
--- FAIL: TestMultiRegionDataDriven (214.59s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2499/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (135.41s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ fd133ba3af3954c93584a501cf68dfa3131e586c:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4163406203
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4163406203
--- FAIL: TestMultiRegionDataDriven (207.03s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2997/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (133.67s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ d978988afcfc78c02c23a3836ba90e4a4faeeb25:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3003480903
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3003480903
--- FAIL: TestMultiRegionDataDriven (207.38s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2971/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (132.43s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ fc04d8e1eabf6b58d32e9792fd11f97c493480ad:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4153465453
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4153465453
--- FAIL: TestMultiRegionDataDriven (400.01s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3734/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (189.22s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 773649a2b08605ba2031dce9b3c8858b15803ca2:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4165274069
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4165274069
--- FAIL: TestMultiRegionDataDriven (215.90s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3020/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (135.38s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ ea0e966fe05a788fd123a08cec8ef5e935fa5496:

             1.759ms      0.045ms    event:sql/conn_executor_exec.go:689 [n4,client=127.0.0.1:49502,user=root] executing after 0 retries, last retry reason: <nil>
             1.794ms      0.035ms    event:sql/conn_executor_exec.go:689 [n4,client=127.0.0.1:49502,user=root] execution starts: distributed engine
             1.833ms      0.038ms    event:sql/distsql_running.go:1671 [n4,client=127.0.0.1:49502,user=root] creating DistSQL plan with isLocal=true
             1.903ms      0.070ms    event:sql/distsql_running.go:628 [n4,client=127.0.0.1:49502,user=root] running DistSQL plan
             1.938ms      0.035ms        === operation:flow _verbose:1 node:4 client:127.0.0.1:49502 user:root
             2.023ms      0.085ms        event:sql/colflow/vectorized_flow.go:241 [n4,client=127.0.0.1:49502,user=root] setting up vectorized flow
             2.121ms      0.098ms        event:sql/colflow/vectorized_flow.go:293 [n4,client=127.0.0.1:49502,user=root] vectorized flow setup succeeded
             2.153ms      0.033ms        event:sql/flowinfra/flow.go:387 [n4,client=127.0.0.1:49502,user=root] starting (0 processors, 0 startables) asynchronously
             2.177ms      0.024ms        event:sql/colflow/vectorized_flow.go:319 [n4,client=127.0.0.1:49502,user=root] running the batch flow coordinator in the flow's goroutine
             6.580ms      4.403ms        event:component:<flow_id:<4de11b53-5926-466e-8c53-73a9863650be> type:FLOW id:0 sql_instance_id:4 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<> tuples_read:<> batch_requests_issued:<> kv_time:<> contention_time:<> num_interface_steps:<> num_internal_steps:<> num_interface_seeks:<> num_internal_seeks:<> > exec:<exec_time:<> max_allocated_mem:<> max_allocated_disk:<> consumed_r_u:<> > output:<num_batches:<> num_tuples:<> > flow_stats:<max_mem_usage:<value_plus_one:20481 > max_disk_usage:<value_plus_one:1 > consumed_r_u:<> >
             6.501ms     -0.079ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"4de11b53-5926-466e-8c53-73a9863650be","type":"FLOW","id":0,"sqlInstanceId":4},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"0"},"tuplesRead":{"valuePlusOne":"0"},"batchRequestsIssued":{"valuePlusOne":"0"},"kvTime":{"valuePlusOne":"0s"},"contentionTime":{"valuePlusOne":"0s"},"numInterfaceSteps":{"valuePlusOne":"0"},"numInternalSteps":{"valuePlusOne":"0"},"numInterfaceSeeks":{"valuePlusOne":"0"},"numInternalSeeks":{"valuePlusOne":"0"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"0"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}},"output":{"numBatches":{"valuePlusOne":"0"},"numTuples":{"valuePlusOne":"0"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"20481"},"maxDiskUsage":{"valuePlusOne":"1"},"consumedRU":{"valuePlusOne":"0"}}}
             2.185ms     -4.316ms        === operation:batch flow coordinator _unfinished:1 _verbose:1 node:4 client:127.0.0.1:49502 user:root cockroach.flowid:4de11b53-5926-466e-8c53-73a9863650be cockroach.processorid:0
             6.108ms      3.923ms        event:component:<flow_id:<4de11b53-5926-466e-8c53-73a9863650be> type:PROCESSOR id:0 sql_instance_id:4 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<value_plus_one:12 > tuples_read:<value_plus_one:2 > batch_requests_issued:<value_plus_one:2 > kv_time:<value_plus_one:<nanos:3479869 > > contention_time:<value_plus_one:<nanos:1 > > num_interface_steps:<value_plus_one:1 > num_internal_steps:<value_plus_one:1 > num_interface_seeks:<value_plus_one:3 > num_internal_seeks:<value_plus_one:3 > > exec:<exec_time:<> max_allocated_mem:<value_plus_one:20481 > max_allocated_disk:<> consumed_r_u:<value_plus_one:1 > > output:<num_batches:<value_plus_one:2 > num_tuples:<value_plus_one:2 > > flow_stats:<max_mem_usage:<> max_disk_usage:<> consumed_r_u:<> >
             6.223ms      0.115ms        event:component:<flow_id:<00000000-0000-0000-0000-000000000000> type:UNSET id:0 sql_instance_id:0 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<> tuples_read:<> batch_requests_issued:<> kv_time:<> contention_time:<> num_interface_steps:<> num_internal_steps:<> num_interface_seeks:<> num_internal_seeks:<> > exec:<exec_time:<> max_allocated_mem:<> max_allocated_disk:<> consumed_r_u:<> > output:<num_batches:<> num_tuples:<> > flow_stats:<max_mem_usage:<> max_disk_usage:<> consumed_r_u:<> >
             5.923ms     -0.299ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"4de11b53-5926-466e-8c53-73a9863650be","type":"PROCESSOR","id":0,"sqlInstanceId":4},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"12"},"tuplesRead":{"valuePlusOne":"2"},"batchRequestsIssued":{"valuePlusOne":"2"},"kvTime":{"valuePlusOne":"0.003479869s"},"contentionTime":{"valuePlusOne":"0.000000001s"},"numInterfaceSteps":{"valuePlusOne":"1"},"numInternalSteps":{"valuePlusOne":"1"},"numInterfaceSeeks":{"valuePlusOne":"3"},"numInternalSeeks":{"valuePlusOne":"3"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"20481"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"1"}},"output":{"numBatches":{"valuePlusOne":"2"},"numTuples":{"valuePlusOne":"2"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}}}
             6.140ms      0.216ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"00000000-0000-0000-0000-000000000000","type":"UNSET","id":0,"sqlInstanceId":0},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"0"},"tuplesRead":{"valuePlusOne":"0"},"batchRequestsIssued":{"valuePlusOne":"0"},"kvTime":{"valuePlusOne":"0s"},"contentionTime":{"valuePlusOne":"0s"},"numInterfaceSteps":{"valuePlusOne":"0"},"numInternalSteps":{"valuePlusOne":"0"},"numInterfaceSeeks":{"valuePlusOne":"0"},"numInternalSeeks":{"valuePlusOne":"0"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"0"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}},"output":{"numBatches":{"valuePlusOne":"0"},"numTuples":{"valuePlusOne":"0"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}}}
             2.238ms     -3.901ms        === operation:colbatchscan _unfinished:1 _verbose:1 node:4 client:127.0.0.1:49502 user:root
             2.238ms      0.000ms        [/cockroach.roachpb.Internal/Batch: {count: 2, duration 3ms}]
             2.238ms      0.000ms        [txn coordinator send: {count: 1, duration 3ms}]
             2.238ms      0.000ms        [dist sender send: {count: 1, duration 3ms}]
             2.542ms      0.304ms        event:sql/row/kv_batch_fetcher.go:406 [n4,client=127.0.0.1:49502,user=root] Scan /Table/108/1/1/0
             2.579ms      0.037ms            === operation:txn coordinator send _verbose:1 node:4 client:127.0.0.1:49502 user:root txnID:d4b9092a-b40d-40a0-b0a1-a13904f770db
             2.579ms      0.000ms            [/cockroach.roachpb.Internal/Batch: {count: 2, duration 3ms}]
             2.579ms      0.000ms            [dist sender send: {count: 1, duration 3ms}]
             2.653ms      0.073ms                === operation:dist sender send _verbose:1 node:4 client:127.0.0.1:49502 user:root txn:d4b9092a
             2.653ms      0.000ms                [/cockroach.roachpb.Internal/Batch: {count: 2, duration 3ms}]
             2.799ms      0.147ms                event:kv/kvclient/kvcoord/range_iter.go:183 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] querying next range at /Table/108/1/1/0
             2.937ms      0.137ms                event:kv/kvclient/kvcoord/range_iter.go:216 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] key: /Table/108/1/1/0, desc: r53:/{Table/108-Max} [(n1,s1):1, (n2,s2):9, (n3,s3):5, (n5,s5):6NON_VOTER, next=12, gen=28]
             3.012ms      0.076ms                event:kv/kvclient/kvcoord/dist_sender.go:1988 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] routing to nearest replica; leaseholder not required
             3.170ms      0.158ms                event:kv/kvclient/kvcoord/dist_sender.go:2046 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] r53: sending batch 1 Get to (n1,s1):1
             3.241ms      0.071ms                event:rpc/nodedialer/nodedialer.go:163 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] sending request to 127.0.0.1:42471
             3.279ms      0.037ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:4 client:127.0.0.1:49502 user:root txn:d4b9092a span.kind:client node:4
             3.611ms      0.333ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:1 span.kind:server request:Get [/Table/108/1/1/0,/Min), [txn: d4b9092a], [can-forward-ts]
             3.711ms      0.100ms                    event:server/node.go:1074 [n1] node received request: 1 Get
             3.847ms      0.136ms                    event:kv/kvserver/store_send.go:166 [n1,s1] executing Get [/Table/108/1/1/0,/Min), [txn: d4b9092a], [can-forward-ts]
             3.942ms      0.095ms                    event:kv/kvserver/replica_send.go:174 [n1,s1,r53/1:/{Table/108-Max}] read-only path
             4.028ms      0.085ms                    event:kv/kvserver/concurrency/concurrency_manager.go:193 [n1,s1,r53/1:/{Table/108-Max}] sequencing request
             4.065ms      0.038ms                    event:kv/kvserver/concurrency/concurrency_manager.go:274 [n1,s1,r53/1:/{Table/108-Max}] acquiring latches
             4.119ms      0.054ms                    event:kv/kvserver/concurrency/concurrency_manager.go:318 [n1,s1,r53/1:/{Table/108-Max}] scanning lock table for conflicting locks
             4.244ms      0.125ms                    event:kv/kvserver/replica_read.go:269 [n1,s1,r53/1:/{Table/108-Max}] executing read-only batch
             4.423ms      0.179ms                    event:scan stats: stepped 0 times (0 internal); seeked 2 times (2 internal); block-bytes: (total 12 KiB, cached 12 KiB); points: (count 2 , key-bytes 27 B, value-bytes 85 B, tombstoned: 0 )
             4.581ms      0.158ms                    event:kv/kvserver/replica_evaluate.go:528 [n1,s1,r53/1:/{Table/108-Max}] evaluated Get command header:<key:"\364\211\211\210" > , txn="sql txn" meta={id=d4b9092a key=/Min pri=0.05801534 epo=0 ts=1672040252.178044973,0 min=1672040252.178044973,0 seq=0} lock=false stat=PENDING rts=1672040252.178044973,0 wto=false gul=1672040252.678044973,0 : resp=header:<num_keys:1 num_bytes:7 > value:<raw_bytes:"\375\311[\010\n#\002" timestamp:<wall_time:1672040236935388680 > > , err=<nil>
             4.714ms      0.133ms                    event:kv/kvserver/replica_read.go:204 [n1,s1,r53/1:/{Table/108-Max}] read completed
             4.354ms     -0.360ms                    structured:{"@type":"type.googleapis.com/cockroach.roachpb.ScanStats","numInterfaceSeeks":"2","numInternalSeeks":"2","numInterfaceSteps":"0","numInternalSteps":"0","blockBytes":"11995","blockBytesInCache":"11995","keyBytes":"27","valueBytes":"85","pointCount":"2","pointsCoveredByRangeTombstones":"0"}
             5.544ms      2.965ms            event:kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:572 [n4,client=127.0.0.1:49502,user=root,txn=d4b9092a] recording span to refresh: /Table/108/1/1/0
             6.721ms      4.818ms    event:sql/conn_executor_exec.go:689 [n4,client=127.0.0.1:49502,user=root] execution ends
             6.748ms      0.027ms    event:sql/conn_executor_exec.go:689 [n4,client=127.0.0.1:49502,user=root] rows affected: 1
             7.136ms      0.388ms        === operation:commit sql txn _verbose:1 node:4 client:127.0.0.1:49502 user:root
             7.516ms      0.768ms    event:sql/conn_executor_exec.go:2164 [n4,client=127.0.0.1:49502,user=root] AutoCommit. err: <nil>
    --- FAIL: TestMultiRegionDataDriven/global_tables (58.72s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ ea0e966fe05a788fd123a08cec8ef5e935fa5496:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1870660309
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1870660309
--- FAIL: TestMultiRegionDataDriven (200.94s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2686/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.12s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ afab335620e28c5d8c849ba09e50ec12c2836b3f:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4239224628
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4239224628
--- FAIL: TestMultiRegionDataDriven (418.75s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3827/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (191.78s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1091230165
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1091230165
--- FAIL: TestMultiRegionDataDriven (572.28s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3159/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (230.42s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 00ed5143845ec05797d16e6ab61d179cf51775f2:

=== RUN   TestMultiRegionDataDriven
I230106 07:54:44.922543 1 (gostd) rand.go:176  [-] 1  random seed: 4186920347575873814
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1681677763
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1681677763
--- FAIL: TestMultiRegionDataDriven (391.70s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3746/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (178.60s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ cce52c9eab1210c6cd508f6b99b2bad0ba3e1e56:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven226811927
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven226811927
--- FAIL: TestMultiRegionDataDriven (472.71s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3744/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (255.27s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 95b59e058c3dc01888204f46b6ff5a10962481e5:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven15522368
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven15522368
--- FAIL: TestMultiRegionDataDriven (387.48s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3741/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (180.06s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 81a114c2bc2ef8ef76fe3809d0469319c7e82635:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3378265953
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3378265953
--- FAIL: TestMultiRegionDataDriven (375.29s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2750/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (172.94s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ f15ad13ff5b197f881f333085a2fda170857b50b:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven718282129
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven718282129
--- FAIL: TestMultiRegionDataDriven (569.40s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/877/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (221.45s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ ab9f14774b18845d0fd0867aa8e35a60b2acc91c:

=== RUN   TestMultiRegionDataDriven
I230120 08:20:24.163501 1 (gostd) rand.go:176  [-] 1  random seed: -1836899639072009080
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven2259539114
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven2259539114
--- FAIL: TestMultiRegionDataDriven (414.88s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3745/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (176.95s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ b4d99da9798bb81dd338dea801056006296ec393:

=== RUN   TestMultiRegionDataDriven
I230122 08:10:59.240129 1 (gostd) rand.go:176  [-] 1  random seed: 396977897907459761
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1842661069
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1842661069
--- FAIL: TestMultiRegionDataDriven (515.65s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3490/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (236.28s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 888d521c163d8e8324dfe00a11ef9c2729d5f002:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1524340156
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1524340156
--- FAIL: TestMultiRegionDataDriven (544.51s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1446/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (225.87s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 799d8dbba62fa9c942a780584c71f8ceab784381:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3066130366
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3066130366
--- FAIL: TestMultiRegionDataDriven (212.11s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3060/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (133.70s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 799d8dbba62fa9c942a780584c71f8ceab784381:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven89741057
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven89741057
--- FAIL: TestMultiRegionDataDriven (576.97s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2909/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (256.78s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 848bd2439016f00a0e6de71dad838fb59bc9de4b:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1953378458
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1953378458
--- FAIL: TestMultiRegionDataDriven (198.76s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3301/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.05s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@blathers-crl blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Jan 31, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 84bb3f75313b900eba8493628f8e2807249f9129:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1322872826
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1322872826
--- FAIL: TestMultiRegionDataDriven (353.15s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2910/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (171.35s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 6ab0496d81d303bae3854ee810fe2233ce18791f:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1178485333
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1178485333
--- FAIL: TestMultiRegionDataDriven (201.07s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3321/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.07s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 9bbf9e626b4d159e0c19e1e78ef1923f79929920:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3499721863
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3499721863
--- FAIL: TestMultiRegionDataDriven (196.26s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (128.73s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ 31eb3ce6ceb18123fd2e7eac464b25e203d7de0f:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven2246921699
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven2246921699
--- FAIL: TestMultiRegionDataDriven (220.56s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3314/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (136.62s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss
Copy link
Collaborator

rafiss commented Feb 21, 2023

@e-mbrown can you skip this test during the stress CI job?

craig bot pushed a commit that referenced this issue Feb 22, 2023
96235: sem/tree: add support for producing vectorized data from strings r=cucaroach a=cucaroach

tree.ValueHandler exposes raw machine type hooks that are used by
vec_handler to build coldata.Vec's.

Epic: CRDB-18892
Informs: #91831
Release note: None


96460: upgrades,systemschema: add covering index to system.privileges on `path` r=andyyang890 a=e-mbrown

Resolves #93525

Queries to the `system.privileges` table are usually filtered by path. With a primary key using `(user,path)` all those queries were full scans.  This commit changes the primary key to be `(path,user)`.

Release note: None

97272: *: update to use full package name for `option go_package` in protos r=rail,stevendanna,RaduBerinde a=rickystewart

Everywhere in the entire tree we use `option go_package` statements like
the following:

```
option go_package = "build";
```

We do this *instead of* using the entire fully-qualified package name,
like:

```
option go_package = "github.com/cockroachdb/cockroach/pkg/build";
```

This is apparently just an error. All the documentation I have seen
(https://github.com/cockroachdb/gogoproto/blob/master/README) suggests
that the fully-qualified package name should be used here. Further, this
caused the `make` build to break after a refactor as the code generator
doesn't know how to import a package like `"roachpb"` (since it should
instead be importing `"github.com/cockroachdb/cockroach/pkg/roachpb"`).

Correct this problem *everywhere* and update the `Makefile` to set
`paths=source_relative` to tell `protoc` that output files should be
placed next to input files.

Epic: none
Release note: None

97443: kv: configure transaction uncertainty interval for reverse scan r=arulajmani a=nvanbenschoten

This commit configures the transaction uncertainty intervals for reverse scans. This was initially missed in 261fc35, which broke the use of observed timestamps for reverse scans. In 0ee3bbd, we started using this new plumbing path for the entire uncertainty interval, so the pessimization became a correctness bug that could lead to a loss of real-time ordering between a write and a reverse scan in cases with moderate but bounded clock skew between nodes.

The commit adds testing for this case and the ScanRequest case. I should have exercised these cases back in 8445150 when I noticed there was a complete absence of end-to-end testing for uncertainty interval errors. Unfortunately, that commit only added testing for the GetRequest case, allowing the bug to slip in for ReverseScanRequest.

Release note (bug fix): Transaction uncertainty intervals are correctly configured for reverse scans again, ensuring that reverse scans cannot serve stale reads when clocks in a cluster are skewed.

Epic: None

97473: ccl/multiregionccl: Skip stress testing flaky secondary region test r=rafiss a=e-mbrown

Informs: #92235
Release note: None

97484: release: fetch all tags for pick sha r=celiala a=rail

Previously, the pick sha step relies on querying existing tags, but when a release is created on cherry-pick branch (staging-*), we don't have those tags in the local git checkout.

This PR explicitly fetches all tags to have the cherry-pick release tags available locally.

Fixes: RE-386
Epic: none
Release note: None

Co-authored-by: Tommy Reilly <[email protected]>
Co-authored-by: e-mbrown <[email protected]>
Co-authored-by: Ricky Stewart <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Rail Aliiev <[email protected]>
craig bot pushed a commit that referenced this issue Mar 21, 2023
98207: sql: add REPLICATION and MANAGETENANT system privileges  r=msbutler a=stevendanna

This adds two new system privileges:

- `REPLICATION`: Allows the user to call the internal functions that
produce a cross-cluster replication stream.

- `MANAGETENANT`: Allows the user to create and manage
  tenants.

A user with the MANAGETENANT privileges is now able to execute
the following statements:

- SHOW TENANT
- SHOW TENANTS
- CREATE TENANT
- CREATE TENANT FROM REPLICATION STREAM
- DROP TENANT (if it is part of an active stream)
- ALTER TENANT

A user with the REPLICATION privileges is able to call the following
functions:

- crdb_internal.start_replication_stream
- crdb_internal.replication_stream_progress
- crdb_internal.stream_partition
- crdb_internal.replication_stream_spec
- crdb_internal.complete_replication_stream

Fixes #95425

Release note: None

98785: roachtest:  de-flake multitenant-fairness/read-heavy/skewed r=irfansharif a=irfansharif

Fixes #97448 (possibly).
Fixes #78691.

These tests run under severe CPU overload, and we see the workload
getting observing the following errors:

  ERROR: liveness session expired 571.043163ms before transaction

The SQL liveness lease extension work ends up getting severely starved,
despite extending leases by 40s every 5s. It turns out for tenant
SQL liveness work, we were using admissionpb.NormalPri, so such
starvation was possible. This wasn't true for the system tenant where we
bypassed AC altogether.

Release note: None

98821: sql: TestRandomSyntaxGeneration fixes r=cucaroach a=cucaroach

### sql: fix some TestRandomSyntaxGeneration bugs

The RSG works by calling format on the AST's it generates so its good
at finding Format bugs.

Fix a missing separator in ShowBackupOptions.  Example:

```
SHOW BACKUP 'family' IN ('string', 'placeholder', 'placeholder', 'placeholder', 'string', 'placeholder', 'string', 'placeholder') WITH incremental_location = 'nullif', privilegesdebug_dump_metadata_sst
```

Fix bad construction in ShowTenant.  Example:

```
SHOW TENANT [B'10010'] WITH REPLICATION STATUS WITH CAPABILITIES
```

Epic: none
Release note: None

### copy: fix copy grammar to match PG

Previously COPY would allow a wide range of syntax in the COPY
TO substatement.  Now like PG we limit it to a few things.

PG grammar is:
```
PreparableStmt:
                        SelectStmt
                        | InsertStmt
                        | UpdateStmt
                        | DeleteStmt
                        | MergeStmt
```

And now we do something similar. This prevents the wheels from coming
off when RSG generates EXPLAIN's in the substatement for instance.

Release note: none
Epic: none


99109: kvserver: fortify TestReplicaClosedTimestamp r=erikgrinaker a=tbg

This test was flaky until ~Feb 2nd. This has since resolved, likely as a
result of some other change, however there's an easy way to make the
test a bit more resilient by widening a critical section.

Closes #93864

Epic: none
Release note: None


99122: ccl/multiregionccl: skip flaky secondary_region test r=matthewtodd a=matthewtodd

Part of #92235.
Part of #98020.

It [flaked][1] this morning, after last night's [other skip][2] landed.

[1]: #98020 (comment)
[2]: #99031

Release note: None

Co-authored-by: Steven Danna <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
Co-authored-by: Tommy Reilly <[email protected]>
Co-authored-by: Tobias Grieger <[email protected]>
Co-authored-by: Matthew Todd <[email protected]>
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on release-22.2 @ a875d58204554ea945300d183e5176bff6d4e703:

             2.037ms      0.042ms        event:component:<flow_id:<00000000-0000-0000-0000-000000000000> type:UNSET id:0 sql_instance_id:0 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<> tuples_read:<> batch_requests_issued:<> kv_time:<> contention_time:<> num_interface_steps:<> num_internal_steps:<> num_interface_seeks:<> num_internal_seeks:<> num_gets:<> num_scans:<> num_reverse_scans:<> > exec:<exec_time:<> max_allocated_mem:<> max_allocated_disk:<> consumed_r_u:<> > output:<num_batches:<> num_tuples:<> > flow_stats:<max_mem_usage:<> max_disk_usage:<> consumed_r_u:<> >
             1.929ms     -0.107ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"3b7cf17a-9867-43d0-b955-7ba013ee2bee","type":"PROCESSOR","id":0,"sqlInstanceId":5},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"12"},"tuplesRead":{"valuePlusOne":"2"},"batchRequestsIssued":{"valuePlusOne":"2"},"kvTime":{"valuePlusOne":"0.001343001s"},"contentionTime":{"valuePlusOne":"0.000000001s"},"numInterfaceSteps":{"valuePlusOne":"1"},"numInternalSteps":{"valuePlusOne":"1"},"numInterfaceSeeks":{"valuePlusOne":"3"},"numInternalSeeks":{"valuePlusOne":"3"},"numGets":{"valuePlusOne":"2"},"numScans":{"valuePlusOne":"1"},"numReverseScans":{"valuePlusOne":"1"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"20481"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"1"}},"output":{"numBatches":{"valuePlusOne":"2"},"numTuples":{"valuePlusOne":"2"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}}}
             1.997ms      0.068ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"00000000-0000-0000-0000-000000000000","type":"UNSET","id":0,"sqlInstanceId":0},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"0"},"tuplesRead":{"valuePlusOne":"0"},"batchRequestsIssued":{"valuePlusOne":"0"},"kvTime":{"valuePlusOne":"0s"},"contentionTime":{"valuePlusOne":"0s"},"numInterfaceSteps":{"valuePlusOne":"0"},"numInternalSteps":{"valuePlusOne":"0"},"numInterfaceSeeks":{"valuePlusOne":"0"},"numInternalSeeks":{"valuePlusOne":"0"},"numGets":{"valuePlusOne":"0"},"numScans":{"valuePlusOne":"0"},"numReverseScans":{"valuePlusOne":"0"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"0"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}},"output":{"numBatches":{"valuePlusOne":"0"},"numTuples":{"valuePlusOne":"0"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRU":{"valuePlusOne":"0"}}}
             0.560ms     -1.437ms        === operation:colbatchscan _unfinished:1 _verbose:1 node:5 client:127.0.0.1:43090 user:root
             0.560ms      0.000ms        [txn coordinator send: {count: 1, duration 1ms}]
             0.560ms      0.000ms        [dist sender send: {count: 1, duration 1ms}]
             0.560ms      0.000ms        [/cockroach.roachpb.Internal/Batch: {count: 3, duration 1ms}]
             0.579ms      0.019ms        event:sql/row/kv_batch_fetcher.go:443 [n5,client=127.0.0.1:43090,user=root] Scan /Table/108/1/1/0
             0.582ms      0.003ms            === operation:txn coordinator send _verbose:1 node:5 client:127.0.0.1:43090 user:root txnID:a9779d54-48d6-4390-8ac8-992c5f58f9c8
             0.582ms      0.000ms            [dist sender send: {count: 1, duration 1ms}]
             0.582ms      0.000ms            [/cockroach.roachpb.Internal/Batch: {count: 3, duration 1ms}]
             0.588ms      0.007ms                === operation:dist sender send _verbose:1 node:5 client:127.0.0.1:43090 user:root txn:a9779d54
             0.588ms      0.000ms                [/cockroach.roachpb.Internal/Batch: {count: 3, duration 1ms}]
             0.614ms      0.025ms                event:kv/kvclient/kvcoord/range_iter.go:183 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] querying next range at /Table/108/1/1/0
             0.654ms      0.040ms                event:kv/kvclient/kvcoord/range_iter.go:216 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] key: /Table/108/1/1/0, desc: r53:/{Table/108-Max} [(n3,s3):1, (n1,s1):9, (n2,s2):5, (n4,s4):6NON_VOTER, (n5,s5):11NON_VOTER, next=12, gen=31]
             0.670ms      0.016ms                event:kv/kvclient/kvcoord/dist_sender.go:1991 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] routing to nearest replica; leaseholder not required
             0.696ms      0.026ms                event:kv/kvclient/kvcoord/dist_sender.go:2049 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] r53: sending batch 1 Get to (n5,s5):11NON_VOTER
             0.704ms      0.008ms                event:rpc/nodedialer/nodedialer.go:159 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] sending request to local client
             0.710ms      0.007ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:5 span.kind:server request:Get [/Table/108/1/1/0,/Min), [txn: a9779d54], [can-forward-ts]
             0.717ms      0.007ms                    event:server/node.go:1090 [n5] node received request: 1 Get
             0.743ms      0.026ms                    event:kv/kvserver/store_send.go:166 [n5,s5] executing Get [/Table/108/1/1/0,/Min), [txn: a9779d54], [can-forward-ts]
             0.807ms      0.064ms                    event:server/node.go:1111 [n5] error from stores.Send: r53 was not found on s5: "sql txn" meta={id=a9779d54 key=/Min pri=0.00014794 epo=0 ts=1683450818.815366410,0 min=1683450818.815366410,0 seq=0} lock=false stat=PENDING rts=1683450818.815366410,0 wto=false gul=1683450819.315366410,0
             0.873ms      0.169ms                event:kv/kvclient/kvcoord/dist_sender.go:2161 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] r53 was not found on s5: "sql txn" meta={id=a9779d54 key=/Min pri=0.00014794 epo=0 ts=1683450818.815366410,0 min=1683450818.815366410,0 seq=0} lock=false stat=PENDING rts=1683450818.815366410,0 wto=false gul=1683450819.315366410,0
             0.950ms      0.077ms                event:kv/kvclient/kvcoord/dist_sender.go:2299 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] application error: r53 was not found on s5: "sql txn" meta={id=a9779d54 key=/Min pri=0.00014794 epo=0 ts=1683450818.815366410,0 min=1683450818.815366410,0 seq=0} lock=false stat=PENDING rts=1683450818.815366410,0 wto=false gul=1683450819.315366410,0
             0.966ms      0.016ms                event:kv/kvclient/kvcoord/dist_sender.go:2052 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] trying next peer (n1,s1):9
             0.976ms      0.010ms                event:rpc/nodedialer/nodedialer.go:163 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] sending request to 127.0.0.1:36721
             0.985ms      0.009ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:5 client:127.0.0.1:43090 user:root txn:a9779d54 span.kind:client node:5
             1.170ms      0.185ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:1 span.kind:server request:Get [/Table/108/1/1/0,/Min), [txn: a9779d54], [can-forward-ts]
             1.188ms      0.018ms                    event:server/node.go:1090 [n1] node received request: 1 Get
             1.226ms      0.038ms                    event:kv/kvserver/store_send.go:166 [n1,s1] executing Get [/Table/108/1/1/0,/Min), [txn: a9779d54], [can-forward-ts]
             1.242ms      0.016ms                    event:kv/kvserver/replica_send.go:174 [n1,s1,r53/9:/{Table/108-Max}] read-only path
             1.257ms      0.015ms                    event:kv/kvserver/concurrency/concurrency_manager.go:193 [n1,s1,r53/9:/{Table/108-Max}] sequencing request
             1.264ms      0.007ms                    event:kv/kvserver/concurrency/concurrency_manager.go:274 [n1,s1,r53/9:/{Table/108-Max}] acquiring latches
             1.276ms      0.012ms                    event:kv/kvserver/concurrency/concurrency_manager.go:318 [n1,s1,r53/9:/{Table/108-Max}] scanning lock table for conflicting locks
             1.292ms      0.016ms                    event:kv/kvserver/replica_read.go:269 [n1,s1,r53/9:/{Table/108-Max}] executing read-only batch
             1.394ms      0.101ms                    event:kv/kvserver/replica_evaluate.go:544 [n1,s1,r53/9:/{Table/108-Max}] evaluated Get command header:<key:"\364\211\211\210" > , txn="sql txn" meta={id=a9779d54 key=/Min pri=0.00014794 epo=0 ts=1683450818.815366410,0 min=1683450818.815366410,0 seq=0} lock=false stat=PENDING rts=1683450818.815366410,0 wto=false gul=1683450819.315366410,0 : resp=header:<num_keys:1 num_bytes:7 > value:<raw_bytes:"\375\311[\010\n#\002" timestamp:<wall_time:1683450801232667026 > > , err=<nil>
             1.423ms      0.029ms                    event:scan stats: stepped 0 times (0 internal); seeked 2 times (2 internal); block-bytes: (total 11 KiB, cached 11 KiB); points: (count 2 , key-bytes 27 B, value-bytes 87 B, tombstoned: 0 ) ranges: (count 0 ), (contained-points 0 , skipped-points 0 ) evaluated requests: 1 gets, 0 scans, 0 reverse scans
             1.460ms      0.037ms                    event:kv/kvserver/replica_read.go:204 [n1,s1,r53/9:/{Table/108-Max}] read completed
             1.480ms      0.020ms                    event:kv/kvserver/replica_send.go:314 [n1,s1,r53/9:/{Table/108-Max}] client had stale range info; returning an update
             1.397ms     -0.083ms                    structured:{"@type":"type.googleapis.com/cockroach.roachpb.ScanStats","numInterfaceSeeks":"2","numInternalSeeks":"2","numInterfaceSteps":"0","numInternalSteps":"0","blockBytes":"11312","blockBytesInCache":"11312","keyBytes":"27","valueBytes":"87","pointCount":"2","pointsCoveredByRangeTombstones":"0","rangeKeyCount":"0","rangeKeyContainedPoints":"0","rangeKeySkippedPoints":"0","numGets":"1","numScans":"0","numReverseScans":"0"}
             1.772ms      0.796ms                event:kv/kvclient/kvcoord/dist_sender.go:2172 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] received updated range info: [desc: r53:/{Table/108-Max} [(n3,s3):1, (n1,s1):9, (n2,s2):5, (n4,s4):6NON_VOTER, next=12, gen=32], lease: repl=(n1,s1):9 seq=5 start=1683450817.255743529,0 epo=1 pro=1683450817.256597055,0, closed_timestamp_policy: LEAD_FOR_GLOBAL_READS]
             1.810ms      0.038ms                event:kv/kvclient/rangecache/range_cache.go:938 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] evict cached descriptor: desc=desc:r53:/{Table/108-Max} [(n3,s3):1, (n1,s1):9, (n2,s2):5, (n4,s4):6NON_VOTER, (n5,s5):11NON_VOTER, next=12, gen=31], lease:repl=(n1,s1):9 seq=5 start=1683450817.255743529,0 epo=1 pro=1683450817.256597055,0
             1.825ms      0.016ms                event:kv/kvclient/rangecache/range_cache.go:555 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] evicting cached range descriptor with 1 replacements
             1.846ms      0.021ms                event:kv/kvclient/rangecache/range_cache.go:1107 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] clearing entries overlapping r53:/{Table/108-Max} [(n3,s3):1, (n1,s1):9, (n2,s2):5, (n4,s4):6NON_VOTER, next=12, gen=32]
             1.875ms      1.294ms            event:kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:572 [n5,client=127.0.0.1:43090,user=root,txn=a9779d54] recording span to refresh: /Table/108/1/1/0
             2.150ms      1.667ms    event:sql/conn_executor_exec.go:694 [n5,client=127.0.0.1:43090,user=root] execution ends
             2.158ms      0.009ms    event:sql/conn_executor_exec.go:694 [n5,client=127.0.0.1:43090,user=root] rows affected: 1
             2.234ms      0.076ms        === operation:commit sql txn _verbose:1 node:5 client:127.0.0.1:43090 user:root
             2.259ms      0.101ms    event:sql/conn_executor_exec.go:2189 [n5,client=127.0.0.1:43090,user=root] AutoCommit. err: <nil>
    --- FAIL: TestMultiRegionDataDriven/global_tables (29.80s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@chengxiong-ruan
Copy link
Contributor

closing in favor of #98020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Development

No branches or pull requests

4 participants