Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccl/multiregionccl: TestMultiRegionDataDriven failed [secondary_region] #92370

Closed
cockroach-teamcity opened this issue Nov 23, 2022 · 16 comments
Closed

ccl/multiregionccl: TestMultiRegionDataDriven failed [secondary_region] #92370

cockroach-teamcity opened this issue Nov 23, 2022 · 16 comments
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 23, 2022 •
edited by cockroach-jira-scripts
Loading

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 0d9669a8669cf5abe22b61e1f1faf9a520f2f372:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven814050627
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven814050627
--- FAIL: TestMultiRegionDataDriven (197.34s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/395/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (127.47s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/multiregion

This test on roachdash | Improve this report!

Jira issue: CRDB-21724
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Nov 23, 2022
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Nov 23, 2022
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ b5be006bedd7d3cedc3fb3d2248df168e3d64be2:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4245211614
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4245211614
--- FAIL: TestMultiRegionDataDriven (207.56s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2684/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (134.27s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 1a6e9f885baa124d5ff2996adb966ea15a1a9b2b:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven151145817
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven151145817
--- FAIL: TestMultiRegionDataDriven (676.11s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3440/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (270.03s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Nov 27, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 1a6e9f885baa124d5ff2996adb966ea15a1a9b2b:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven262764324
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven262764324
--- FAIL: TestMultiRegionDataDriven (723.70s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/823/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25: 
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (343.38s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@knz
Copy link
Contributor

knz commented Nov 28, 2022

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/7732956?showRootCauses=false&expandBuildChangesSection=true&expandBuildDeploymentsSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 146556e19f5e4fdc8c3e6a623b280cc33aee4d18:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4274581580
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4274581580
--- FAIL: TestMultiRegionDataDriven (786.34s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3192/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (316.22s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 2a1bed15513905dd4172f66bfc2421326775fd09:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven120364447
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven120364447
--- FAIL: TestMultiRegionDataDriven (703.75s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3033/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (271.52s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ c4bde8b72cdd4016845ae70ef5162b3f11fab1fb:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4283092946
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:423: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven4283092946
--- FAIL: TestMultiRegionDataDriven (206.10s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/583/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.31s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3791979706
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:424: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3791979706
--- FAIL: TestMultiRegionDataDriven (205.94s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/247/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (132.36s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4ec5a5f43980583eff8b8f8dff1b23090c7e2304:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven532547850
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:424: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven532547850
--- FAIL: TestMultiRegionDataDriven (488.07s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3029/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (223.25s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 90e24ab4313f59fdc825443c03a1f4dc04bcef09:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven4124632496
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:424: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven4124632496
--- FAIL: TestMultiRegionDataDriven (241.98s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8546/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (139.46s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ d4f63d7e7f77a60819ce3fcec14b173e3e6acac8:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3106725753
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:424: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven3106725753
--- FAIL: TestMultiRegionDataDriven (212.99s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2796/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (137.78s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 3e26d85118ef73133c00b04b17449c27c31b8bc4:

        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
        wait-for-zone-config-changes [7 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:29:
        trace-sql [1 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        served locally: true
        served via follower read: false
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:36:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:41:
        trace-sql [1 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        served locally: false
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:47:
        stop-server [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:50:
        stop-server [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4406/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:54:
         
        expected:
        
        found:
        expected leaseholder 0 but got 1
    --- FAIL: TestMultiRegionDataDriven/secondary_region (336.93s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 31365e21dc606cdc1e4302c86192ffc5a6cf1255:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven191230017
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:424: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven191230017
--- FAIL: TestMultiRegionDataDriven (212.03s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:125: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2829/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.30s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@blathers-crl blathers-crl bot added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Feb 22, 2023
@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 528cd197bf43793672d0660dcabdd89676c0f753:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven3002652871
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven3002652871
--- FAIL: TestMultiRegionDataDriven (233.01s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/6509/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:
         
        expected:
        
        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (142.33s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 8c54290f5f1ad68976e59a5f623126ae9d734bf0:

            19.110ms      0.809ms    event:sql/conn_executor_exec.go:748 [n3,client=127.0.0.1:53448,user=root] executing after 0 retries, last retry reason: <nil>
            19.154ms      0.044ms    event:sql/conn_executor_exec.go:748 [n3,client=127.0.0.1:53448,user=root] execution starts: distributed engine
            21.676ms      2.522ms    event:sql/distsql_running.go:1795 [n3,client=127.0.0.1:53448,user=root] creating DistSQL plan with isLocal=true
            21.730ms      0.054ms    event:sql/distsql_running.go:795 [n3,client=127.0.0.1:53448,user=root] running DistSQL plan
            21.762ms      0.032ms        === operation:flow _verbose:1 node:3 client:127.0.0.1:53448 user:root
            23.718ms      1.957ms        event:sql/colflow/vectorized_flow.go:232 [n3,client=127.0.0.1:53448,user=root] setting up vectorized flow
            27.354ms      3.636ms        event:sql/colflow/vectorized_flow.go:282 [n3,client=127.0.0.1:53448,user=root] vectorized flow setup succeeded
            27.403ms      0.049ms        event:sql/flowinfra/flow.go:402 [n3,client=127.0.0.1:53448,user=root] starting (0 processors, 0 startables) asynchronously
            27.441ms      0.038ms        event:sql/colflow/vectorized_flow.go:308 [n3,client=127.0.0.1:53448,user=root] running the batch flow coordinator in the flow's goroutine
            68.015ms     40.574ms        event:component:<flow_id:<cc9ec214-ef1c-4334-b818-9d1d0c04172d> type:FLOW id:0 sql_instance_id:3 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<> tuples_read:<> batch_requests_issued:<> kv_time:<> contention_time:<> num_interface_steps:<> num_internal_steps:<> num_interface_seeks:<> num_internal_seeks:<> block_bytes:<> block_bytes_in_cache:<> key_bytes:<> value_bytes:<> point_count:<> points_covered_by_range_tombstones:<> range_key_count:<> range_key_contained_points:<> range_key_skipped_points:<> kv_cpu_time:<> num_gets:<> num_scans:<> num_reverse_scans:<> > exec:<exec_time:<> max_allocated_mem:<> max_allocated_disk:<> consumed_ru:<> cpu_time:<> > output:<num_batches:<> num_tuples:<> > flow_stats:<max_mem_usage:<value_plus_one:20481 > max_disk_usage:<value_plus_one:1 > consumed_ru:<> >
            67.877ms     -0.137ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"cc9ec214-ef1c-4334-b818-9d1d0c04172d","type":"FLOW","id":0,"sqlInstanceId":3},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"0"},"tuplesRead":{"valuePlusOne":"0"},"batchRequestsIssued":{"valuePlusOne":"0"},"kvTime":{"valuePlusOne":"0s"},"contentionTime":{"valuePlusOne":"0s"},"numInterfaceSteps":{"valuePlusOne":"0"},"numInternalSteps":{"valuePlusOne":"0"},"numInterfaceSeeks":{"valuePlusOne":"0"},"numInternalSeeks":{"valuePlusOne":"0"},"blockBytes":{"valuePlusOne":"0"},"blockBytesInCache":{"valuePlusOne":"0"},"keyBytes":{"valuePlusOne":"0"},"valueBytes":{"valuePlusOne":"0"},"pointCount":{"valuePlusOne":"0"},"pointsCoveredByRangeTombstones":{"valuePlusOne":"0"},"rangeKeyCount":{"valuePlusOne":"0"},"rangeKeyContainedPoints":{"valuePlusOne":"0"},"rangeKeySkippedPoints":{"valuePlusOne":"0"},"contentionEvents":[],"kvCpuTime":{"valuePlusOne":"0s"},"numGets":{"valuePlusOne":"0"},"numScans":{"valuePlusOne":"0"},"numReverseScans":{"valuePlusOne":"0"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"0"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRu":{"valuePlusOne":"0"},"cpuTime":{"valuePlusOne":"0s"}},"output":{"numBatches":{"valuePlusOne":"0"},"numTuples":{"valuePlusOne":"0"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"20481"},"maxDiskUsage":{"valuePlusOne":"1"},"consumedRu":{"valuePlusOne":"0"}}}
            27.458ms    -40.419ms        === operation:batch flow coordinator _unfinished:1 _verbose:1 node:3 client:127.0.0.1:53448 user:root cockroach.flowid:cc9ec214-ef1c-4334-b818-9d1d0c04172d cockroach.processorid:0
            65.350ms     37.891ms        event:component:<flow_id:<cc9ec214-ef1c-4334-b818-9d1d0c04172d> type:PROCESSOR id:0 sql_instance_id:3 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<value_plus_one:12 > tuples_read:<value_plus_one:2 > batch_requests_issued:<value_plus_one:2 > kv_time:<value_plus_one:<nanos:35131941 > > contention_time:<value_plus_one:<nanos:1 > > num_interface_steps:<value_plus_one:1 > num_internal_steps:<value_plus_one:1 > num_interface_seeks:<value_plus_one:2 > num_internal_seeks:<value_plus_one:2 > block_bytes:<value_plus_one:98 > block_bytes_in_cache:<value_plus_one:98 > key_bytes:<value_plus_one:19 > value_bytes:<value_plus_one:15 > point_count:<value_plus_one:2 > points_covered_by_range_tombstones:<value_plus_one:1 > range_key_count:<value_plus_one:1 > range_key_contained_points:<value_plus_one:1 > range_key_skipped_points:<value_plus_one:1 > kv_cpu_time:<> num_gets:<value_plus_one:2 > num_scans:<value_plus_one:1 > num_reverse_scans:<value_plus_one:1 > > exec:<exec_time:<> max_allocated_mem:<value_plus_one:20481 > max_allocated_disk:<> consumed_ru:<value_plus_one:1 > cpu_time:<value_plus_one:<nanos:317903 > > > output:<num_batches:<value_plus_one:2 > num_tuples:<value_plus_one:2 > > flow_stats:<max_mem_usage:<> max_disk_usage:<> consumed_ru:<> >
            65.486ms      0.136ms        event:component:<flow_id:<00000000-0000-0000-0000-000000000000> type:UNSET id:0 sql_instance_id:0 > net_rx:<latency:<> wait_time:<> deserialization_time:<> tuples_received:<> bytes_received:<> messages_received:<> > net_tx:<tuples_sent:<> bytes_sent:<> messages_sent:<> > kv:<bytes_read:<> tuples_read:<> batch_requests_issued:<> kv_time:<> contention_time:<> num_interface_steps:<> num_internal_steps:<> num_interface_seeks:<> num_internal_seeks:<> block_bytes:<> block_bytes_in_cache:<> key_bytes:<> value_bytes:<> point_count:<> points_covered_by_range_tombstones:<> range_key_count:<> range_key_contained_points:<> range_key_skipped_points:<> kv_cpu_time:<> num_gets:<> num_scans:<> num_reverse_scans:<> > exec:<exec_time:<> max_allocated_mem:<> max_allocated_disk:<> consumed_ru:<> cpu_time:<> > output:<num_batches:<> num_tuples:<> > flow_stats:<max_mem_usage:<> max_disk_usage:<> consumed_ru:<> >
            65.204ms     -0.281ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"cc9ec214-ef1c-4334-b818-9d1d0c04172d","type":"PROCESSOR","id":0,"sqlInstanceId":3},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"12"},"tuplesRead":{"valuePlusOne":"2"},"batchRequestsIssued":{"valuePlusOne":"2"},"kvTime":{"valuePlusOne":"0.035131941s"},"contentionTime":{"valuePlusOne":"0.000000001s"},"numInterfaceSteps":{"valuePlusOne":"1"},"numInternalSteps":{"valuePlusOne":"1"},"numInterfaceSeeks":{"valuePlusOne":"2"},"numInternalSeeks":{"valuePlusOne":"2"},"blockBytes":{"valuePlusOne":"98"},"blockBytesInCache":{"valuePlusOne":"98"},"keyBytes":{"valuePlusOne":"19"},"valueBytes":{"valuePlusOne":"15"},"pointCount":{"valuePlusOne":"2"},"pointsCoveredByRangeTombstones":{"valuePlusOne":"1"},"rangeKeyCount":{"valuePlusOne":"1"},"rangeKeyContainedPoints":{"valuePlusOne":"1"},"rangeKeySkippedPoints":{"valuePlusOne":"1"},"contentionEvents":[],"kvCpuTime":{"valuePlusOne":"0s"},"numGets":{"valuePlusOne":"2"},"numScans":{"valuePlusOne":"1"},"numReverseScans":{"valuePlusOne":"1"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"20481"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRu":{"valuePlusOne":"1"},"cpuTime":{"valuePlusOne":"0.000317903s"}},"output":{"numBatches":{"valuePlusOne":"2"},"numTuples":{"valuePlusOne":"2"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRu":{"valuePlusOne":"0"}}}
            65.372ms      0.168ms        structured:{"@type":"type.googleapis.com/cockroach.sql.distsqlrun.ComponentStats","component":{"flowId":"00000000-0000-0000-0000-000000000000","type":"UNSET","id":0,"sqlInstanceId":0},"netRx":{"latency":{"valuePlusOne":"0s"},"waitTime":{"valuePlusOne":"0s"},"deserializationTime":{"valuePlusOne":"0s"},"tuplesReceived":{"valuePlusOne":"0"},"bytesReceived":{"valuePlusOne":"0"},"messagesReceived":{"valuePlusOne":"0"}},"netTx":{"tuplesSent":{"valuePlusOne":"0"},"bytesSent":{"valuePlusOne":"0"},"messagesSent":{"valuePlusOne":"0"}},"kv":{"bytesRead":{"valuePlusOne":"0"},"tuplesRead":{"valuePlusOne":"0"},"batchRequestsIssued":{"valuePlusOne":"0"},"kvTime":{"valuePlusOne":"0s"},"contentionTime":{"valuePlusOne":"0s"},"numInterfaceSteps":{"valuePlusOne":"0"},"numInternalSteps":{"valuePlusOne":"0"},"numInterfaceSeeks":{"valuePlusOne":"0"},"numInternalSeeks":{"valuePlusOne":"0"},"blockBytes":{"valuePlusOne":"0"},"blockBytesInCache":{"valuePlusOne":"0"},"keyBytes":{"valuePlusOne":"0"},"valueBytes":{"valuePlusOne":"0"},"pointCount":{"valuePlusOne":"0"},"pointsCoveredByRangeTombstones":{"valuePlusOne":"0"},"rangeKeyCount":{"valuePlusOne":"0"},"rangeKeyContainedPoints":{"valuePlusOne":"0"},"rangeKeySkippedPoints":{"valuePlusOne":"0"},"contentionEvents":[],"kvCpuTime":{"valuePlusOne":"0s"},"numGets":{"valuePlusOne":"0"},"numScans":{"valuePlusOne":"0"},"numReverseScans":{"valuePlusOne":"0"}},"exec":{"execTime":{"valuePlusOne":"0s"},"maxAllocatedMem":{"valuePlusOne":"0"},"maxAllocatedDisk":{"valuePlusOne":"0"},"consumedRu":{"valuePlusOne":"0"},"cpuTime":{"valuePlusOne":"0s"}},"output":{"numBatches":{"valuePlusOne":"0"},"numTuples":{"valuePlusOne":"0"}},"inputs":[],"flowStats":{"maxMemUsage":{"valuePlusOne":"0"},"maxDiskUsage":{"valuePlusOne":"0"},"consumedRu":{"valuePlusOne":"0"}}}
            28.260ms    -37.112ms        === operation:colbatchscan _unfinished:1 _verbose:1 node:3 client:127.0.0.1:53448 user:root
            28.260ms      0.000ms        [txn coordinator send: {count: 1, duration 33ms}]
            28.260ms      0.000ms        [dist sender send: {count: 1, duration 24ms}]
            28.260ms      0.000ms        [/cockroach.roachpb.Internal/Batch: {count: 1, duration 13ms}]
            29.622ms      1.362ms        event:sql/row/kv_batch_fetcher.go:435 [n3,client=127.0.0.1:53448,user=root] Scan /Table/108/1/1/0
            29.683ms      0.061ms            === operation:txn coordinator send _verbose:1 node:3 client:127.0.0.1:53448 user:root txnID:89b28a6d-b4f8-4449-87e9-11c378ebbaf5
            29.683ms      0.000ms            [dist sender send: {count: 1, duration 24ms}]
            29.683ms      0.000ms            [/cockroach.roachpb.Internal/Batch: {count: 1, duration 13ms}]
            37.687ms      8.005ms                === operation:dist sender send _verbose:1 node:3 client:127.0.0.1:53448 user:root txn:89b28a6d
            37.687ms      0.000ms                [/cockroach.roachpb.Internal/Batch: {count: 1, duration 13ms}]
            47.330ms      9.643ms                event:kv/kvclient/kvcoord/range_iter.go:188 [n3,client=127.0.0.1:53448,user=root,txn=89b28a6d] querying next range at /Table/108/1/1/0
            47.468ms      0.138ms                event:kv/kvclient/kvcoord/range_iter.go:225 [n3,client=127.0.0.1:53448,user=root,txn=89b28a6d] key: /Table/108/1/1/0, desc: r59:/{Table/108-Max} [(n1,s1):1, (n3,s3):9, (n2,s2):12, (n8,s8):13NON_VOTER, (n7,s7):10NON_VOTER, next=14, gen=33]
            47.674ms      0.206ms                event:kv/kvclient/kvcoord/dist_sender.go:2105 [n3,client=127.0.0.1:53448,user=root,txn=89b28a6d] r59: sending batch 1 Get to (n3,s3):9
            47.735ms      0.061ms                event:rpc/nodedialer/nodedialer.go:157 [n3,client=127.0.0.1:53448,user=root,txn=89b28a6d] sending request to local client
            47.782ms      0.047ms                    === operation:/cockroach.roachpb.Internal/Batch _verbose:1 node:3 span.kind:server request:Get [/Table/108/1/1/0,/Min), [txn: 89b28a6d], [can-forward-ts]
            47.881ms      0.099ms                    event:server/node.go:1132 [n3] node received request: 1 Get
            49.066ms      1.185ms                    event:kv/kvserver/store_send.go:167 [n3,s3] executing Get [/Table/108/1/1/0,/Min), [txn: 89b28a6d], [can-forward-ts]
            50.694ms      1.628ms                    event:kv/kvserver/replica_send.go:179 [n3,s3,r59/9:/{Table/108-Max}] read-only path
            50.797ms      0.103ms                    event:kv/kvserver/concurrency/concurrency_manager.go:194 [n3,s3,r59/9:/{Table/108-Max}] sequencing request
            50.862ms      0.065ms                    event:kv/kvserver/concurrency/concurrency_manager.go:275 [n3,s3,r59/9:/{Table/108-Max}] acquiring latches
            51.212ms      0.350ms                    event:kv/kvserver/concurrency/concurrency_manager.go:319 [n3,s3,r59/9:/{Table/108-Max}] scanning lock table for conflicting locks
            54.618ms      3.406ms                    event:kv/kvserver/replica_read.go:296 [n3,s3,r59/9:/{Table/108-Max}] can drop latches early for batch (Get [/Table/108/1/1/0,/Min), [txn: 89b28a6d], [can-forward-ts]); scanning lock table first to detect conflicts
            54.991ms      0.372ms                    event:kv/kvserver/replica_read.go:113 [n3,s3,r59/9:/{Table/108-Max}] lock table scan complete without conflicts; dropping latches early
            59.198ms      4.208ms                    event:kv/kvserver/replica_read.go:408 [n3,s3,r59/9:/{Table/108-Max}] executing read-only batch
            59.383ms      0.185ms                    event:kv/kvserver/replica_evaluate.go:550 [n3,s3,r59/9:/{Table/108-Max}] evaluated Get command header:<key:"\364\211\211\210" > , txn="sql txn" meta={id=89b28a6d key=/Min pri=0.01281379 epo=0 ts=1677825020.750555123,0 min=1677825020.750555123,0 seq=0} lock=false stat=PENDING rts=1677825020.750555123,0 wto=false gul=1677825021.250555123,0 : resp=header:<num_keys:1 num_bytes:7 > value:<raw_bytes:"\375\311[\010\n#\002" timestamp:<wall_time:1677824984377001974 logical:2 > > , err=<nil>
            60.020ms      0.636ms                    event:scan stats: stepped 0 times (0 internal); seeked 1 times (1 internal); block-bytes: (total 97 B, cached 97 B); points: (count 1 , key-bytes 18 B, value-bytes 14 B, tombstoned: 0 ) ranges: (count 0 ), (contained-points 0 , skipped-points 0 ) evaluated requests: 1 gets, 0 scans, 0 reverse scans
            60.892ms      0.873ms                    event:kv/kvserver/replica_read.go:221 [n3,s3,r59/9:/{Table/108-Max}] read completed
            59.932ms     -0.960ms                    structured:{"@type":"type.googleapis.com/cockroach.roachpb.ScanStats","numInterfaceSeeks":"1","numInternalSeeks":"1","numInterfaceSteps":"0","numInternalSteps":"0","blockBytes":"97","blockBytesInCache":"97","keyBytes":"18","valueBytes":"14","pointCount":"1","pointsCoveredByRangeTombstones":"0","rangeKeyCount":"0","rangeKeyContainedPoints":"0","rangeKeySkippedPoints":"0","separatedPointCount":"0","separatedPointValueBytes":"0","separatedPointValueBytesFetched":"0","numGets":"1","numScans":"0","numReverseScans":"0"}
            62.196ms     32.513ms            event:kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:576 [n3,client=127.0.0.1:53448,user=root,txn=89b28a6d] recording span to refresh: /Table/108/1/1/0
            70.669ms     48.939ms    event:sql/conn_executor_exec.go:748 [n3,client=127.0.0.1:53448,user=root] execution ends
            71.558ms      0.889ms    event:sql/conn_executor_exec.go:748 [n3,client=127.0.0.1:53448,user=root] rows affected: 1
            77.085ms      5.527ms        === operation:commit sql txn _verbose:1 node:3 client:127.0.0.1:53448 user:root
            81.494ms      9.936ms    event:sql/conn_executor_exec.go:2291 [n3,client=127.0.0.1:53448,user=root] AutoCommit. err: <nil>
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (171.13s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss changed the title ccl/multiregionccl: TestMultiRegionDataDriven failed ccl/multiregionccl: TestMultiRegionDataDriven failed [secondary_region] Mar 3, 2023
@chengxiong-ruan
Copy link
Contributor

closing in favor of #98020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. skipped-test T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
4 participants
@knz @rafiss @chengxiong-ruan @cockroach-teamcity