Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test: TestCCLLogic_regional_by_row_query_behavior failed #86284

Closed
cockroach-teamcity opened this issue Aug 17, 2022 · 2 comments · Fixed by #87391
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Aug 17, 2022

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestCCLLogic_regional_by_row_query_behavior failed with artifacts on master @ b173a16715e71e94115820374da1eb350b3b459d:

=== RUN   TestCCLLogic_regional_by_row_query_behavior/regressions
    logic.go:2631: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3156/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior:2484: SELECT
            message
        FROM
            [SHOW KV TRACE FOR SESSION] WITH ORDINALITY
        WHERE
            message LIKE 'fetched:%' OR message LIKE 'output row%'
        ORDER BY
            "ordinality" ASC
        expected:
            fetched: /regional_by_row_table_as4/a_idx/?/3/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/6/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/9/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/12/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/15/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/18/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/21/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/24/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/27/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/30/? -> <undecoded>
            output row: [10]
            
        but found (query options: "") :
            fetched: /regional_by_row_table_as4/a_idx/?/3/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/6/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/9/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/12/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/15/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/18/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/21/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/24/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/27/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/30/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/33/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/36/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/39/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/42/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/45/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/48/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/51/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/54/? -> <undecoded>
            output row: [10]
    logic.go:1884: 
         /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3156/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior:2506: too many errors encountered, skipping the rest of the input
[05:26:54] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3156/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior with config multiregion-9node-3region-3azs: 166 tests, 2 failures
[05:26:56] --- total progress: 166 statements
--- total: 166 tests, 2 failures
    --- FAIL: TestCCLLogic_regional_by_row_query_behavior/regressions (1.34s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-18664

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 17, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Aug 17, 2022
@blathers-crl blathers-crl bot added the T-sql-queries SQL Queries Team label Aug 18, 2022
@cockroach-teamcity
Copy link
Member Author

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test.TestCCLLogic_regional_by_row_query_behavior failed with artifacts on master @ 869d44cb46ba1900202eb9e33a3353eecf06b269:

[05:43:45] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior: 6 statements
[05:43:47] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior: 103 statements
[05:43:49] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior: 112 statements
=== CONT  TestCCLLogic_regional_by_row_query_behavior
    logic.go:3896: -- test log scope end --
--- FAIL: TestCCLLogic_regional_by_row_query_behavior (22.75s)
=== RUN   TestCCLLogic_regional_by_row_query_behavior/regressions
    logic.go:2719: 
         
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior:2484: SELECT
            message
        FROM
            [SHOW KV TRACE FOR SESSION] WITH ORDINALITY
        WHERE
            message LIKE 'fetched:%' OR message LIKE 'output row%'
        ORDER BY
            "ordinality" ASC
        expected:
            fetched: /regional_by_row_table_as4/a_idx/?/3/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/6/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/9/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/12/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/15/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/18/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/21/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/24/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/27/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/30/? -> <undecoded>
            output row: [10]
            
        but found (query options: "") :
            fetched: /regional_by_row_table_as4/a_idx/?/3/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/6/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/9/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/12/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/15/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/18/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/21/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/24/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/27/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/30/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/33/? -> <undecoded>
            fetched: /regional_by_row_table_as4/a_idx/?/36/? -> <undecoded>
            output row: [10]
    logic.go:1972: 
         /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior:2506: too many errors encountered, skipping the rest of the input
[05:43:51] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs/multiregion-9node-3region-3azs_test_/multiregion-9node-3region-3azs_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row_query_behavior with config multiregion-9node-3region-3azs: 166 tests, 2 failures
[05:43:54] --- total progress: 166 statements
--- total: 166 tests, 2 failures
    --- FAIL: TestCCLLogic_regional_by_row_query_behavior/regressions (1.29s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@rytaft
Copy link
Collaborator

rytaft commented Aug 23, 2022

@msirek can you please take a look at this? This is one of the test cases for locality optimized search with LIMIT

msirek pushed a commit to msirek/cockroach that referenced this issue Sep 5, 2022
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
likely caused by zone configs not being made available between
table creation time and query execution time. A sleep test directive
is added to allow zone configs time to become available.

Release justification: low risk fix for test flake

Release note: None
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 8, 2022
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
likely caused by 2 issues:

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic settings cause the query plan to change and not choose
     locality-optimized search, which this test is dependent on.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` is repurposed
to do what it says, skip the test if metamorphic settings are enabled
in the current build. The pre-existing flag of the same name is renamed
to `!metamorphic-batch-sizes` to avoid confusion, since it resets any
batch size related metamorphic settings back to their default production
values when running in metamorphic mode.

Release justification: low risk fix for test flake

Release note: None
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 8, 2022
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
caused by 2 issues:

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic testing modify the `coldata-batch-size` setting,
     causing a different number of kv reads to be done, which affects
     the kv trace results in this test.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` which resets
batch size related settings that were modified in metamorphic mode back
to their default production values, is updated to also cover the
`coldata-batch-size` setting. The flag is renamed to
`!metamorphic-batch-sizes` to better reflect its actual effect.

Release justification: low risk fix for test flake

Release note: None
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 8, 2022
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
caused by 2 issues:

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic testing modify the `coldata-batch-size` setting,
     causing a different number of kv reads to be done, which affects
     the kv trace results in this test.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` which resets
batch size related settings that were modified in metamorphic mode back
to their default production values, is updated to also cover the
`coldata-batch-size` setting. The flag is renamed to
`!metamorphic-batch-sizes` to better reflect its actual effect.

Release justification: low risk fix for test flake

Release note: None
msirek pushed a commit to msirek/cockroach that referenced this issue Sep 8, 2022
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
caused by 2 issues:

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic testing modify the `coldata-batch-size` setting,
     causing a different number of kv reads to be done, which affects
     the kv trace results in this test.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` which resets
batch size related settings that were modified in metamorphic mode back
to their default production values, is updated to also cover the
`coldata-batch-size` setting. The flag is renamed to
`!metamorphic-batch-sizes` to better reflect its actual effect.

Release justification: low risk fix for test flake

Release note: None
craig bot pushed a commit that referenced this issue Sep 18, 2022
87391: logictestccl: Fix regional_by_row_query_behavior flake r=msirek a=msirek

Fixes #86284

This fixes a test flake in regional_by_row_query_behavior which is
caused by 2 issues:       

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic testing modify the `coldata-batch-size` setting, 
     causing a different number of kv reads to be done, which affects    
     the kv trace results in this test.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` which resets 
batch size related settings that were modified in metamorphic mode back                          
to their default production values, is updated to also cover the
`coldata-batch-size` setting. The flag is renamed to
`!metamorphic-batch-sizes` to better reflect its actual effect.

Release justification: low risk fix for test flake

Release note: None

Co-authored-by: Mark Sirek <[email protected]>
@craig craig bot closed this as completed in 70f85cd Sep 18, 2022
msirek pushed a commit to msirek/cockroach that referenced this issue Jun 29, 2023
Fixes cockroachdb#86284

This fixes a test flake in regional_by_row_query_behavior which is
caused by 2 issues:

  1. Zone configs not becoming available in `SystemConfig` between
     table creation time and query execution time.

  2. Metamorphic testing modify the `coldata-batch-size` setting,
     causing a different number of kv reads to be done, which affects
     the kv trace results in this test.

The flake can be reproduced via the command:
```
./dev testlogic ccl --files=regional_by_row_query_behavior \
                    --config=multiregion-9node-3region-3azs --stress
```
For point 1, the `retry` logic test directive is enhanced to purge the
zone config cache in the SystemConfig when a failing test is retried,
allowing the updated zone config to be read.

For point 2, the logic test blocking flag `!metamorphic` which resets
batch size related settings that were modified in metamorphic mode back
to their default production values, is updated to also cover the
`coldata-batch-size` setting. The flag is renamed to
`!metamorphic-batch-sizes` to better reflect its actual effect.

Release justification: low risk fix for test flake

Release note: None
@mgartner mgartner moved this to Done in SQL Queries Jul 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-sql-queries SQL Queries Team
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants