Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/sql/logictest/tests/fakedist/fakedist_test: TestLogic_zone_config_system_tenant failed #130405

Closed
cockroach-teamcity opened this issue Sep 10, 2024 · 5 comments · Fixed by #130453
Closed

pkg/sql/logictest/tests/fakedist/fakedist_test: TestLogic_zone_config_system_tenant failed #130405

cockroach-teamcity opened this issue Sep 10, 2024 · 5 comments · Fixed by #130453
Assignees
@michae2
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 10, 2024 •
edited by cockroach-jira-scripts
Loading

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_zone_config_system_tenant failed on master @ 7571549d8772a087ab4577d1e770e02582ba5877:

=== RUN   TestLogic_zone_config_system_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_zone_config_system_tenant1995980402
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
[10:47:04] setting distsql_workmem='29065B';
    logic.go:4371: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_zone_config_system_tenant1995980402
--- FAIL: TestLogic_zone_config_system_tenant (67.87s)
=== RUN   TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl
[10:47:09] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 31 statements
[10:47:12] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 33 statements
    logic.go:3012: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant:198: SELECT
          crdb_internal.pretty_key(start_key, -1),
          crdb_internal.pb_to_json('cockroach.roachpb.SpanConfig', config)->'gcPolicy'->>'ttlSeconds'
        FROM system.span_configurations
        WHERE start_key >= (SELECT crdb_internal.table_span(100)[1])
        ORDER BY start_key
        expected:
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  90001
            /Table/112  14400
            
        but found (query options: "retry") :
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  1
            /Table/112  1
            
[10:47:59] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 35 statements
[10:47:59] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant with config fakedist: 35 tests, 1 failures
[10:48:04] --- total progress: 35 statements
--- total: 35 tests, 1 failures
    --- FAIL: TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl (52.46s)

Parameters:

  • attempt=1
  • race=true
  • run=3
  • shard=24
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-42039
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team labels Sep 10, 2024
@github-project-automation github-project-automation bot moved this to Triage in SQL Queries Sep 10, 2024
@michae2
Copy link
Collaborator

michae2 commented Sep 10, 2024

Same test as #121351 but that was fixed.

@michae2 michae2 mentioned this issue Sep 10, 2024
Closed
@michae2
Copy link
Collaborator

michae2 commented Sep 10, 2024

Looks like we need to apply #121585 again, maybe another accidental --rewrite?

@michae2 michae2 self-assigned this Sep 10, 2024
@michae2 michae2 mentioned this issue Sep 10, 2024
Merged
@cockroach-teamcity
Copy link
Member Author

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_zone_config_system_tenant failed on master @ 31794d8d89e3e117e1654cb13ad109da04de5445:

=== RUN   TestLogic_zone_config_system_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_zone_config_system_tenant586894767
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    logic.go:4371: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_zone_config_system_tenant586894767
--- FAIL: TestLogic_zone_config_system_tenant (67.75s)
=== RUN   TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl
[10:47:43] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 31 statements
[10:47:46] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 33 statements
    logic.go:3012: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant:198: SELECT
          crdb_internal.pretty_key(start_key, -1),
          crdb_internal.pb_to_json('cockroach.roachpb.SpanConfig', config)->'gcPolicy'->>'ttlSeconds'
        FROM system.span_configurations
        WHERE start_key >= (SELECT crdb_internal.table_span(100)[1])
        ORDER BY start_key
        expected:
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  90001
            /Table/112  14400
            
        but found (query options: "retry") :
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  1
            /Table/112  1
            
[10:48:32] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 35 statements
[10:48:32] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant with config fakedist: 35 tests, 1 failures
[10:48:38] --- total progress: 35 statements
--- total: 35 tests, 1 failures
    --- FAIL: TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl (52.43s)

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=24
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_zone_config_system_tenant failed on master @ 3f8226eb7f09eb93c26dda4cd47e023a8f56ea23:

=== RUN   TestLogic_zone_config_system_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_zone_config_system_tenant1952375573
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
[10:07:49] setting distsql_workmem='53419B';
    logic.go:4371: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_zone_config_system_tenant1952375573
--- FAIL: TestLogic_zone_config_system_tenant (68.49s)
=== RUN   TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl
[10:07:54] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 31 statements
[10:07:56] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 32 statements
[10:07:59] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 34 statements
    logic.go:3012: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant:198: SELECT
          crdb_internal.pretty_key(start_key, -1),
          crdb_internal.pb_to_json('cockroach.roachpb.SpanConfig', config)->'gcPolicy'->>'ttlSeconds'
        FROM system.span_configurations
        WHERE start_key >= (SELECT crdb_internal.table_span(100)[1])
        ORDER BY start_key
        expected:
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  90001
            /Table/112  14400
            
        but found (query options: "retry") :
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  1
            /Table/112  1
            
[10:08:44] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 35 statements
[10:08:44] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant with config fakedist: 35 tests, 1 failures
[10:08:50] --- total progress: 35 statements
--- total: 35 tests, 1 failures
    --- FAIL: TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl (53.42s)

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=25
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

pkg/sql/logictest/tests/fakedist/fakedist_test.TestLogic_zone_config_system_tenant failed on master @ 128fcab4c07413513a05aea1d1494943f4bc3092:

=== RUN   TestLogic_zone_config_system_tenant
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_zone_config_system_tenant3767271489
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
[09:57:44] setting distsql_workmem='44833B';
    logic.go:4371: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_zone_config_system_tenant3767271489
--- FAIL: TestLogic_zone_config_system_tenant (68.27s)
=== RUN   TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl
[09:57:49] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 31 statements
[09:57:52] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 33 statements
    logic.go:3012: 
         
        /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant:198: SELECT
          crdb_internal.pretty_key(start_key, -1),
          crdb_internal.pb_to_json('cockroach.roachpb.SpanConfig', config)->'gcPolicy'->>'ttlSeconds'
        FROM system.span_configurations
        WHERE start_key >= (SELECT crdb_internal.table_span(100)[1])
        ORDER BY start_key
        expected:
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  90001
            /Table/112  14400
            
        but found (query options: "retry") :
            /Table/106  14400
            /Table/107  14400
            /Table/110  90001
            /Table/111  1
            /Table/112  1
            
[09:58:39] --- progress: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant: 35 statements
[09:58:39] --- done: /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/fakedist/fakedist_test_/fakedist_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/zone_config_system_tenant with config fakedist: 35 tests, 1 failures
[09:58:45] --- total progress: 35 statements
--- total: 35 tests, 1 failures
    --- FAIL: TestLogic_zone_config_system_tenant/dropped_relation_gc_ttl (52.75s)

Parameters:

  • attempt=1
  • race=true
  • run=2
  • shard=25
Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Sep 18, 2024
@michae2 @jbowens @rafiss
Merge #130453 #130572 #130906
68fc953 
130453: logictest: revert incorrect test assertion update r=rafiss a=michae2

(Deja vu: this is #121556 all over again.)

103bd54 incorrectly updated the test expectations, likely because the `--rewrite` flag was used on an assertion that has the retry directive.

This commit undoes that change.

Fixes: #130405

Release note: None

130572: storage: GC range keys by unsetting identical suffixes r=jbowens a=jbowens

In CockroachDB's key encoding some keys have multiple logically equivalent but physically distinct encodings. Most notably, in CockroachDB versions 23.2 and earlier keys written to global tables encoded MVCC timestamps with a 'synthetic bit.' In #101938 CockroachDB stopped encoding and decoding this synthetic bit, transparently ignoring it.

In #129592 we observed the existence of a bug in the CockroachDB comparator when comparing two MVCC timestamp suffixes, specifically outside the context of a full MVCC key. The comparator failed to consider a timestamp with the synthetic bit and a timestamp without the synthetic bit as logically equivalent. There are limited instances where Pebble uses the comparator to compare "bare suffixes," and all instances are constrained to the implementation of range keys.

In #129592 it was observed that the comparator bug could prevent the garbage collection of MVCC delete range tombstones (the single use of range keys within CockroachDB). A cluster running 23.2 or earlier may write a MVCC delete range tombstone with a timestamp encoding the synthetic bit. If the cluster subsequently upgraded to 24.1 or later, the code path to clear range keys stopped understanding synthetic bits and wrote range key unset tombstones without the synthetic bit set. Due to the comparator bug, Pebble did not consider these timestamp suffixes equal and the unset was ineffective.

We initially attempted to fix this issue by fixing the comparator, but inadvertently introduced the possibility of replica divergence #130533 by changing the semantics of LSM state below raft.

This commit works around this comparator bug by adapting ClearMVCCRangeKey to write range key unsets using the verbatim suffix that was read from the storage engine. To avoid reverting #101938 and re-introducing knowledge of the synthetic bit, the MVCCRangeKey data structures are adapted to retain a copy of the encoded timestamp suffix when reading range keys from storage engine iterators. If later an attempt is made to clear the range key through ClearMVCCRangeKey, this encoded timestamp suffix is used instead of re-encoding the timestamp. Through avoiding the decoding/encoding roundtrip, ClearMVCCRangeKey ensures that the suffixes it writes are identical to the range keys that exist on disk, even if they encode a synthetic bit.

Release note (bug fix): Fixes a bug that could result in the inability to garbage collect a MVCC range tombstone within a global table.
Epic: none
Informs #129592.

130906: sql: deflake TestValidationWithProtectedTS r=rafiss a=rafiss

This test does not work if ranges get split, so we disable the split queue.

fixes #130715
Release note: None

Co-authored-by: Michael Erickson <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
@craig craig bot closed this as completed in ef2f74c Sep 18, 2024
@github-project-automation github-project-automation bot moved this from Triage to Done in SQL Queries Sep 18, 2024
This was referenced Oct 2, 2024
Closed
Closed
Closed
@yuzefovich yuzefovich mentioned this issue Nov 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michae2 michae2

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-queries SQL Queries Team
Projects
SQL Queries
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

logictest: revert incorrect test assertion update michae2/cockroach
2 participants
@cockroach-teamcity @michae2