Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #112173

Closed
cockroach-teamcity opened this issue Oct 11, 2023 · 3 comments
Closed

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #112173

cockroach-teamcity opened this issue Oct 11, 2023 · 3 comments
Assignees
@miraradeva
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Oct 11, 2023 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos546319761
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 3142170241867687704
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2310399351
    kvnemesis.go:177: error applying x.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
          txn.SetIsoLevel(isolation.Snapshot)
          {
            b := &kv.Batch{}
            txn.Run(ctx, b) // <nil>
          }
          txn.GetForShareSkipLockedGuaranteedDurability(ctx, tk(17793418098181237768)) // @1697020960.393543824,0 (v51, <nil>)
          b := &kv.Batch{}
          b.ScanForShareSkipLockedGuaranteedDurability(tk(10556256394920442937), tk(16426098256482324624)) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          b.Del(tk(10944655808608756795) /* @s61 */) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          b.ReverseScanForShare(tk(3882210548237825871), tk(17494002221326605701)) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          txn.CommitInBatch(ctx, b) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          return nil
        }) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.ScanForShareSkipLockedGuaranteedDurability(ctx, tk(10556256394920442937), tk(16426098256482324624), 0) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.Del(ctx, tk(10944655808608756795) /* @s61 */) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.ReverseScanForShare(ctx, tk(3882210548237825871), tk(17494002221326605701), 0) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:212: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2310399351/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2310399351/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2310399351/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2310399351/kvs-scan.txt
    kvnemesis_test.go:310: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:310
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:247
        	Error:      	Should be zero, but was 4
        	Test:       	TestKVNemesisSingleNode_ReproposalChaos
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos546319761
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (57.65s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-32272
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Oct 11, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Oct 11, 2023
@cockroach-teamcity cockroach-teamcity mentioned this issue Oct 12, 2023
Closed
@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos146269668
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 8769722708159990638
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3574408815
    kvnemesis.go:177: error applying x.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
          txn.SetIsoLevel(isolation.Serializable)
          txn.GetForShareSkipLocked(ctx, tk(14120673188490173951)) // @1697278130.390779268,0 (<nil>, <nil>)
          txn.GetForShareSkipLocked(ctx, tk(8507834913077395951)) // @1697278130.390779268,0 (<nil>, <nil>)
          txn.GetForShareGuaranteedDurability(ctx, tk(8505607097540422098)) // @1697278130.390779268,0 (<nil>, <nil>)
          b := &kv.Batch{}
          b.ScanForShareGuaranteedDurability(tk(2851684828326249295), tk(13533668357537142482)) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          b.Del(tk(3751271083413883216) /* @s48 */) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          txn.CommitInBatch(ctx, b) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
          return nil
        }) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.ScanForShareGuaranteedDurability(ctx, tk(2851684828326249295), tk(13533668357537142482), 0) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.Del(ctx, tk(3751271083413883216) /* @s48 */) // cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Shared at seq number 0, already held at higher seq number 1
    kvnemesis.go:212: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3574408815/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3574408815/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3574408815/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3574408815/kvs-scan.txt
    kvnemesis_test.go:310: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:310
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:247
        	Error:      	Should be zero, but was 3
        	Test:       	TestKVNemesisSingleNode_ReproposalChaos
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos146269668
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (69.76s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 87a94f6ac59d4e01c822bb2128db858176600dd6:

=== RUN   TestKVNemesisSingleNode_ReproposalChaos
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos1696286987
    test_log_scope.go:81: use -show-logs to present logs inline
    kvnemesis_test.go:283: seed: 1260789548671771309
    kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3047848113
    kvnemesis.go:177: error applying x.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
          txn.SetIsoLevel(isolation.ReadCommitted)
          txn.ScanForShareSkipLocked(ctx, tk(5619852505528097051), tk(15210795031924798523), 0) // @1697364611.200620038,0 <nil>
          txn.GetForShareSkipLocked(ctx, tk(12383819912099597458)) // @1697364611.206989038,0 (<nil>, <nil>)
          txn.ReverseScanForUpdateSkipLockedGuaranteedDurability(ctx, tk(8322580655986029036), tk(14321354221370821891), 0) // @1697364611.210007568,0 <nil>
          b := &kv.Batch{}
          b.ScanForUpdateSkipLockedGuaranteedDurability(tk(10542629340055425569), tk(16584549434796921561)) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
          b.Put(tk(15518666153367270410), sv(62)) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
          txn.CommitInBatch(ctx, b) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
          return nil
        }) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.ScanForUpdateSkipLockedGuaranteedDurability(ctx, tk(10542629340055425569), tk(16584549434796921561), 0) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
    kvnemesis.go:177: error applying x.Put(ctx, tk(15518666153367270410), sv(62)) // cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1: cannot acquire lock with strength Exclusive at seq number 0, already held at higher seq number 1
    kvnemesis.go:212: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3047848113/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3047848113/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3047848113/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3047848113/kvs-scan.txt
    kvnemesis_test.go:310: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:310
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:247
        	Error:      	Should be zero, but was 3
        	Test:       	TestKVNemesisSingleNode_ReproposalChaos
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (69.69s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@andrewbaptist andrewbaptist added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Oct 19, 2023
nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Oct 23, 2023
@nvanbenschoten
kv: tolerate lock acquisition replay with writes in same batch
e2cfb79 
Informs cockroachdb#112221.
Informs cockroachdb#112174.
Informs cockroachdb#112173.
Informs cockroachdb#111984.
Informs cockroachdb#111893.
Informs cockroachdb#111564.
Informs cockroachdb#111530.

This commit fixes the handling of replayed batches that contain a
replicated lock acquisition and a later write to the same key. In such
cases, the write at the higher sequence number should not be detected as
an error during the replay. Instead, it should simply be ignored.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue Oct 23, 2023
Merged
craig bot pushed a commit that referenced this issue Oct 23, 2023
@DrewKimball @maryliag @nvanbenschoten
Merge #112725 #112891 #112893
a3d5317 
112725: sql: disallow refcursor comparisons during type-checking r=DrewKimball a=DrewKimball

This patch adds back the comparison functions for the `REFCURSOR` data type, since there are various points within the codebase where we rely on their existence. Users are still not allowed to use the comparison functions, but now this is checked during type-checking of the AST, rather than at execution-time. This should avoid internal errors from places in planning and execution that expect comparison overloads to exist, but also maintains parity with postgres.

Fixes #112365
Fixes #112642
Fixes #112362
Fixes #112368

Release note: None

112891: sql: fix for crash with null elements on jsonb_array_to_string_array r=maryliag a=maryliag

Fixes #112829

The builtin `jsonb_array_to_string_array` was crashing, when it had `null` elements. This commit handles this case, removing them from the final array.

Release note: None

112893: kv: tolerate lock acquisition replay with writes in same batch r=arulajmani a=nvanbenschoten

Informs #112221.
Informs #112174.
Informs #112173.
Informs #111984.
Informs #111893.
Informs #111564.
Informs #111530.

This commit fixes the handling of replayed batches that contain a replicated lock acquisition and a later write to the same key. In such cases, the write at the higher sequence number should not be detected as an error during the replay. Instead, it should simply be ignored.

Release note: None

Co-authored-by: Drew Kimball <[email protected]>
Co-authored-by: maryliag <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
blathers-crl bot pushed a commit that referenced this issue Oct 23, 2023
@nvanbenschoten
kv: tolerate lock acquisition replay with writes in same batch
f0390ea 
Informs #112221.
Informs #112174.
Informs #112173.
Informs #111984.
Informs #111893.
Informs #111564.
Informs #111530.

This commit fixes the handling of replayed batches that contain a
replicated lock acquisition and a later write to the same key. In such
cases, the write at the higher sequence number should not be detected as
an error during the replay. Instead, it should simply be ignored.

Release note: None
@blathers-crl blathers-crl bot mentioned this issue Oct 23, 2023
Merged
@nvanbenschoten
Copy link
Member

Fixed by #112893.

@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@miraradeva miraradeva

Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team
Projects
No open projects
KV
Archived in project
Milestone
23.2
Development

No branches or pull requests
4 participants
@nvanbenschoten @cockroach-teamcity @andrewbaptist @miraradeva