Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage/metamorphic: TestPebbleEquivalence failed #113806

Closed
cockroach-teamcity opened this issue Nov 4, 2023 · 4 comments · Fixed by #113955
Closed

storage/metamorphic: TestPebbleEquivalence failed #113806

cockroach-teamcity opened this issue Nov 4, 2023 · 4 comments · Fixed by #113955
Assignees
@itsbilal
Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 4, 2023 •
edited by cockroach-jira-scripts
Loading

storage/metamorphic.TestPebbleEquivalence failed with artifacts on master @ 0496181c886c305f261598f90f0b226f9940fdac:

=== RUN   TestPebbleEquivalence
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/logTestPebbleEquivalence736555959
    test_log_scope.go:81: use -show-logs to present logs inline
first run output file: /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence2459347338/output.meta
    meta_test.go:177: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/logTestPebbleEquivalence736555959
--- FAIL: TestPebbleEquivalence (88.35s)
=== RUN   TestPebbleEquivalence/engine/random=3
check file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence2459347338/output.meta
output file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_random=31958581041/output.meta
store path = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_random=34082699037/store
    generator.go:417: mismatching output at line 3942, operation index 3828: expected error: conflicting locks on "fpyfokmwldyanoh", "gqmhifphuvitld", "gsbbcwwpaebrt", "gsquhoskp", "haibktlhi"
        , got truncated range to delete = "dapisequstrvglr" - "pnraowaljkodaip", deleted keys = "dbvtpramvcaw", "encwkcqwpldpwbg", "fjsadflg", "fttowbultcdhvjg", "gsofxqcpydjmp", "jjtxqcrbbji", "jvadyvovotjr", "jwdgoadlcbhwyw", "jwqtcmmnrx", "ljgwwvclmbrtde", "nmkcurmuiucsr", "oyqllujts"
    --- FAIL: TestPebbleEquivalence/engine/random=3 (1.53s)
=== RUN   TestPebbleEquivalence/engine/standard=7
check file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence2459347338/output.meta
output file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_standard=72565263700/output.meta
store path = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_standard=71191460795/store
    generator.go:417: mismatching output at line 3942, operation index 3828: expected error: conflicting locks on "fpyfokmwldyanoh", "gqmhifphuvitld", "gsbbcwwpaebrt", "gsquhoskp", "haibktlhi"
        , got truncated range to delete = "dapisequstrvglr" - "pnraowaljkodaip", deleted keys = "dbvtpramvcaw", "encwkcqwpldpwbg", "fjsadflg", "fttowbultcdhvjg", "gsofxqcpydjmp", "jjtxqcrbbji", "jvadyvovotjr", "jwdgoadlcbhwyw", "jwqtcmmnrx", "ljgwwvclmbrtde", "nmkcurmuiucsr", "oyqllujts"
    --- FAIL: TestPebbleEquivalence/engine/standard=7 (0.59s)
=== RUN   TestPebbleEquivalence/engine/random=0
check file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence2459347338/output.meta
output file = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_random=02012281150/output.meta
store path = /artifacts/tmp/_tmp/ed77015fcf5315e89955a74e01226740/TestPebbleEquivalence_engine_random=01830834771/store
    generator.go:417: mismatching output at line 3942, operation index 3828: expected error: conflicting locks on "fpyfokmwldyanoh", "gqmhifphuvitld", "gsbbcwwpaebrt", "gsquhoskp", "haibktlhi"
        , got truncated range to delete = "dapisequstrvglr" - "pnraowaljkodaip", deleted keys = "dbvtpramvcaw", "encwkcqwpldpwbg", "fjsadflg", "fttowbultcdhvjg", "gsofxqcpydjmp", "jjtxqcrbbji", "jvadyvovotjr", "jwdgoadlcbhwyw", "jwqtcmmnrx", "ljgwwvclmbrtde", "nmkcurmuiucsr", "oyqllujts"
    --- FAIL: TestPebbleEquivalence/engine/random=0 (0.58s)
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-33169
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team labels Nov 4, 2023
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Nov 4, 2023
@blathers-crl blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Nov 4, 2023
@jbowens
Copy link
Collaborator

jbowens commented Nov 4, 2023

@itsbilal - fallout from #113698 ?

@cockroach-teamcity
Copy link
Member Author

storage/metamorphic.TestPebbleEquivalence failed with artifacts on master @ a831e248ba2650f2e68facfd79d9a1cdba8114b1:

=== RUN   TestPebbleEquivalence
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/logTestPebbleEquivalence3557191963
    test_log_scope.go:81: use -show-logs to present logs inline
first run output file: /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence3610142892/output.meta
    meta_test.go:177: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/logTestPebbleEquivalence3557191963
--- FAIL: TestPebbleEquivalence (130.94s)
=== RUN   TestPebbleEquivalence/engine/random=0
check file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence3610142892/output.meta
output file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_random=01096607222/output.meta
store path = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_random=03032600823/store
    generator.go:417: mismatching output at line 13863, operation index 13749: expected error: conflicting locks on "ioryisoep"
        , got kvs = [{"fgtialbvfyn" {[] 0.000001228,0}} {"fousrmbovbmmswc" {[] 0.000001249,0}} {"ioryisoep" {[] 0.000001249,0}} {"keanusfkm" {[0 0 0 0 3 110 109 104 117 119 105 114 120 109 107] 0.000001249,0}} {"kotqaviwnsmqnle" {[0 0 0 0 3 105 114 113 108 102 114 110 117 117] 0.000001253,0}} {"lqkcaphyueywndm" {[0 0 0 0 3 116 117 112 121 100 101] 0.000001253,0}} {"nbtvwvebjjvq" {[0 0 0 0 3 105 99 115 109 121 108 117 112 99] 0.000001247,0}} {"ohaeyccr" {[0 0 0 0 3 114 104 118 104 114 110 104 104 103 112 103] 0.000001249,0}} {"pkhkiknp" {[0 0 0 0 3 114 97 105 118 113 117 106 106 105 104 111 114] 0.000001222,0}} {"ppnlsfmyutu" {[0 0 0 0 3 103 120 106 101 106 108 103] 0.000001249,0}} {"pvdmfbkopustpa" {[] 0.000001253,0}} {"qanmuungsckvxm" {[0 0 0 0 3 102 111 101 113 112 100 102 97 105 108 104 97 117 106 119] 0.000001205,0}} {"qctoqiea" {[] 0.000001159,0}} {"qdilljekhqrcx" {[] 0.000001249,0}} {"qdjycutxiwcsrhi" {[0 0 0 0 3 104 117 113 116 107 120 120 113 110 115 117 111 106 121 119] 0.000001247,0}} {"qhdlbwqkcdlnxu" {[0 0 0 0 3 111 110 106 106 98] 0.000001222,0}} {"qryctepeatw" {[0 0 0 0 3 116 98 109 107 100 100 112 103 108 103 111 109 115 104 100] 0.000001175,0}} {"qtsmsirw" {[0 0 0 0 3 115 103 99 103 108 107 97 97 120 106 109] 0.000001214,0}} {"quxlcmuhwnsbv" {[0 0 0 0 3 97 108 119 101 114 109 103 118 112 112 110 103 107 100 99] 0.000001228,0}} {"rcafhqxrgkcqq" {[] 0.000001222,0}} {"rdvmqmakmc" {[0 0 0 0 3 97 108 105 97 116 105 111 104] 0.000001247,0}} {"rjsqrdyrw" {[0 0 0 0 3 111 114 119 109 116 106] 0.000001222,0}} {"rlfnibovc" {[0 0 0 0 3 104 116 102 107 100 100] 0.000001195,0}} {"rljfmbipixltj" {[0 0 0 0 3 113 110 114 111 113 104 101 105] 0.000001205,0}} {"rmuskafuy" {[0 0 0 0 3 102 102 118 121 102 103] 0.000001231,0}} {"rqrdkxsrccyi" {[105 110 103 101 115 116 101 100] 0.000000963,0}} {"rscmdcioyatkc" {[0 0 0 0 3 105 119 105 100 121 116] 0.000001253,0}} {"rvcjxwjs" {[0 0 0 0 3 105 107 115 113 120 117 108] 0.000001253,0}} {"rvkaquufllvrf" {[0 0 0 0 3 118 107 117 105 101 112 97 105 112 116 119 97 117] 0.000001159,0}} {"sbpdekejp" {[0 0 0 0 3 121 100 100 121 97 109 116 103 118 107] 0.000001231,0}}], intents = [], resumeSpan = s{dtvglon-iergcdkddhxa}, numBytes = 1209, numKeys = 30
    --- FAIL: TestPebbleEquivalence/engine/random=0 (13.12s)
=== RUN   TestPebbleEquivalence/engine/random=8
check file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence3610142892/output.meta
output file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_random=8541657352/output.meta
store path = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_random=81153915273/store
    generator.go:417: mismatching output at line 13863, operation index 13749: expected error: conflicting locks on "ioryisoep"
        , got kvs = [{"fgtialbvfyn" {[] 0.000001228,0}} {"fousrmbovbmmswc" {[] 0.000001249,0}} {"ioryisoep" {[] 0.000001249,0}} {"keanusfkm" {[0 0 0 0 3 110 109 104 117 119 105 114 120 109 107] 0.000001249,0}} {"kotqaviwnsmqnle" {[0 0 0 0 3 105 114 113 108 102 114 110 117 117] 0.000001253,0}} {"lqkcaphyueywndm" {[0 0 0 0 3 116 117 112 121 100 101] 0.000001253,0}} {"nbtvwvebjjvq" {[0 0 0 0 3 105 99 115 109 121 108 117 112 99] 0.000001247,0}} {"ohaeyccr" {[0 0 0 0 3 114 104 118 104 114 110 104 104 103 112 103] 0.000001249,0}} {"pkhkiknp" {[0 0 0 0 3 114 97 105 118 113 117 106 106 105 104 111 114] 0.000001222,0}} {"ppnlsfmyutu" {[0 0 0 0 3 103 120 106 101 106 108 103] 0.000001249,0}} {"pvdmfbkopustpa" {[] 0.000001253,0}} {"qanmuungsckvxm" {[0 0 0 0 3 102 111 101 113 112 100 102 97 105 108 104 97 117 106 119] 0.000001205,0}} {"qctoqiea" {[] 0.000001159,0}} {"qdilljekhqrcx" {[] 0.000001249,0}} {"qdjycutxiwcsrhi" {[0 0 0 0 3 104 117 113 116 107 120 120 113 110 115 117 111 106 121 119] 0.000001247,0}} {"qhdlbwqkcdlnxu" {[0 0 0 0 3 111 110 106 106 98] 0.000001222,0}} {"qryctepeatw" {[0 0 0 0 3 116 98 109 107 100 100 112 103 108 103 111 109 115 104 100] 0.000001175,0}} {"qtsmsirw" {[0 0 0 0 3 115 103 99 103 108 107 97 97 120 106 109] 0.000001214,0}} {"quxlcmuhwnsbv" {[0 0 0 0 3 97 108 119 101 114 109 103 118 112 112 110 103 107 100 99] 0.000001228,0}} {"rcafhqxrgkcqq" {[] 0.000001222,0}} {"rdvmqmakmc" {[0 0 0 0 3 97 108 105 97 116 105 111 104] 0.000001247,0}} {"rjsqrdyrw" {[0 0 0 0 3 111 114 119 109 116 106] 0.000001222,0}} {"rlfnibovc" {[0 0 0 0 3 104 116 102 107 100 100] 0.000001195,0}} {"rljfmbipixltj" {[0 0 0 0 3 113 110 114 111 113 104 101 105] 0.000001205,0}} {"rmuskafuy" {[0 0 0 0 3 102 102 118 121 102 103] 0.000001231,0}} {"rqrdkxsrccyi" {[105 110 103 101 115 116 101 100] 0.000000963,0}} {"rscmdcioyatkc" {[0 0 0 0 3 105 119 105 100 121 116] 0.000001253,0}} {"rvcjxwjs" {[0 0 0 0 3 105 107 115 113 120 117 108] 0.000001253,0}} {"rvkaquufllvrf" {[0 0 0 0 3 118 107 117 105 101 112 97 105 112 116 119 97 117] 0.000001159,0}} {"sbpdekejp" {[0 0 0 0 3 121 100 100 121 97 109 116 103 118 107] 0.000001231,0}}], intents = [], resumeSpan = s{dtvglon-iergcdkddhxa}, numBytes = 1209, numKeys = 30
    --- FAIL: TestPebbleEquivalence/engine/random=8 (11.70s)
=== RUN   TestPebbleEquivalence/engine/standard=7
check file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence3610142892/output.meta
output file = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_standard=72322459676/output.meta
store path = /artifacts/tmp/_tmp/3cb480549dd56d78f96a16434f955825/TestPebbleEquivalence_engine_standard=73264641453/store
    generator.go:417: mismatching output at line 13863, operation index 13749: expected error: conflicting locks on "ioryisoep"
        , got kvs = [{"fgtialbvfyn" {[] 0.000001228,0}} {"fousrmbovbmmswc" {[] 0.000001249,0}} {"ioryisoep" {[] 0.000001249,0}} {"keanusfkm" {[0 0 0 0 3 110 109 104 117 119 105 114 120 109 107] 0.000001249,0}} {"kotqaviwnsmqnle" {[0 0 0 0 3 105 114 113 108 102 114 110 117 117] 0.000001253,0}} {"lqkcaphyueywndm" {[0 0 0 0 3 116 117 112 121 100 101] 0.000001253,0}} {"nbtvwvebjjvq" {[0 0 0 0 3 105 99 115 109 121 108 117 112 99] 0.000001247,0}} {"ohaeyccr" {[0 0 0 0 3 114 104 118 104 114 110 104 104 103 112 103] 0.000001249,0}} {"pkhkiknp" {[0 0 0 0 3 114 97 105 118 113 117 106 106 105 104 111 114] 0.000001222,0}} {"ppnlsfmyutu" {[0 0 0 0 3 103 120 106 101 106 108 103] 0.000001249,0}} {"pvdmfbkopustpa" {[] 0.000001253,0}} {"qanmuungsckvxm" {[0 0 0 0 3 102 111 101 113 112 100 102 97 105 108 104 97 117 106 119] 0.000001205,0}} {"qctoqiea" {[] 0.000001159,0}} {"qdilljekhqrcx" {[] 0.000001249,0}} {"qdjycutxiwcsrhi" {[0 0 0 0 3 104 117 113 116 107 120 120 113 110 115 117 111 106 121 119] 0.000001247,0}} {"qhdlbwqkcdlnxu" {[0 0 0 0 3 111 110 106 106 98] 0.000001222,0}} {"qryctepeatw" {[0 0 0 0 3 116 98 109 107 100 100 112 103 108 103 111 109 115 104 100] 0.000001175,0}} {"qtsmsirw" {[0 0 0 0 3 115 103 99 103 108 107 97 97 120 106 109] 0.000001214,0}} {"quxlcmuhwnsbv" {[0 0 0 0 3 97 108 119 101 114 109 103 118 112 112 110 103 107 100 99] 0.000001228,0}} {"rcafhqxrgkcqq" {[] 0.000001222,0}} {"rdvmqmakmc" {[0 0 0 0 3 97 108 105 97 116 105 111 104] 0.000001247,0}} {"rjsqrdyrw" {[0 0 0 0 3 111 114 119 109 116 106] 0.000001222,0}} {"rlfnibovc" {[0 0 0 0 3 104 116 102 107 100 100] 0.000001195,0}} {"rljfmbipixltj" {[0 0 0 0 3 113 110 114 111 113 104 101 105] 0.000001205,0}} {"rmuskafuy" {[0 0 0 0 3 102 102 118 121 102 103] 0.000001231,0}} {"rqrdkxsrccyi" {[105 110 103 101 115 116 101 100] 0.000000963,0}} {"rscmdcioyatkc" {[0 0 0 0 3 105 119 105 100 121 116] 0.000001253,0}} {"rvcjxwjs" {[0 0 0 0 3 105 107 115 113 120 117 108] 0.000001253,0}} {"rvkaquufllvrf" {[0 0 0 0 3 118 107 117 105 101 112 97 105 112 116 119 97 117] 0.000001159,0}} {"sbpdekejp" {[0 0 0 0 3 121 100 100 121 97 109 116 103 118 107] 0.000001231,0}}], intents = [], resumeSpan = s{dtvglon-iergcdkddhxa}, numBytes = 1209, numKeys = 30
    --- FAIL: TestPebbleEquivalence/engine/standard=7 (4.29s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

@itsbilal itsbilal self-assigned this Nov 6, 2023
@itsbilal itsbilal removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Nov 6, 2023
@itsbilal
Copy link
Contributor

itsbilal commented Nov 6, 2023

On closer inspection, this is looking like fallout from #112732 which also landed on Friday.

@itsbilal
Copy link
Contributor

itsbilal commented Nov 7, 2023

Nevermind. I did something dumb in #113698. Fixing.

@itsbilal itsbilal mentioned this issue Nov 7, 2023
Merged
craig bot pushed a commit that referenced this issue Nov 7, 2023
@itsbilal
Merge #113955
f8c553c 
113955: storage/metamorphic: use unused keys in MVCCAcquireLock r=RaduBerinde a=itsbilal

Previously, we were overwriting the txnGenerator's view of what transaction was writing to what key, by having MVCCAcquireLock work on all mvcc keys, not just unused ones. This would generate operations that would break expected usage of the MVCC API and result in non-determinism.

This change fixes it by having MVCCAcquireLock use keys that don't have conflicting transactions. Note that MVCCCheckForAcquireLock will continue to run on the whole keyspace as it doesn't need to track writes.

Fixes #113806.

Epic: none

Release note: None

Co-authored-by: Bilal Akhtar <[email protected]>
@craig craig bot closed this as completed in 1f16ff2 Nov 7, 2023
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@itsbilal itsbilal

Labels
A-storage Relating to our storage engine (Pebble) on-disk storage. branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

storage/metamorphic: use unused keys in MVCCAcquireLock itsbilal/cockroach
3 participants
@itsbilal @jbowens @cockroach-teamcity