kv/kvnemesis: TestKVNemesisSingleNode failed #99010

cockroach-teamcity · 2023-03-20T08:07:01Z

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ de5b08a418390a803357ba4b74a55a4eec26e16b:

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2060427476
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 6463069194924432758
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis306452052
    kvnemesis.go:165: range 86 (/Table/100/"37159e1ad40c2773") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679299535647728864 IntentAge:0 GCBytesAge:5991 LiveBytes:0 LiveCount:0 KeyBytes:213 KeyCount:5 ValBytes:143 ValCount:9 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:69 RangeValCount:3 RangeValBytes:31 SysBytes:4590 SysCount:39 AbortSpanBytes:2818}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679299535647728864 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:6 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis306452052/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis306452052/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis306452052/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis306452052/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:148
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisSingleNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2060427476
--- FAIL: TestKVNemesisSingleNode (24.64s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv _{This test on roachdash | Improve this report!

Jira issue: CRDB-25655}

The text was updated successfully, but these errors were encountered:

erikgrinaker · 2023-03-20T08:51:58Z

See #93896.

cockroach-teamcity · 2023-03-22T08:44:02Z

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ 78bb7696847e1d8abf5d6a199c6dd3414a568691:

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2529120921
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 5521694330780283284
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2946656328
    kvnemesis.go:165: range 100 (/Table/100/"15f4e59789426c4d") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679474556240215977 IntentAge:0 GCBytesAge:9841 LiveBytes:106 LiveCount:2 KeyBytes:462 KeyCount:10 ValBytes:366 ValCount:21 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:4262 SysCount:34 AbortSpanBytes:2386}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679474556240215977 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-6 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2946656328/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2946656328/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2946656328/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2946656328/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:148
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisSingleNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2529120921
--- FAIL: TestKVNemesisSingleNode (29.86s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-25T09:01:51Z

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ 0fcc33bc2870961b9387999d8a9fed97fccbb2ae:

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2442833695
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 995057339477242240
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1883274959
    kvnemesis.go:165: range 122 (/Table/100/"0ac6a91c57f6e6a6") RANGE_CONSISTENT_STATS_INCORRECT:
        stats: {ContainsEstimates:0 LastUpdateNanos:1679734855722795870 IntentAge:0 GCBytesAge:70080 LiveBytes:53 LiveCount:1 KeyBytes:4785 KeyCount:105 ValBytes:3330 ValCount:215 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:17 RangeKeyBytes:1150 RangeValCount:48 RangeValBytes:495 SysBytes:5617 SysCount:21 AbortSpanBytes:693}
        delta (stats-computed): {ContainsEstimates:0 LastUpdateNanos:1679734855722795870 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:-6 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1883274959/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1883274959/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1883274959/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1883274959/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:148
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisSingleNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2442833695
--- FAIL: TestKVNemesisSingleNode (19.45s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: How To Investigate a Go Test Failure (internal)

_{This test on roachdash | Improve this report!}

cockroach-teamcity · 2023-03-27T07:20:30Z

kv/kvnemesis.TestKVNemesisSingleNode failed with artifacts on master @ 2bd2c806ab3044569b09e0a205b5bc0452ad4e2b:

=== RUN   TestKVNemesisSingleNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2323803564
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:180: seed: 3755574968338271651
    kvnemesis_test.go:124: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3596276386
    kvnemesis.go:165: error applying x.AddSSTable(ctx, tk(10906530712285504299), tk(12340570697873214426), ... /* @s416 */) // 1549 bytes
        // ^-- tk(10906530712285504299) -> sv(s416): /Table/100/"975bc7e5c5aa632b"/0,1 -> /BYTES/v416
        // ^-- tk(10974481074854236397) -> sv(s416): /Table/100/"984d30646cef10ed"/0,1 -> /BYTES/v416
        // ^-- [tk(11312515042409405318), tk(11315988874302194829)) -> sv(s416): /Table/100/"9{cfe207eab135b86"-d0a77ed2f0e048d"} -> /<empty>
        // ^-- tk(11331132117531587368) -> sv(s416): /Table/100/"9d4044a0407d7328"/0,1 -> /<empty>
        // ^-- tk(11343225235233570111) -> sv(s416): /Table/100/"9d6b3b40a157fd3f"/0,1 -> /BYTES/v416
        // ^-- tk(11404188119824406742) -> sv(s416): /Table/100/"9e43d0ac49dd18d6"/0,1 -> /<empty>
        // ^-- tk(11689754648867613221) -> sv(s416): /Table/100/"a23a59eb0d39aa25"/0,1 -> /BYTES/v416
        // ^-- tk(11780948592989728782) -> sv(s416): /Table/100/"a37e5653c24a700e"/0,1 -> /BYTES/v416
        // ^-- tk(12134961105016806227) -> sv(s416): /Table/100/"a8680ad4cbb55b53"/0,1 -> /BYTES/v416
        // ^-- tk(12306289562062562168) -> sv(s416): /Table/100/"aac8b927b9d62378"/0,1 -> /BYTES/v416
        // ^-- [tk(12321816705424378089), tk(12323913456553089929)) -> sv(s416): /Table/100/"a{affe302824b48e9"-b0755fe6c80db89"} -> /<empty>
        // ^-- tk(12340570697873214425) -> sv(s416): /Table/100/"ab4283aaa453cbd9"/0,1 -> /BYTES/v416: checking for key collisions: expected engine iter to be ahead of sst iter
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3596276386/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3596276386/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3596276386/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3596276386/kvs-scan.txt
    kvnemesis_test.go:207: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:207
        	            				github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:148
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisSingleNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode2323803564
--- FAIL: TestKVNemesisSingleNode (8.11s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

kv/kvnemesis: TestKVNemesisSingleNode failed #99566 kv/kvnemesis: TestKVNemesisSingleNode failed [C-test-failure GA-blocker O-robot T-storage branch-release-23.1]

_{This test on roachdash | Improve this report!}

erikgrinaker · 2023-03-27T08:04:48Z

Last failure mode here is same as #99566. cc @itsbilal

erikgrinaker · 2023-03-27T15:41:20Z

Reopening to track checking for key collisions: expected engine iter to be ahead of sst iter.

99541: ui: update sort setting on search criteria r=maryliag a=maryliag Fixes #99397 When a new search criteria is applied, the table on Statement and Transactions are sorted to match the value selected by the search criteria. When a new column is selected on the table, a warning is displayed to let the users know they're looking into a subset od the data. If the new column selected is one of the options on the search criteria, we give a suggestion to update the search criteria with that value instead. <img width="1229" alt="Screenshot 2023-03-24 at 8 27 52 PM" src="https://user-images.githubusercontent.com/1017486/227667529-14c9dc5c-dc00-4dd9-b12d-78058c381fc4.png"> https://www.loom.com/share/e123e4033d95434a8eef20c257c83bbd This commit adds the counter per page on the Statement and Transaction tables. <img width="705" alt="Screenshot 2023-03-24 at 8 28 01 PM" src="https://user-images.githubusercontent.com/1017486/227667540-bc7aae81-2121-46c1-ad01-94ceda34960b.png"> This commit also adds analytics to the apply button, sending the information about each criteria. Release note (ui change): Add a warning when a user select a sorting column on Statement and Transaction tables that were not the original selected sorting on the search criteria. 99674: sql: add explicit column family assignments to persisted stats upgrades r=ericharmeling a=ericharmeling Fixes #99270. Part of #99260. This commit adds explicit column family assignments to the computed column `ADD COLUMN` statements for system.statement_statistics and system.transaction_statistics version upgrades. Epic: none Release note: None 99695: storage: CheckSSTConflicts: fix instance of iterator mismatch r=RaduBerinde a=itsbilal Previously, in one case, we'd let the sst iterator advance ahead of the engine iterator, which violates an invariant in this function. This change fixes that case. Fixes #99566. Fixes #99010. Epic: none Release note: None Co-authored-by: maryliag <[email protected]> Co-authored-by: Eric Harmeling <[email protected]> Co-authored-by: Bilal Akhtar <[email protected]>

Previously, in one case, we'd let the sst iterator advance ahead of the engine iterator, which violates an invariant in this function. This change fixes that case. Fixes #99566. Fixes #99010. Epic: none Release note: None

cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Mar 20, 2023

cockroach-teamcity added this to the 23.1 milestone Mar 20, 2023

blathers-crl bot added the T-kv KV Team label Mar 20, 2023

erikgrinaker self-assigned this Mar 20, 2023

erikgrinaker mentioned this issue Mar 20, 2023

kvserver: reduce SysBytes MVCC stats race during merges #99017

Merged

cockroach-teamcity mentioned this issue Mar 25, 2023

kv/kvnemesis: TestKVNemesisSingleNode failed #99566

Closed

craig bot closed this as completed in 3c3d2a5 Mar 27, 2023

erikgrinaker reopened this Mar 27, 2023

erikgrinaker assigned itsbilal and unassigned erikgrinaker Mar 27, 2023

erikgrinaker added the T-storage Storage Team label Mar 27, 2023

blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Mar 27, 2023

erikgrinaker added GA-blocker and removed A-storage Relating to our storage engine (Pebble) on-disk storage. labels Mar 27, 2023

blathers-crl bot added the A-storage Relating to our storage engine (Pebble) on-disk storage. label Mar 27, 2023

erikgrinaker removed A-storage Relating to our storage engine (Pebble) on-disk storage. T-kv KV Team labels Mar 27, 2023

itsbilal mentioned this issue Mar 27, 2023

storage: CheckSSTConflicts: fix instance of iterator mismatch #99695

Merged

craig bot closed this as completed in 49507f3 Mar 28, 2023

blathers-crl bot mentioned this issue Mar 28, 2023

release-23.1: storage: CheckSSTConflicts: fix instance of iterator mismatch #99755

Merged

jbowens added this to [Deprecated] Storage Jun 4, 2024

jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kv/kvnemesis: TestKVNemesisSingleNode failed #99010

kv/kvnemesis: TestKVNemesisSingleNode failed #99010

cockroach-teamcity commented Mar 20, 2023 •

edited by cockroach-jira-scripts

Loading

erikgrinaker commented Mar 20, 2023

cockroach-teamcity commented Mar 22, 2023

cockroach-teamcity commented Mar 25, 2023

cockroach-teamcity commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

kv/kvnemesis: TestKVNemesisSingleNode failed #99010

kv/kvnemesis: TestKVNemesisSingleNode failed #99010

Comments

cockroach-teamcity commented Mar 20, 2023 • edited by cockroach-jira-scripts Loading

erikgrinaker commented Mar 20, 2023

cockroach-teamcity commented Mar 22, 2023

cockroach-teamcity commented Mar 25, 2023

cockroach-teamcity commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

erikgrinaker commented Mar 27, 2023

cockroach-teamcity commented Mar 20, 2023 •

edited by cockroach-jira-scripts

Loading