Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv/kvnemesis: TestKVNemesisMultiNode failed #93312

Closed
cockroach-teamcity opened this issue Dec 9, 2022 · 2 comments
Closed

kv/kvnemesis: TestKVNemesisMultiNode failed #93312

cockroach-teamcity opened this issue Dec 9, 2022 · 2 comments
Assignees
@erikgrinaker
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 9, 2022 •
edited by cockroach-jira-scripts
Loading

kv/kvnemesis.TestKVNemesisMultiNode failed with artifacts on master @ 3910ac409195ab6ab2ec3e9f6eda553048afa23f:

=== RUN   TestKVNemesisMultiNode
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1824305477
    test_log_scope.go:79: use -show-logs to present logs inline
    kvnemesis_test.go:171: seed: 4260519248299330720
    kvnemesis_test.go:115: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4090788434
    kvnemesis.go:165: range 367 (/Table/100/"576094dc79cc0f50") RANGE_CONSISTENT_STATS_INCORRECT: stats: {ContainsEstimates:0 LastUpdateNanos:1670581562071848089 IntentAge:0 GCBytesAge:22800 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:60 RangeValCount:2 RangeValBytes:20 SysBytes:4314 SysCount:51 AbortSpanBytes:3927}
        stats - recomputation: {ContainsEstimates:0 LastUpdateNanos:1670581562071848089 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:10 SysCount:0 AbortSpanBytes:0}
    kvnemesis.go:185: failures(verbose): /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4090788434/failures
        repro steps: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4090788434/repro.go
        rangefeed KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4090788434/kvs-rangefeed.txt
        scan KVs: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis4090788434/kvs-scan.txt
    kvnemesis_test.go:198: 
        	Error Trace:	/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:198
        	            				/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/kv/kvnemesis/kvnemesis_test_/kvnemesis_test.runfiles/com_github_cockroachdb_cockroach/pkg/kv/kvnemesis/kvnemesis_test.go:151
        	Error:      	Should be zero, but was 1
        	Test:       	TestKVNemesisMultiNode
        	Messages:   	kvnemesis detected failures
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisMultiNode1824305477
--- FAIL: TestKVNemesisMultiNode (448.74s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-22268
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Dec 9, 2022
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Dec 9, 2022
@blathers-crl blathers-crl bot added the T-kv KV Team label Dec 9, 2022
@erikgrinaker erikgrinaker self-assigned this Dec 9, 2022
@erikgrinaker
Copy link
Contributor

@tbg A few meta-points: the artifacts here are huge (~30GB), we should maybe consider storing them in a more compact form? Also, the repro.go uses tk(123) rather than the actual encoded key, while the traces, scans, and rangefeed dumps use the hex-encoded representation, so we may want to use the hex form everywhere to make it easier to grep.

@erikgrinaker erikgrinaker mentioned this issue Dec 9, 2022
Closed
@erikgrinaker
Copy link
Contributor

erikgrinaker commented Dec 9, 2022 •
edited
Loading

Heh, I was freaking out because I read this as the range having lost the range keys and other data, but the recomputation lists the delta, not the absolute stats. So there's a 10-byte discrepancy in SysBytes, which is hardly a crisis. We've also seen this come up elsewhere without a good explanation. Not critical, so I'm going to close this out for now, but will submit a PR to make the error message clearer.

@erikgrinaker erikgrinaker mentioned this issue Dec 9, 2022
Merged
craig bot pushed a commit that referenced this issue Dec 9, 2022
@srosenberg @erikgrinaker @nicktrav
Merge #93299 #93333 #93341
1a36338 
93299: roachprod: bug fixes for azure.Extend r=dbist a=srosenberg

roachprod extend is intended to extend the lifetime of an existing cluster. This functionality was previously broken in Azure.

The changes fix the two existing bugs, namely the use of the fully-qualified subscription path (instead of the id), and the truncation of all the existing VM tags--Tags in
compute.VirtualMachineUpdate must include _all_ tags, not just the modified one.

Epic: none

Release note: None

93333: kvserver: clarify consistency checker stats delta message r=erikgrinaker a=erikgrinaker

The error message on consistency checker stats mismatches could be misleading. E.g. a test failure said:

```
RANGE_CONSISTENT_STATS_INCORRECT: stats: {ContainsEstimates:0 LastUpdateNanos:1670581562071848089 IntentAge:0 GCBytesAge:22800 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:1 RangeKeyBytes:60 RangeValCount:2 RangeValBytes:20 SysBytes:4314 SysCount:51 AbortSpanBytes:3927}
stats - recomputation: {ContainsEstimates:0 LastUpdateNanos:1670581562071848089 IntentAge:0 GCBytesAge:0 LiveBytes:0 LiveCount:0 KeyBytes:0 KeyCount:0 ValBytes:0 ValCount:0 IntentBytes:0 IntentCount:0 SeparatedIntentCount:0 RangeKeyCount:0 RangeKeyBytes:0 RangeValCount:0 RangeValBytes:0 SysBytes:10 SysCount:0 AbortSpanBytes:0}
```

This could be read as "recomputed stats", i.e. the absolute value of the recomputed stats, indicating possible data loss. However, it is the delta of the stats minus the recomputed stats, i.e. a 10-byte discrepancy in `SysBytes`. This patch clarifies the message.

Touches #93312.

Epic: none
Release note: None

93341: vendor: bump Pebble to 463b049c1dd1 r=jbowens a=nicktrav

```
463b049c tool: tweak the log context regexp
5f03b000 internal: Allow building on riscv64 architecture
```

Release note: None.

Epic: None.

Co-authored-by: Stan Rosenberg <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Co-authored-by: Nick Travers <[email protected]>
@erikgrinaker erikgrinaker mentioned this issue Dec 19, 2022
Merged
craig bot pushed a commit that referenced this issue Dec 19, 2022
@msbutler @erikgrinaker
Merge #93837 #93897
a34ab92 
93837: backupccl: deflake TestClusterRestoreFailCleanup r=stevendanna a=msbutler

This test occasionally flakes due to #86806. To prevent the flakiness, this patch manually sets the kv.bulkio.write_metadata_sst.enabled cluster setting to false. When #86806  gets addressed, this patch should be reverted.

Epic: None

Release note: None

93897: kvnemesis: ignore `SysBytes:10` MVCC stats discrepancy r=erikgrinaker a=erikgrinaker

Resolves #93890.
Touches #93896.
Touches #93312.
Touches #86542.

Epic: none
Release note: None

Co-authored-by: Michael Butler <[email protected]>
Co-authored-by: Erik Grinaker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erikgrinaker erikgrinaker

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. T-kv KV Team
Projects
None yet
Milestone
23.1
Development

No branches or pull requests
2 participants
@erikgrinaker @cockroach-teamcity