Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: clearrange/checks=false/rangeTs=false failed #97612

Closed
cockroach-teamcity opened this issue Feb 24, 2023 · 3 comments · Fixed by #97564
Closed

roachtest: clearrange/checks=false/rangeTs=false failed #97612

cockroach-teamcity opened this issue Feb 24, 2023 · 3 comments · Fixed by #97564
Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Milestone
23.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 24, 2023 •
edited by cockroach-jira-scripts
Loading

roachtest.clearrange/checks=false/rangeTs=false failed with artifacts on master @ 0d3393b0623a5c258b25725f64f3689e2f54667b:

test artifacts and logs in: /artifacts/clearrange/checks=false/rangeTs=false/run_1
(cluster.go:1956).Run: output in run_105101.069200958_n1-10_cockroach-workload-r: ./cockroach workload run kv --concurrency=32 --duration=1h returned: context canceled
(monitor.go:127).Wait: monitor failure: monitor command failure: unexpected node event: 10: dead (exit status 7)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-24786
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 24, 2023
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Feb 24, 2023
@blathers-crl blathers-crl bot added the T-storage Storage Team label Feb 24, 2023
@jbowens
Copy link
Collaborator

jbowens commented Feb 24, 2023 •
edited
Loading

n10 died with a fatal error in (*replicaStateMachine).ApplySideEffects

F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165  finishing proposal with outstanding reproposal at a higher max lease index: ‹Term:8 Index:185037 Type:EntryNormal Data:"\000eT\376\356\277\036\024\0340uj$B\n\010\366\361\243\367\354\251\257\243\027\222\001\025\t\366\370\350\316N\275F\027 D(\002048\002@\020H\002r0\n.\000\000\000\000\000\000\000\000\001\000\000\000\001\027\366p\211\200\236\2617\226\333\316/\303\210\000\027F\275N\316\350\370\366\t\010Z\233\345\251\n&\001\355 \246\245\013\212\001\n\010\203\304\361\342\341\251\257\243\027"›
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !goroutine 456 [running]:
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x1)
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0x89
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc000dedc00, {{{0xc001cd6060, 0x24}, {0x0, 0x0}, {0x590b7d2, 0x1}, {0x5a8ce17, 0x2}}, 0x1746bd4ecfd4b4e7, ...})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:262 +0x97
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x6d3c548, 0xc00cabe540}, 0x2, 0x4, 0x0, 0x0?, {0x5a37b3a, 0x4f}, {0xc010c74000, 0x1, ...})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:106 +0x645
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/util/log/log_channels_generated.go:848
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc00a626680, {0x6d3c548, 0xc00cabe540}, {0x6d7d6d8?, 0xc002c7d408})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:241 +0x925
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f49c8652548?, 0xc00a626a50}, 0xc010c747d0)
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:210 +0x158
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc010c74cc8, {0x6d3c548, 0xc018f5e600}, {0x6d6d200, 0xc00a6269f0})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:310 +0x1f8
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc010c74cc8, {0x6d3c548, 0xc018f5e600})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1004 +0x13b9
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:699 +0x17b
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0016f8a80, 0x6d3c548?)
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:644 +0x145
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc000953800, {0x6d3c548, 0xc007e12570})
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:333 +0x235
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 +0x146
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F230224 10:54:17.434010 456 kv/kvserver/replica_application_state_machine.go:241 ⋮ [T1,n10,s10,r2479/2:‹/Table/112/1/-{74874…-68474…}›,raft] 165 !	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:461 +0x43b

@blathers-crl blathers-crl bot added the T-kv KV Team label Feb 24, 2023
@pav-kv
Copy link
Collaborator

pav-kv commented Feb 24, 2023

Same issue as #97102. Should be fixed by #97564.

@pav-kv pav-kv added the A-kv-replication Relating to Raft, consensus, and coordination. label Feb 24, 2023
@pav-kv pav-kv added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. and removed T-kv-replication labels Feb 24, 2023
@blathers-crl
Copy link

blathers-crl bot commented Feb 24, 2023

cc @cockroachdb/replication

@pav-kv pav-kv mentioned this issue Feb 24, 2023
Merged
@craig craig bot closed this as completed in 13c58f6 Feb 26, 2023
@exalate-issue-sync exalate-issue-sync bot removed the T-kv KV Team label Feb 26, 2023
@jbowens jbowens moved this to Done in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-replication Relating to Raft, consensus, and coordination. branch-master Failures and bugs on the master branch. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-storage Storage Team
Projects
No open projects
[Deprecated] Storage
Archived in project
Milestone
23.1
Development

Successfully merging a pull request may close this issue.

kvserver: narrow down 'finishing a proposal with outstanding reproposal' tbg/cockroach
3 participants
@jbowens @pav-kv @cockroach-teamcity