Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: version/mixed/nodes=5 failed [should stop after beta1] #70252

Closed
cockroach-teamcity opened this issue Sep 15, 2021 · 6 comments · Fixed by #70432
Closed

roachtest: version/mixed/nodes=5 failed [should stop after beta1] #70252

cockroach-teamcity opened this issue Sep 15, 2021 · 6 comments · Fixed by #70432
Assignees
@irfansharif
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 2baefd353239ddc4e7f4299b8930001675b99aa9:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/version/mixed/nodes=5/run_1
	monitor.go:128,version.go:211,version.go:224,test_runner.go:777: monitor failure: unexpected node event: 3: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:211
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:224
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 3: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1253,context.go:89,cluster.go:1241,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3445998-1631686180-39-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 25637
		3: dead (exit status 7)
		2: 24163
		6: skipped
		4: 21358
		5: 21692
		Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 3: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 15, 2021
@tbg
Copy link
Member

tbg commented Sep 15, 2021

@irfansharif a v21.1.8 node crashed. I think (but you should check) that this is happening while rolling back from master to 21.1.8 (without having bumped the cluster version). I think we can safely assume this is fallout from #69887, please take a look as soon as you can. We're not backporting this but since it's entirely unexpected, it's still weird and we better find out what's causing it, it might have an effect on the release yet.
Also, note that master still identifies as 21.1 since we don't have a 21.2 binary yet that we could use as a predecessorVersion of 22.1. Maybe that is the whole reason for this crash? We ripped out things thinking the only version we would need to interop with was 21.2, but here we are running mixed with 21.1.

F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86  on-disk and in-memory state diverged: [UsingAppliedStateKey: true != false]
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !goroutine 66848 [running]:
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x8794801, 0x4a88e7, 0x7a77e6c, 0x7fef1e728788)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb9
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0xc00118e5c0, 0xc0020d3320, 0x24, 0x3, 0x0, 0x0, 0x0, 0x16a4fb1c79d0e563, 0x400000000, 0x0, ...)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:274 +0xbd2
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(0x5aab160, 0xc00310b200, 0x1, 0x4, 0x4ceff6f, 0x28, 0xc002438030, 0x1, 0x1)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/channels.go:58 +0x198
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_channels_generated.go:834
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).assertStateRaftMuLockedReplicaMuRLocked(0xc001d34000, 0x5aab160, 0xc00310b200, 0x5b173a0, 0xc0011c9ee0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica.go:1230 +0x7c8
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot(0xc001d34000, 0x5aab160, 0xc00310b200, 0xe4171aab40bbda8, 0xe4e306b9c0ea2296, 0xc002640500, 0x0, 0x0, 0x0, 0xc001babb00, ...)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raftstorage.go:1066 +0x11aa
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(0xc001d34000, 0x5aab160, 0xc00310b200, 0xe4171aab40bbda8, 0xe4e306b9c0ea2296, 0xc002640500, 0x0, 0x0, 0x0, 0xc001babb00, ...)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_raft.go:573 +0x1df8
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1(0x5aab160, 0xc000d11230, 0xc001d34000, 0x0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:358 +0x329
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest(0xc0009aea00, 0x5aab160, 0xc000d11230, 0xc001c9c298, 0xc002439388, 0x0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:219 +0x128
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest(0xc0009aea00, 0x5aab160, 0xc000d11230, 0xc001c9c240, 0xe4171aab40bbda8, 0xe4e306b9c0ea2296, 0xc002640500, 0x0, 0x0, 0x0, ...)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:280 +0x15e
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc0009aea00, 0x5aab160, 0xc000d11230, 0xc001c9c240, 0x7fef1b9c29c8, 0xc003bd20f0, 0x0, 0x0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_snapshot.go:810 +0x57d
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1(0x5aab160, 0xc000d11230, 0x1cfc94ee63, 0xc04893ca30781358)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:83 +0x18d
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000dc2780, 0x5aab160, 0xc000d11230, 0x4cb7016, 0x1e, 0xc003243c38, 0x0, 0x0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:328 +0xb2
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc0009aea00, 0xc001c9c240, 0x7fef1b9c2998, 0xc003bd20f0, 0xc003bd20f0, 0x474f6c0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:73 +0xe5
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1.1(0x5b0fa20, 0xc003bd20f0, 0xc000b941c0, 0x5aab160, 0xc000d11170, 0xc00164324d, 0x5aab0e0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:412 +0x13d
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot.func1(0x5aab160, 0xc000d11170)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/raft_transport.go:413 +0x5d
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2(0xc000dc2780, 0x5aab160, 0xc000d11170, 0x0, 0x0, 0xc000d111d0)
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:446 +0xf3
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
F210915 11:24:24.736351 66848 kv/kvserver/replica.go:1230 ⋮ [n3,s3,r45/5:‹/Table/6{2-3}›] 86 !	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:438 +0x22b

@tbg tbg added GA-blocker and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 15, 2021
@tbg
Copy link
Member

tbg commented Sep 15, 2021

Labeling as GA-blocker, but this is just being conservative. It looks very likely that the fact that the mixed version tests are marrying incompatible versions at the moment is at fault here. This will rectify once the first beta is out.

@adityamaru adityamaru mentioned this issue Sep 15, 2021
Closed
@irfansharif irfansharif self-assigned this Sep 15, 2021
@irfansharif
Copy link
Contributor

I'll take a look today if I get time during L2, thanks for catching it.

@ajwerner ajwerner mentioned this issue Sep 17, 2021
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ 78c6771c7e9f7ba6431f44b067f27e0857341374:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/version/mixed/nodes=5/run_1
	monitor.go:128,version.go:211,version.go:224,test_runner.go:777: monitor failure: unexpected node event: 2: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:211
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:224
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 2: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1253,context.go:89,cluster.go:1241,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3462983-1631945971-37-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 18373
		2: dead (exit status 7)
		3: 19211
		6: skipped
		5: 17622
		1: 21440
		Error: UNCLASSIFIED_PROBLEM: 2: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 2: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ d49fadb6f1c67d99ce91b719bac44b5640fa8e01:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/version/mixed/nodes=5/run_1
	monitor.go:128,version.go:211,version.go:224,test_runner.go:777: monitor failure: unexpected node event: 1: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:211
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:224
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 1: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1253,context.go:89,cluster.go:1241,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3464900-1632032301-39-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: dead (exit status 7)
		2: 24211
		4: 21616
		3: 24940
		5: 21533
		6: skipped
		Error: UNCLASSIFIED_PROBLEM: 1: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 1: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

@nvanbenschoten nvanbenschoten mentioned this issue Sep 20, 2021
Closed
@tbg tbg changed the title roachtest: version/mixed/nodes=5 failed roachtest: version/mixed/nodes=5 failed [should stop after beta1] Sep 20, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.version/mixed/nodes=5 failed with artifacts on master @ d9278939d4be4d1109e9ed0a84c6e9cd2a8705bb:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/version/mixed/nodes=5/run_1
	monitor.go:128,version.go:211,version.go:224,test_runner.go:777: monitor failure: unexpected node event: 5: dead (exit status 7)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:211
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerVersion.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/version.go:224
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (2) monitor failure
		Wraps: (3) unexpected node event: 5: dead (exit status 7)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString

	cluster.go:1253,context.go:89,cluster.go:1241,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3466946-1632119031-38-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 5: dead (exit status 7)
		3: 21501
		2: 20695
		4: 21692
		6: skipped
		1: 21640
		Error: UNCLASSIFIED_PROBLEM: 5: dead (exit status 7)
		(1) UNCLASSIFIED_PROBLEM
		Wraps: (2) attached stack trace
		  -- stack trace:
		  | main.glob..func14
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
		  | main.wrap.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
		  | github.com/spf13/cobra.(*Command).execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
		  | github.com/spf13/cobra.(*Command).ExecuteC
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
		  | github.com/spf13/cobra.(*Command).Execute
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
		  | main.main
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:225
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (3) 5: dead (exit status 7)
		Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: roachtest README

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

irfansharif added a commit to irfansharif/cockroach that referenced this issue Sep 20, 2021
@irfansharif
Revert "kv,migration: rm code handling legacy raft truncated state"
22cf7b8 
This reverts commit 6464de2. That PR
broke few of our roachtests since we haven't release the 21.2 beta yet.
For our roachtests that exercised the upgrade path, we were effectively
upgrading from 21.1 to 22.1 code (as of that PR) that asserted on the
completion of the long running migration removing the legacy raft
truncated state -- something that would only happen when going through
21.2. Given that, we temporarily revert cockroachdb#69887 while our beta gets
prepared. cockroachdb#69887 (or rather, the revert of this commit) will be
re-introduced to master once cockroachdb#69826 lands.

Fixes cockroachdb#70244.
Fixes cockroachdb#70252.
Fixes cockroachdb#70253.
Fixes cockroachdb#70283.
Fixes cockroachdb#70350.
Fixes cockroachdb#70390.

Release note: None
@irfansharif irfansharif mentioned this issue Sep 20, 2021
Merged
craig bot pushed a commit that referenced this issue Sep 20, 2021
@rimadeodhar @otan
@maryliag @irfansharif
Merge #70325 #70347 #70374 #70432 #70436
bfdcfa0 
70325: vendor: Add dependency on prometheus r=dhartunian a=rimadeodhar

This PR adds an external dependency on prometheus. We need
the promql library in order to enforce validity of promql
expressions which will be contained in upcoming alerting
and aggregation rules. These rule implementations are
upcoming as a part of the new metrics upgrade.

Resolves #69796

Release note: None

70347: pgcode: use XC instead of CDB r=knz a=otan

Release note (sql change): Change the pgerror code XC instead of CD
for CockroachDB specific errors. This is because the "C" class is
reserved for the SQL standard. The pgcode `CDB00` used for
unsatisfiable bounded staleness is now `XCUBS`.

70374: ui: updates the jobs table styling r=maryliag a=maryliag

This commit updates the style of the table on the Jobs page
and adds tooltips to its columns.

Resolves #70149

Before
<img width="924" alt="Screen Shot 2021-09-17 at 3 35 04 PM" src="https://user-images.githubusercontent.com/1017486/133844066-3168bec7-db52-4194-9f97-c7b10628d98e.png">

After
<img width="880" alt="Screen Shot 2021-09-17 at 3 35 52 PM" src="https://user-images.githubusercontent.com/1017486/133844146-86e94611-ca99-4764-a97e-c8ca4e09f269.png">


Release note (ui change): Updating job table style to
match all other tables on the console.

70432: Revert "kv,migration: rm code handling legacy raft truncated state" r=irfansharif a=irfansharif

This reverts commit 6464de2. That PR
broke few of our roachtests since we haven't release the 21.2 beta yet.
For our roachtests that exercised the upgrade path, we were effectively
upgrading from 21.1 to 22.1 code (as of that PR) that asserted on the
completion of the long running migration removing the legacy raft
truncated state -- something that would only happen when going through
21.2. Given that, we temporarily revert #69887 while our beta gets
prepared. #69887 (or rather, the revert of _this_ commit) will be
re-introduced to master once #69826 lands.

Fixes #70244.
Fixes #70252.
Fixes #70253.
Fixes #70283.
Fixes #70350.
Fixes #70390.

Release note: None

70436: spanconfig: fix an erroneous usage of timeutil.Timer r=irfansharif a=irfansharif

The contract for timeutil.Timer indicates that we should only be
setting .Read when reading from the timer channel, not unconditionally
before a call to .Reset().

Release note: None

Co-authored-by: rimadeodhar <[email protected]>
Co-authored-by: Oliver Tan <[email protected]>
Co-authored-by: Marylia Gutierrez <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
@craig craig bot closed this as completed in ef1dd6f Sep 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@irfansharif irfansharif

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). GA-blocker O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert "kv,migration: rm code handling legacy raft truncated state" irfansharif/cockroach
3 participants
@tbg @irfansharif @cockroach-teamcity