Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: restore/nodeShutdown/worker failed #88469

Closed
cockroach-teamcity opened this issue Sep 22, 2022 · 5 comments
Closed

roachtest: restore/nodeShutdown/worker failed #88469

cockroach-teamcity opened this issue Sep 22, 2022 · 5 comments
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Milestone
22.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Sep 22, 2022 •
edited by cockroach-jira-scripts
Loading

roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.2 @ a33d71dcd904c771d1297323d8d206b8b59d40bf:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:908: monitor failure: monitor command failure: unexpected node event: 4: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 4: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1039,test_runner.go:938: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-19941
@cockroach-teamcity cockroach-teamcity added branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Sep 22, 2022
@cockroach-teamcity cockroach-teamcity added this to the 22.2 milestone Sep 22, 2022
@adityamaru
Copy link
Contributor

Hmm its another stuck AdminSplit:

goroutine 8333 [select, 598 minutes]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc005d9e480)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:324 +0x7c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00d1f2bb0, {0x4f9b800?, 0xc0073fe080}, 0x0?)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:969 +0xbb
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x64?)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:834 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc005d9e240, 0xc000d112b0, 0xc000d11280)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:692 +0xd3
google.golang.org/grpc.(*clientStream).RecvMsg(0xc005d9e240, {0x4f9b800?, 0xc0073fe080?})
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:833 +0x11f
google.golang.org/grpc.invoke({0x5fa0c08?, 0xc0079010b0?}, {0x519b1a3?, 0x2?}, {0x50c85a0, 0xc009e29590}, {0x4f9b800, 0xc0073fe080}, 0x0?, {0xc005c3f060, ...})
	google.golang.org/grpc/external/org_golang_google_grpc/call.go:73 +0xd7
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2({0x5fa0c08, 0xc0079010b0}, {0x519b1a3, 0x21}, {0x50c85a0, 0xc009e29590}, {0x4f9b800, 0xc0073fe080}, 0x5944e9ae0221aee9?, 0x542f320, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:247 +0x38a
google.golang.org/grpc.(*ClientConn).Invoke(0x5ec9bede55232c09?, {0x5fa0c08?, 0xc0079010b0?}, {0x519b1a3?, 0x0?}, {0x50c85a0?, 0xc009e29590?}, {0x4f9b800?, 0xc0073fe080?}, {0x0, ...})
	google.golang.org/grpc/external/org_golang_google_grpc/call.go:35 +0x223
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc00830a550, {0x5fa0c08, 0xc0079010b0}, 0x0?, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9295 +0xc9
github.com/cockroachdb/cockroach/pkg/rpc/nodedialer.TracingInternalClient.Batch({{0x5ff31d0?, 0xc00830a550?}}, {0x5fa0c08, 0xc0079010b0}, 0xc009e29590, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/nodedialer/nodedialer.go:328 +0x171
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc006a4b740, {_, _}, _, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc006a4b740, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717aca14bfac5, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:190 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0013a2500, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717aca14bfac5, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2069 +0x110f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717aca14bfac5, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1613 +0x805
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717aca14bfac5, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1216 +0x433
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0013a2500, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717aca14bfac5, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:840 +0x629
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc000f35ad8, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xda
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000f35a70, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1000 +0x156
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(0x4ee865?, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:983 +0x85
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x5fa0b60, 0xc006bde980}, 0x17171af2d203c66b?, 0xc0042a6680)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:844 +0x119
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc000f35a70, {0x5fa0b60, 0xc006bde980}, 0xc000100400?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:867 +0x74
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminSplit(0x5fa0b60?, {0x5fa0b60, 0xc006bde980}, {0x4ea86c0, 0xc0063e0648}, {0x5165d21?, 0x18?, 0x0?}, {0x0, 0x0, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:597 +0xb2
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.dbSplitAndScatterer.split({0xc000f35a70?, 0xc002363200?}, {0x5fa0b60, 0xc006bde980}, {{0xc0009b5f20?}, {0xc0009b5f20?}, 0x0?}, {0xc005e94d48, 0x6, 0x8})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:99 +0x2ac
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.runSplitAndScatter.func1({0x5fa0b60, 0xc006bde980})
	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/split_and_scatter_processor.go:368 +0x264
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1()
	github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x25
golang.org/x/sync/errgroup.(*Group).Go.func1()
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:74 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:71 +0xa5

The stack on node 4:

goroutine 5234 [select]:
github.com/cockroachdb/cockroach/pkg/util/retry.(*Retry).Next(0xc0016c1e18)
	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:127 +0x13e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminTransferLease(0xc00625a000, {0x5fa0c08, 0xc007dd49f0}, 0x3)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_range_lease.go:951 +0x26a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch(0xc00625a000, {0x5fa0c08, 0xc007dd4990}, 0xc00c6751d0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:936 +0x79d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc00625a000, {_, _}, {{{0x1717385c0868380f, 0x0, 0x0}, 0xc00fa49b40, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x845
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc000f8aa80, {_, _}, {{{0x1717385c0868380f, 0x0, 0x0}, 0xc00fa49b40, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:204 +0x929
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc000d26300?, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:227 +0x1f3
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00041e000, {0x5fa0c08, 0xc007dd4930}, {0x2a61746164617465?}, 0xc00c6750e0)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1090 +0x578
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00041e000, {0x5fa0c08, 0xc007dd48a0}, 0xc00c6750e0)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1139 +0xfd
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x5fa0c08?, 0xc007dd48a0?}, {0x50c85a0?, 0xc00c6750e0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:724 +0x4b
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x5fa0c08, 0xc007dd48a0}, {0x50c85a0, 0xc00c6750e0}, 0xc00054c7a0, 0xc00052d998)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:95 +0x254
github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1({0x5fa0c08?, 0xc007dd48a0?}, {0x50c85a0?, 0xc00c6750e0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795 +0x3a
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x5fa0c08, 0xc007dd48a0}, {0x50c85a0, 0xc00c6750e0}, 0xc0016c4178?, 0xc00054c840)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:263 +0x83
github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1({0x5fa0c08?, 0xc007dd48a0?}, {0x50c85a0?, 0xc00c6750e0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795 +0x3a
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x5fa0c08?, 0xc007dd48a0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:232 +0x39
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0012105a0, {0x5fa0c08, 0xc007dd48a0}, {0x7f37639c5580?, 0x0?}, 0xc0016c4240)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:341 +0xd1
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x5fa0c08?, 0xc007dd48a0?}, {0x50c85a0?, 0xc00c6750e0?}, 0xc0016c42c0?, 0xc36975?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:230 +0x95
github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1({0x5fa0c08?, 0xc007dd48a0?}, {0x50c85a0?, 0xc00c6750e0?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:795 +0x3a
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x5fa0c08?, 0xc007dd48a0?}, {0xc007dd48a0?, 0x4?}, {0x50c85a0?, 0xc00c6750e0?}, {0x4f9b800?, 0xc00e7feb80}, 0x203001?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:734 +0x54
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2({0x5fa0c08, 0xc007dd48a0}, {0x519b1a3, 0x21}, {0x50c85a0, 0xc00c6750e0}, {0x4f9b800, 0xc00e7feb80}, 0xc0016c4470?, 0xc000cb9790, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:225 +0x58d
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0x5fa0c08, 0xc007dd48a0}, {0x519b1a3, 0x21}, {0x50c85a0, 0xc00c6750e0}, {0x4f9b800, 0xc00e7feb80}, 0x4f04aa?, {0x0, ...})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:879 +0x13e
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0x5fa0c08, 0xc007dd4840}, 0xc00c6750e0, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:764 +0xf8
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:887
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00ef4e780, {_, _}, _, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00ef4e780, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:190 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc00134a500, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2069 +0x110f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1613 +0x805
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(_, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1216 +0x433
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc00134a500, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x1717385c08675c80, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:840 +0x629
github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send(0xc001211928, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:223 +0xda
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc0012118c0, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:1000 +0x156
github.com/cockroachdb/cockroach/pkg/kv.(*DB).send(0xc0016c68f8?, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/db.go:983 +0x85
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill({0x5fa0c08, 0xc004ef14a0}, 0x8?, 0xc0060d9700)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:844 +0x119
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run(0xc0012118c0, {0x5fa0c08, 0xc004ef14a0}, 0x3?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:867 +0x74
github.com/cockroachdb/cockroach/pkg/kv.(*DB).AdminTransferLease(0xc005fc4f80?, {0x5fa0c08, 0xc004ef14a0}, {0x4e1da80, 0xc006ad7140}, 0x16c6b00?)
	github.com/cockroachdb/cockroach/pkg/kv/db.go:653 +0x76
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeTransferLeaseDuringLeaveJoint(0xc00625a000, {0x5fa0c08, 0xc004ef14a0}, 0xc0078851f0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:1227 +0x228
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).maybeLeaveAtomicChangeReplicas(0xc00625a000, {0x5fa0c08, 0xc004ef14a0}, 0xc0078851f0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:1258 +0xec
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).adminSplitWithDescriptor(0xc00625a000, {0x5fa0c08, 0xc004ef14a0}, {{{0xc004c7d2b0, 0x6, 0x8}, {0x0, 0x0, 0x0}, 0x0}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:315 +0x94
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminSplit.func1(0x171717ac56acde66?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:80 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminCommandWithDescriptor(0x4?, {0x5fa0c08, 0xc004ef14a0}, 0xc0016c7590)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:548 +0x1ec
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).AdminSplit(0x0?, {0x5fa0c08?, 0xc004ef14a0?}, {{{0xc004c7d2b0, 0x6, 0x8}, {0x0, 0x0, 0x0}, 0x0}, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:78 +0x15a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch(0xc00625a000, {0x5fa0c08, 0xc004ef1440}, 0xc00185af00)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:922 +0x568
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc00625a000, {_, _}, {{{0x171717ad3b8868f6, 0x0, 0x0}, 0xc004c7d2c0, {0x171717ad3b84dfc8, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x845
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc000f8aa80, {_, _}, {{{0x171717ad3b8868f6, 0x0, 0x0}, 0xc004c7d2c0, {0x171717ad3b84dfc8, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:204 +0x929
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc000d26300?, {_, _}, {{{0x0, 0x0, 0x0}, 0x0, {0x171717ad3b84dfc8, 0x0, 0x0}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:227 +0x1f3
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00041e000, {0x5fa0c08, 0xc004ef13e0}, {0xc0058aeff0?}, 0xc00185ae10)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1090 +0x578
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00041e000, {0x5fa0c08, 0xc004ef1320}, 0xc00185ae10)
	github.com/cockroachdb/cockroach/pkg/server/node.go:1139 +0xfd
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1({0x5fa0c08, 0xc004ef1320}, {0x50c85a0?, 0xc00185ae10})
	github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9580 +0x78
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x5fa0c08, 0xc004ef1320}, {0x50c85a0, 0xc00185ae10}, 0xc001322540, 0xc005258ea0)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:95 +0x254
google.golang.org/grpc.chainUnaryInterceptors.func1.1({0x5fa0c08?, 0xc004ef1320?}, {0x50c85a0?, 0xc00185ae10?})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1117 +0x5b
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x5fa0c08, 0xc004ef1320}, {0x50c85a0, 0xc00185ae10}, 0xc004a39920?, 0xc007084fc0)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:263 +0x83
google.golang.org/grpc.chainUnaryInterceptors.func1.1({0x5fa0c08?, 0xc004ef1320?}, {0x50c85a0?, 0xc00185ae10?})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1120 +0x83
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x5fa0c08?, 0xc004ef1320?})
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:232 +0x39
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0012105a0, {0x5fa0c08, 0xc004ef1320}, {0x7f3792cc6a68?, 0x40?}, 0xc004a399e8)
	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:341 +0xd1
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x5fa0c08?, 0xc004ef1320?}, {0x50c85a0?, 0xc00185ae10?}, 0x4def700?, 0x7f3792cc6a01?)
	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:230 +0x95
google.golang.org/grpc.chainUnaryInterceptors.func1.1({0x5fa0c08?, 0xc004ef1320?}, {0x50c85a0?, 0xc00185ae10?})
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1120 +0x83
google.golang.org/grpc.chainUnaryInterceptors.func1({0x5fa0c08, 0xc004ef1320}, {0x50c85a0, 0xc00185ae10}, 0xc001322540, 0xc005258ea0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1122 +0x12b
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler({0x508f020?, 0xc00041e000}, {0x5fa0c08, 0xc004ef1320}, 0xc0041b8cc0, 0xc000145080)
	github.com/cockroachdb/cockroach/pkg/roachpb/bazel-out/k8-opt/bin/pkg/roachpb/roachpb_go_proto_/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9582 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0009ffa40, {0x5feab80, 0xc0018984e0}, 0xc006c7bb00, 0xc0008c4480, 0x8c2c580, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1283 +0xcfe
google.golang.org/grpc.(*Server).handleStream(0xc0009ffa40, {0x5feab80, 0xc0018984e0}, 0xc006c7bb00, 0x0)
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1620 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:922 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc/external/org_golang_google_grpc/server.go:920 +0x28a

Is similar to #87837.

@blathers-crl blathers-crl bot added the T-kv KV Team label Sep 23, 2022
@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 24, 2022
Closed
@nvanbenschoten
Copy link
Member 

E220922 06:30:34.493096 77157 kv/kvserver/queue.go:1119 ⋮ [n1,merge,s1,r95/3:‹/Table/106/1/{3693933-4272459}›] 1502  refusing to transfer lease to (n3,s3):4VOTER_INCOMING because target may need a Raft snapshot: ‹replica in StateProbe›

This is the same as #88667.

@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 27, 2022
Closed
@cockroach-teamcity
Copy link
Member Author

roachtest.restore/nodeShutdown/worker failed with artifacts on release-22.2 @ 860584a59dee73d7a66ce882c668cef6eb2556f7:

test artifacts and logs in: /artifacts/restore/nodeShutdown/worker/run_1
	monitor.go:127,jobs.go:154,restore.go:304,test_runner.go:908: monitor failure: monitor command failure: unexpected node event: 2: dead (exit status 137)
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	main/pkg/cmd/roachtest/monitor.go:115
		  | main.(*monitorImpl).Wait
		  | 	main/pkg/cmd/roachtest/monitor.go:123
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.jobSurvivesNodeShutdown
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/jobs.go:154
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerRestoreNodeShutdown.func2
		  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/restore.go:304
		  | [...repeated from below...]
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func3
		  | 	main/pkg/cmd/roachtest/monitor.go:202
		  | runtime.goexit
		  | 	GOROOT/src/runtime/asm_amd64.s:1594
		Wraps: (4) monitor command failure
		Wraps: (5) unexpected node event: 2: dead (exit status 137)
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString

	test_runner.go:1039,test_runner.go:938: test timed out (0s)

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@stevendanna
Copy link
Collaborator

I'm removing release-blocker here. We can track this with #88667.

@stevendanna stevendanna removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Oct 3, 2022
@exalate-issue-sync exalate-issue-sync bot added blocks-22.2.0-beta.2 release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team and removed T-disaster-recovery T-kv KV Team labels Oct 4, 2022
@nvanbenschoten
Copy link
Member

Fixed by #89621 and #89621.

@lunevalex lunevalex added the X-duplicate Closed as a duplicate of another issue. label Dec 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-22.2 Used to mark GA and release blockers, technical advisories, and bugs for 22.2 C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Projects
No open projects
Disaster Recovery Backlog
Archived in project
Milestone
22.2
Development

No branches or pull requests
6 participants
@stevendanna @celiala @nvanbenschoten @adityamaru @cockroach-teamcity @lunevalex