Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: restart/down-for-2m failed #44060

Closed
cockroach-teamcity opened this issue Jan 16, 2020 · 17 comments
Closed

roachtest: restart/down-for-2m failed #44060

cockroach-teamcity opened this issue Jan 16, 2020 · 17 comments
Assignees
@knz
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
20.1

Comments

@cockroach-teamcity
Copy link
Member

(roachtest).restart/down-for-2m failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200116-1693614/restart/down-for-2m/run_1
	cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1693614-1579160767-17-n3cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: dead
		3: 7441
		2: 6619
		Error:  1: dead
Repro

Artifacts: /restart/down-for-2m

make stressrace TESTS=restart/down-for-2m PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

powered by pkg/cmd/internal/issues
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Jan 16, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.1 milestone Jan 16, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on release-19.1@a8444e28d3e7bf07c8ee0c6d5809e7466b57f6f9:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200118-1698230/restart/down-for-2m/run_1
	cluster.go:1821,restart.go:33,restart.go:92,test_runner.go:716: error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:399
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1855
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1819
		    main.runRestart
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/restart.go:33
		    main.registerRestart.func1
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/restart.go:92
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:716
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1698230-1579331271-17-n3cpu4:1 -- ./cockroach workload fixtures import tpcc --warehouses=100 --fks=false --checks=false returned:
		    stderr:
		    
		    stdout:
		    I200118 10:58:27.311522 1 ccl/workloadccl/cliccl/fixtures.go:324  starting import of 9 tables
		    I200118 10:58:52.797209 15 ccl/workloadccl/fixture.go:516  imported district (25s, 1000 rows, 0 index entries, 99 KiB)
		    I200118 10:58:53.757239 14 ccl/workloadccl/fixture.go:516  imported warehouse (26s, 100 rows, 0 index entries, 5.1 KiB)
		    I200118 10:58:55.015078 69 ccl/workloadccl/fixture.go:516  imported item (28s, 100000 rows, 0 index entries, 7.8 MiB)
		    I200118 10:59:02.825894 68 ccl/workloadccl/fixture.go:516  imported new_order (36s, 900000 rows, 0 index entries, 11 MiB)
		    I200118 11:01:58.124374 66 ccl/workloadccl/fixture.go:516  imported history (3m31s, 3000000 rows, 6000000 index entries, 380 MiB)
		    I200118 11:02:45.053210 67 ccl/workloadccl/fixture.go:516  imported order (4m18s, 3000000 rows, 6000000 index entries, 166 MiB)
		    I200118 11:02:45.498747 16 ccl/workloadccl/fixture.go:516  imported customer (4m18s, 3000000 rows, 3000000 index entries, 1.7 GiB)
		    I200118 11:05:12.218435 70 ccl/workloadccl/fixture.go:516  imported stock (6m45s, 10000000 rows, 10000000 index entries, 3.1 GiB)
		    I200118 11:06:43.113244 71 ccl/workloadccl/fixture.go:516  imported order_line (8m16s, 30005985 rows, 30005985 index entries, 2.1 GiB)
		    Error: importing fixture: importing table new_order: pq: failed to insert stats: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" id=ba736317 key=/Table/SystemConfigSpan/Start rw=true pri=0.00111265 stat=ABORTED epo=0 ts=1579345142.861409681,0 orig=1579345142.861409681,0 max=1579345143.361409681,0 wto=false seq=2
		    Error:  exit status 1:
		  - exit status 1
Repro

Artifacts: /restart/down-for-2m
roachdash

powered by pkg/cmd/internal/issues

@tbg tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@e671a4ef97cbc6cf5d22f8f322fd45733d302094:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/restart/down-for-2m/run_1
	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1749385-1581667748-18-n3cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: dead
		2: 7015
		3: 7675
		Error:  1: dead
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@tbg tbg assigned andreimatei and unassigned tbg Feb 18, 2020
@tbg
Copy link
Member

tbg commented Feb 18, 2020 

F200214 08:57:57.676196 366646 kv/txn_interceptor_span_refresher.go:231  [n1,txn=5faaeb50] unexpected WriteTooOld request. ba: [txn: 5faaeb50], EndTxn(commit:false tsflex:false) [/Min]  (txn: "sql txn" meta={id=5faaeb50 key=/Table/58/1/84/1/-2109/0 pri=0.01347382 epo=0 ts=1581670676.146405442,2 min=1581670676.146405442,0 seq=19} rw=true stat=PENDING rts=1581670676.146405442,0 wto=true max=1581670676.646405442,0)
goroutine 366646 [running]:

cc @andreimatei

@andreimatei
Copy link
Contributor

This last crash should be fixed by #45102

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@954fe69d554162aec0fbc001aad1fe5103d8df13:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200306-1790595/restart/down-for-2m/run_1
	test_runner.go:756: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@752dea867f3aeb142e98c22f8d320ce19041aa8d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200307-1793930/restart/down-for-2m/run_1
	test_runner.go:756: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@dfa5bd527ae7d7373dd03c62118df87a87a77130:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200308-1795062/restart/down-for-2m/run_1
	test_runner.go:756: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/restart/down-for-2m/run_1
	test_runner.go:756: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@72c4a1bd411f2f82bf9aaa22883821a946614148:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200310-1799071/restart/down-for-2m/run_1
	test_runner.go:779: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@69dc87d68addedf2fabfb2b14c098cfb35b5f3d0:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200313-1806460/restart/down-for-2m/run_1
	test_runner.go:785: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@33d71472dc01cbc5064b3c5e1fcd666a33f606de:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200315-1808695/restart/down-for-2m/run_1
	test_runner.go:785: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@nvanbenschoten nvanbenschoten assigned knz and unassigned andreimatei Mar 17, 2020
@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@5a3d0c9539a671f0e55b680d3021b18dde9d190d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200317-1811809/restart/down-for-2m/run_1
	test_runner.go:785: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(roachtest).restart/down-for-2m failed on master@b5f030223fbcf22e806c48a3c46e74a73a54f50f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200318-1814553/restart/down-for-2m/run_1
	test_runner.go:800: test timed out (10h0m0s)
More

Artifacts: /restart/down-for-2m

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 19, 2020
Closed
@nvanbenschoten
Copy link
Member

This doesn't look like anything we're aware of. In the last two failures, we see node 3 stall while being started:

11:46:02 cluster.go:335: > /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1811809-1584430043-20-n3cpu4:3
teamcity-1811809-1584430043-20-n3cpu4: starting..
21:29:56 test_runner.go:773: [w0] dumped stacks to __stacks
21:29:56 test.go:325: test failure: 	test_runner.go:785: test timed out (10h0m0s)

I don't see anything in the logs that immediately makes it clear what's going wrong. We'll need to take a closer look into this.

@tbg
Copy link
Member

tbg commented Mar 24, 2020

The test is stuck here:

cockroach/pkg/cmd/roachtest/restart.go

Lines 75 to 77 in 8e4d7b2

if _, err := restartNodeDB.Exec(`SELECT count(*) FROM tpcc.order_line`); err != nil {
t.Fatal(err)
}

#45887 was supposed to give us the server side of the hang, but it doesn't seem to have done that (the roachprod logs confirm that we did send SIGSEGV).

@tbg
Copy link
Member

tbg commented Mar 24, 2020

From debug zip

queries:

15fd61346c76560c0000000000000003	3	15fd61346a6132250000000000000003	root	2020-03-18 11:05:21.689483+00:00	SELECT count(*) FROM tpcc.order_line	35.231.52.48:53572		true	executing

def something stuck on the KV side:

goroutine 379395 [select, 589 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc01587b200, 0xc00da84a20, 0x1)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc

[...]

github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb4, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1630 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendRPC(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb4, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendSingleRange(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1401 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1230 +0x8d6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc0000f0c60, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:737 +0x785
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc019edcd10, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc019edcc38, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:225 +0x9b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc019edcc38, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:183 +0x180
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc019edcb88, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:224 +0x159
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc019edcb68, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc019edc900, 0x4b7f940, 0xc0229169c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:504 +0x406
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000506580, 0x4b7f880, 0xc007320700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x122
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc022c98b40, 0x4b7f880, 0xc007320700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:894 +0x11e
github.com/cockroachdb/cockroach/pkg/sql/row.makeKVBatchFetcher.func1(0x4b7f880, 0xc007320700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:184 +0x76
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc00b00ea90, 0x4b7f880, 0xc007320700, 0xc0185fcc68, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:315 +0x6b1
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc00b00ea90, 0x4b7f880, 0xc007320700, 0xc031e43801, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:399 +0x1af
github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV(0xc022379700, 0x4b7f880, 0xc007320700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:87 +0x114
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextBatch(0xc00046b900, 0x4b7f880, 0xc007320700, 0xc007320a00, 0x11044, 0x20800, 0x9)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:600 +0x8e
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextAdapter(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:582
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Allocator).PerformOperation(0xc019c55880, 0xc026bbcc80, 0xa, 0xa, 0xc000affa30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/allocator.go:191 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).NextBatch(0xc00046b900, 0x4b7f880, 0xc007320700, 0x22ad102, 0x0, 0xc016856cc0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:574 +0xa2
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*colBatchScan).Next(0xc0165c73b0, 0x4b7f880, 0xc007320700, 0x6e08da0, 0xc00b00eb60)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colbatch_scan.go:68 +0x47
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*CancelChecker).Next(0xc019a23860, 0x4b7f880, 0xc007320700, 0xc00b00ebe8, 0xc007320701)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cancel_checker.go:47 +0x69
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*simpleProjectOp).Next(0xc0165c7400, 0x4b7f880, 0xc007320700, 0x4bebf80, 0xc007320e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/simple_project.go:110 +0x62
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*countOp).Next(0xc019a238f0, 0x4b7f880, 0xc007320700, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/count.go:55 +0x79
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*deselectorOp).Next(0xc0165c7540, 0x4b7f880, 0xc007320700, 0x79958a, 0x4a7a4d4)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/deselector.go:51 +0x73
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:220 +0x5f
github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror.CatchVectorizedRuntimeError(0xc029635da0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror/error.go:95 +0x5f
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches(0xc022c98cf0, 0x4b7f940, 0xc019a23950, 0x7fdb1da4b9c8, 0xc02a8ef630, 0xc02a8ef4d0, 0xc022c98cf0, 0xc016856f00, 0x7753c5)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:227 +0xbb
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).runWithStream(0xc022c98cf0, 0x4b7f940, 0xc019a23950, 0x7fdb1da4b9c8, 0xc02a8ef630, 0xc02a8ef4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:307 +0x104
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run(0xc022c98cf0, 0x4b7f880, 0xc007320700, 0x4af7600, 0xc0000ed630, 0x305eaa2400000003, 0x6596348cf541e8d4, 0x903ca8a, 0x4, 0xc02a8ef4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:167 +0x586
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1(0x4b7f880, 0xc007320700, 0xc02a8ef4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:544 +0x9a
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1(0xc019c55b00, 0x4b7f880, 0xc007320700, 0xc02a8ef4d0, 0xc026ce6790)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1090 +0x44
created by github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1089 +0x6f

There's nothing stuck in a (*Node).Batch-anchored stack trace. There are a few active ones (surprisingly maybe, though one of them is just a sampler). Confusingly there's this one which clearly belongs to the COUNT(*) somehow but isn't the server-side part of the one stuck above.

goroutine 380090 [syscall]:
github.com/cockroachdb/cockroach/pkg/storage._Cfunc_MVCCScan(0x7ff977871780, 0xc016d12bd0, 0x9, 0xc0076eda70, 0x8, 0x15fd61346c7591c0, 0x0, 0x2710, 0xa00000, 0xc0034ab9e0, ...)
	_cgo_gotypes.go:1566 +0x84
github.com/cockroachdb/cockroach/pkg/storage.(*rocksDBIterator).MVCCScan.func1(0xc0045f3248, 0xc016d12bd0, 0x9, 0x9, 0xc0076eda70, 0x8, 0x8, 0x15fd61346c7591c0, 0xc000000000, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:2493 +0x28f
github.com/cockroachdb/cockroach/pkg/storage.(*rocksDBIterator).MVCCScan(0xc0045f3248, 0xc016d12bd0, 0x9, 0x9, 0xc0076eda70, 0x8, 0x8, 0x15fd61346c7591c0, 0xc000000000, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:2493 +0x16e
github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes(0x4b7f940, 0xc009e43800, 0x4bff3a0, 0xc0045f3248, 0xc016d12bd0, 0x9, 0x9, 0xc0076eda70, 0x8, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2327 +0x921
github.com/cockroachdb/cockroach/pkg/storage.MVCCScanToBytes(0x4b7f940, 0xc009e43800, 0x4bc4ba0, 0xc0045f31d0, 0xc016d12bd0, 0x9, 0x9, 0xc0076eda70, 0x8, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2541 +0x222
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan(0x4b7f940, 0xc009e43800, 0x4bc4ba0, 0xc0045f31d0, 0x4c0d820, 0xc003223500, 0x15fd61346c7591c0, 0x0, 0x200000002, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_scan.go:53 +0x23d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x4b7f940, 0xc009e43800, 0x0, 0x0, 0x0, 0x4bfe4a0, 0xc0045f31d0, 0x4c0d820, 0xc003223500, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:450 +0x5fd
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x4b7f940, 0xc009e43800, 0x0, 0x0, 0x4bfe4a0, 0xc0045f31d0, 0x4c0d820, 0xc003223500, 0x0, 0xc0070d4c80, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:237 +0x3c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes(0xc003223500, 0x4b7f940, 0xc009e43800, 0x4bfe4a0, 0xc0045f31d0, 0x4c0d820, 0xc003223500, 0xc0070d4c80, 0xc0095af8c0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_read.go:112 +0xee
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch(0xc003223500, 0x4b7f940, 0xc009e43800, 0xc0070d4c80, 0xc005890070, 0x0, 0x0, 0xc0057a6f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_read.go:84 +0x407
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc003223500, 0x4b7f940, 0xc009e43800, 0xc0070d4c80, 0x4478760, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:215 +0x315
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc003223500, 0x4b7f940, 0xc009e437d0, 0x88, 0xc0070d4c80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc003223500, 0x4b7f940, 0xc009e437d0, 0x15fd61346c7591c0, 0x0, 0x200000002, 0x3, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc000988000, 0x4b7f940, 0xc009e43770, 0x15fd61346c7591c0, 0x0, 0x200000002, 0x3, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc000a2a3c0, 0x4b7f940, 0xc009e43770, 0x0, 0x0, 0x200000002, 0x3, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:188 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4b7f940, 0xc009e43770, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0005b6000, 0x4b7f940, 0xc009e43770, 0x42e026b, 0x10, 0xc0030c2780, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000752500, 0x4b7f940, 0xc009e43770, 0xc0070d4b80, 0xc009e43770, 0x80, 0x80)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000752500, 0x4b7f940, 0xc009e43740, 0xc0070d4b80, 0xc008974750, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:537
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc007312f80, 0x4b7f940, 0xc009e43740, 0x2, 0x4b3d100, 0xc000318570, 0x0, 0x0, 0x200000002, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc007312f80, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x200000002, 0x3, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1630 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendRPC(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x88, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendSingleRange(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1401 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1230 +0x8d6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send(0xc000bd0c60, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:737 +0x785
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked(0xc004ae8410, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc004ae8338, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:225 +0x9b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked(0xc004ae8338, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:183 +0x180
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked(0xc004ae8288, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:224 +0x159
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked(0xc004ae8268, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send(0xc004ae8000, 0x4b7f940, 0xc008974750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:504 +0x406
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender(0xc000548680, 0x4b7f880, 0xc007313e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/db.go:739 +0x122
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send(0xc0060ec750, 0x4b7f880, 0xc007313e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn.go:894 +0x11e
github.com/cockroachdb/cockroach/pkg/sql/row.makeKVBatchFetcher.func1(0x4b7f880, 0xc007313e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:184 +0x76
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc0037eb6c0, 0x4b7f880, 0xc007313e40, 0xc00bd9b9e8, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:315 +0x6b1
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc0037eb6c0, 0x4b7f880, 0xc007313e40, 0xc014f7a301, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:399 +0x1af
github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV(0xc00361b780, 0x4b7f880, 0xc007313e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:87 +0x114
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextBatch(0xc000b03900, 0x4b7f880, 0xc007313e40, 0xc004c02140, 0x11044, 0x20800, 0x9)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:600 +0x8e
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).nextAdapter(...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:582
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Allocator).PerformOperation(0xc005fa2840, 0xc00ed61b80, 0xa, 0xa, 0xc0030c5a30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/allocator.go:191 +0x57
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*cFetcher).NextBatch(0xc000b03900, 0x4b7f880, 0xc007313e40, 0x22ad102, 0x0, 0xc004ce5260, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cfetcher.go:574 +0xa2
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*colBatchScan).Next(0xc0083becd0, 0x4b7f880, 0xc007313e40, 0x6e08da0, 0xc0037eb860)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/colbatch_scan.go:68 +0x47
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*CancelChecker).Next(0xc0098cd6e0, 0x4b7f880, 0xc007313e40, 0xc0037eb8e8, 0xc007313e01)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/cancel_checker.go:47 +0x69
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*simpleProjectOp).Next(0xc0083bed20, 0x4b7f880, 0xc007313e40, 0x4bebf80, 0xc030879100)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/simple_project.go:110 +0x62
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*countOp).Next(0xc0098cd770, 0x4b7f880, 0xc007313e40, 0x0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/count.go:55 +0x79
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*deselectorOp).Next(0xc0083bee60, 0x4b7f880, 0xc007313e40, 0x79958a, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/deselector.go:51 +0x73
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches.func1()
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:220 +0x5f
github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror.CatchVectorizedRuntimeError(0xc005ab3da0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colexec/execerror/error.go:95 +0x5f
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).sendBatches(0xc0060ec870, 0x4b7f940, 0xc003cbe510, 0x7ff970fa5dc0, 0xc00e8c8ea0, 0xc01199d7d0, 0xc0060ec870, 0xc0088a43c0, 0x7753c5)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:227 +0xbb
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).runWithStream(0xc0060ec870, 0x4b7f940, 0xc003cbe510, 0x7ff970fa5dc0, 0xc00e8c8ea0, 0xc01199d7d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:307 +0x104
github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc.(*Outbox).Run(0xc0060ec870, 0x4b7f880, 0xc007313e40, 0x4af7600, 0xc0002be870, 0x305eaa2400000003, 0x6596348cf541e8d4, 0x903ca8a, 0x5, 0xc01199d7d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc/outbox.go:167 +0x586
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreator).setupRemoteOutputStream.func1(0x4b7f880, 0xc007313e40, 0xc01199d7d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:544 +0x9a
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1.1(0xc005fa2ac0, 0x4b7f880, 0xc007313e40, 0xc01199d7d0, 0xc004715f30)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1090 +0x44
created by github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlowCreatorHelper).accumulateAsyncComponent.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:1089 +0x6f

We're probably caught in some infinite retry loop. The only other goroutine anchored in (*Node).Batch (and thus the counterpart of the stuck goroutine at the top) must be

goroutine 1586 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage.(*rocksDBIterator).MVCCScan(0xc0034ad068, 0xc01273e028, 0x8, 0x8, 0xc01273e050, 0x8, 0x8, 0x15fd61346c7591c0, 0xc000000000, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:2505 +0x2a0
github.com/cockroachdb/cockroach/pkg/storage.mvccScanToBytes(0x4b7f940, 0xc0043f6540, 0x4bff3a0, 0xc0034ad068, 0xc01273e028, 0x8, 0x8, 0xc01273e050, 0x8, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2327 +0x921
github.com/cockroachdb/cockroach/pkg/storage.MVCCScanToBytes(0x4b7f940, 0xc0043f6540, 0x4bc4ba0, 0xc0034acff0, 0xc01273e028, 0x8, 0x8, 0xc01273e050, 0x8, 0x8, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:2541 +0x222
github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval.Scan(0x4b7f940, 0xc0043f6540, 0x4bc4ba0, 0xc0034acff0, 0x4c0d820, 0xc00060f800, 0x15fd61346c7591c0, 0x0, 0x300000003, 0x2, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/cmd_scan.go:53 +0x23d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateCommand(0x4b7f940, 0xc0043f6540, 0x0, 0x0, 0x0, 0x4bfe4a0, 0xc0034acff0, 0x4c0d820, 0xc00060f800, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:450 +0x5fd
github.com/cockroachdb/cockroach/pkg/kv/kvserver.evaluateBatch(0x4b7f940, 0xc0043f6540, 0x0, 0x0, 0x4bfe4a0, 0xc0034acff0, 0x4c0d820, 0xc00060f800, 0x0, 0xc000c27380, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_evaluate.go:237 +0x3c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatchWithServersideRefreshes(0xc00060f800, 0x4b7f940, 0xc0043f6540, 0x4bfe4a0, 0xc0034acff0, 0x4c0d820, 0xc00060f800, 0xc000c27380, 0xc008930600, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_read.go:112 +0xee
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeReadOnlyBatch(0xc00060f800, 0x4b7f940, 0xc0043f6540, 0xc000c27380, 0xc000615340, 0x0, 0x0, 0xc0046084b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_read.go:84 +0x407
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00060f800, 0x4b7f940, 0xc0043f6540, 0xc000c27380, 0x4478760, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:215 +0x315
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00060f800, 0x4b7f940, 0xc0043f6510, 0xb4, 0xc000c27380, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00060f800, 0x4b7f940, 0xc0043f6510, 0x15fd61346c7591c0, 0x0, 0x300000003, 0x2, 0x0, 0xb4, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0006e6e00, 0x4b7f940, 0xc0043f6480, 0x15fd61346c7591c0, 0x0, 0x300000003, 0x2, 0x0, 0xb4, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc00011de00, 0x4b7f940, 0xc0043f6480, 0x0, 0x0, 0x300000003, 0x2, 0x0, 0xb4, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:188 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4b7f940, 0xc0043f6450, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000c68000, 0x4b7f940, 0xc0043f6450, 0x42e026b, 0x10, 0xc000bd35f0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000154500, 0x4b7f940, 0xc0043f6450, 0xc000c27280, 0xc0043f6450, 0xc00004e000, 0x3dfd7c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000154500, 0x4b7f940, 0xc0043f6420, 0xc000c27280, 0xc000154500, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x4b7f940, 0xc0043f6420, 0x4290480, 0xc000c27280, 0x4290480, 0xc000c27280, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10121 +0x86
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x4b7f940, 0xc0043f6420, 0x4290480, 0xc000c27280, 0xc00adea360, 0xc00adea380, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9ef
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x4b7f940, 0xc0043f6420, 0x4290480, 0xc000c27280, 0xc00adea360, 0xc00adea380, 0xc006b1aa78, 0x95fdd8, 0x40713e0, 0xc0043f6420)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:224 +0xb6
github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x42528c0, 0xc000154500, 0x4b7f940, 0xc0043f6420, 0xc008930540, 0xc00034af00, 0x4b7f940, 0xc0043f6420, 0xc00c39e0e0, 0xd4)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:10123 +0x14b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0008c0000, 0x4bd9600, 0xc0007dd500, 0xc0053f6000, 0xc00067b1a0, 0x6ddd980, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x460
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0008c0000, 0x4bd9600, 0xc0007dd500, 0xc0053f6000, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xd97
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002650a0, 0xc0008c0000, 0x4bd9600, 0xc0007dd500, 0xc0053f6000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0xbb
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1

which is similarly not stuck, but has likely been retrying for a long time.

@tbg
Copy link
Member

tbg commented Mar 31, 2020

Closing due to inactivity. We're not sure what fixed this, but it used to fail very regularly.

@tbg tbg closed this as completed Mar 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knz knz

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
20.1
Development

No branches or pull requests
5 participants
@andreimatei @knz @tbg @nvanbenschoten @cockroach-teamcity