Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com/cockroachdb/cockroach/pkg/ccl/backupccl: TestBackupRestoreNegativePrimaryKey failed under stress #26273

Closed
cockroach-teamcity opened this issue May 31, 2018 · 1 comment
Closed

github.com/cockroachdb/cockroach/pkg/ccl/backupccl: TestBackupRestoreNegativePrimaryKey failed under stress #26273

cockroach-teamcity opened this issue May 31, 2018 · 1 comment
Assignees
@benesch
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/4bcf507fc2c930f602d09e3561a1cb728223096b

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=689034&tab=buildLog


I180531 06:53:14.334600 28206 storage/replica_command.go:863  [n1,s1,r26/1:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/-150 [r28]
I180531 06:53:14.458943 28065 storage/replica_command.go:863  [n1,s1,r25/1:/Table/55{-/1/-350}] initiating a split of this range at key /Table/55/1/-500 [r29]
I180531 06:53:14.532268 28302 storage/replica_command.go:863  [n1,s1,r26/1:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/-300 [r30]
I180531 06:53:15.618454 26313 server/status/runtime.go:219  [n2] runtime stats: 1.5 GiB RSS, 1197 goroutines, 98 MiB/18 MiB/146 MiB GO alloc/idle/total, 189 MiB/226 MiB CGO alloc/total, 859.52cgo/sec, 0.94/0.06 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:17.479134 25672 server/status/runtime.go:219  [n1] runtime stats: 1.5 GiB RSS, 1203 goroutines, 108 MiB/12 MiB/146 MiB GO alloc/idle/total, 190 MiB/226 MiB CGO alloc/total, 878.32cgo/sec, 0.98/0.06 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:17.507362 26652 server/status/runtime.go:219  [n3] runtime stats: 1.5 GiB RSS, 1203 goroutines, 108 MiB/12 MiB/146 MiB GO alloc/idle/total, 190 MiB/226 MiB CGO alloc/total, 880.60cgo/sec, 0.98/0.06 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:17.970434 26476 storage/replica_proposal.go:202  [n2,s2,r18/2:/Table/2{1-2}] new range lease repl=(n2,s2):2 seq=4 start=1527749597.601966073,0 epo=1 pro=1527749597.602014905,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749565.942857390,0
I180531 06:53:17.987135 25604 storage/replica_proposal.go:202  [n1,s1,r25/1:/Table/55{-/1/-350}] new range lease repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:18.409550 28671 storage/replica_command.go:863  [n1,s1,r29/1:/Table/55/1/-{500-350}] initiating a split of this range at key /Table/55/1/-450 [r31]
W180531 06:53:18.645122 26654 server/node.go:802  [n3,summaries] health alerts detected: {Alerts:[{StoreID:0 Category:METRICS Description:round-trip-latency-p90 Value:1.006632959e+09}]}
I180531 06:53:18.699829 25620 storage/replica_proposal.go:202  [n1,s1,r26/1:/{Table/55/1/-…-Max}] new range lease repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:19.727708 26519 storage/replica_proposal.go:202  [n2,s2,r25/3:/Table/55{-/1/-500}] new range lease repl=(n2,s2):3 seq=4 start=1527749599.086553459,0 epo=1 pro=1527749599.086596353,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:19.747655 28735 storage/replica_command.go:863  [n1,s1,r28/1:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/100 [r32]
I180531 06:53:19.874010 28712 storage/replica_command.go:863  [n1,s1,r28/1:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/-100 [r33]
I180531 06:53:19.969760 28302 storage/replica_command.go:863  [n1,s1,r26/1:/Table/55/1/-{350-150}] initiating a split of this range at key /Table/55/1/-300 [r34]
I180531 06:53:20.666253 26351 gossip/gossip.go:1307  [n2] node has connected to cluster via gossip
I180531 06:53:20.774834 26351 storage/stores.go:241  [n2] wrote 2 node addresses to persistent storage
I180531 06:53:21.974656 18436 server/status/runtime.go:219  [n2] runtime stats: 1.6 GiB RSS, 1213 goroutines, 95 MiB/24 MiB/146 MiB GO alloc/idle/total, 197 MiB/234 MiB CGO alloc/total, 863.48cgo/sec, 1.01/0.05 %(u/s)time, 0.02 %gc (4x)
I180531 06:53:22.007485 26706 gossip/gossip.go:1307  [n3] node has connected to cluster via gossip
I180531 06:53:22.008666 26706 storage/stores.go:241  [n3] wrote 2 node addresses to persistent storage
I180531 06:53:22.155953 26798 storage/replica_proposal.go:202  [n3,s3,r28/2:/{Table/55/1/-…-Max}] new range lease repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:22.386138 25640 storage/replica_proposal.go:202  [n1,s1,r29/1:/Table/55/1/-{500-350}] new range lease repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
W180531 06:53:23.094862 25600 storage/store.go:3633  [n1,s1] handle raft ready: 0.5s [processed=0]
I180531 06:53:23.309897 18756 server/status/runtime.go:219  [n3] runtime stats: 1.6 GiB RSS, 1205 goroutines, 83 MiB/30 MiB/146 MiB GO alloc/idle/total, 196 MiB/234 MiB CGO alloc/total, 878.54cgo/sec, 1.04/0.05 %(u/s)time, 0.02 %gc (3x)
W180531 06:53:23.333142 28797 storage/replica.go:3131  [n1,s1,r29/1:/Table/55/1/-{500-450}] context canceled before proposing: 1 HeartbeatTxn
I180531 06:53:23.648283 29093 storage/replica_command.go:863  [n1,s1,r31/1:/Table/55/1/-{450-350}] initiating a split of this range at key /Table/55/1/-400 [r35]
I180531 06:53:24.220816 17894 gossip/gossip.go:488  [n1] gossip status (ok, 3 nodes)
gossip client (0/3 cur/max conns)
gossip server (2/3 cur/max conns, infos 1236/113 sent/received, bytes 841587B/150195B sent/received)
  3: 127.0.0.1:37861 (1m51s)
  2: 127.0.0.1:34717 (1m52s)
I180531 06:53:24.239033 17897 server/status/runtime.go:219  [n1] runtime stats: 1.6 GiB RSS, 1209 goroutines, 100 MiB/15 MiB/146 MiB GO alloc/idle/total, 196 MiB/235 MiB CGO alloc/total, 846.57cgo/sec, 1.02/0.05 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:24.269884 26514 storage/replica_proposal.go:202  [n2,s2,r5/3:/System/ts{d-e}] new range lease repl=(n2,s2):3 seq=4 start=1527749603.501587574,0 epo=1 pro=1527749603.501637634,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749565.927614436,0
I180531 06:53:24.360370 25603 storage/replica_proposal.go:202  [n1,s1,r26/1:/Table/55/1/-{350-150}] new range lease repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:25.577699 26313 server/status/runtime.go:219  [n2] runtime stats: 1.6 GiB RSS, 1211 goroutines, 68 MiB/44 MiB/147 MiB GO alloc/idle/total, 196 MiB/235 MiB CGO alloc/total, 832.50cgo/sec, 1.01/0.05 %(u/s)time, 0.03 %gc (4x)
I180531 06:53:26.708395 26776 storage/replica_proposal.go:202  [n3,s3,r34/2:/Table/55/1/-{300-150}] new range lease repl=(n3,s3):2 seq=4 start=1527749606.174492941,0 epo=1 pro=1527749606.174511156,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
W180531 06:53:26.885090 18681 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 1.1s
I180531 06:53:27.310679 25672 server/status/runtime.go:219  [n1] runtime stats: 1.6 GiB RSS, 1235 goroutines, 105 MiB/11 MiB/147 MiB GO alloc/idle/total, 198 MiB/235 MiB CGO alloc/total, 838.52cgo/sec, 1.01/0.05 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:27.338365 26478 storage/replica_proposal.go:202  [n2,s2,r16/2:/Table/{19-20}] new range lease repl=(n2,s2):2 seq=4 start=1527749606.240403839,0 epo=1 pro=1527749606.240422806,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.140764518,0
I180531 06:53:27.396315 26652 server/status/runtime.go:219  [n3] runtime stats: 1.6 GiB RSS, 1236 goroutines, 105 MiB/10 MiB/147 MiB GO alloc/idle/total, 199 MiB/235 MiB CGO alloc/total, 835.28cgo/sec, 1.01/0.05 %(u/s)time, 0.02 %gc (3x)
W180531 06:53:27.862287 26450 storage/node_liveness.go:528  [n2,hb] slow heartbeat took 1.8s
W180531 06:53:28.362874 25609 storage/store.go:3633  [n1,s1] handle raft ready: 0.5s [processed=0]
I180531 06:53:28.628663 29346 storage/replica_command.go:863  [n3,s3,r34/2:/Table/55/1/-{300-150}] initiating a split of this range at key /Table/55/1/-250 [r51]
W180531 06:53:29.128691 25681 storage/node_liveness.go:528  [n1,hb] slow heartbeat took 2.3s
I180531 06:53:29.307774 26791 storage/replica_proposal.go:202  [n3,s3,r28/2:/{Table/55/1/-…-Max}] new range lease repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0 following repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0
I180531 06:53:29.401996 26522 storage/replica_proposal.go:202  [n2,s2,r22/3:/Table/5{1-2}] new range lease repl=(n2,s2):3 seq=4 start=1527749608.724869569,0 epo=1 pro=1527749608.724926853,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
W180531 06:53:29.543147 26821 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 1.7s
W180531 06:53:29.729678 29137 storage/replica.go:3131  [n3,s3,r28/2:/Table/55/1/-1{50-00}] context canceled before proposing: 1 HeartbeatTxn
I180531 06:53:29.904873 25634 storage/replica_proposal.go:202  [n1,s1,r31/1:/Table/55/1/-{450-350}] new range lease repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.259929510,0
I180531 06:53:30.218515 29554 storage/replica_command.go:863  [n3,s3,r33/2:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/-50 [r52]
I180531 06:53:31.055612 29576 storage/replica_command.go:863  [n3,s3,r33/2:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/100 [r53]
W180531 06:53:31.918995 26450 storage/node_liveness.go:528  [n2,hb] slow heartbeat took 1.4s
I180531 06:53:31.977223 18369 gossip/gossip.go:488  [n2] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:38599 (2m0s: infos 50/797 sent/received, bytes 15317B/745487B sent/received)
gossip server (0/3 cur/max conns, infos 50/797 sent/received, bytes 15317B/745487B sent/received)
I180531 06:53:32.030353 18436 server/status/runtime.go:219  [n2] runtime stats: 1.6 GiB RSS, 1220 goroutines, 77 MiB/38 MiB/148 MiB GO alloc/idle/total, 200 MiB/238 MiB CGO alloc/total, 826.29cgo/sec, 0.92/0.07 %(u/s)time, 0.03 %gc (3x)
I180531 06:53:32.566199 26791 storage/replica_proposal.go:202  [n3,s3,r8/3:/Table/1{1-2}] new range lease repl=(n3,s3):3 seq=4 start=1527749611.639759238,0 epo=1 pro=1527749611.639798796,0 following repl=(n1,s1):1 seq=3 start=1527749557.225801828,0 epo=1 pro=1527749566.406650134,0
I180531 06:53:33.314326 18756 server/status/runtime.go:219  [n3] runtime stats: 1.6 GiB RSS, 1218 goroutines, 97 MiB/19 MiB/148 MiB GO alloc/idle/total, 200 MiB/238 MiB CGO alloc/total, 750.36cgo/sec, 0.90/0.07 %(u/s)time, 0.03 %gc (3x)
I180531 06:53:33.333052 18657 gossip/gossip.go:488  [n3] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:38599 (2m1s: infos 63/550 sent/received, bytes 134878B/159340B sent/received)
gossip server (0/3 cur/max conns, infos 63/550 sent/received, bytes 134878B/159340B sent/received)
W180531 06:53:33.671008 25681 storage/node_liveness.go:528  [n1,hb] slow heartbeat took 2.2s
I180531 06:53:34.297690 17897 server/status/runtime.go:219  [n1] runtime stats: 1.6 GiB RSS, 1220 goroutines, 113 MiB/7.0 MiB/148 MiB GO alloc/idle/total, 200 MiB/239 MiB CGO alloc/total, 698.00cgo/sec, 0.90/0.07 %(u/s)time, 0.03 %gc (4x)
W180531 06:53:34.313290 25641 storage/store.go:3633  [n1,s1] handle raft ready: 0.6s [processed=0]
W180531 06:53:34.873643 26821 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 2.5s
I180531 06:53:35.496030 26798 storage/replica_proposal.go:202  [n3,s3,r34/2:/Table/55/1/-{300-150}] new range lease repl=(n3,s3):2 seq=4 start=1527749606.174492941,0 epo=1 pro=1527749606.174511156,0 following repl=(n3,s3):2 seq=4 start=1527749606.174492941,0 epo=1 pro=1527749606.174511156,0
I180531 06:53:35.586538 26313 server/status/runtime.go:219  [n2] runtime stats: 1.6 GiB RSS, 1210 goroutines, 82 MiB/34 MiB/148 MiB GO alloc/idle/total, 199 MiB/239 MiB CGO alloc/total, 700.18cgo/sec, 0.90/0.08 %(u/s)time, 0.02 %gc (3x)
I180531 06:53:36.081548 29793 storage/replica_command.go:863  [n3,s3,r51/2:/Table/55/1/-{250-150}] initiating a split of this range at key /Table/55/1/-200 [r54]
I180531 06:53:37.329326 25669 gossip/gossip.go:488  [n1] gossip status (ok, 3 nodes)
gossip client (0/3 cur/max conns)
gossip server (2/3 cur/max conns, infos 571/120 sent/received, bytes 178999B/31877B sent/received)
  2: 127.0.0.1:36401 (52s)
  3: 127.0.0.1:34291 (50s)
I180531 06:53:37.425791 26652 server/status/runtime.go:219  [n3] runtime stats: 1.6 GiB RSS, 1219 goroutines, 114 MiB/6.8 MiB/148 MiB GO alloc/idle/total, 200 MiB/239 MiB CGO alloc/total, 705.91cgo/sec, 0.90/0.07 %(u/s)time, 0.03 %gc (4x)
I180531 06:53:37.519275 25672 server/status/runtime.go:219  [n1] runtime stats: 1.6 GiB RSS, 1221 goroutines, 114 MiB/7.0 MiB/148 MiB GO alloc/idle/total, 200 MiB/239 MiB CGO alloc/total, 707.34cgo/sec, 0.89/0.07 %(u/s)time, 0.03 %gc (4x)
I180531 06:53:37.991013 18351 storage/node_liveness.go:773  [n2,hb] retrying liveness update after storage.errRetryLiveness: result is ambiguous (context finished during distsender send: context deadline exceeded)
W180531 06:53:37.991470 18351 storage/node_liveness.go:528  [n2,hb] slow heartbeat took 4.5s
W180531 06:53:37.991720 18351 storage/node_liveness.go:465  [n2,hb] failed node liveness heartbeat: context deadline exceeded
I180531 06:53:38.194582 26771 storage/replica_proposal.go:202  [n3,s3,r33/2:/{Table/55/1/-…-Max}] new range lease repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0 following repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0
I180531 06:53:38.977545 30113 storage/replica_command.go:863  [n3,s3,r52/2:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/100 [r55]
I180531 06:53:39.027020 30211 storage/replica_command.go:863  [n3,s3,r52/2:/{Table/55/1/-…-Max}] initiating a split of this range at key /Table/55/1/0 [r56]
W180531 06:53:39.277486 17922 storage/node_liveness.go:528  [n1,hb] slow heartbeat took 4.6s
W180531 06:53:39.297557 18681 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 4.5s
W180531 06:53:40.445334 17922 storage/node_liveness.go:528  [n1,hb] slow heartbeat took 1.2s
W180531 06:53:40.546808 18681 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 1.2s
W180531 06:53:40.698257 26450 storage/node_liveness.go:528  [n2,hb] slow heartbeat took 1.1s
I180531 06:53:41.371346 26788 storage/replica_proposal.go:202  [n3,s3,r51/2:/Table/55/1/-{250-150}] new range lease repl=(n3,s3):2 seq=4 start=1527749606.174492941,0 epo=1 pro=1527749606.174511156,0 following repl=(n3,s3):2 seq=4 start=1527749606.174492941,0 epo=1 pro=1527749606.174511156,0
I180531 06:53:41.717399 26811 storage/replica_proposal.go:202  [n3,s3,r52/2:/{Table/55/1/-…-Max}] new range lease repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0 following repl=(n3,s3):2 seq=4 start=1527749601.533792366,0 epo=1 pro=1527749601.533849887,0
W180531 06:53:41.809466 30339 storage/replica.go:3131  [n3,s3,r51/2:/Table/55/1/-2{50-00}] context canceled before proposing: 1 HeartbeatTxn
I180531 06:53:42.015100 18436 server/status/runtime.go:219  [n2] runtime stats: 1.6 GiB RSS, 1216 goroutines, 92 MiB/29 MiB/153 MiB GO alloc/idle/total, 202 MiB/241 MiB CGO alloc/total, 785.01cgo/sec, 0.94/0.06 %(u/s)time, 0.03 %gc (3x)
W180531 06:53:42.242710 30312 storage/replica.go:3131  [n3,s3,r52/2:/Table/55/1/{-50-100}] context canceled before proposing: 1 HeartbeatTxn
I180531 06:53:42.564890 30211 storage/replica_command.go:863  [n3,s3,r52/2:/Table/55/1/{-50-100}] initiating a split of this range at key /Table/55/1/0 [r57]
I180531 06:53:42.621298 30382 storage/replica_command.go:863  [n3,s3,r55/2:/{Table/55/1/1…-Max}] initiating a split of this range at key /Table/55/1/500 [r58]
I180531 06:53:43.116111 30348 storage/replica_command.go:863  [n3,s3,r55/2:/{Table/55/1/1…-Max}] initiating a split of this range at key /Table/55/1/200 [r59]
W180531 06:53:43.130404 18438 server/node.go:802  [n2,summaries] health alerts detected: {Alerts:[{StoreID:0 Category:METRICS Description:liveness.heartbeatfailures Value:1}]}
I180531 06:53:43.501342 18756 server/status/runtime.go:219  [n3] runtime stats: 1.6 GiB RSS, 1212 goroutines, 114 MiB/11 MiB/153 MiB GO alloc/idle/total, 202 MiB/242 MiB CGO alloc/total, 823.41cgo/sec, 0.94/0.06 %(u/s)time, 0.03 %gc (4x)
F180531 06:53:43.615989 17802 storage/replica.go:3873  [n1,s1,r3/1:/System/NodeLiveness{-Max}] while purging index %d: while purging index 145: IO error: /go/src/github.com/cockroachdb/cockroach/artifacts/cockroach-auxiliary890459387/sideloading/3/3.1/i145.t9: File not found
goroutine 17802 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x5473c01, 0xc4204e4a20, 0x0, 0x12)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:956 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x4aacf80, 0xc400000004, 0x450c00d, 0x12, 0xf21, 0xc424456700, 0xdc)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:834 +0x7f4
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x37a3f40, 0xc424ce8cf0, 0xc400000004, 0x2, 0x3205d55, 0x6, 0xc425bafb88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x31e
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x37a3f40, 0xc424ce8cf0, 0x1, 0x4, 0x3205d55, 0x6, 0xc425bafb88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x37a3f40, 0xc424ce8cf0, 0x3205d55, 0x6, 0xc425bafb88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:149 +0x90
github.com/cockroachdb/cockroach/pkg/storage.fatalOnRaftReadyErr(0x37a3f40, 0xc424ce8cf0, 0x3225a2b, 0x16, 0x377e6c0, 0xc423827960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3873 +0x219
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x37a3f40, 0xc424ce8cf0, 0xc42214a000, 0x37a3f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3576 +0x215
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc4214d6000, 0x37a3f40, 0xc424ce8cf0, 0xc420d5e508, 0xc425bafe98, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3178 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4214d6000, 0x37a3f40, 0xc422d84120, 0x3)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3563 +0x302
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422774090, 0x37a3f40, 0xc422d84120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x2f2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422d84120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420c860b0, 0xc4231c9b00, 0xc420c860a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1 [chan receive, 2 minutes]:
testing.(*T).Run(0xc420128960, 0x3243a6e, 0x23, 0x3332a30, 0xc4205c5c01)
	/usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc420128960)
	/usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc420128960, 0xc4205c5d68)
	/usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc4203426e0, 0x4856b00, 0x2b, 0x2b, 0xbebc048f53feceb0)
	/usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc420156980, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/ccl/backupccl.TestMain(0xc420156980)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/main_test.go:31 +0x180
main.main()
	_testmain.go:136 +0x22b

goroutine 20 [syscall, 5 minutes]:
os/signal.signal_recv(0x90aab1)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 11 [semacquire]:
sync.runtime_SemacquireMutex(0x4aacfd8, 0x900000000)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0x4aacfd4)
	/usr/local/go/src/sync/mutex.go:134 +0x172
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0x4aacfd4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1163 +0x160
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x1b5

goroutine 12 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1cd

goroutine 34 [select, 5 minutes, locked to thread]:
runtime.gopark(0x3338128, 0x0, 0x32074dd, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc420091f50, 0xc4200c62a0)
	/usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 14 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc42038d100)
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:144 +0x18f
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x9b

goroutine 18627 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader(0xc426dec300, 0x60, 0x30f2920)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:240 +0x155
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).RecvCompress(0xc426dec300, 0x58, 0xc4266634d0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:251 +0x3d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc426a79930, 0x30969c0, 0xc420e72ea0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:525 +0xadb
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc420ddc100, 0x30969c0, 0xc420e72ea0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:405 +0x67
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x37a3f00, 0xc425bbb4a0, 0x3233f63, 0x1d, 0x30bacc0, 0xc4237a6240, 0x30969c0, 0xc420e72ea0, 0xc4209c6c00, 0xc422c11e00, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:83 +0x241
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x37a3f00, 0xc425bbb4a0, 0x3233f63, 0x1d, 0x30bacc0, 0xc4237a6240, 0x30969c0, 0xc420e72ea0, 0xc4209c6c00, 0x3337448, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:44 +0x10aa
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc4209c6c00, 0x37a3f00, 0xc425bbb4a0, 0x3233f63, 0x1d, 0x30bacc0, 0xc4237a6240, 0x30969c0, 0xc420e72ea0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x199
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.Invoke(0x37a3f00, 0xc425bbb4a0, 0x3233f63, 0x1d, 0x30bacc0, 0xc4237a6240, 0x30969c0, 0xc420e72ea0, 0xc4209c6c00, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:60 +0xe0
github.com/cockroachdb/cockroach/pkg/rpc.(*heartbeatClient).Ping(0xc420d5c310, 0x37a3f00, 0xc425bbb4a0, 0xc4237a6240, 0x0, 0x0, 0x0, 0x1533a82fa8df6f92, 0x29b063a, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/heartbeat.pb.go:125 +0x104
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc422307cc0, 0xc420365720, 0xc42037cdd0, 0xf, 0xc422399aa0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:592 +0x705
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x37a3f40, 0xc42160f500)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420427880, 0xc421d85560, 0xc4203dbcb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 17503 [IO wait]:
internal/poll.runtime_pollWait(0x7f2cccf51340, 0x72, 0x4863f58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc42078a418, 0x72, 0x3781400, 0x4863f58, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc42078a418, 0xc424fbbc00, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc42078a400, 0xc424fbbc00, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc42078a400, 0xc424fbbc00, 0x400, 0x400, 0x0, 0xe4, 0x3f3)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420d94110, 0xc424fbbc00, 0x400, 0x400, 0xc4285fe220, 0x8b719d, 0xc42003a000)
	/usr/local/go/src/net/net.go:176 +0x85
crypto/tls.(*block).readFromUntil(0xc4265f8930, 0x7f2cc683e1a0, 0xc420d94110, 0x5, 0xc420d94110, 0x3f3)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc420c11c00, 0x3338217, 0xc420c11d20, 0xc424fbb800)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc420c11c00, 0xc4211be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc420decea0, 0xc42338c5a0, 0x5, 0x200, 0x3332138, 0x8d67c8, 0xc420062a00)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x377b000, 0xc420decea0, 0xc42338c5a0, 0x5, 0x200, 0x5, 0xc422f881a8, 0xc422f881b0, 0x8e7544)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x377b000, 0xc420decea0, 0xc42338c5a0, 0x5, 0x200, 0x18, 0x2ea50c0, 0xc4285fe8e8)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc42338c580, 0xc424e35a00, 0xc4285fe8e8, 0x8bb168, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:947 +0x206
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc42338c580, 0xc424e35a00, 0x3e)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:997 +0x47
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(0xc42338c580, 0x4, 0x3a)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1018 +0xa2
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).readBindResponse(0xc42338c580)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1633 +0x3d
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*stmt).exec(0xc420b3c000, 0xc4215dc380, 0x1, 0x1)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1270 +0x1519
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).query(0xc42338c580, 0x3234394, 0x1d, 0xc4215dc380, 0x1, 0x1, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:855 +0x4a8
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).QueryContext(0xc42338c580, 0x37a3ec0, 0xc4200dc050, 0x3234394, 0x1d, 0xc4265f8450, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:21 +0x232
database/sql.ctxDriverQuery(0x37a3ec0, 0xc4200dc050, 0x7f2cccf49f20, 0xc42338c580, 0x0, 0x0, 0x3234394, 0x1d, 0xc4265f8450, 0x1, ...)
	/usr/local/go/src/database/sql/ctxutil.go:48 +0x2e3
database/sql.(*DB).queryDC.func1()
	/usr/local/go/src/database/sql/sql.go:1464 +0x2da
database/sql.withLock(0x3784800, 0xc420a9e680, 0xc4285ff098)
	/usr/local/go/src/database/sql/sql.go:3032 +0x75
database/sql.(*DB).queryDC(0xc4216d88c0, 0x37a3ec0, 0xc4200dc050, 0x0, 0x0, 0xc420a9e680, 0xc4215dc370, 0x3234394, 0x1d, 0xc4285ff790, ...)
	/usr/local/go/src/database/sql/sql.go:1459 +0x952
database/sql.(*DB).query(0xc4216d88c0, 0x37a3ec0, 0xc4200dc050, 0x3234394, 0x1d, 0xc4285ff790, 0x1, 0x1, 0xc420064f01, 0xc420064f00, ...)
	/usr/local/go/src/database/sql/sql.go:1442 +0x184
database/sql.(*DB).QueryContext(0xc4216d88c0, 0x37a3ec0, 0xc4200dc050, 0x3234394, 0x1d, 0xc4285ff790, 0x1, 0x1, 0xc422c143c0, 0xc440000000, ...)
	/usr/local/go/src/database/sql/sql.go:1419 +0xe3
database/sql.(*DB).QueryRowContext(0xc4216d88c0, 0x37a3ec0, 0xc4200dc050, 0x3234394, 0x1d, 0xc4285ff790, 0x1, 0x1, 0xa006d9)
	/usr/local/go/src/database/sql/sql.go:1520 +0xac
database/sql.(*DB).QueryRow(0xc4216d88c0, 0x3234394, 0x1d, 0xc4285ff790, 0x1, 0x1, 0x10)
	/usr/local/go/src/database/sql/sql.go:1531 +0x9d
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryRow(0xc4285ff720, 0x37d1260, 0xc422c143c0, 0x3234394, 0x1d, 0xc4285ff790, 0x1, 0x1, 0x3789e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:89 +0xa0
github.com/cockroachdb/cockroach/pkg/ccl/backupccl_test.backupAndRestore(0x37a3ec0, 0xc4200dc050, 0xc422c143c0, 0xc42409b340, 0x321948b, 0x10, 0x3e8)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:328 +0xad5
github.com/cockroachdb/cockroach/pkg/ccl/backupccl_test.TestBackupRestoreNegativePrimaryKey(0xc422c143c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_test.go:266 +0x2aa
testing.tRunner(0xc422c143c0, 0x3332a30)
	/usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x565

goroutine 19992 [semacquire, 1 minutes]:
sync.runtime_notifyListWait(0xc420a1c190, 0x5)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420a1c180)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).curCmd(0xc420a066e0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:424 +0xe1
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc421186c00, 0x37a3e80, 0xc420a1c200, 0xc4204d71b0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1023 +0x5b4
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc4214e9b00, 0x37a3e80, 0xc420a1c200, 0xc426529011, 0x4, 0xc42652901b, 0x4, 0x0, 0x0, 0x3788f40, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:401 +0x1e1
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func3(0xc4214e9b00, 0x37a3e80, 0xc420a1c200, 0xc420369180, 0x5400, 0x15000, 0xc420fe55c0, 0xc4204d71b0, 0xc4204d7190, 0xc42184a580, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:266 +0x1aa
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:265 +0x147b

goroutine 17798 [semacquire]:
sync.runtime_notifyListWait(0xc4237f5210, 0x2bcfe12400001245)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4237f5200)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422774090, 0x37a3f40, 0xc422d840c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422d840c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420c86030, 0xc4231c9b00, 0xc420c86020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18535 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc421d85560, 0xc422f73a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:122 +0x11d
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:119 +0x57

goroutine 26240 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x37a3f40, 0xc4214b90e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420406940, 0xc4227753b0, 0xc420443540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26498 [semacquire]:
sync.runtime_notifyListWait(0xc4211d3990, 0xb83fc74c00000a51)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4211d3980)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420e57ef0, 0x37a3f40, 0xc4209d6a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc4209d6a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42130d0c0, 0xc4227753b0, 0xc42130d0b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 17562 [semacquire]:
sync.runtime_notifyListWait(0xc4237f5210, 0x1b1c7faf00001246)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4237f5200)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422774090, 0x37a3f40, 0xc422d842d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422d842d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423532070, 0xc4231c9b00, 0xc423532060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 25424 [semacquire]:
sync.runtime_notifyListWait(0xc420e79ad0, 0xf0ea8b300000fd8)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420e79ac0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422dea360, 0x37a3f40, 0xc422b3f650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422b3f650)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b799d0, 0xc4201fdd40, 0xc423b799c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18682 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func1(0x37a3f40, 0xc423354a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:259 +0x16a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422b24350, 0xc421d85560, 0xc4233544b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18845 [semacquire]:
sync.runtime_notifyListWait(0xc424d6e310, 0xed5f97f800000f33)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc424d6e300)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424592480, 0x37a3f40, 0xc424ce8900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc424ce8900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420c87250, 0xc421d85560, 0xc420c87240)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 25691 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4242a7400, 0xc4250d4000, 0xc420c3a4c0, 0xf, 0xc4230900c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x37a3f40, 0xc422478e70)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ce600, 0xc4201fdd40, 0xc42171e540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 25609 [semacquire]:
sync.runtime_notifyListWait(0xc420e79ad0, 0xf0ea8b300000fc8)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420e79ac0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422dea360, 0x37a3f40, 0xc420b02f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc420b02f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b79cf0, 0xc4201fdd40, 0xc423b79ce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 27882 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc4209c22d0, 0x37ba240, 0xc420382150, 0x3786d80, 0xc4209c22d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:359 +0x2db
github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x31ba5e0, 0xc4209c22d0, 0x37b4540, 0xc4216ee420, 0x37a3f40, 0xc42119cbd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:415 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x31ba5e0, 0xc4209c22d0, 0x37b4540, 0xc4216ee420, 0xc4218f76e0, 0x33355f8, 0x37a42c0, 0xc422448480)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:182 +0x16e
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc42113c000, 0x37bcfa0, 0xc420cdd200, 0xc423b6c100, 0xc424d77440, 0x4844320, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1167 +0x995
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc42113c000, 0x37bcfa0, 0xc420cdd200, 0xc423b6c100, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1253 +0x12f9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4252cb590, 0xc42113c000, 0x37bcfa0, 0xc420cdd200, 0xc423b6c100)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:680 +0xad
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:678 +0xb9

goroutine 17942 [IO wait]:
internal/poll.runtime_pollWait(0x7f2cccf51000, 0x72, 0x4863f58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc421198198, 0x72, 0x3781400, 0x4863f58, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc421198198, 0xc420642800, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc421198180, 0xc420642800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc421198180, 0xc420642800, 0x400, 0x400, 0xc41fe0d0ef, 0xc41fe0d0ee, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420d94140, 0xc420642800, 0x400, 0x400, 0xc4209476f0, 0x8b719d, 0xc42003a000)
	/usr/local/go/src/net/net.go:176 +0x85
crypto/tls.(*block).readFromUntil(0xc420ca4b40, 0x7f2cc683e1a0, 0xc420d94140, 0x5, 0xc420d94140, 0x7f2ccd035458)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc425a3ea80, 0x3338217, 0xc425a3eba0, 0x37)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc425a3ea80, 0xc422cf8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc425043c20, 0xc4205c9538, 0x9, 0x9, 0x0, 0x0, 0xc420947c58)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x377b000, 0xc425043c20, 0xc4205c9538, 0x9, 0x9, 0x9, 0x8ba9fa, 0xc420082c00, 0x10)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x377b000, 0xc425043c20, 0xc4205c9538, 0x9, 0x9, 0xc420acb348, 0x2, 0x90bb92)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4205c9538, 0x9, 0x9, 0x377b000, 0xc425043c20, 0x0, 0x0, 0xc420fdeb40, 0x3788d00)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4205c9500, 0xc421009620, 0xc421009600, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4217d7200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x1fe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x10fc

goroutine 26311 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc420f0a690, 0xc42126a1c0, 0xc420307c30, 0x33384c8, 0xc4227e50e0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:887 +0x20f
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x37a3f40, 0xc424ec2f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:961 +0xdb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420307c40, 0xc4227753b0, 0xc420b62180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18351 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader(0xc421012300, 0x60, 0x30f2920)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:240 +0x155
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).RecvCompress(0xc421012300, 0x58, 0xc42614a6b0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:251 +0x3d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc422646750, 0x30f5720, 0xc422677080, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:525 +0xadb
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc426e0b080, 0x30f5720, 0xc422677080, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:405 +0x67
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x37a3f40, 0xc426549ef0, 0x323e002, 0x21, 0x31cd660, 0xc420e419b0, 0x30f5720, 0xc422677080, 0xc4204ecf00, 0xc422c21900, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:83 +0x241
github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x37a3f40, 0xc426549ef0, 0x323e002, 0x21, 0x31cd660, 0xc420e419b0, 0x30f5720, 0xc422677080, 0xc4204ecf00, 0x3337448, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:44 +0x10aa
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc4204ecf00, 0x37a3f40, 0xc426549ef0, 0x323e002, 0x21, 0x31cd660, 0xc420e419b0, 0x30f5720, 0xc422677080, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x199
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.Invoke(0x37a3f40, 0xc426549ef0, 0x323e002, 0x21, 0x31cd660, 0xc420e419b0, 0x30f5720, 0xc422677080, 0xc4204ecf00, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:60 +0xe0
github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc420e84ce8, 0x37a3f40, 0xc426549ef0, 0xc420e419b0, 0x0, 0x0, 0x0, 0x1, 0x5473fd0, 0x311f120)
	/go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:3734 +0x104
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send.func1(0xc42464d9d0, 0xc420e419a0, 0x37a3f40, 0xc426549ef0, 0x0, 0x0, 0xc426e9d098)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:253 +0x383
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).send(0xc42464d9d0, 0x37a3f40, 0xc426549ef0, 0xc426dee440, 0xf, 0x0, 0x0, 0x100000001, 0x1, 0x3, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:273 +0xa1
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc42464d9d0, 0x37a3f40, 0xc426549ef0, 0xc422677020)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:207 +0x498
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc426551680, 0x37a3f40, 0xc426549ef0, 0xc4265516d0, 0x3, 0xc4220b27c0, 0x1, 0x1, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1274 +0x45d
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc426551680, 0x37a3f40, 0xc426549ef0, 0x3, 0xc4220b27c0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:393 +0x2bf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc426551680, 0x37a3f40, 0xc426549ef0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421330400, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x266
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc426551680, 0x37a3f40, 0xc426549ef0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421330400, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1052 +0x3de
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc426551680, 0x37a3f40, 0xc426549ef0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc421330400, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:744 +0x1718
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc426551680, 0x37a3f40, 0xc426549ef0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc423829f00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:655 +0x566
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc4227b5500, 0x37a3f40, 0xc426549e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc423829f00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:490 +0x2c5
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc42171f620, 0x37a3e80, 0xc4221a1280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc423829f00, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:553 +0x107
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc42353c340, 0x37a3e80, 0xc4221a1280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:973 +0x66f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send-fm(0x37a3e80, 0xc4221a1280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:500 +0x97
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x37a3e80, 0xc4221a1280, 0xc426e9f498, 0xc423c7c300, 0xc420fd9ea0, 0xc426e9f5b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:468 +0x13e
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc42353c340, 0x37a3e80, 0xc4221a1280, 0xc423c7c300, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:500 +0xfe
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLivenessAttempt.func1(0x37a3e80, 0xc4221a1280, 0xc42353c340, 0x10, 0x2fc9a60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:821 +0x57d
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x37a3e80, 0xc4221a1280, 0xc42353c340, 0xc420ac6650, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:516 +0x5f
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc42353c340, 0x37a3e80, 0xc4221a1280, 0xc423530101, 0xc424e1cee0, 0x10000000092aa4e, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:748 +0x129
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc42171f620, 0x37a3e80, 0xc4221a1280, 0xc4218f0ca0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:515 +0x193
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLivenessAttempt(0xc421809f00, 0x37a3f40, 0xc420cba900, 0x2, 0x1, 0x1533a832280236ce, 0x0, 0x0, 0x0, 0xc420b6c8a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:797 +0x198
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).updateLiveness(0xc421809f00, 0x37a3f40, 0xc420cba900, 0x2, 0x1, 0x1533a832280236ce, 0x0, 0x0, 0x0, 0xc420b6c8a0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:770 +0x2b0
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).heartbeatInternal(0xc421809f00, 0x37a3f40, 0xc420cba900, 0xc420b6c8a0, 0x4000000000000000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:576 +0x530
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1.1(0xc421809f00, 0xc426e9fe80, 0x2faf080, 0x3b9aca00, 0x4000000000000000, 0x0, 0x0, 0xc421802840, 0xc42614de37)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:460 +0x37d
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x37a3f40, 0xc4246752f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:471 +0x34f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc425f66940, 0xc4216675f0, 0xc424f78f00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26303 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x37a3f40, 0xc424d81d10)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204d6940, 0xc4227753b0, 0xc4204d6920)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18131 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc4231bdb00, 0x33359a0, 0xc4231c9b00, 0x37a3f40, 0xc422e17920)
	<autogenerated>:1 +0x80
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc421280be8, 0x37a3f40, 0xc422e17920, 0xc4231c9b00, 0x37a17c0, 0xc4231bdb00, 0xc422e17950, 0xc420a1bea8, 0xa4dd93)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:119 +0x61
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func18(0x37a3f40, 0xc422e17920)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1580 +0x2cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422367720, 0xc4231c9b00, 0xc424aa59c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 25595 [semacquire]:
sync.runtime_notifyListWait(0xc420e79ad0, 0x4512739900000fef)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420e79ac0)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc422dea360, 0x37a3f40, 0xc42157d7d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc42157d7d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b79b30, 0xc4201fdd40, 0xc423b79b20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26845 [semacquire]:
sync.runtime_notifyListWait(0xc421b4d790, 0x12810630000009ea)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421b4d780)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42354f170, 0x37a3f40, 0xc422b3e900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422b3e900)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b799d0, 0xc422884000, 0xc423b799c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 97 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4209cf440)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 98 [chan receive, 5 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4209cf560)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 26846 [semacquire]:
sync.runtime_notifyListWait(0xc421b4d790, 0x253f43da000009ed)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421b4d780)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42354f170, 0x37a3f40, 0xc422b3ecf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc422b3ecf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b799f0, 0xc422884000, 0xc423b799e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26533 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x37a3f40, 0xc423415440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1562 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42288fee0, 0xc4227753b0, 0xc4242f13b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 25674 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x37a3f40, 0xc426527860)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:775 +0x25d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc422367940, 0xc4201fdd40, 0xc4221eace0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26806 [semacquire]:
sync.runtime_notifyListWait(0xc421b4d790, 0x419ea982000009ec)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421b4d780)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42354f170, 0x37a3f40, 0xc420b03980)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc420b03980)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423517d90, 0xc422884000, 0xc423517d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 27028 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc420ac94a0, 0xc4218f7290, 0x5, 0x5, 0xc42045c340, 0x8e7930, 0x2982730)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:142 +0x19c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc420ac94a0, 0xc4218f7290, 0x5, 0x5, 0x8e6bec, 0xc42045c340, 0xc422957970)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:131 +0x82
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc426925c50, 0xc4218f7290, 0x5, 0x5, 0x10190d3, 0xc42045c340, 0x1fffe0000555ee)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:394 +0x71
io.ReadAtLeast(0x377ee60, 0xc426925c50, 0xc4218f7290, 0x5, 0x5, 0x5, 0xc422957a48, 0x1043379, 0xc4261eac00)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x377ee60, 0xc426925c50, 0xc4218f7290, 0x5, 0x5, 0xc42087ffa0, 0xc42087ffa0, 0xc422957ad0)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc426cae200, 0xc4218f7290, 0x5, 0x5, 0x8ae8ed, 0xc42154ae10, 0x8e7930)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:378 +0x15d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4218f7280, 0x7fffffff, 0xb, 0x0, 0x92aef6, 0xc421b4d790, 0x0, 0xe)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:452 +0x99
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4218f7280, 0x7f2cc69bb6e8, 0x5473fd0, 0xc426cae200, 0x0, 0x0, 0x30a2ea0, 0xc42040dfc0, 0x7fffffff, 0xc4209c5c80, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:578 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc42244b810, 0x30a2ea0, 0xc42040dfc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:757 +0x1e9
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc4215d4e60, 0x37a3f40, 0xc42654b170, 0xc4265f5688)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:434 +0x83
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x37ba240, 0xc4215d4e60, 0xc4209c22d0, 0x37a3f40, 0xc42654b170, 0x8d61ed, 0xc422001d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:330 +0x109
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x37a3f40, 0xc42654b170)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:353 +0x75
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4215d4e90, 0xc422884000, 0xc423b6b4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18916 [select, 2 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc42109a9c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x205
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x4d

goroutine 18546 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x37a3f40, 0xc423354d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4215d42b0, 0xc4216675f0, 0xc4218f1e20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 18849 [semacquire]:
sync.runtime_notifyListWait(0xc424d6e310, 0x8a264efd00000f2e)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc424d6e300)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424592480, 0x37a3f40, 0xc424ce89f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37a3f40, 0xc424ce89f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420c872d0, 0xc421d85560, 0xc420c872c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 26654 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x37a3f40, 0xc420b03b00)
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone May 31, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 31, 2018
@windchan7
Copy link
Contributor

Fixed by #26259.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benesch benesch

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
3 participants
@benesch @cockroach-teamcity @windchan7