Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com/cockroachdb/cockroach/pkg/ccl/importccl: TestMultiNodeExportStmt failed under stress #26121

Closed
cockroach-teamcity opened this issue May 26, 2018 · 7 comments
Closed

github.com/cockroachdb/cockroach/pkg/ccl/importccl: TestMultiNodeExportStmt failed under stress #26121

cockroach-teamcity opened this issue May 26, 2018 · 7 comments
Assignees
@windchan7
Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/cbbc13cd30b14bc16cd8ebe56be093f264465ebe

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=680975&tab=buildLog


I180526 06:59:06.648120 79979 storage/replica.go:3320  [n2,s2,r30/2:/Table/53/1/{40-60}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n3,s3):4 (n2,s2):2 (n4,s4):3] next=5
E180526 06:59:06.823688 80020 storage/queue.go:778  [replicate,n2,s2,r28/2:/{Table/55-Max}] no removable replicas from range that needs a removal: [2*:29]
I180526 06:59:06.826615 74942 storage/store.go:3508  [n1,s1,r30/1:/Table/53/1/{40-60}] added to replica GC queue (peer suggestion)
I180526 06:59:06.845856 80021 storage/replica_raftstorage.go:520  [replicate,n2,s2,r30/2:/Table/53/1/{40-60}] generated preemptive snapshot 80321d4b at index 28
I180526 06:59:06.865999 74942 storage/store.go:3508  [n1,s1,r30/1:/Table/53/1/{40-60}] added to replica GC queue (peer suggestion)
I180526 06:59:06.869649 80021 storage/store_snapshot.go:605  [replicate,n2,s2,r30/2:/Table/53/1/{40-60}] streamed snapshot to (n5,s5):?: kv pairs: 31, log entries: 18, rate-limit: 2.0 MiB/sec, 22ms
I180526 06:59:06.888680 80005 storage/store.go:2492  [replicaGC,n1,s1,r30/1:/Table/53/1/{40-60}] removing replica
I180526 06:59:06.891945 80005 storage/replica.go:829  [replicaGC,n1,s1,r30/1:/Table/53/1/{40-60}] removed 27 (20+7) keys in 2ms [clear=0ms commit=2ms]
I180526 06:59:06.948210 80022 storage/replica_raftstorage.go:730  [n5,s5,r30/?:{-}] applying preemptive snapshot at index 28 (id=80321d4b, encoded size=8521, 1 rocksdb batches, 18 log entries)
I180526 06:59:07.047275 80022 storage/replica_raftstorage.go:736  [n5,s5,r30/?:/Table/53/1/{40-60}] applied preemptive snapshot in 99ms [clear=0ms batch=0ms entries=3ms commit=0ms]
W180526 06:59:07.057044 74187 server/node.go:802  [n2,summaries] health alerts detected: {Alerts:[{StoreID:2 Category:METRICS Description:queue.replicate.process.failure Value:2}]}
I180526 06:59:07.066844 80021 storage/replica_command.go:1777  [replicate,n2,s2,r30/2:/Table/53/1/{40-60}] change replicas (ADD_REPLICA (n5,s5):5): read existing descriptor r30:/Table/53/1/{40-60} [(n3,s3):4, (n2,s2):2, (n4,s4):3, next=5]
I180526 06:59:07.099450 80037 storage/replica.go:3320  [n1,s1,r23/1:/Table/5{2-3}] proposing ADD_REPLICA((n3,s3):4): updated=[(n1,s1):1 (n2,s2):2 (n4,s4):3 (n3,s3):4] next=5
I180526 06:59:07.482084 74449 gossip/gossip.go:488  [n3] gossip status (ok, 5 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:36221 (1m0s: infos 50/616 sent/received, bytes 16491B/219992B sent/received)
gossip server (1/3 cur/max conns, infos 1086/664 sent/received, bytes 316172B/238303B sent/received)
  5: 127.0.0.1:38739 (56s)
I180526 06:59:07.520560 79983 storage/replica.go:3320  [n3,s3,r24/4:/Table/53{-/1/20}] proposing REMOVE_REPLICA((n2,s2):2): updated=[(n3,s3):4 (n5,s5):5 (n4,s4):3] next=6
I180526 06:59:07.528996 74548 server/status/runtime.go:219  [n3] runtime stats: 1.7 GiB RSS, 1125 goroutines, 93 MiB/18 MiB/140 MiB GO alloc/idle/total, 168 MiB/207 MiB CGO alloc/total, 892.67cgo/sec, 1.10/0.08 %(u/s)time, 0.02 %gc (4x)
I180526 06:59:07.918074 77390 storage/store.go:3508  [n2,s2,r24/2:/Table/53{-/1/20}] added to replica GC queue (peer suggestion)
I180526 06:59:08.078358 74269 storage/replica_proposal.go:202  [n2,s2,r23/2:/Table/5{2-3}] new range lease repl=(n2,s2):2 seq=4 start=1527317947.348901188,0 epo=1 pro=1527317947.348916637,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
I180526 06:59:08.115320 80162 storage/replica_raftstorage.go:520  [replicate,n1,s1,r31/1:/Table/53/1/{60-80}] generated preemptive snapshot e08f6639 at index 21
I180526 06:59:08.353044 73672 server/status/runtime.go:219  [n1] runtime stats: 1.7 GiB RSS, 1133 goroutines, 74 MiB/40 MiB/140 MiB GO alloc/idle/total, 168 MiB/207 MiB CGO alloc/total, 878.87cgo/sec, 1.09/0.07 %(u/s)time, 0.03 %gc (5x)
I180526 06:59:08.385627 80162 storage/store_snapshot.go:605  [replicate,n1,s1,r31/1:/Table/53/1/{60-80}] streamed snapshot to (n5,s5):?: kv pairs: 29, log entries: 11, rate-limit: 2.0 MiB/sec, 189ms
I180526 06:59:08.410173 73576 storage/replica_proposal.go:202  [n1,s1,r35/1:/Table/5{3/1/140-4}] new range lease repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
I180526 06:59:08.424687 80136 storage/replica_raftstorage.go:730  [n5,s5,r31/?:{-}] applying preemptive snapshot at index 21 (id=e08f6639, encoded size=6353, 1 rocksdb batches, 11 log entries)
I180526 06:59:08.617120 80136 storage/replica_raftstorage.go:736  [n5,s5,r31/?:/Table/53/1/{60-80}] applied preemptive snapshot in 192ms [clear=0ms batch=0ms entries=140ms commit=51ms]
I180526 06:59:08.810772 80162 storage/replica_command.go:1777  [replicate,n1,s1,r31/1:/Table/53/1/{60-80}] change replicas (ADD_REPLICA (n5,s5):4): read existing descriptor r31:/Table/53/1/{60-80} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:09.008327 74874 gossip/gossip.go:488  [n4] gossip status (ok, 5 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:36221 (1m1s: infos 44/799 sent/received, bytes 14431B/167676B sent/received)
gossip server (0/3 cur/max conns, infos 44/799 sent/received, bytes 14431B/167676B sent/received)
I180526 06:59:09.139631 74877 server/status/runtime.go:219  [n4] runtime stats: 1.7 GiB RSS, 1133 goroutines, 74 MiB/35 MiB/140 MiB GO alloc/idle/total, 169 MiB/207 MiB CGO alloc/total, 902.29cgo/sec, 1.07/0.08 %(u/s)time, 0.03 %gc (4x)
I180526 06:59:09.219095 80212 storage/replica_command.go:863  [n1,s1,r36/1:/Table/5{3/1/160-4}] initiating a split of this range at key /Table/53/1/180 [r37]
I180526 06:59:09.843883 80249 storage/replica.go:3320  [n2,s2,r30/2:/Table/53/1/{40-60}] proposing ADD_REPLICA((n5,s5):5): updated=[(n3,s3):4 (n2,s2):2 (n4,s4):3 (n5,s5):5] next=6
I180526 06:59:10.124190 80041 storage/store.go:2492  [replicaGC,n2,s2,r24/2:/Table/53{-/1/20}] removing replica
I180526 06:59:10.193612 80041 storage/replica.go:829  [replicaGC,n2,s2,r24/2:/Table/53{-/1/20}] removed 27 (20+7) keys in 36ms [clear=0ms commit=35ms]
I180526 06:59:10.422330 80253 storage/replica_command.go:1777  [replicate,n2,s2,r23/2:/Table/5{2-3}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r23:/Table/5{2-3} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n3,s3):4, next=5]
I180526 06:59:10.516046 75348 server/status/runtime.go:219  [n5] runtime stats: 1.7 GiB RSS, 1129 goroutines, 73 MiB/38 MiB/140 MiB GO alloc/idle/total, 168 MiB/208 MiB CGO alloc/total, 872.07cgo/sec, 1.09/0.08 %(u/s)time, 0.03 %gc (5x)
I180526 06:59:10.519614 75265 gossip/gossip.go:488  [n5] gossip status (ok, 5 nodes)
gossip client (1/3 cur/max conns)
  3: 127.0.0.1:37929 (1m0s: infos 49/1084 sent/received, bytes 18677B/309368B sent/received)
gossip server (0/3 cur/max conns, infos 55/1109 sent/received, bytes 19415B/320127B sent/received)
I180526 06:59:10.577910 80299 storage/replica_command.go:1777  [replicate,n5,s5,r18/3:/Table/2{1-2}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r18:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, (n5,s5):3, (n3,s3):4, next=5]
I180526 06:59:10.713816 80261 storage/replica.go:3320  [n1,s1,r31/1:/Table/53/1/{60-80}] proposing ADD_REPLICA((n5,s5):4): updated=[(n1,s1):1 (n2,s2):2 (n4,s4):3 (n5,s5):4] next=5
E180526 06:59:10.762218 80317 storage/queue.go:778  [replicate,n4,s4,r26/3:/Table/5{4-5}] no removable replicas from range that needs a removal: [raft progress unknown]
I180526 06:59:11.692872 80349 storage/replica_raftstorage.go:520  [replicate,n1,s1,r32/1:/Table/53/1/{80-100}] generated preemptive snapshot 738880bd at index 19
W180526 06:59:11.740701 75350 server/node.go:802  [n5,summaries] health alerts detected: {Alerts:[{StoreID:5 Category:METRICS Description:queue.replicate.process.failure Value:1}]}
I180526 06:59:11.801674 74972 storage/replica_proposal.go:202  [n4,s4,r31/3:/Table/53/1/{60-80}] new range lease repl=(n4,s4):3 seq=4 start=1527317951.201637246,0 epo=1 pro=1527317951.201677438,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
I180526 06:59:12.043727 73633 storage/replica_proposal.go:202  [n1,s1,r36/1:/Table/5{3/1/160-4}] new range lease repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
I180526 06:59:12.084415 80349 storage/store_snapshot.go:605  [replicate,n1,s1,r32/1:/Table/53/1/{80-100}] streamed snapshot to (n3,s3):?: kv pairs: 28, log entries: 9, rate-limit: 2.0 MiB/sec, 287ms
I180526 06:59:12.344594 80409 storage/replica_raftstorage.go:730  [n3,s3,r32/?:{-}] applying preemptive snapshot at index 19 (id=738880bd, encoded size=5376, 1 rocksdb batches, 9 log entries)
I180526 06:59:12.419356 80503 storage/replica_raftstorage.go:520  [n1,s1,r34/1:/Table/53/1/1{20-40}] generated preemptive snapshot 444778d8 at index 16
I180526 06:59:12.530493 80505 storage/replica_raftstorage.go:520  [n1,s1,r29/1:/Table/53/1/{20-40}] generated preemptive snapshot 93d949d1 at index 18
I180526 06:59:12.531964 80508 storage/replica_raftstorage.go:520  [n1,s1,r33/1:/Table/53/1/1{00-20}] generated preemptive snapshot df26a211 at index 18
I180526 06:59:12.536088 80510 storage/replica_raftstorage.go:520  [n1,s1,r35/1:/Table/53/1/1{40-60}] generated preemptive snapshot 7d4c61fc at index 17
I180526 06:59:12.538456 80409 storage/replica_raftstorage.go:736  [n3,s3,r32/?:/Table/53/1/{80-100}] applied preemptive snapshot in 194ms [clear=0ms batch=0ms entries=86ms commit=10ms]
I180526 06:59:12.671772 80513 storage/replica_raftstorage.go:520  [n1,s1,r36/1:/Table/53/1/1{60-80}] generated preemptive snapshot f794f3af at index 17
I180526 06:59:12.929277 80508 storage/replica_raftstorage.go:520  [n1,s1,r33/1:/Table/53/1/1{00-20}] generated preemptive snapshot 56e45740 at index 18
I180526 06:59:13.111154 80505 storage/store_snapshot.go:605  [n1,s1,r29/1:/Table/53/1/{20-40}] streamed snapshot to (n3,s3):?: kv pairs: 28, log entries: 8, rate-limit: 2.0 MiB/sec, 561ms
I180526 06:59:13.183858 80503 storage/replica_raftstorage.go:520  [n1,s1,r34/1:/Table/53/1/1{20-40}] generated preemptive snapshot b6fe8313 at index 16
W180526 06:59:13.234411 74887 storage/node_liveness.go:528  [n4,hb] slow heartbeat took 1.2s
I180526 06:59:13.270517 75015 storage/replica_proposal.go:202  [n4,s4,r30/3:/Table/53/1/{40-60}] new range lease repl=(n4,s4):3 seq=5 start=1527317952.541640477,0 epo=1 pro=1527317952.541656115,0 following repl=(n2,s2):2 seq=4 start=1527317939.354336593,0 epo=1 pro=1527317939.354354388,0
I180526 06:59:13.322009 80535 storage/replica_raftstorage.go:730  [n3,s3,r29/?:{-}] applying preemptive snapshot at index 18 (id=93d949d1, encoded size=5261, 1 rocksdb batches, 8 log entries)
I180526 06:59:13.431586 80510 storage/replica_raftstorage.go:520  [n1,s1,r35/1:/Table/53/1/1{40-60}] generated preemptive snapshot f7002b9d at index 17
I180526 06:59:13.342824 80508 storage/store_snapshot.go:605  [n1,s1,r33/1:/Table/53/1/1{00-20}] streamed snapshot to (n5,s5):?: kv pairs: 28, log entries: 8, rate-limit: 2.0 MiB/sec, 401ms
I180526 06:59:13.469440 80535 storage/replica_raftstorage.go:736  [n3,s3,r29/?:/Table/53/1/{20-40}] applied preemptive snapshot in 51ms [clear=0ms batch=0ms entries=1ms commit=48ms]
I180526 06:59:13.472833 80543 storage/replica_raftstorage.go:730  [n5,s5,r33/?:{-}] applying preemptive snapshot at index 18 (id=56e45740, encoded size=5379, 1 rocksdb batches, 8 log entries)
I180526 06:59:13.476931 80543 storage/replica_raftstorage.go:736  [n5,s5,r33/?:/Table/53/1/1{00-20}] applied preemptive snapshot in 4ms [clear=0ms batch=0ms entries=2ms commit=1ms]
I180526 06:59:13.550890 80513 storage/replica_raftstorage.go:520  [n1,s1,r36/1:/Table/53/1/1{60-80}] generated preemptive snapshot eba8bcf5 at index 18
I180526 06:59:13.559987 74225 storage/replica_proposal.go:202  [n2,s2,r32/2:/Table/53/1/{80-100}] new range lease repl=(n2,s2):2 seq=4 start=1527317952.512450770,0 epo=1 pro=1527317952.512468861,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
I180526 06:59:13.842369 80349 storage/replica_command.go:1777  [replicate,n1,s1,r32/1:/Table/53/1/{80-100}] change replicas (ADD_REPLICA (n3,s3):4): read existing descriptor r32:/Table/53/1/{80-100} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:13.890435 80505 storage/replica_command.go:1777  [n1,s1,r29/1:/Table/53/1/{20-40}] change replicas (ADD_REPLICA (n3,s3):4): read existing descriptor r29:/Table/53/1/{20-40} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:13.900115 80508 storage/replica_command.go:1777  [n1,s1,r33/1:/Table/53/1/1{00-20}] change replicas (ADD_REPLICA (n5,s5):4): read existing descriptor r33:/Table/53/1/1{00-20} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:13.924285 80510 storage/store_snapshot.go:605  [n1,s1,r35/1:/Table/53/1/1{40-60}] streamed snapshot to (n5,s5):?: kv pairs: 28, log entries: 7, rate-limit: 2.0 MiB/sec, 374ms
I180526 06:59:13.965087 80519 storage/replica_command.go:1777  [n4,s4,r31/3:/Table/53/1/{60-80}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r31:/Table/53/1/{60-80} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n5,s5):4, next=5]
I180526 06:59:14.090672 80573 storage/replica_raftstorage.go:730  [n5,s5,r35/?:{-}] applying preemptive snapshot at index 17 (id=f7002b9d, encoded size=5391, 1 rocksdb batches, 7 log entries)
I180526 06:59:14.348831 80587 storage/replica_raftstorage.go:520  [n1,s1,r37/1:/Table/5{3/1/180-4}] generated preemptive snapshot 88ee8b9b at index 10
I180526 06:59:14.443951 80573 storage/replica_raftstorage.go:736  [n5,s5,r35/?:/Table/53/1/1{40-60}] applied preemptive snapshot in 353ms [clear=0ms batch=0ms entries=1ms commit=287ms]
I180526 06:59:14.697377 74907 storage/replica_proposal.go:202  [n4,s4,r34/3:/Table/53/1/1{20-40}] new range lease repl=(n4,s4):3 seq=4 start=1527317953.613081791,0 epo=1 pro=1527317953.613099447,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
W180526 06:59:14.841713 73637 storage/store.go:3633  [n1,s1] handle raft ready: 0.5s [processed=0]
I180526 06:59:14.850749 80587 storage/store_snapshot.go:605  [n1,s1,r37/1:/Table/5{3/1/180-4}] streamed snapshot to (n3,s3):?: kv pairs: 27, log entries: 0, rate-limit: 2.0 MiB/sec, 425ms
I180526 06:59:14.871258 80689 storage/replica_raftstorage.go:730  [n3,s3,r37/?:{-}] applying preemptive snapshot at index 10 (id=88ee8b9b, encoded size=2890, 1 rocksdb batches, 0 log entries)
I180526 06:59:14.992411 80689 storage/replica_raftstorage.go:736  [n3,s3,r37/?:/Table/5{3/1/180-4}] applied preemptive snapshot in 2ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I180526 06:59:15.014744 74264 storage/replica_proposal.go:202  [n2,s2,r30/2:/Table/53/1/{40-60}] new range lease repl=(n2,s2):2 seq=6 start=1527317954.668792213,0 epo=1 pro=1527317954.668860253,0 following repl=(n4,s4):3 seq=5 start=1527317952.541640477,0 epo=1 pro=1527317952.541656115,0
I180526 06:59:15.123508 74223 storage/replica_proposal.go:202  [n2,s2,r36/2:/Table/53/1/1{60-80}] new range lease repl=(n2,s2):2 seq=4 start=1527317954.082981111,0 epo=1 pro=1527317954.082997319,0 following repl=(n1,s1):1 seq=3 start=1527317878.215532986,0 epo=1 pro=1527317886.780181338,0
W180526 06:59:15.133421 75357 storage/node_liveness.go:528  [n5,hb] slow heartbeat took 1.6s
I180526 06:59:15.273612 80510 storage/replica_command.go:1777  [n1,s1,r35/1:/Table/53/1/1{40-60}] change replicas (ADD_REPLICA (n5,s5):4): read existing descriptor r35:/Table/53/1/1{40-60} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:15.449453 80587 storage/replica_command.go:1777  [n1,s1,r37/1:/Table/5{3/1/180-4}] change replicas (ADD_REPLICA (n3,s3):4): read existing descriptor r37:/Table/5{3/1/180-4} [(n1,s1):1, (n2,s2):2, (n4,s4):3, next=4]
I180526 06:59:15.916879 80736 storage/replica_command.go:1777  [replicate,n4,s4,r26/3:/Table/5{4-5}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r26:/Table/5{4-5} [(n1,s1):1, (n2,s2):2, (n4,s4):3, (n5,s5):4, next=5]
W180526 06:59:16.084255 74290 storage/node_liveness.go:528  [n2,hb] slow heartbeat took 2.2s
I180526 06:59:16.353628 74185 server/status/runtime.go:219  [n2] runtime stats: 1.7 GiB RSS, 1156 goroutines, 92 MiB/17 MiB/140 MiB GO alloc/idle/total, 171 MiB/211 MiB CGO alloc/total, 769.16cgo/sec, 1.07/0.07 %(u/s)time, 0.02 %gc (4x)
I180526 06:59:16.427785 80855 storage/store.go:2492  [replicaGC,n3,s3,r32/?:/Table/53/1/{80-100}] removing replica
I180526 06:59:16.460107 80855 storage/replica.go:829  [replicaGC,n3,s3,r32/?:/Table/53/1/{80-100}] removed 26 (20+6) keys in 1ms [clear=0ms commit=1ms]
W180526 06:59:16.492549 74557 storage/node_liveness.go:528  [n3,hb] slow heartbeat took 1.5s
W180526 06:59:16.548889 73681 storage/node_liveness.go:528  [n1,hb] slow heartbeat took 1.6s
I180526 06:59:16.656269 80833 storage/replica.go:3320  [n5,s5,r18/3:/Table/2{1-2}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n3,s3):4 (n2,s2):2 (n5,s5):3] next=5
F180526 06:59:16.852384 74964 storage/replica.go:3873  [n4,s4,r26/3:/Table/5{4-5}] while purging index %d: while purging index 32: IO error: /go/src/github.com/cockroachdb/cockroach/artifacts/cockroach-auxiliary985447670/sideloading/26/26.3/i32.t9: File not found
goroutine 74964 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x54b0301, 0xc4204c0240, 0x0, 0x12)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:956 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x4ae9680, 0xc400000004, 0x454828d, 0x12, 0xf21, 0xc422d800d0, 0xd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:834 +0x7f4
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x37c4140, 0xc4222091a0, 0xc400000004, 0x2, 0x3228e05, 0x6, 0xc420b35b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x31e
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x37c4140, 0xc4222091a0, 0x1, 0x4, 0x3228e05, 0x6, 0xc420b35b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x37c4140, 0xc4222091a0, 0x3228e05, 0x6, 0xc420b35b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:149 +0x90
github.com/cockroachdb/cockroach/pkg/storage.fatalOnRaftReadyErr(0x37c4140, 0xc4222091a0, 0x3249a5d, 0x16, 0x379e800, 0xc4213a2f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3873 +0x219
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x37c4140, 0xc4222091a0, 0xc42104a380, 0x37c4140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3576 +0x215
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc4236f4000, 0x37c4140, 0xc4222091a0, 0xc424f1ac08, 0xc420b35e98, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3178 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4236f4000, 0x37c4140, 0xc421540b70, 0x1a)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3563 +0x302
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4246b2120, 0x37c4140, 0xc421540b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x2f2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc421540b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424b623b0, 0xc42577e2d0, 0xc424b623a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1 [chan receive, 1 minutes]:
testing.(*T).Run(0xc420236960, 0x324a45f, 0x17, 0x3354f88, 0xc420683c01)
	/usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc420236960)
	/usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc420236960, 0xc420683d68)
	/usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc4204b2320, 0x488aac0, 0x12, 0x12, 0xbeba5e9aa565e6a7)
	/usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc4206da180, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestMain(0xc4206da180)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/main_test.go:31 +0x180
main.main()
	_testmain.go:82 +0x22b

goroutine 6 [syscall, 12 minutes]:
os/signal.signal_recv(0x90ab41)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 15 [semacquire]:
sync.runtime_SemacquireMutex(0x4ae96d8, 0x1100000000)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0x4ae96d4)
	/usr/local/go/src/sync/mutex.go:134 +0x172
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*Mutex).Lock(0x4ae96d4)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/mutex_sync.go:32 +0x3b
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1163 +0x160
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x1b5

goroutine 16 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1cd

goroutine 34 [select, 12 minutes, locked to thread]:
runtime.gopark(0x335a528, 0x0, 0x322a599, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc42053ef50, 0xc420518060)
	/usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 50 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc4203b8d40)
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:144 +0x18f
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x9b

goroutine 76438 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc425b783c0, 0x37da000, 0xc4202ed3e0, 0x37a6f40, 0xc425b783c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:359 +0x2db
github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x31dd4c0, 0xc425b783c0, 0x37d4540, 0xc4216e8f20, 0x37c4140, 0xc4211f53e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:415 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x31dd4c0, 0xc425b783c0, 0x37d4540, 0xc4216e8f20, 0xc420364740, 0x33579d0, 0x37c4480, 0xc4206f6200)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:182 +0x16e
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc424ae2a80, 0x37dcdc0, 0xc420581c00, 0xc4205c5100, 0xc420a7b920, 0x48803a0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1167 +0x995
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc424ae2a80, 0x37dcdc0, 0xc420581c00, 0xc4205c5100, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1253 +0x12f9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc425b19df0, 0xc424ae2a80, 0x37dcdc0, 0xc420581c00, 0xc4205c5100)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:680 +0xad
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:678 +0xb9

goroutine 74990 [semacquire]:
sync.runtime_notifyListWait(0xc422074a10, 0x70f7801400000af9)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc422074a00)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4246b2120, 0x37c4140, 0xc421541dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc421541dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424b627f0, 0xc42577e2d0, 0xc424b627d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 73985 [semacquire, 1 minutes]:
sync.runtime_SemacquireMutex(0xc4227d3a44, 0x900000000)
	/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc4227d3a40)
	/usr/local/go/src/sync/mutex.go:134 +0x172
sync.(*Once).Do(0xc4227d3a40, 0xc421a3a750)
	/usr/local/go/src/sync/once.go:40 +0x63
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func15(0x37c4140, 0xc424f88ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1388 +0xb9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42039c7f0, 0xc421c35d40, 0xc4204fc280)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74838 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc4240a7860, 0xc423b2f050, 0x5, 0x5, 0xc425a6c4e0, 0x8e79c0, 0x29a1010)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:142 +0x19c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4240a7860, 0xc423b2f050, 0x5, 0x5, 0x8e6c7c, 0xc425a6c4e0, 0xc4216378c8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:131 +0x82
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc421137230, 0xc423b2f050, 0x5, 0x5, 0xfe8bb3, 0xc425a6c4e0, 0x1fffe000022673)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:394 +0x71
io.ReadAtLeast(0x379efa0, 0xc421137230, 0xc423b2f050, 0x5, 0x5, 0x5, 0xc4216379a0, 0x10113c9, 0xc4220e2000)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x379efa0, 0xc421137230, 0xc423b2f050, 0x5, 0x5, 0xc423f913a0, 0x1, 0x15321f96f26002cc)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4211f6d00, 0xc423b2f050, 0x5, 0x5, 0x50, 0xc42228c780, 0xc421637b08)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:378 +0x15d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc423b2f040, 0x7fffffff, 0xb, 0x0, 0xc423bf9500, 0xc42160fb00, 0x0, 0xe)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:452 +0x99
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc423b2f040, 0x7f44405c3bc0, 0x54b06d0, 0xc4211f6d00, 0x0, 0x0, 0x30b44a0, 0xc42228c7d0, 0x7fffffff, 0xc426c68960, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:578 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc42260c4e0, 0x30b44a0, 0xc42228c7d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:539 +0x27c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc420ab1800, 0x30b44a0, 0xc42228c7d0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:405 +0x67
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc424b62860, 0x37c4140, 0xc420ac9170, 0xc423bfa780)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:191 +0x83
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x37d97c0, 0xc424b62860, 0xc420b01a00, 0x37c4140, 0xc420ac9170, 0xc423bfa780, 0xc4245fa200, 0x30)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x43
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x37c4140, 0xc420ac9170)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xe6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424b628f0, 0xc42577e2d0, 0xc420b06440)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75221 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc425dd6140, 0xc420d38c30, 0xc4203df880, 0xf, 0xc42493d7a0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x37c4140, 0xc420ffc7e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202d2000, 0xc421b346c0, 0xc420ffc6f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74271 [semacquire]:
sync.runtime_notifyListWait(0xc421142710, 0xd7c8b14500000b16)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421142700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42577e120, 0x37c4140, 0xc421885290)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc421885290)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203845b0, 0xc421c35d40, 0xc4203845a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 73630 [semacquire]:
sync.runtime_notifyListWait(0xc423103710, 0x1eeb4b76000014a1)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc423103700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424ef3320, 0x37c4140, 0xc42242a630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc42242a630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f78550, 0xc42556b170, 0xc420f78540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75111 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc426cd5500, 0xc420d88e00, 0x37c4140, 0xc42537b920)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:321 +0x120
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0x1200

goroutine 75273 [IO wait]:
internal/poll.runtime_pollWait(0x7f4444962610, 0x72, 0x489fd58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc420d89018, 0x72, 0x37a1500, 0x489fd58, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc420d89018, 0xc425606000, 0x2000, 0x2000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc420d89000, 0xc425606000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc420d89000, 0xc425606000, 0x2000, 0x2000, 0xc4218ee4a0, 0x67, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc422fe64c0, 0xc425606000, 0x2000, 0x2000, 0x5a, 0x1ff3, 0x335a408)
	/usr/local/go/src/net/net.go:176 +0x85
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc424436460, 0xc425606000, 0x2000, 0x2000, 0xc42033eb40, 0xc4225776b0, 0xbdcc7a)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x241
crypto/tls.(*block).readFromUntil(0xc420779b60, 0x7f44410257d8, 0xc424436460, 0x5, 0xc424436460, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc4218ee380, 0x335a617, 0xc4218ee4a0, 0xc42560600d)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc4218ee380, 0xc424b98000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc4254a9aa0, 0xc42029b6f8, 0x9, 0x9, 0x92aade, 0xc421b55d48, 0x90bd4c)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x379b140, 0xc4254a9aa0, 0xc42029b6f8, 0x9, 0x9, 0x9, 0xc42052b400, 0x60, 0x0)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x379b140, 0xc4254a9aa0, 0xc42029b6f8, 0x9, 0x9, 0x51, 0x51, 0x90bc22)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc42029b6f8, 0x9, 0x9, 0x379b140, 0xc4254a9aa0, 0x0, 0x0, 0x51, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42029b6c0, 0xc4261b8690, 0xc4261b8690, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4216ba200, 0xc425784e40, 0x3359790)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:427 +0xd6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc421a55500, 0x37dcdc0, 0xc4216ba200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:676 +0x16f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:631 +0x54
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4238c5ac0, 0xc421a55500, 0x7f4444915740, 0xc4216ba200)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:639 +0x35
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0x7c7

goroutine 75365 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc42360e780)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x205
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x4d

goroutine 74122 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc42379a2c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc42379a2c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 75089 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).waitOnHeader(0xc422557800, 0x8bc2d2, 0xc421414900)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:240 +0x155
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).RecvCompress(0xc422557800, 0xc421414900, 0xc420a6d5e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:251 +0x3d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc420ac2b60, 0x30c2200, 0xc421a40c90, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:525 +0xadb
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc4206e2d80, 0x30c2200, 0xc421a40c90, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:405 +0x67
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc4236f6de0, 0xc420d3f000, 0xc420a6d6a8, 0xa4de23)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:366 +0x83
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x37d9fa0, 0xc4236f6de0, 0xc420376b80, 0xc425b783c0, 0x37c4140, 0xc421a40c60, 0x8d627d, 0xc4219cec80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:491 +0x5a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x37c4140, 0xc421a40c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:506 +0x7e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4236f6e20, 0xc42577e2d0, 0xc4239ac960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74944 [IO wait]:
internal/poll.runtime_pollWait(0x7f4441069f08, 0x72, 0x489fd58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc421ea9118, 0x72, 0x37a1500, 0x489fd58, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc421ea9118, 0xc4218f1400, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc421ea9100, 0xc4218f1400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc421ea9100, 0xc4218f1400, 0x400, 0x400, 0xc41fd54d3b, 0xc41fd54d3a, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc420b244b0, 0xc4218f1400, 0x400, 0x400, 0xc4209eb6f0, 0x8b722d, 0xc42003a000)
	/usr/local/go/src/net/net.go:176 +0x85
crypto/tls.(*block).readFromUntil(0xc42113b0b0, 0x7f44405c3be8, 0xc420b244b0, 0x5, 0xc420b244b0, 0x7f4444a72000)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc422db9880, 0x335a617, 0xc422db99a0, 0x37)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc422db9880, 0xc422034000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc420876660, 0xc4211b3378, 0x9, 0x9, 0x0, 0xc422d5bb50, 0xc4209ebc58)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x379b140, 0xc420876660, 0xc4211b3378, 0x9, 0x9, 0x9, 0x8baa8a, 0xc4204d6000, 0x10)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x379b140, 0xc420876660, 0xc4211b3378, 0x9, 0x9, 0xc420cd7b08, 0x2, 0x90bc22)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4211b3378, 0x9, 0x9, 0x379b140, 0xc420876660, 0x0, 0x0, 0xc422208b40, 0x37a8f00)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4211b3340, 0xc420f68c60, 0xc420f68c00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420b158c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x1fe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x10fc

goroutine 74261 [semacquire]:
sync.runtime_notifyListWait(0xc421142710, 0xd7c8b14500000af3)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421142700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42577e120, 0x37c4140, 0xc4215a9b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc4215a9b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420384440, 0xc421c35d40, 0xc420384430)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74904 [semacquire]:
sync.runtime_notifyListWait(0xc422074a10, 0x1cf8fd5600000b04)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc422074a00)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4246b2120, 0x37c4140, 0xc426341ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc426341ad0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424b62230, 0xc42577e2d0, 0xc424b62220)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 73619 [semacquire]:
sync.runtime_notifyListWait(0xc423103710, 0x5091b19000001499)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc423103700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424ef3320, 0x37c4140, 0xc42242a270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc42242a270)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f78370, 0xc42556b170, 0xc420f78340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75216 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x37c4140, 0xc420ffc3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1161 +0x5a5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b7bd20, 0xc421b346c0, 0xc423b7bd10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74070 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc421f1e600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:869 +0x31d
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:281 +0x1651

goroutine 74893 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x37c4140, 0xc42242b6b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:321 +0x6c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423ef5d20, 0xc421b346c0, 0xc424720e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74684 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x37c4140, 0xc4269b1200)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420446e00, 0xc42577e2d0, 0xc420446df0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 73585 [semacquire]:
sync.runtime_notifyListWait(0xc423103710, 0x997bcd4d00001498)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc423103700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424ef3320, 0x37c4140, 0xc424f88120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc424f88120)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030bd80, 0xc42556b170, 0xc42030bd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74733 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x37c4140, 0xc4215406f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420358470, 0xc42577e2d0, 0xc420358460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 86 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208410e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 87 [chan receive, 12 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc420841200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 75407 [semacquire]:
sync.runtime_notifyListWait(0xc4212add90, 0x196282ff000006a5)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4212add80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42242e900, 0x37c4140, 0xc420d4ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc420d4ab40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42039c470, 0xc421b346c0, 0xc42039c460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75193 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x37c4140, 0xc421512d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b7b6a0, 0xc421b346c0, 0xc423b7b690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74472 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc42421af00, 0xc42087c400, 0x37c4140, 0xc4217eadb0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:321 +0x120
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0x1200

goroutine 75196 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x37c4140, 0xc4217871a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b7b720, 0xc421b346c0, 0xc423b7b710)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74825 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4231a62c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc4231a62c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 75400 [runnable]:
sync.runtime_notifyListWait(0xc4212add90, 0x196282ff00000691)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc4212add80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42242e900, 0x37c4140, 0xc42360fe60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc42360fe60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc424b63e30, 0xc421b346c0, 0xc424b63e20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75099 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc424a81700)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1d4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x294

goroutine 74209 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc421eeeee0, 0xc420050128, 0x2f0fec0, 0x4871a10, 0x31e39e0)
	<autogenerated>:1 +0x80
net/http.(*Server).Serve(0xc420bb8c30, 0x37c1880, 0xc421eeeee0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2770 +0x1f6
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x37c4140, 0xc420efaff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1101 +0x146
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420368860, 0xc423b78090, 0xc420a7a000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74330 [semacquire, 1 minutes]:
sync.runtime_notifyListWait(0xc421cfaae8, 0x0)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421cfaad8)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc421cfa9c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:648 +0xb3
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:635 +0x607

goroutine 75360 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc421629340)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc421629340)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 74242 [semacquire]:
sync.runtime_notifyListWait(0xc421142710, 0xd7c8b14500000aec)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421142700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42577e120, 0x37c4140, 0xc4215a95c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc4215a95c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420384100, 0xc421c35d40, 0xc4203840f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75203 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x37c4140, 0xc4215130e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc423b7b8b0, 0xc421b346c0, 0xc42031a8e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74298 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x37c4140, 0xc4217ebf50)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f790c0, 0xc421c35d40, 0xc4247212c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75013 [semacquire]:
sync.runtime_notifyListWait(0xc422074a10, 0x6a466b3a00000aed)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc422074a00)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4246b2120, 0x37c4140, 0xc4216ea8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc4216ea8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420462c80, 0xc42577e2d0, 0xc420462c70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 75014 [semacquire]:
sync.runtime_notifyListWait(0xc422074a10, 0x408b9edc00000b01)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc422074a00)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4246b2120, 0x37c4140, 0xc4216ea8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc4216ea8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420462ca0, 0xc42577e2d0, 0xc420462c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 73633 [semacquire]:
sync.runtime_notifyListWait(0xc423103710, 0xcff358fb000014a2)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc423103700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc424ef3320, 0x37c4140, 0xc42242a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc42242a5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f785b0, 0xc42556b170, 0xc420f785a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74473 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).read(0xc42639b5e0, 0xc420311590, 0x5, 0x5, 0xc420378fa0, 0x8e79c0, 0x29a1010)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:142 +0x19c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42639b5e0, 0xc420311590, 0x5, 0x5, 0x8e6c7c, 0xc420378fa0, 0xc4209e18c8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:131 +0x82
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc4217eaed0, 0xc420311590, 0x5, 0x5, 0xfe8bb3, 0xc420378fa0, 0x1fffe000026897)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:394 +0x71
io.ReadAtLeast(0x379efa0, 0xc4217eaed0, 0xc420311590, 0x5, 0x5, 0x5, 0xc4209e19a0, 0x10113c9, 0xc421cffd40)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x379efa0, 0xc4217eaed0, 0xc420311590, 0x5, 0x5, 0x17, 0x20, 0x15321f96f26002cc)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc420dd4100, 0xc420311590, 0x5, 0x5, 0xc420f06728, 0xc421a0ead0, 0xc4216d2340)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:378 +0x15d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc420311580, 0x7fffffff, 0xb, 0x0, 0xc420bc9e00, 0xc420a5a000, 0x0, 0xe)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:452 +0x99
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc420311580, 0x7f44405c3bc0, 0x54b06d0, 0xc420dd4100, 0x0, 0x0, 0x30b44a0, 0xc425565ef0, 0x7fffffff, 0xc4253b5a40, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:578 +0x61
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc422708c30, 0x30b44a0, 0xc425565ef0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:539 +0x27c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42087c400, 0x30b44a0, 0xc425565ef0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:405 +0x67
github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc4236f68d0, 0x37c4140, 0xc4217eafc0, 0xc4222d4c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:191 +0x83
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x37d97c0, 0xc4236f68d0, 0xc4237a3930, 0x37c4140, 0xc4217eafc0, 0xc4222d4c00, 0xc4210276c0, 0x30)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x43
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x37c4140, 0xc4217eafc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xe6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4236f6950, 0xc423b78090, 0xc421ff0340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74706 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc425cf7540, 0xc424f2c910, 0xc423c05500, 0xf, 0xc4231c01e0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x37c4140, 0xc420efa120)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4227983d0, 0xc42556b170, 0xc42491ad80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74313 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x37c4140, 0xc421885470)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420384790, 0xc421c35d40, 0xc420c64690)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 74232 [semacquire]:
sync.runtime_notifyListWait(0xc421142710, 0x948a4c3a00000b02)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421142700)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc42577e120, 0x37c4140, 0xc4215a93e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x37c4140, 0xc4215a93e0)
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone May 26, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 26, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b5edd2a76f1cdea7c24ce97ce27d663a02fadfbb

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=686572&tab=buildLog

@tbg tbg assigned windchan7 and unassigned dt May 30, 2018
@tbg
Copy link
Member

tbg commented May 30, 2018

@windchan7 this was recently introduced by you, could you send a fix? The problem is that the error you get back isn't os.IsNotExist because it's produced by RocksDB. You need a "smarter" way to find out existence of the file. Ideally DeleteFile can be changed to return the right error (and this should be noted on the interface). We also need a unit test (which is clearly missing today).

cockroach/pkg/storage/replica_sideload_disk.go

Lines 117 to 121 in b5edd2a

if err := ss.eng.DeleteFile(filename); err != nil {
if os.IsNotExist(err) {
return errSideloadedFileNotFound
}
return err

@tbg tbg mentioned this issue May 30, 2018
Closed
@tbg tbg added the A-kv-client Relating to the KV client and the KV interface. label May 30, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/89690e007f3f4bab973a107754f471d140d5373f

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=687188&tab=buildLog

@windchan7
Copy link
Contributor

windchan7 commented May 30, 2018 •
edited
Loading

Yes, I believe I had a sanity check against the returned error from RocksDB, here:https://github.com/cockroachdb/cockroach/blob/master/pkg/storage/engine/rocksdb.go#L2657. If the error on the RocksDB side is not found error, I will return No such file or directory status back to RocksDB's DeleteFile, which accordingly return os.ErrNotExist.

Anyway, I will see what's going on.

@andreimatei
Copy link
Contributor

Reproed in ~1h on a gce worker with

make stressrace PKG=./pkg/ccl/backupccl/ TESTS=TestBackupRestoreNegativePrimaryKey STRESSFLAGS='-ignore "race detected" -maxfails=20 -p 20' 2>&1 | tee ~/out

@windchan7 windchan7 mentioned this issue May 30, 2018
Merged
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/4bcf507fc2c930f602d09e3561a1cb728223096b

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=689041&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/81c8044c7a19281610180527d0cefc4094673fa0

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=689660&tab=buildLog

craig bot pushed a commit that referenced this issue May 31, 2018
@windchan7
Merge #26259
29c0135 
26259: storage: fix RocksDB error and go os error discrepancy r=tschottdorf a=windchan7

Because error returned from RocksDB is different from go's os error, we need to
bridge the gap by examining error message.

Fixes #26121 
Release note: None

Co-authored-by: Victor Chen <[email protected]>
@craig craig bot closed this as completed in #26259 May 31, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@windchan7 windchan7

Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
5 participants
@dt @andreimatei @tbg @cockroach-teamcity @windchan7