Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: testrace/TestImportData, test/TestImportData #29106

Closed
cockroach-teamcity opened this issue Aug 27, 2018 · 8 comments
Closed

teamcity: failed tests on master: testrace/TestImportData, test/TestImportData #29106

cockroach-teamcity opened this issue Aug 27, 2018 · 8 comments
Assignees
@nvanbenschoten
Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#863418:

--- FAIL: test/TestImportData/CSV:_good_bytes_encoding (0.000s)
Test ended in panic.

------- Stdout: -------
I180827 12:31:47.918364 665 sql/event_log.go:126  [n1,client=127.0.0.1:34184,user=root] Event: "create_database", target: 63, info: {DatabaseName:d5 Statement:CREATE DATABASE d5 User:root}
I180827 12:31:47.924067 1443 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r36]
I180827 12:31:47.950600 1465 ccl/importccl/read_import_proc.go:74  [n1,import-distsql] could not fetch file size; falling back to per-file progress: bad ContentLength: -1
I180827 12:31:47.975532 1324 ccl/importccl/read_import_proc.go:74  [n1,import-distsql] could not fetch file size; falling back to per-file progress: bad ContentLength: -1
I180827 12:31:47.981787 1414 storage/replica_command.go:298  [n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/65 [r37]
W180827 12:31:47.982279 320 sql/schema_changer.go:1304  [n1] Error executing schema change: descriptor not found
W180827 12:31:48.489355 320 sql/schema_changer.go:1304  [n1] Error executing schema change: descriptor not found
W180827 12:31:49.506035 320 sql/schema_changer.go:1304  [n1] Error executing schema change: descriptor not found
W180827 12:31:49.506236 320 sql/schema_changer.go:1304  [n1] Error executing schema change: descriptor not found
I180827 12:31:55.939820 275 storage/replica_proposal.go:214  [n1,s1,r35/1:/Table/62{-/2/"STRI…}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373115.939272759,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:56.096446 137 storage/replica_proposal.go:214  [n1,s1,r13/1:/Table/1{6-7}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373116.095893165,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:57.090113 317 server/status/runtime.go:433  [n1] runtime stats: 168 MiB RSS, 255 goroutines, 19 MiB/1.9 MiB/33 MiB GO alloc/idle/total, 27 MiB/34 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (15x)
I180827 12:31:57.096649 226 storage/replica_proposal.go:214  [n1,s1,r25/1:/Table/5{4-5}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373117.096251458,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:57.124608 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:31:57.175081 227 storage/replica_proposal.go:214  [n1,s1,r5/1:/System/ts{d-e}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373117.174498421,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:57.175752 278 storage/replica_proposal.go:214  [n1,s1,r4/1:/System/{NodeLive…-tsd}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373117.172768849,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:58.097320 237 storage/replica_proposal.go:214  [n1,s1,r31/1:/Table/6{0-1}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373118.096735480,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:31:59.097619 244 storage/replica_proposal.go:214  [n1,s1,r33/1:/{Table/62/2/"…-Max}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373119.097109076,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:32:07.090620 317 server/status/runtime.go:433  [n1] runtime stats: 172 MiB RSS, 257 goroutines, 14 MiB/6.9 MiB/36 MiB GO alloc/idle/total, 27 MiB/34 MiB CGO alloc/total, 51.90cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:32:07.124533 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:32:17.089817 317 server/status/runtime.go:433  [n1] runtime stats: 174 MiB RSS, 256 goroutines, 20 MiB/1.8 MiB/36 MiB GO alloc/idle/total, 28 MiB/36 MiB CGO alloc/total, 76.71cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:32:17.091184 244 storage/replica_proposal.go:214  [n1,s1,r7/1:/Table/{SystemCon…-11}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373137.090726349,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:32:17.093721 241 storage/replica_proposal.go:214  [n1,s1,r12/1:/Table/1{5-6}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373137.093355270,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:32:17.126759 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:32:19.116024 259 storage/replica_proposal.go:214  [n1,s1,r6/1:/{System/tse-Table/System…}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373139.115508594,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:32:27.090446 317 server/status/runtime.go:433  [n1] runtime stats: 175 MiB RSS, 255 goroutines, 16 MiB/7.6 MiB/38 MiB GO alloc/idle/total, 28 MiB/36 MiB CGO alloc/total, 42.10cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:32:27.124537 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:32:37.089759 317 server/status/runtime.go:433  [n1] runtime stats: 176 MiB RSS, 255 goroutines, 22 MiB/2.5 MiB/38 MiB GO alloc/idle/total, 28 MiB/36 MiB CGO alloc/total, 33.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:32:37.124568 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:32:47.089650 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:32:47.090736 317 server/status/runtime.go:433  [n1] runtime stats: 176 MiB RSS, 255 goroutines, 16 MiB/8.6 MiB/39 MiB GO alloc/idle/total, 28 MiB/36 MiB CGO alloc/total, 28.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:32:47.124561 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:32:47.924440 1443 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:32:47.924603 1674 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r38]
I180827 12:32:57.089858 317 server/status/runtime.go:433  [n1] runtime stats: 176 MiB RSS, 256 goroutines, 13 MiB/12 MiB/39 MiB GO alloc/idle/total, 28 MiB/37 MiB CGO alloc/total, 25.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:32:57.124578 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:32:59.098313 1562 storage/consistency_queue.go:125  [consistencyChecker,n1,s1,r33/1:/{Table/62/2/"…-Max}] context deadline exceeded
E180827 12:32:59.098342 1562 storage/queue.go:788  [consistencyChecker,n1,s1,r33/1:/{Table/62/2/"…-Max}] context deadline exceeded
I180827 12:32:59.099029 242 storage/replica_proposal.go:214  [n1,s1,r10/1:/Table/1{3-4}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373179.098541204,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:33:07.090532 317 server/status/runtime.go:433  [n1] runtime stats: 176 MiB RSS, 255 goroutines, 19 MiB/6.3 MiB/39 MiB GO alloc/idle/total, 28 MiB/37 MiB CGO alloc/total, 48.40cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:33:07.128420 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:33:17.090171 317 server/status/runtime.go:433  [n1] runtime stats: 179 MiB RSS, 256 goroutines, 14 MiB/10 MiB/39 MiB GO alloc/idle/total, 30 MiB/38 MiB CGO alloc/total, 19.50cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:33:17.129277 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:33:27.090308 317 server/status/runtime.go:433  [n1] runtime stats: 181 MiB RSS, 255 goroutines, 20 MiB/5.0 MiB/39 MiB GO alloc/idle/total, 30 MiB/38 MiB CGO alloc/total, 26.20cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:33:27.124580 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:33:37.090363 317 server/status/runtime.go:433  [n1] runtime stats: 181 MiB RSS, 255 goroutines, 15 MiB/10 MiB/39 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 20.20cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:33:37.124582 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:33:47.088875 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:33:47.090478 317 server/status/runtime.go:433  [n1] runtime stats: 181 MiB RSS, 255 goroutines, 20 MiB/4.9 MiB/39 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 19.20cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:33:47.124557 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:33:47.911322 1658 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
I180827 12:33:47.911401 1658 storage/queue.go:873  [n1,merge] purgatory is now empty
E180827 12:33:47.927839 1674 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:33:47.927994 1779 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r39]
I180827 12:33:57.094559 317 server/status/runtime.go:433  [n1] runtime stats: 181 MiB RSS, 257 goroutines, 16 MiB/9.6 MiB/40 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 26.09cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:33:57.125352 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:34:07.090283 317 server/status/runtime.go:433  [n1] runtime stats: 181 MiB RSS, 257 goroutines, 22 MiB/4.6 MiB/40 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 22.81cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:34:07.124598 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:34:17.077358 237 storage/replica_proposal.go:214  [n1,s1,r8/1:/Table/1{1-2}] new range lease repl=(n1,s1):1 seq=3 start=1535373107.086003437,0 epo=1 pro=1535373257.076947291,0 following repl=(n1,s1):1 seq=2 start=1535373107.086003437,0 exp=1535373116.086712670,0 pro=1535373107.086731875,0
I180827 12:34:17.090439 317 server/status/runtime.go:433  [n1] runtime stats: 182 MiB RSS, 257 goroutines, 14 MiB/13 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 37.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:34:17.124595 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:34:27.090257 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 257 goroutines, 20 MiB/7.7 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 20.70cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:34:27.124602 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:34:37.090494 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 257 goroutines, 14 MiB/12 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 21.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:34:37.124563 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:34:47.090157 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 257 goroutines, 20 MiB/7.0 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 22.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:34:47.091101 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:34:47.124589 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:34:47.911714 1762 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
E180827 12:34:47.928304 1779 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:34:47.928492 1850 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r40]
I180827 12:34:57.105845 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 259 goroutines, 14 MiB/13 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 26.66cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:34:57.124584 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:35:07.090266 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 259 goroutines, 20 MiB/7.8 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 19.23cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:35:07.129279 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:35:17.090383 317 server/status/runtime.go:433  [n1] runtime stats: 183 MiB RSS, 259 goroutines, 14 MiB/13 MiB/41 MiB GO alloc/idle/total, 30 MiB/39 MiB CGO alloc/total, 21.20cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:35:17.128065 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:35:27.097811 317 server/status/runtime.go:433  [n1] runtime stats: 185 MiB RSS, 259 goroutines, 19 MiB/8.0 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 22.98cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:35:27.124570 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:35:37.089799 317 server/status/runtime.go:433  [n1] runtime stats: 185 MiB RSS, 259 goroutines, 15 MiB/12 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 21.52cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:35:37.124843 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:35:47.088881 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:35:47.090650 317 server/status/runtime.go:433  [n1] runtime stats: 187 MiB RSS, 259 goroutines, 20 MiB/6.8 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 21.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:35:47.124530 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:35:47.912080 1813 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
E180827 12:35:47.928723 1850 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:35:47.928892 1904 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r41]
I180827 12:35:57.090112 317 server/status/runtime.go:433  [n1] runtime stats: 187 MiB RSS, 261 goroutines, 15 MiB/12 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 25.00cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:35:57.125200 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:36:07.090283 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 261 goroutines, 20 MiB/6.8 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 19.30cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:36:07.129330 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:36:17.091582 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 261 goroutines, 13 MiB/13 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 23.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:36:17.133307 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:36:27.089772 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 261 goroutines, 19 MiB/8.1 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 23.90cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:36:27.124538 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:36:37.090107 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 261 goroutines, 15 MiB/11 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 23.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:36:37.124542 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:36:47.088876 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:36:47.093066 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 261 goroutines, 22 MiB/5.9 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 31.89cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:36:47.124609 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:36:47.912447 1901 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
E180827 12:36:47.929113 1904 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:36:47.929258 1947 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r42]
I180827 12:36:57.090288 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 263 goroutines, 17 MiB/9.6 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 21.91cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:36:57.124583 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:37:07.095746 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 263 goroutines, 23 MiB/4.9 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 22.79cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:37:07.124546 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:37:17.090204 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 263 goroutines, 17 MiB/9.4 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 21.21cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:37:17.124550 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:37:27.093913 317 server/status/runtime.go:433  [n1] runtime stats: 188 MiB RSS, 263 goroutines, 14 MiB/12 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 25.39cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:37:27.124587 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:37:37.089665 317 server/status/runtime.go:433  [n1] runtime stats: 190 MiB RSS, 263 goroutines, 20 MiB/7.5 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 22.71cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:37:37.124588 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:37:47.088996 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:37:47.089636 317 server/status/runtime.go:433  [n1] runtime stats: 194 MiB RSS, 263 goroutines, 14 MiB/12 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 19.30cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:37:47.124545 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:37:47.912771 1986 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
E180827 12:37:47.929497 1947 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:37:47.929629 2043 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r43]
I180827 12:37:57.089985 317 server/status/runtime.go:433  [n1] runtime stats: 194 MiB RSS, 265 goroutines, 21 MiB/6.4 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 23.90cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:37:57.124549 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:38:07.090337 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 265 goroutines, 14 MiB/12 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 21.80cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:38:07.124578 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:38:17.089656 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 265 goroutines, 20 MiB/7.0 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 20.10cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:38:17.124596 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:38:27.089807 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 265 goroutines, 14 MiB/13 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 23.00cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:38:27.124584 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:38:37.089750 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 265 goroutines, 19 MiB/7.9 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 24.70cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:38:37.124710 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:38:47.089216 314 gossip/gossip.go:537  [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I180827 12:38:47.089713 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 265 goroutines, 14 MiB/13 MiB/41 MiB GO alloc/idle/total, 34 MiB/43 MiB CGO alloc/total, 19.20cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:38:47.124722 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
E180827 12:38:47.913307 1214 storage/queue.go:788  [merge,n1,s1,r35/1:/Table/62{-/2/"STRI…}] context deadline exceeded
E180827 12:38:47.929965 2043 storage/queue.go:788  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] unable to split [n1,s1,r33/1:/{Table/62/2/"…-Max}] at key "/Table/63": split at key /Table/63 failed: context deadline exceeded
I180827 12:38:47.930111 2147 storage/replica_command.go:298  [split,n1,s1,r33/1:/{Table/62/2/"…-Max}] initiating a split of this range at key /Table/63 [r44]
I180827 12:38:57.089922 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 267 goroutines, 20 MiB/7.7 MiB/41 MiB GO alloc/idle/total, 34 MiB/44 MiB CGO alloc/total, 22.60cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:38:57.124902 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:39:07.089719 317 server/status/runtime.go:433  [n1] runtime stats: 196 MiB RSS, 267 goroutines, 15 MiB/11 MiB/41 MiB GO alloc/idle/total, 34 MiB/44 MiB CGO alloc/total, 19.20cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:39:07.124710 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:39:17.089729 317 server/status/runtime.go:433  [n1] runtime stats: 195 MiB RSS, 267 goroutines, 11 MiB/14 MiB/40 MiB GO alloc/idle/total, 34 MiB/44 MiB CGO alloc/total, 33.80cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:39:17.124734 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:39:27.089661 317 server/status/runtime.go:433  [n1] runtime stats: 195 MiB RSS, 267 goroutines, 18 MiB/8.7 MiB/40 MiB GO alloc/idle/total, 34 MiB/44 MiB CGO alloc/total, 26.60cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (0x)
I180827 12:39:27.124702 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180827 12:39:37.089861 317 server/status/runtime.go:433  [n1] runtime stats: 195 MiB RSS, 267 goroutines, 13 MiB/12 MiB/40 MiB GO alloc/idle/total, 34 MiB/44 MiB CGO alloc/total, 20.20cgo/sec, 0.01/0.00 %(u/s)time, 0.00 %gc (1x)
I180827 12:39:37.124764 319 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
panic: test timed out after 8m0s

goroutine 2186 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:1240 +0xfc
created by time.goFunc
	/usr/local/go/src/time/sleep.go:172 +0x44

goroutine 1 [chan receive, 7 minutes]:
testing.(*T).Run(0xc420547950, 0x299826c, 0xe, 0x2ac5d58, 0x975a01)
	/usr/local/go/src/testing/testing.go:825 +0x301
testing.runTests.func1(0xc4205472c0)
	/usr/local/go/src/testing/testing.go:1063 +0x64
testing.tRunner(0xc4205472c0, 0xc42027bdd8)
	/usr/local/go/src/testing/testing.go:777 +0xd0
testing.runTests(0xc42051e640, 0x3fe4ac0, 0x17, 0x17, 0xbed919acc04d9c2e)
	/usr/local/go/src/testing/testing.go:1061 +0x2c4
testing.(*M).Run(0xc420184280, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x171
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestMain(0xc420184280)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/main_test.go:31 +0xda
main.main()
	_testmain.go:92 +0x151

goroutine 20 [syscall, 8 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 42 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1178 +0xf1
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:596 +0x126

goroutine 43 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:603 +0xab
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:597 +0x13e

goroutine 21 [select, 8 minutes, locked to thread]:
runtime.gopark(0x2acb5e8, 0x0, 0x29895b6, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420077f50, 0xc4200c55c0)
	/usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 47 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc42038c200)
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:144 +0x11d
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x58

goroutine 98 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2eb2180, 0xc420be4180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:101 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202eb060, 0xc4206661b0, 0xc42037e020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 26 [select]:
net.(*pipe).read(0xc420185300, 0xc4205d2000, 0x400, 0x400, 0x8, 0x8, 0x3f3)
	/usr/local/go/src/net/pipe.go:164 +0x1d7
net.(*pipe).Read(0xc420185300, 0xc4205d2000, 0x400, 0x400, 0x283ef20, 0x0, 0x7f2872b24000)
	/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*block).readFromUntil(0xc420c081e0, 0x7f2872b24000, 0xc420185300, 0x5, 0xc420185300, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x96
crypto/tls.(*Conn).readRecord(0xc420374380, 0x2acb717, 0xc4203744a0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:595 +0xe0
crypto/tls.(*Conn).Read(0xc420374380, 0xc420cb0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x100
bufio.(*Reader).Read(0xc420bf6a20, 0xc4202b1a78, 0x9, 0x9, 0xc420c06300, 0x4, 0xc420ab4d98)
	/usr/local/go/src/bufio/bufio.go:216 +0x238
io.ReadAtLeast(0x2e85b40, 0xc420bf6a20, 0xc4202b1a78, 0x9, 0x9, 0x9, 0xc420ab4e10, 0x3, 0x18)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2e85b40, 0xc420bf6a20, 0xc4202b1a78, 0x9, 0x9, 0x93ca10, 0xc4200c5380, 0xc420ab4e58)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4202b1a78, 0x9, 0x9, 0x2e85b40, 0xc420bf6a20, 0x0, 0x0, 0x1007f2872c12d90, 0xc4204dcc90)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4202b1a40, 0xc420501e60, 0xc420501e60, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc42084e000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x115
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0xb3f

goroutine 24 [select, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420a7a000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x23b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420a7a000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x1b5
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0xe1

goroutine 57 [chan receive, 7 minutes]:
testing.(*T).Run(0xc421190000, 0xc420412040, 0x18, 0xc42150c380, 0x2)
	/usr/local/go/src/testing/testing.go:825 +0x301
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestImportData(0xc420547950)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/import_stmt_test.go:596 +0x2d5b
testing.tRunner(0xc420547950, 0x2ac5d58)
	/usr/local/go/src/testing/testing.go:777 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x2e0

goroutine 58 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc4206661b0, 0xc4204b66e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:180 +0x108
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
	/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:177 +0x49

goroutine 59 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x2eb2180, 0xc4204e61b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:329 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420537d00, 0xc4206661b0, 0xc42051e160)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 60 [semacquire, 7 minutes]:
sync.runtime_notifyListWait(0xc42067e2d0, 0xc400000000)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc42067e2c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc42067e1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:654 +0x72
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:642 +0x418

goroutine 61 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6210)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddc10, 0xc4206661b0, 0xc4204ddc00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 62 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e61e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddc30, 0xc4206661b0, 0xc4204ddc20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 63 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6270)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddc50, 0xc4206661b0, 0xc4204ddc40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 64 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e62a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddc70, 0xc4206661b0, 0xc4204ddc60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 65 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e62d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddc90, 0xc4206661b0, 0xc4204ddc80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 66 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6300)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddcb0, 0xc4206661b0, 0xc4204ddca0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 67 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6330)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddcd0, 0xc4206661b0, 0xc4204ddcc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 68 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6360)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddcf0, 0xc4206661b0, 0xc4204ddce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 69 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6390)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddd10, 0xc4206661b0, 0xc4204ddd00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 70 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6240)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddd30, 0xc4206661b0, 0xc4204ddd20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 71 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e63f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddd50, 0xc4206661b0, 0xc4204ddd40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 72 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6420)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddd80, 0xc4206661b0, 0xc4204ddd70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 73 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6450)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204dddb0, 0xc4206661b0, 0xc4204ddd90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 74 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e63c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddde0, 0xc4206661b0, 0xc4204dddc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 75 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e64b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204dde10, 0xc4206661b0, 0xc4204dddf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 76 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x2eb2180, 0xc4204e6480)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:88 +0x151
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204dde30, 0xc4206661b0, 0xc4204dde20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 77 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42049cdc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 78 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc42049cf00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1230 +0x68
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:80 +0x288

goroutine 79 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x2eb2180, 0xc420be41b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1627 +0x1d1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddf10, 0xc4206661b0, 0xc4209e27b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 80 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x2eb2180, 0xc420be41e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1703 +0x160
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddf30, 0xc4206661b0, 0xc4204ddf20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 81 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x2acb950, 0xc420540000, 0xc42065a3f0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:185 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
	/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:178 +0xaf

goroutine 82 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x2eb2180, 0xc420bf0000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:101 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204ddfc0, 0xc4206661b0, 0xc42035f460)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 83 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x2eb2180, 0xc420be4210)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1117 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204741b0, 0xc4206661b0, 0xc42035f5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 84 [IO wait, 7 minutes]:
internal/poll.runtime_pollWait(0x7f2872bb6e30, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420185218, 0x72, 0xc420be8200, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420185218, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420185200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc420185200, 0x8, 0x0, 0x80)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e428, 0x911cf9, 0xc420be8230, 0xc420bcaf00)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e428, 0x2ac9288, 0xc420a58840, 0xc4202ec020, 0x0)
	/usr/local/go/src/net/tcpsock.go:259 +0x49
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420a58840, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x2eb2180, 0xc420be4240)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1129 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204741c0, 0xc4206661b0, 0xc42035f600)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 85 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42035f5c0, 0xc4200d8050, 0x2615160, 0x3fc9910, 0x29383c0)
	<autogenerated>:1 +0x5b
net/http.(*Server).Serve(0xc420bea000, 0x2eb0a40, 0xc42035f5c0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2770 +0x1a5
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x2eb2180, 0xc420be4000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1141 +0x119
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204741d0, 0xc4206661b0, 0xc4209e2a20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 86 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42035f5e0, 0xff, 0xc420bcbe88, 0xa0e88a, 0x2815320)
	<autogenerated>:1 +0x5b
crypto/tls.(*listener).Accept(0xc42035f620, 0xc4200d8050, 0x2615160, 0x3fc9910, 0x29383c0)
	/usr/local/go/src/crypto/tls/tls.go:52 +0x37
net/http.(*Server).Serve(0xc420253520, 0x2e9c380, 0xc42035f620, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2770 +0x1a5
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x2eb2180, 0xc420be4270)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1148 +0x42
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204741e0, 0xc4206661b0, 0xc42035f640)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 87 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x2eb2180, 0xc420bf0060)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1152 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204741f0, 0xc4206661b0, 0xc420a58880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 88 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc42035f4c0, 0x2aca968, 0xc42023ac40, 0x2eccaa0, 0xc4210686e0)
	<autogenerated>:1 +0x5b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc42023ac40, 0x2eb0a40, 0xc42035f4c0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:544 +0x21a
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x2eb2180, 0xc420bf0030)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1164 +0x43
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420474200, 0xc4206661b0, 0xc42035f660)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 89 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func9(0x2eb2180, 0xc420be4300)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1260 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204742e0, 0xc4206661b0, 0xc4209e2d50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 23 [select, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420bf81c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x14a
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x14f

goroutine 91 [select, 7 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc4209e2de0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x182
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f

goroutine 92 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x2eb2180, 0xc420c08000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1290 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204743b0, 0xc4206661b0, 0xc42035f960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 31 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2eb2180, 0xc420c091a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:596 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204dcce0, 0xc4206661b0, 0xc420383d00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 93 [semacquire]:
sync.runtime_notifyListWait(0xc42067e470, 0xc400000442)
	/usr/local/go/src/runtime/sema.go:510 +0x10b
sync.(*Cond).Wait(0xc42067e460)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc42067e340)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:654 +0x72
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:642 +0x418

goroutine 94 [chan receive, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2eb2180, 0xc4206c0db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:369 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b65f0, 0xc4206661b0, 0xc42038e120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 95 [select, 7 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2eb2180, 0xc4206c0e10)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1230 +0x418
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b6610, 0xc4206661b0, 0xc4202b6600)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 96 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2eb2180, 0xc4206cdbf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1262 +0x246
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b6640, 0xc4206661b0, 0xc4202b6620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:199 +0xe9
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:192 +0xad

goroutine 2096 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc420a72580, 0x2eb2180, 0xc420cfdb6

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone Aug 27, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 27, 2018
@RaduBerinde
Copy link
Member

https://teamcity.cockroachdb.com/viewLog.html?buildId=863693&buildTypeId=Cockroach_UnitTests

@couchand
Copy link
Contributor

https://teamcity.cockroachdb.com/viewLog.html?tab=buildLog&logTab=tree&filter=debug&expand=all&buildId=866558&_focus=182296

@maddyblue
Copy link
Contributor

Stuck in AdminSplit for 7 minutes (8 minute test timeout).

@maddyblue maddyblue assigned tbg and unassigned maddyblue Sep 5, 2018
@tbg tbg assigned petermattis and unassigned tbg Sep 6, 2018
@tbg
Copy link
Member

tbg commented Sep 6, 2018

@petermattis I likely won't have time to look into this today due to #29252 and will be on vacation. Could you take a look or reassign?

@petermattis
Copy link
Collaborator

I've been leaning on @nvanbenschoten regarding split problems recently. Nathan, can you take a look and triage?

@nvanbenschoten
Copy link
Member

Sure, I'll take a look.

@nvanbenschoten
Copy link
Member

@benesch, looks like the split got stuck waiting on a merge:

cockroach/pkg/storage/store.go

Lines 3192 to 3201 in 02a3f9d

if mergeCompleteCh != nil {
select {
case <-mergeCompleteCh:
// Merge complete. Retry the command.
case <-ctx.Done():
return nil, roachpb.NewError(ctx.Err())
case <-s.stopper.ShouldQuiesce():
return nil, roachpb.NewError(&roachpb.NodeUnavailableError{})
}
}

This was seen over a week ago, so there's a good chance that you've already fixed the issue. Have you observed any split/merge deadlocks that would materialize like this? Perhaps #29324 (which was not yet merged during this failure)?

@benesch
Copy link
Contributor

benesch commented Sep 6, 2018

Whoops, sorry, missed this one. Unless I'm very unlucky, this was fixed by #29324.

@benesch benesch closed this as completed Sep 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nvanbenschoten nvanbenschoten

Labels
A-disaster-recovery C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
8 participants
@maddyblue @couchand @benesch @tbg @nvanbenschoten @petermattis @RaduBerinde @cockroach-teamcity