Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github.com/cockroachdb/cockroach/pkg/ccl/importccl: TestImportCSVStmt/schema-in-query-transform-only failed under stress #26036

Closed
cockroach-teamcity opened this issue May 24, 2018 · 6 comments
Closed

github.com/cockroachdb/cockroach/pkg/ccl/importccl: TestImportCSVStmt/schema-in-query-transform-only failed under stress #26036

cockroach-teamcity opened this issue May 24, 2018 · 6 comments
Assignees
@windchan7
Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/9c197a359cf35d495425365cd22c3169e8cc0335

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=677154&tab=buildLog


I180524 11:23:25.399855 28485 storage/replica_command.go:863  [split,n3,s3,r171/2:/Table/6{0-1/1/741}] initiating a split of this range at key /Table/61 [r90]
I180524 11:23:25.853695 2428 storage/replica_proposal.go:202  [n3,s3,r171/2:/Table/6{0-1/1/741}] new range lease repl=(n3,s3):2 seq=4 start=1527160963.640948736,0 epo=1 pro=1527160963.640996467,0 following repl=(n3,s3):2 seq=4 start=1527160963.640948736,0 epo=1 pro=1527160963.640996467,0
I180524 11:23:28.012263 1787 gossip/gossip.go:488  [n1] gossip status (ok, 3 nodes)
gossip client (0/3 cur/max conns)
gossip server (2/3 cur/max conns, infos 2135/635 sent/received, bytes 792625B/203263B sent/received)
  2: 127.0.0.1:44219 (3m56s)
  3: 127.0.0.1:32877 (3m55s)
I180524 11:23:28.475586 1790 server/status/runtime.go:219  [n1] runtime stats: 1.4 GiB RSS, 651 goroutines, 84 MiB/13 MiB/119 MiB GO alloc/idle/total, 207 MiB/247 MiB CGO alloc/total, 1419.12cgo/sec, 0.93/0.08 %(u/s)time, 0.03 %gc (7x)
I180524 11:23:32.308962 2312 server/status/runtime.go:219  [n2] runtime stats: 1.4 GiB RSS, 635 goroutines, 69 MiB/26 MiB/119 MiB GO alloc/idle/total, 205 MiB/248 MiB CGO alloc/total, 1472.05cgo/sec, 0.84/0.07 %(u/s)time, 0.03 %gc (5x)
I180524 11:23:32.317387 2309 gossip/gossip.go:488  [n2] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:35955 (4m0s: infos 218/1490 sent/received, bytes 71570B/570656B sent/received)
gossip server (0/3 cur/max conns, infos 218/1490 sent/received, bytes 71570B/570656B sent/received)
I180524 11:23:33.064584 2257 gossip/gossip.go:488  [n3] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:35955 (4m0s: infos 420/678 sent/received, bytes 132182B/268358B sent/received)
gossip server (0/3 cur/max conns, infos 420/678 sent/received, bytes 132182B/268358B sent/received)
I180524 11:23:33.100389 2564 server/status/runtime.go:219  [n3] runtime stats: 1.4 GiB RSS, 608 goroutines, 81 MiB/18 MiB/119 MiB GO alloc/idle/total, 203 MiB/248 MiB CGO alloc/total, 1370.89cgo/sec, 0.85/0.07 %(u/s)time, 0.02 %gc (5x)
F180524 11:23:37.087494 2638 storage/replica.go:3873  [n3,s3,r1/3:/{Min-System/}] while purging index %d: while purging index 512: IO error: /go/src/github.com/cockroachdb/cockroach/artifacts/cockroach-auxiliary798717125/sideloading/1/1.3/i512.t16: File not found
goroutine 2638 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x53efb01, 0xc4204be8a0, 0x0, 0x12)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:956 +0xdd
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x4a28e60, 0xc400000004, 0x44a15e8, 0x12, 0xf21, 0xc421202620, 0xd1)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:834 +0x7f4
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x3742f40, 0xc42162fa70, 0xc400000004, 0x2, 0x31b87db, 0x6, 0xc422f51b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:154 +0x31e
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x3742f40, 0xc42162fa70, 0x1, 0x4, 0x31b87db, 0x6, 0xc422f51b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:55 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x3742f40, 0xc42162fa70, 0x31b87db, 0x6, 0xc422f51b88, 0x2, 0x2)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:149 +0x90
github.com/cockroachdb/cockroach/pkg/storage.fatalOnRaftReadyErr(0x3742f40, 0xc42162fa70, 0x31d88e4, 0x16, 0x371e980, 0xc4240145c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3873 +0x219
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x3742f40, 0xc42162fa70, 0xc42164f500, 0x3742f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3576 +0x215
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc4218d8000, 0x3742f40, 0xc42162fa70, 0xc42489e700, 0xc422f51e98, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3178 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc4218d8000, 0x3742f40, 0xc4216aa7b0, 0x1)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3563 +0x302
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420eea240, 0x3742f40, 0xc4216aa7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x2f2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4216aa7b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203f29f0, 0xc421037200, 0xc4203f29e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc42022a960, 0x31cd1aa, 0x11, 0x32e3640, 0xc420603c01)
	/usr/local/go/src/testing/testing.go:825 +0x597
testing.runTests.func1(0xc42022a960)
	/usr/local/go/src/testing/testing.go:1063 +0xa5
testing.tRunner(0xc42022a960, 0xc420603d68)
	/usr/local/go/src/testing/testing.go:777 +0x16e
testing.runTests(0xc420313480, 0x47e29e0, 0xf, 0xf, 0xbeb9c5cd7aef8556)
	/usr/local/go/src/testing/testing.go:1061 +0x4e2
testing.(*M).Run(0xc420326d80, 0x0)
	/usr/local/go/src/testing/testing.go:978 +0x2ce
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestMain(0xc420326d80)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/main_test.go:31 +0x180
main.main()
	_testmain.go:76 +0x22b

goroutine 6 [syscall, 4 minutes]:
os/signal.signal_recv(0x90ab41)
	/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x30
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x4f

goroutine 8 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1158 +0x132
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:591 +0x1b5

goroutine 9 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:598 +0xf2
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:592 +0x1cd

goroutine 50 [select, 4 minutes, locked to thread]:
runtime.gopark(0x32e8cd8, 0x0, 0x31b9ed3, 0x6, 0x18, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0xf9
runtime.selectgo(0xc420091f50, 0xc4200b82a0)
	/usr/local/go/src/runtime/select.go:392 +0x11d4
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:549 +0x19f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 10 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc42038f080)
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:144 +0x18f
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
	/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x9b

goroutine 1557 [chan receive]:
testing.(*T).Run(0xc42122ba40, 0x31eacfb, 0x1e, 0xc42054cde0, 0xc4213ce001)
	/usr/local/go/src/testing/testing.go:825 +0x597
github.com/cockroachdb/cockroach/pkg/ccl/importccl.TestImportCSVStmt(0xc42122ba40)
	/go/src/github.com/cockroachdb/cockroach/pkg/ccl/importccl/import_stmt_test.go:417 +0x1216
testing.tRunner(0xc42122ba40, 0x32e3640)
	/usr/local/go/src/testing/testing.go:777 +0x16e
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:824 +0x565

goroutine 2726 [IO wait]:
internal/poll.runtime_pollWait(0x7f139c0468a0, 0x72, 0x47f8010)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc421209118, 0x72, 0x3721700, 0x47f8010, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc421209118, 0xc4246b8000, 0x800, 0x800)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc421209100, 0xc4246b8000, 0x800, 0x800, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc421209100, 0xc4246b8000, 0x800, 0x800, 0xc41fea8da4, 0xc41fea8da3, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42063c3a8, 0xc4246b8000, 0x800, 0x800, 0xc425f6d6f0, 0x8b722d, 0xc42003a000)
	/usr/local/go/src/net/net.go:176 +0x85
crypto/tls.(*block).readFromUntil(0xc42113fd40, 0x7f139d21e808, 0xc42063c3a8, 0x5, 0xc42063c3a8, 0x7f139d3116c8)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc42154ce00, 0x32e8e17, 0xc42154cf20, 0x37)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc42154ce00, 0xc421cac000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc421bc31a0, 0xc420bb0578, 0x9, 0x9, 0x0, 0x0, 0xc425f6dc58)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x371b2e0, 0xc421bc31a0, 0xc420bb0578, 0x9, 0x9, 0x9, 0x8baa8a, 0xc420082c00, 0x10)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x371b2e0, 0xc421bc31a0, 0xc420bb0578, 0x9, 0x9, 0xc42141aca8, 0x2, 0x90bc22)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420bb0578, 0x9, 0x9, 0x371b2e0, 0xc421bc31a0, 0x0, 0x0, 0xc424797410, 0x3728c80)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420bb0540, 0xc424491da0, 0xc424491d00, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc421940fc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x1fe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x10fc

goroutine 2572 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearStmtStats.func1(0x3742f40, 0xc42113e270)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:608 +0x2bd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a1df0, 0xc421037200, 0xc421166c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1518 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc42110ff80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1d4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x294

goroutine 1746 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x8ba78759000042a5)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42182c6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d68e0, 0xc42082afc0, 0xc4203d68c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1624 [runnable]:
sync.runtime_notifyListWait(0xc421184c88, 0x11dd)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc421184c78)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc421184b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:648 +0xb3
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:635 +0x607

goroutine 1623 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x3742f40, 0xc420fc05a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1249 +0x84
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030af40, 0xc42082afc0, 0xc4203b5760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1626 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3742f40, 0xc422088bd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1161 +0x5a5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030b160, 0xc42082afc0, 0xc42030b150)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1625 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x3742f40, 0xc420fc05d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:371 +0x83
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030b130, 0xc42082afc0, 0xc4203b5e40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2014 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc421e60000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1373 +0x2ac
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc421e60000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:949 +0x247
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:940 +0x173

goroutine 2231 [select]:
net.(*pipe).read(0xc421c13980, 0xc4212bfc00, 0x400, 0x400, 0x16a5, 0x2000, 0xc4208196f0)
	/usr/local/go/src/net/pipe.go:164 +0x290
net.(*pipe).Read(0xc421c13980, 0xc4212bfc00, 0x400, 0x400, 0xc420082c00, 0xc420819720, 0xbdcc7a)
	/usr/local/go/src/net/pipe.go:147 +0x61
crypto/tls.(*block).readFromUntil(0xc4211ad740, 0x7f139d1b9550, 0xc421c13980, 0x5, 0xc421c13980, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc4213dfc00, 0x32e8e17, 0xc4213dfd20, 0xc4212bfc0d)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc4213dfc00, 0xc421e1c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc421630d20, 0xc4204d5458, 0x9, 0x9, 0x7f138460de88, 0xc4215355c0, 0x8e79c0)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x371b2e0, 0xc421630d20, 0xc4204d5458, 0x9, 0x9, 0x9, 0xc4215355c0, 0xc420d34a80, 0x8abcf5)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x371b2e0, 0xc421630d20, 0xc4204d5458, 0x9, 0x9, 0x8e792a, 0xc4208c8600, 0x90bc22)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4204d5458, 0x9, 0x9, 0x371b2e0, 0xc421630d20, 0x0, 0x0, 0xc4215355c0, 0xc420819da7)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4204d5420, 0xc422e52260, 0xc422e52200, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420fbd680)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1129 +0x1fe
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x10fc

goroutine 2259 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc42108da40, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc4214f4d80, 0xc4203d66b0, 0xc42108da40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc42120bb00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x101
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0x14cf

goroutine 1627 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x3742f40, 0xc420fc0600)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1190 +0x2fe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030b1d0, 0xc42082afc0, 0xc42030b1c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 6278 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc420c6ca00, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc42322daa0, 0xc421798c40, 0xc420c6ca00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc421fad600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0x1ab
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0x162c

goroutine 2275 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x3742f40, 0xc4211acae0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:156 +0x63
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420095bb0, 0xc42082b950, 0xc4216ee960)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2313 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x3742f40, 0xc421d67c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:126 +0x18d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b180, 0xc42082b950, 0xc42035b170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2314 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x3742f40, 0xc42094ff20)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:775 +0x25d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b190, 0xc42082b950, 0xc420359760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2312 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3742f40, 0xc4211ac060)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1772 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b160, 0xc42082b950, 0xc420359740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2315 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3742f40, 0xc4213f63c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1124 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b1e0, 0xc42082b950, 0xc420359820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1631 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc421920fc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1176 +0x1ad
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:309 +0x1558

goroutine 80 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208a1200)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 28 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc4208e2000)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1210 +0x8b
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:82 +0x4bb

goroutine 1738 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x52dcfb8000042b3)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42182c510)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6770, 0xc42082afc0, 0xc4203d6760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1767 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x56c65ff50000428d)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42113e8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42113e8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6cc0, 0xc42082afc0, 0xc4203d6cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1628 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc4206a00c0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc421482240, 0xc42036c550, 0xc4206a00c0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc4208ac000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x101
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0x14cf

goroutine 2565 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x3742f40, 0xc42147c3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:126 +0x18d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a14d0, 0xc421037200, 0xc4203a14c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2566 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x3742f40, 0xc421e28270)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:775 +0x25d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a14e0, 0xc421037200, 0xc421166b80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1692 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x3742f40, 0xc4212ec300)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:296 +0x1af
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420342c00, 0xc42082afc0, 0xc420418b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1691 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x3742f40, 0xc4212ec330)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:125 +0x1a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420342bf0, 0xc42082afc0, 0xc420342be0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1742 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x4294238f00004297)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42182c5d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6820, 0xc42082afc0, 0xc4203d6810)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2347 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0xf4af02b50000333c)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4215d1350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4215d1350)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a2fb0, 0xc42082b950, 0xc4202a2fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2562 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc42130d9f0, 0xc4218363f0, 0xc4203a13c0, 0x32e9078, 0xc420fef680, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:887 +0x20f
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x3742f40, 0xc421430b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:961 +0xdb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a13d0, 0xc421037200, 0xc4219249f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2713 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420c6c600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x1d4
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x294

goroutine 1696 [select]:
github.com/cockroachdb/cockroach/pkg/sql/jobs.(*Registry).Start.func2(0x3742f40, 0xc42113ea50)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/jobs/registry.go:270 +0x146
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420342ce0, 0xc42082afc0, 0xc4214341e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2567 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3742f40, 0xc421bc57d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1124 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203a14f0, 0xc421037200, 0xc421166ba0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2303 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3742f40, 0xc421b9a390)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035bbc0, 0xc421037200, 0xc42035bbb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2310 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc420d82e10, 0xc4206d87e0, 0xc42035b0d0, 0x32e9078, 0xc4208672c0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:887 +0x20f
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x3742f40, 0xc4211ac000)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:961 +0xdb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b0e0, 0xc42082b950, 0xc42094fce0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2727 [runnable]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).NewStream.func2(0xc40000081b, 0x8e79c0, 0x6000293eac0, 0x18)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:536 +0x1b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).originateStream(0xc421bc3260, 0xc4245d0140, 0xc420561d90, 0xc422829d98)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:488 +0xd2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).headerHandler(0xc421bc3260, 0xc422ae4eb0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:483 +0xaf7
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).handle(0xc421bc3260, 0x2daafa0, 0xc422ae4eb0, 0xc422ae4eb0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:631 +0x690
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc421bc3260, 0xc4203f3c50, 0xc420561d80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:388 +0x9b
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc421940fc0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x101
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0x14cf

goroutine 1688 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3742f40, 0xc4218bf4d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1124 +0x632
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420342ba0, 0xc42082afc0, 0xc420418b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2282 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0x4854d00c00003346)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4211acbd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4211acbd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420095ca0, 0xc42082b950, 0xc420095c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1813 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc420df4f00, 0xc4219ceeb0, 0xc4202a8650, 0xf, 0xc42056efc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x3742f40, 0xc4206777d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203f2620, 0xc42082afc0, 0xc420677710)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1777 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc4206b8000, 0x3742f40, 0xc4214a4630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3724 +0x1a6
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x3742f40, 0xc4214a4630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3679 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6eb0, 0xc42082afc0, 0xc4203d6ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1408 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x52dcfb8000042b8)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc4202e3560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4202e3560)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6390, 0xc42082afc0, 0xc4203d6380)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2348 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0xf4af02b500003314)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4215d1380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4215d1380)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a2ff0, 0xc42082b950, 0xc4202a2fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2699 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc420df4f00, 0xc421b1d4f0, 0xc421d620e0, 0xf, 0xc4213a2d80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:575 +0x5a1
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.1.1(0x3742f40, 0xc42170ec60)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:507 +0x8f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204fa0c0, 0xc42082afc0, 0xc42170ec00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1607 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3742f40, 0xc422088990)
	/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:86 +0x1aa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42030a860, 0xc42082afc0, 0xc42030a850)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2345 [runnable]:
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).term(0xc4203a41c0, 0x9d, 0x5, 0xc4212b0030, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/log.go:219 +0x297
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).matchTerm(0xc4203a41c0, 0x9d, 0x7, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/log.go:272 +0x47
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raftLog).maybeAppend(0xc4203a41c0, 0x9d, 0x7, 0x9d, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/log.go:77 +0x57
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).handleAppendEntries(0xc42153c500, 0x3, 0x2, 0x3, 0x7, 0x7, 0x9d, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/raft.go:1222 +0x11b
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.stepFollower(0xc42153c500, 0x3, 0x2, 0x3, 0x7, 0x7, 0x9d, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/raft.go:1173 +0x115
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*raft).Step(0xc42153c500, 0x3, 0x2, 0x3, 0x7, 0x7, 0x9d, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/raft.go:898 +0x165a
github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft.(*RawNode).Step(0xc42100a6c0, 0x3, 0x2, 0x3, 0x7, 0x7, 0x9d, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/coreos/etcd/raft/rawnode.go:195 +0x159
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).stepRaftGroup.func1(0xc42100a6c0, 0x3742ec0, 0xc420050130, 0x3742f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3443 +0x100
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).withRaftGroupLocked(0xc42089f880, 0xc425471ae8, 0xc42089f978, 0xa4e9cd)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:560 +0x18d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).withRaftGroup(0xc42089f880, 0xc425471ae8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:577 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).stepRaftGroup(0xc42089f880, 0xc423381860, 0x30, 0x30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3438 +0xa2
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftRequestWithReplica(0xc421bf0480, 0x3742f40, 0xc4211e0d80, 0xc42089f880, 0xc423381860, 0x92e1cc)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3232 +0xce
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue.func1(0x3742f40, 0xc4211e0d80, 0xc42089f880, 0x3742f40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3569 +0x102
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc421bf0480, 0x3742f40, 0xc4211e0d80, 0xc423381860, 0xc425471e98, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3178 +0x169
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRequestQueue(0xc421bf0480, 0x3742f40, 0xc4215d1710, 0x5)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3563 +0x302
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4215d1710)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:226 +0x2f2
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4215d1710)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a2f60, 0xc42082b950, 0xc4202a2f50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2289 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0x4628d34200003323)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4211acd20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4211acd20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420095e80, 0xc42082b950, 0xc420095e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2078 [select, 4 minutes]:
database/sql.(*DB).connectionOpener(0xc4217500a0, 0x3742e80, 0xc4214d72c0)
	/usr/local/go/src/database/sql/sql.go:935 +0x13f
created by database/sql.OpenDB
	/usr/local/go/src/database/sql/sql.go:634 +0x2bd

goroutine 2123 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3742f40, 0xc420f3a360)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420324eb0, 0xc42082b950, 0xc4204118e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2346 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0x38a7ac8e0000334c)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4215d12f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4215d12f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a2f80, 0xc42082b950, 0xc4202a2f70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1500 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3742f40, 0xc42182c8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:586 +0x1c8
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420464050, 0xc42082afc0, 0xc4217c00c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2344 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0xb2a5e4e50000334a)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4215d1320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4215d1320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a2ec0, 0xc42082b950, 0xc4202a2ea0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1417 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc4208c6040, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:293 +0x1e9
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc4214822a0, 0xc420324070, 0xc4208c6040)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:370 +0x236
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc420b64000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:276 +0x1ab
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:273 +0x162c

goroutine 2309 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x3742f40, 0xc4211ac030)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:700 +0x357
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035b0a0, 0xc42082b950, 0xc4203596e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1779 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3742f40, 0xc4217dd620)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x499
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6ef0, 0xc42082afc0, 0xc420adf950)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1770 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x56c65ff5000042a2)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42113e870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42113e870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6d50, 0xc42082afc0, 0xc4203d6d40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2273 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc4200a5a00, 0x3742f40, 0xc4218cb140, 0xc42082b950, 0xad14c47, 0xed298979c, 0x0, 0xc4214e0a80, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:212 +0x24e
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc4214e0a80, 0x58)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:274 +0x13c
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420f3d110, 0xc420b40080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:319 +0x2e2
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x3742f40, 0xc4218cb140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:272 +0x392
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420f5e350, 0xc42082b950, 0xc42043ed20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2351 [semacquire]:
sync.runtime_notifyListWait(0xc420c6c150, 0x16a2418b00003327)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420c6c140)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421c64360, 0x3742f40, 0xc4216baa20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc4216baa20)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202a30c0, 0xc42082b950, 0xc4202a30b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2157 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f139d201d60, 0x72, 0xc4219a4c28)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc421c13818, 0x72, 0xc420f5d500, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc421c13818, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Accept(0xc421c13800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:372 +0x2e2
net.(*netFD).accept(0xc421c13800, 0x292ca30, 0x7f1395475508, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x53
net.(*TCPListener).accept(0xc42000e698, 0xc4219a4e10, 0x8, 0xc4219a4e58)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x4e
net.(*TCPListener).Accept(0xc42000e698, 0x32e68a8, 0xc420c6de80, 0x3758f40, 0xc42107a038)
	/usr/local/go/src/net/tcpsock.go:259 +0x50
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc420c6de80, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xc1
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func14.1()
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1368 +0x47
sync.(*Once).Do(0xc4203b9a60, 0xc4208d5750)
	/usr/local/go/src/sync/once.go:44 +0xe2
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func14(0x3742f40, 0xc4216ab050)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1367 +0xb9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4214ff500, 0xc42082b950, 0xc42044d0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1790 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3742f40, 0xc4217dd770)
	/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1772 +0x194
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d7420, 0xc42082afc0, 0xc420378920)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1712 [IO wait]:
internal/poll.runtime_pollWait(0x7f139d201540, 0x72, 0x47f8010)
	/usr/local/go/src/runtime/netpoll.go:173 +0x5e
internal/poll.(*pollDesc).wait(0xc421208a98, 0x72, 0x3721700, 0x47f8010, 0xffffffffffffffff)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xe5
internal/poll.(*pollDesc).waitRead(0xc421208a98, 0xc420e22400, 0x400, 0x400)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x4b
internal/poll.(*FD).Read(0xc421208a80, 0xc420e22400, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:157 +0x22a
net.(*netFD).Read(0xc421208a80, 0xc420e22400, 0x400, 0x400, 0xc41fe4db99, 0x5b8c99, 0xffffffffffffffff)
	/usr/local/go/src/net/fd_unix.go:202 +0x66
net.(*conn).Read(0xc42063c2c0, 0xc420e22400, 0x400, 0x400, 0xc422550040, 0xc420e97618, 0x32e8bb8)
	/usr/local/go/src/net/net.go:176 +0x85
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc420b377c0, 0xc420e22400, 0x400, 0x400, 0x3e8, 0xc420e976b0, 0xbdcc7a)
	/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x241
crypto/tls.(*block).readFromUntil(0xc42113e090, 0x7f139d1ba000, 0xc420b377c0, 0x5, 0xc420b377c0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:493 +0x114
crypto/tls.(*Conn).readRecord(0xc42144a000, 0x32e8e17, 0xc42144a120, 0x105802e)
	/usr/local/go/src/crypto/tls/conn.go:595 +0x134
crypto/tls.(*Conn).Read(0xc42144a000, 0xc4217ce000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1156 +0x160
bufio.(*Reader).Read(0xc420a64420, 0xc421c44038, 0x9, 0x9, 0x92aade, 0xc421c72a48, 0x90bd4c)
	/usr/local/go/src/bufio/bufio.go:216 +0x740
io.ReadAtLeast(0x371b2e0, 0xc420a64420, 0xc421c44038, 0x9, 0x9, 0x9, 0x1, 0xc423648e70, 0x0)
	/usr/local/go/src/io/io.go:309 +0x94
io.ReadFull(0x371b2e0, 0xc420a64420, 0xc421c44038, 0x9, 0x9, 0x0, 0x0, 0x90bc22)
	/usr/local/go/src/io/io.go:327 +0x73
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc421c44038, 0x9, 0x9, 0x371b2e0, 0xc420a64420, 0x0, 0x0, 0x0, 0x371b760)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x8f
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc421c44000, 0xc42113e960, 0xc42113e960, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0x10f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc420542000, 0xc42175a060, 0x32e7f40)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:427 +0xd6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420b2ec40, 0x375ba60, 0xc420542000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:676 +0x16f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:631 +0x54
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc4203cade0, 0xc420b2ec40, 0x7f139d21e698, 0xc420542000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:639 +0x35
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0x7c7

goroutine 1727 [runnable]:
sync.runtime_notifyListWait(0xc420168a90, 0x52dcfb80000427a)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42182c000)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6610, 0xc42082afc0, 0xc4203d6600)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1775 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0xc421268084)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc421268078)
	/usr/local/go/src/sync/waitgroup.go:129 +0xb3
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc421268000, 0x3742f40, 0xc4218150b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:172 +0x3f
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x3742f40, 0xc4218150b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3676 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6e70, 0xc42082afc0, 0xc4203d6e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 2451 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3742f40, 0xc4208e65a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:100 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203880f0, 0xc421037200, 0xc421280020)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1720 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0x8e5088490000429f)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:197 +0x9c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3742f40, 0xc42182c0c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:166 +0x56
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6530, 0xc42082afc0, 0xc4203d6520)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1783 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x3742f40, 0xc421815110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x4b2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203d6f90, 0xc42082afc0, 0xc4203d6f80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0x14b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:186 +0xbb

goroutine 1769 [semacquire]:
sync.runtime_notifyListWait(0xc420168a90, 0xb4e808410000429e)
	/usr/local/go/src/runtime/sema.go:510 +0x11a
sync.(*Cond).Wait(0xc420168a80)
	/usr/local/go/src/sync/cond.go:56 +0x8e
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421268000, 0x3742f40, 0xc42182c7b0)
@cockroach-teamcity cockroach-teamcity added this to the 2.1 milestone May 24, 2018
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels May 24, 2018
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/0530ad4931dfcfa89307652a6eaf95eb82a3d521

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=683149&tab=buildLog

1 similar comment
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/0530ad4931dfcfa89307652a6eaf95eb82a3d521

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=683149&tab=buildLog

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/915a266778aab15c98fc7e18639583fbe6532399

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=684091&tab=buildLog

1 similar comment
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/915a266778aab15c98fc7e18639583fbe6532399

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=684091&tab=buildLog

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue May 29, 2018
@nvanbenschoten
importccl: skip flaky TestImportCSVStmt
dbacc7d 
See cockroachdb#26036.

This has flaked on my PR three times in a row.

Release note: None
@nvanbenschoten nvanbenschoten mentioned this issue May 29, 2018
Merged
@tbg
Copy link
Member

tbg commented May 29, 2018

This fails pretty frequently. @nvanbenschoten is skipping this in #26179.

craig bot pushed a commit that referenced this issue May 29, 2018
@RaduBerinde @nvanbenschoten
Merge #26171 #26172 #26174 #26179
e3345dc 
26171: sql, opt: move explain tests, add corresponding tests in execbuilder r=RaduBerinde a=RaduBerinde

#### sql: consolidate explain tests

Consolidating explain, explain_plan, and explain_types into a single
file.

Release note: None

#### sql: move explain tests to planner and execbuilder tests

Release note: None

#### opt: fix up explain test

This commit shows the diffs between the current planner and the
optimizer. Testcases with `NOEXPAND` and `NONORMALIZE` were removed
(these options don't do anything when the optimizer is used).

Release note: None.


26172: opt, sql: move explain_distsql tests r=RaduBerinde a=RaduBerinde

Moving `explain_distsql` test to `planner_test` and renaming to
`distsql_auto_mode`. Adding corresponding test in execbuilder.

Release note: None

26174: sql, opt: split distsql_agg and enable for optimizer r=RaduBerinde a=RaduBerinde

#### sql: split distsql_agg test

Moving EXPLAINs to a planning_test.

Release note: None

#### opt: copy distsql_agg test to execbuilder

Release note: None

#### opt: distsql_agg test fixup

This commit shows the diffs for distsql_agg planning tests. The
distsql plan differences are mostly due to normalization of
expressions: `a+b+c` is interpreted as `(a+b)+c` which is then
normalized to `c+(a+b)` (we prefer single variable on the lhs).

Release note: None


26179: importccl: skip flaky TestImportCSVStmt r=nvanbenschoten a=nvanbenschoten

See #26036.

This has flaked on my PR three times in a row.

Release note: None

Co-authored-by: Radu Berinde <[email protected]>
Co-authored-by: Nathan VanBenschoten <[email protected]>
@tbg tbg assigned windchan7 and unassigned dt May 30, 2018
@tbg tbg added the A-kv-client Relating to the KV client and the KV interface. label May 30, 2018
@windchan7
Copy link
Contributor

#26259

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@windchan7 windchan7

Labels
A-kv-client Relating to the KV client and the KV interface. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.1
Development

No branches or pull requests
4 participants
@dt @tbg @cockroach-teamcity @windchan7