Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands #18554

Closed
cockroach-teamcity opened this issue Sep 18, 2017 · 3 comments
Closed

teamcity: failed tests on master: test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands #18554

cockroach-teamcity opened this issue Sep 18, 2017 · 3 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
2.0

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#354447:

--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
I170918 18:08:26.053347 44717 storage/store.go:1208  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I170918 18:08:26.053410 44717 gossip/gossip.go:315  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:56124" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W170918 18:08:26.053466 44809 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W170918 18:08:26.053491 44810 storage/store.go:1295  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
W170918 18:08:26.057193 44717 gossip/gossip.go:1261  [n2] no incoming or outgoing connections
I170918 18:08:26.057528 44717 storage/store.go:1208  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I170918 18:08:26.057568 44717 gossip/gossip.go:315  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:52947" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I170918 18:08:26.057824 44702 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:56124
W170918 18:08:26.065994 44717 gossip/gossip.go:1261  [n3] no incoming or outgoing connections
I170918 18:08:26.066156 44717 storage/store.go:1208  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I170918 18:08:26.066210 44717 gossip/gossip.go:315  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:46219" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I170918 18:08:26.066628 45039 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:56124
I170918 18:08:26.073299 45090 storage/replica_raftstorage.go:516  [s1,r1/1:/M{in-ax}] generated preemptive snapshot d827d47a at index 15
I170918 18:08:26.074237 45090 storage/store.go:3500  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 1ms
I170918 18:08:26.074510 45046 storage/replica_raftstorage.go:722  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=d827d47a, encoded size=5681, 1 rocksdb batches, 5 log entries)
I170918 18:08:26.074810 45046 storage/replica_raftstorage.go:728  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170918 18:08:26.075267 45090 storage/replica_command.go:3664  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I170918 18:08:26.076294 44686 storage/replica.go:3056  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I170918 18:08:26.077500 45079 storage/replica_raftstorage.go:516  [s1,r1/1:/M{in-ax}] generated preemptive snapshot d29a54fe at index 17
I170918 18:08:26.078227 45052 storage/replica_raftstorage.go:722  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=d29a54fe, encoded size=7228, 1 rocksdb batches, 7 log entries)
I170918 18:08:26.078572 45052 storage/replica_raftstorage.go:728  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170918 18:08:26.078678 45079 storage/store.go:3500  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms
I170918 18:08:26.079374 45079 storage/replica_command.go:3664  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I170918 18:08:26.080658 45055 storage/raft_transport.go:455  raft transport stream to node 1 established
I170918 18:08:26.082190 45073 storage/replica.go:3056  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W170918 18:08:26.222590 45140 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 18:08:26.223145 45138 storage/store.go:3237  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170918 18:08:26.223564 44688 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
W170918 18:08:26.223589 45140 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 18:08:26.223641 45140 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
I170918 18:08:26.224025 45092 storage/raft_transport.go:455  raft transport stream to node 3 established
W170918 18:08:26.224200 45110 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170918 18:08:26.224314 45108 storage/store.go:3237  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170918 18:08:26.227414 45071 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W170918 18:08:26.227727 45057 storage/store.go:3237  [s2] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W170918 18:08:26.227788 45055 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
I170918 18:08:26.229896 44717 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
SIGQUIT: quit
PC=0x6f5151 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x2f89790, 0x0, 0x0, 0x0, 0xc400000000, 0xc420020600, 0x0, 0x0, 0x7ffcdd2d94e0, 0x6a66ab, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21
runtime.futexsleep(0x2f89790, 0x0, 0xffffffffffffffff)
	/usr/local/go/src/runtime/os_linux.go:45 +0x62
runtime.notesleep(0x2f89790)
	/usr/local/go/src/runtime/lock_futex.go:151 +0x9b
runtime.stopm()
	/usr/local/go/src/runtime/proc.go:1670 +0xe5
runtime.exitsyscall0(0xc420001500)
	/usr/local/go/src/runtime/proc.go:2808 +0x109
runtime.mcall(0x7ffcdd2d9590)
	/usr/local/go/src/runtime/asm_amd64.s:286 +0x5b

goroutine 1 [chan receive, 4 minutes]:
testing.(*T).Run(0xc4204f81e0, 0x1cab33e, 0x1a, 0x1d6ec00, 0x724901)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4204f81e0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4204f81e0, 0xc420691b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc42026ce20, 0x2a98620, 0x1d8, 0x1d8, 0x58)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420cf1f18, 0xc420691d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420691f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1002 +0xdb

goroutine 19 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2f88ac0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1004 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:580 +0xbf

goroutine 20 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45188 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203099e0, 0xc420ccc2d0, 0xc4203099d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44837 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7fdd516c2df0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4202eaa98, 0x72, 0xffffffffffffff00, 0x2aa70a0, 0x2a9bd10)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc4202eaa98, 0xc420c2c000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4202eaa80, 0xc420c2c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc4202eaa80, 0xc420c2c000, 0x8000, 0x8000, 0x69b05c, 0xc4206a8d80, 0x4)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42396c1a8, 0xc420c2c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420cde6c0, 0xc4204d1a78, 0x9, 0x9, 0xc42081ba90, 0xc42081bb18, 0xaeff17)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2a9dae0, 0xc420cde6c0, 0xc4204d1a78, 0x9, 0x9, 0x9, 0xc423409628, 0xc4200868c0, 0x4b)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2a9dae0, 0xc420cde6c0, 0xc4204d1a78, 0x9, 0x9, 0x4b, 0x4b, 0x0)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4204d1a78, 0x9, 0x9, 0x2a9dae0, 0xc420cde6c0, 0x0, 0x4b00000000, 0x4b, 0x4b)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4204d1a40, 0x4b, 0x4b, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc4205ff4d0, 0xc4205ff500, 0xc4205ff500, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4235bc000, 0xc4205ff6b0, 0x1d6ff58)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:377 +0x30d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420292780, 0x2ac3280, 0xc4235bc000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:547 +0x140
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc420292780, 0x2ac5680, 0xc42396c1a8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:540 +0x463
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc420292780, 0x2ac5680, 0xc42396c1a8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:507 +0x497
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:476 +0x5b9

goroutine 44727 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func1(0x2ab9820, 0xc4205ff3e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:49 +0x5f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375490, 0xc423496090, 0xc4206589f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45200 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309da0, 0xc420ccc2d0, 0xc420309d90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45169 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309550, 0xc420ccc2d0, 0xc420309540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45102 [semacquire]:
sync.runtime_notifyListWait(0xc423842650, 0xc400000004)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423842640)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420ccc6c0, 0x2ab9820, 0xc4201da8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4201da8d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309080, 0xc420ccc2d0, 0xc420309070)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44860 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000046)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420968db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420968db0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375890, 0xc4234973b0, 0xc420375880)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44832 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc422a0c000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff

goroutine 44894 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc40000002a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659410)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659410)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375e50, 0xc4234973b0, 0xc420375dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 43977 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc4235e6230, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4235e6220)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4235e6140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45105 [semacquire]:
sync.runtime_notifyListWait(0xc423842650, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423842640)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420ccc6c0, 0x2ab9820, 0xc4201da960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4201da960)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309150, 0xc420ccc2d0, 0xc420309140)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44896 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc40000002c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659470)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659470)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375e90, 0xc4234973b0, 0xc420375e80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44887 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000023)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc4206592c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4206592c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375ca0, 0xc4234973b0, 0xc420375c90)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45100 [semacquire]:
sync.runtime_notifyListWait(0xc423842650, 0xc400000002)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423842640)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420ccc6c0, 0x2ab9820, 0xc4201da870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4201da870)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309000, 0xc420ccc2d0, 0xc420308fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45054 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4209d1270, 0xc42034f510, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc423409710, 0xc42034f510, 0x5, 0x5, 0x70cd5e, 0xc4237af4c0, 0xc420486e80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2aa3220, 0xc423409710, 0xc42034f510, 0x5, 0x5, 0x5, 0xc4235bc000, 0xc4237af400, 0x5)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2aa3220, 0xc423409710, 0xc42034f510, 0x5, 0x5, 0x4, 0xc4238bcc80, 0x6d753a)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4237af400, 0xc42034f510, 0x5, 0x5, 0xc42098cad8, 0xc4238bcc98, 0x709e53)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc42034f500, 0x7fffffff, 0x0, 0x1, 0xc420d89600, 0x0, 0x6, 0xc4206a8d80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc42034f500, 0x2ab9060, 0x2fcb178, 0xc4237af400, 0x2aabc60, 0x2fcb178, 0x1babcc0, 0xc420d89620, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc4206dfe60, 0x1babcc0, 0xc420d89620, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:648 +0x10e
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc4202b4920, 0x2ab9820, 0xc423409770, 0xc420b59c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:397 +0x62
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x2ac5380, 0xc4202b4920, 0xc4208baa50, 0x2ab9820, 0xc423409770, 0x18, 0xd83430)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:326 +0x83
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x2ab9820, 0xc423409770)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:349 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202b4940, 0xc423496090, 0xc423409740)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45212 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309fb0, 0xc420ccc2d0, 0xc420309fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44509 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420620820)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 45195 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309bf0, 0xc420ccc2d0, 0xc420309bc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44731 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7fdd516c27f0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420927418, 0x72, 0xffffffffffffff00, 0x2aa70a0, 0x2a9bd10)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420927418, 0xc4228a2000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420927400, 0xc4228a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420927400, 0xc4228a2000, 0x8000, 0x8000, 0xa9b197, 0xc4201f4e00, 0xc420322230)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc4204be2f0, 0xc4228a2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc4203e2b40, 0xc4201f4c78, 0x9, 0x9, 0x1d70bf0, 0xc42081ed60, 0x69bfc2)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2a9dae0, 0xc4203e2b40, 0xc4201f4c78, 0x9, 0x9, 0x9, 0xc420276690, 0x80000000006, 0x20)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2a9dae0, 0xc4203e2b40, 0xc4201f4c78, 0x9, 0x9, 0x0, 0xc42081ee28, 0x6c1af6)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4201f4c78, 0x9, 0x9, 0x2a9dae0, 0xc4203e2b40, 0x0, 0xc400000000, 0xc42081ee6d, 0xc4235f0990)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4201f4c40, 0x2aaa460, 0xc4235f0990, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420658b70, 0xc4235f0990, 0xc4235f0990, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4232964e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 44733 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc4232964e0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 44963 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2ab9820, 0xc4233bb350)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1120 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420427c50, 0xc423496090, 0xc420427c40)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45068 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func3(0x2ac55c0, 0xc4232964e0, 0xc423296340, 0xc4238466e0, 0xc42359f100)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:255 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:254 +0xe23

goroutine 44895 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc40000002b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659440)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375e70, 0xc4234973b0, 0xc420375e60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 43983 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc42377e400, 0x2ac3fa0, 0xc42022bde0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:198 +0x61d
github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x1c53860, 0xc42377e400, 0x2ac1c00, 0xc422960480, 0x2a9b640, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:211 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420578b40, 0x2ac3280, 0xc42020e160, 0xc42359f000, 0xc4229be960, 0x2a7d760, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:891 +0x943
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420578b40, 0x2ac3280, 0xc42020e160, 0xc42359f000, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:985 +0x14c1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42023e190, 0xc420578b40, 0x2ac3280, 0xc42020e160, 0xc42359f000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:551 +0x9f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:549 +0xa1

goroutine 44728 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7fdd516c2670, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420927218, 0x72, 0xc42346ec00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420927218, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420927200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420927200, 0xc42396c1a8, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4204be2d8, 0xb064f9, 0x6f0420, 0xc42346ee28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4204be2d8, 0x1d6ff20, 0xc420292780, 0x2ac5680, 0xc42396c1a8)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc420292780, 0x2ab7e60, 0xc4204be2d8, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2ab9820, 0xc4205ff440)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375510, 0xc423496090, 0xc420333940)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45211 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309f80, 0xc420ccc2d0, 0xc420309f70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45101 [semacquire]:
sync.runtime_notifyListWait(0xc423842650, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423842640)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420ccc6c0, 0x2ab9820, 0xc4201da8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4201da8a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309060, 0xc420ccc2d0, 0xc420309030)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45186 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309980, 0xc420ccc2d0, 0xc420309970)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45067 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc4208baa50, 0x2, 0xc420986240, 0xc420488780, 0x2ac5320, 0xc420442d20, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:513 +0x52a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess.func1(0xc4208baa50, 0x2, 0x2ab9820, 0xc423641740, 0x1, 0x0, 0xc423677cc0, 0xc420986240, 0xc420488780, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:458 +0x32a
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).connectAndProcess(0xc4208baa50, 0x2ab9820, 0xc423641740, 0x2, 0xc420986240, 0xc420488780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0xd0
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendAsync.func1.1(0x2ab9820, 0xc423641740)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:587 +0x60
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420442ce0, 0xc423496090, 0xc4236416e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44871 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000013)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420658fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420658fc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375a60, 0xc4234973b0, 0xc420375a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44907 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000037)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659680)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659680)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375ff0, 0xc4234973b0, 0xc420375fe0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44905 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000035)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659620)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659620)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375fb0, 0xc4234973b0, 0xc420375fa0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44869 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000011)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420658f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420658f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375a20, 0xc4234973b0, 0xc420375a10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45184 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309940, 0xc420ccc2d0, 0xc420309930)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45191 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309a70, 0xc420ccc2d0, 0xc420309a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44833 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4202d3ba0, 0xc42326a3c0, 0xc42037a4a0, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2b9
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2ab9820, 0xc4233baed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420427ba0, 0xc423496090, 0xc4233bac00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44945 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ab9820, 0xc420968ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ec6b0, 0xc4234973b0, 0xc42040f700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44903 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000033)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc4206595c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc4206595c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375f70, 0xc4234973b0, 0xc420375f60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44725 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc420579c70, 0xc400000009)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420579c60)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc420579b80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45040 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func3(0x2ac55c0, 0xc420620820, 0xc4232961a0, 0xc4237b1760, 0xc4237ae600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:255 +0x1cd
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:254 +0xe23

goroutine 44662 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).controller(0xc42020e160)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:1011 +0x148
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:220 +0x8be

goroutine 45202 [semacquire]:
sync.runtime_notifyListWait(0xc423842650, 0xc40000000a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc423842640)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420ccc6c0, 0x2ab9820, 0xc423880090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc423880090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309de0, 0xc420ccc2d0, 0xc420309dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44839 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc4235bc000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:949 +0x251
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:221 +0x8e3

goroutine 44853 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000042)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420968c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420968c60)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375770, 0xc4234973b0, 0xc420375760)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44850 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x2ab9820, 0xc420824c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1158 +0x269
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375710, 0xc423496090, 0xc420375700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44856 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000045)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420968cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420968cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203757e0, 0xc4234973b0, 0xc4203757c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44889 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000025)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375d00, 0xc4234973b0, 0xc420375cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45175 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420309680, 0xc420ccc2d0, 0xc420309650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44735 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4202d3ba0, 0xc42098cb00, 0xc420336ba0, 0xf, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2b9
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2ab9820, 0xc4205ff740)
	/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375640, 0xc423496090, 0xc420658cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45181 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203098e0, 0xc420ccc2d0, 0xc4203098d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44944 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ab9820, 0xc420968f90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ec6a0, 0xc4234973b0, 0xc42040f6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 43979 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func1(0x2ab9820, 0xc42043b980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:49 +0x5f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42022bc80, 0xc423496090, 0xc42098bb30)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44884 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000020)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659230)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375c20, 0xc4234973b0, 0xc420375c10)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44734 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc422855500)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff

goroutine 44855 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc400000044)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420968cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420968cc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203757b0, 0xc4234973b0, 0xc4203757a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44918 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc420bfe400, 0x2ab9820, 0xc420659a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3688 +0x16d
github.com/cockroachdb/cockroach/pkg/storage.(*Store).(github.com/cockroachdb/cockroach/pkg/storage.coalescedHeartbeatsLoop)-fm(0x2ab9820, 0xc420659a40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3634 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212900, 0xc4234973b0, 0xc4202128f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44925 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2ab9820, 0xc420969020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:366 +0x2a9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212b80, 0xc4234973b0, 0xc42048d3b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44831 [select, 4 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420620b60)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 44879 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc42098cad0, 0xc40000001b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42098cac0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4209ac000, 0x2ab9820, 0xc420659140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2ab9820, 0xc420659140)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420375b80, 0xc4234973b0, 0xc420375b70)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45219 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420328110, 0xc420ccc2d0, 0xc4234f37c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44920 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2ab9820, 0xc420659b00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1307 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420212940, 0xc4234973b0, 0xc4236ff900)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44942 [select, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2ab9820, 0xc420968ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ec680, 0xc4234973b0, 0xc42040f6a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45217 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203280a0, 0xc420ccc2d0, 0xc4234f3720)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 44962 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2ab9820, 0xc4233bb140)
	/go/src/github.c

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Sep 18, 2017
@jordanlewis jordanlewis mentioned this issue Sep 18, 2017
Closed
@petermattis petermattis self-assigned this Sep 26, 2017
@petermattis petermattis added this to the 1.2 milestone Sep 26, 2017
@andreimatei andreimatei mentioned this issue Sep 27, 2017
Closed
@andreimatei
Copy link
Contributor

This keeps happening to me on TC. Peter, do you happen to know what this SIGQUIT is about?

@petermattis
Copy link
Collaborator

@andreimatei No, I don't. It is on my TODO list to investigate. I wasn't able to reproduce it locally when I tried a few days ago.

This was referenced Oct 3, 2017
Closed
Closed
Closed
Closed
@knz
Copy link
Contributor

knz commented Oct 16, 2017

https://teamcity.cockroachdb.com/viewLog.html?tab=buildLog&logTab=tree&filter=debug&expand=all&buildId=379109

tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
storage: de-flake TestRefreshPendingCommands
1b44843 
The test ran a for loop without preemption points. The loop checked a
condition that would only become true after another goroutine had been
scheduled and carried out its job.

If, with only few cores (four in my case) GC kicked in before that other
goroutine got scheduled, that loop would just run hot forever until the
test timed out, and the resulting stack dump looked quite unhelpful.

Add a small sleep so the runtime can preempt the goroutine.

The issue was harder to run into when stressing only the test, since there
was less garbage available at that point. Adding some print statements,
I accidentally made it much more likely.

Previously flaked (got stuck) within <500iters, now ran past 1.5k without
problems.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Touches cockroachdb#19367.
Fixes cockroachdb#18554.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
4 participants
@andreimatei @knz @petermattis @cockroach-teamcity