Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_history.tcl, test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands, testrace/TestNodeLivenessGetIsLiveMap, testrace/TestSchemaChangeEvalContext/ALTER_TABLE_t.test_ADD_COLUMN_x_TIMESTAMP_DEFAULT_current_timestamp;, testrace/TestSchemaChangeEvalContext #19367

Closed
cockroach-teamcity opened this issue Oct 19, 2017 · 1 comment
Closed

teamcity: failed tests on master: acceptance/TestDockerCLI, acceptance/TestDockerCLI/test_history.tcl, test/TestRefreshPendingCommands/#00, test/TestRefreshPendingCommands, testrace/TestNodeLivenessGetIsLiveMap, testrace/TestSchemaChangeEvalContext/ALTER_TABLE_t.test_ADD_COLUMN_x_TIMESTAMP_DEFAULT_current_timestamp;, testrace/TestSchemaChangeEvalContext #19367

cockroach-teamcity opened this issue Oct 19, 2017 · 1 comment
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#382443:

--- FAIL: acceptance/TestDockerCLI (227.390s)
test_log_scope.go:78: test logs captured to: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/acceptance/logTestDockerCLI688382464
test_log_scope.go:61: use -show-logs to present logs inline
--- FAIL: acceptance/TestDockerCLI/test_history.tcl (35.530s)
null
--- FAIL: test/TestRefreshPendingCommands/#00 (0.000s)
Test ended in panic.

------- Stdout: -------
W171019 12:56:02.088825 45564 storage/store.go:1299  [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled
W171019 12:56:02.088857 45565 storage/store.go:1299  [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled
I171019 12:56:02.089163 45462 storage/store.go:1212  [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I171019 12:56:02.089351 45462 gossip/gossip.go:331  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:54633" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
W171019 12:56:02.096229 45462 gossip/gossip.go:1277  [n2] no incoming or outgoing connections
I171019 12:56:02.096430 45462 storage/store.go:1212  [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I171019 12:56:02.096485 45462 gossip/gossip.go:331  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:40263" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171019 12:56:02.096542 45577 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:54633
W171019 12:56:02.167077 45462 gossip/gossip.go:1277  [n3] no incoming or outgoing connections
I171019 12:56:02.167211 45462 storage/store.go:1212  [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I171019 12:56:02.167264 45462 gossip/gossip.go:331  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:60877" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > 
I171019 12:56:02.167425 45326 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:54633
I171019 12:56:02.170494 45843 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot c71edd5a at index 15
I171019 12:56:02.171462 45843 storage/store.go:3549  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 1ms
I171019 12:56:02.171867 45819 storage/replica_raftstorage.go:732  [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=c71edd5a, encoded size=4620, 1 rocksdb batches, 5 log entries)
I171019 12:56:02.172218 45819 storage/replica_raftstorage.go:738  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171019 12:56:02.172578 45843 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I171019 12:56:02.175318 45848 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I171019 12:56:02.176194 45824 storage/replica_raftstorage.go:526  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 8f57dc11 at index 17
I171019 12:56:02.176790 45824 storage/store.go:3549  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 0ms
I171019 12:56:02.176972 45851 storage/replica_raftstorage.go:732  [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=8f57dc11, encoded size=5667, 1 rocksdb batches, 7 log entries)
I171019 12:56:02.177362 45851 storage/replica_raftstorage.go:738  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I171019 12:56:02.177685 45824 storage/replica_command.go:3754  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I171019 12:56:02.178487 45854 storage/raft_transport.go:455  raft transport stream to node 1 established
I171019 12:56:02.179589 45806 storage/replica.go:3071  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W171019 12:56:02.322318 45863 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 12:56:02.322491 45582 storage/store.go:3287  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171019 12:56:02.322514 45861 storage/raft_transport.go:461  raft transport stream to node 3 failed: store 3 was not found
I171019 12:56:02.323077 45875 storage/raft_transport.go:455  raft transport stream to node 3 established
W171019 12:56:02.323347 45706 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 12:56:02.323480 45706 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 12:56:02.323581 45877 storage/store.go:3287  [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W171019 12:56:02.323965 45859 storage/raft_transport.go:281  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W171019 12:56:02.323997 45706 storage/raft_transport.go:281  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W171019 12:56:02.324098 45856 storage/store.go:3287  [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W171019 12:56:02.324119 45854 storage/raft_transport.go:461  raft transport stream to node 1 failed: store 1 was not found
I171019 12:56:02.324559 45462 storage/client_test.go:1194  test clock advanced to: 1.800000125,0
SIGQUIT: quit
PC=0x6f6589 m=0 sigcode=0

goroutine 0 [idle]:
runtime.epollwait(0x4, 0x7ffcd6dce330, 0xffffffff00000080, 0x0, 0xffffffff00000004, 0x7ffc00000000, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:576 +0x19
runtime.netpoll(0xc420026601, 0xc420025301)
	/usr/local/go/src/runtime/netpoll_epoll.go:71 +0x12c
runtime.findrunnable(0xc42001e000, 0x0)
	/usr/local/go/src/runtime/proc.go:2107 +0x5fb
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2245 +0x12c
runtime.goexit0(0xc42037e900)
	/usr/local/go/src/runtime/proc.go:2396 +0x236
runtime.mcall(0x7ffcd6dceac0)
	/usr/local/go/src/runtime/asm_amd64.s:286 +0x5b

goroutine 1 [chan receive]:
testing.(*T).Run(0xc4204e82d0, 0x1d7a7d0, 0x1a, 0x1e41790, 0x725c01)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
testing.runTests.func1(0xc4204e82d0)
	/usr/local/go/src/testing/testing.go:1004 +0x64
testing.tRunner(0xc4204e82d0, 0xc420725b80)
	/usr/local/go/src/testing/testing.go:746 +0xd0
testing.runTests(0xc420497be0, 0x2bec140, 0x1e4, 0x1e4, 0xc420725c98)
	/usr/local/go/src/testing/testing.go:1002 +0x2d8
testing.(*M).Run(0xc420723f18, 0xc420725d80)
	/usr/local/go/src/testing/testing.go:921 +0x111
github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc420725f18)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7
main.main()
	github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1028 +0xdb

goroutine 18 [chan receive, 4 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x30d1060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
	/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf

goroutine 14 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 45598 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func1(0x2c0d800, 0xc4212da5a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:49 +0x5f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204060a0, 0xc4206d2000, 0xc4205d3980)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45909 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000030)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd5020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd5020)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef4c0, 0xc421278bd0, 0xc4203ef4b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45637 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000004a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420f40360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420f40360)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6c30, 0xc420b674d0, 0xc4202c6b60)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45638 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000011)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420f08090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420f08090)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6c60, 0xc420b674d0, 0xc4202c6c50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45957 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1.1(0xc4206b4ea0, 0xc4202a6e20, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077
created by github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077 +0x765

goroutine 45919 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000035)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf500)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef6d0, 0xc421278bd0, 0xc4203ef6c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45640 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000018)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770150)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6d90, 0xc420b674d0, 0xc4202c6cb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45890 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000007)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcef00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcef00)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef1c0, 0xc421278bd0, 0xc4203ef1b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45642 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000001b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc4207701e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc4207701e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6dd0, 0xc420b674d0, 0xc4202c6dc0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45677 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000003e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc4207709f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc4207709f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c77e0, 0xc420b674d0, 0xc4202c77b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45660 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000002d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770570)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7200, 0xc420b674d0, 0xc4202c7180)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45804 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4202f69b0, 0xc4203a8d50, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc4206e0750, 0xc4203a8d50, 0x5, 0x5, 0x70dffe, 0xc4218688c0, 0xc420586140)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55
io.ReadAtLeast(0x2bf6ec0, 0xc4206e0750, 0xc4203a8d50, 0x5, 0x5, 0x5, 0xc420d62000, 0xc421868800, 0x5)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2bf6ec0, 0xc4206e0750, 0xc4203a8d50, 0x5, 0x5, 0x0, 0xc420ebecf0, 0x6ae7b3)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc421868800, 0xc4203a8d50, 0x5, 0x5, 0xc420ebec68, 0x6c0718, 0xc42029a600)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4203a8d40, 0x7fffffff, 0x1133b72, 0xc420ebed10, 0xc420ebed18, 0x1dd, 0x1136653, 0x1133bb8)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4203a8d40, 0x2c0d000, 0x3114fa0, 0xc421868800, 0x2bffa80, 0x3114fa0, 0x1c74ae0, 0xc4206e0810, 0x7fffffff, 0x0, ...)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc42121e580, 0x1c74ae0, 0xc4206e0810, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6
github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc4202fa700, 0xc4206e07e0, 0xc4206e07e0, 0x1b6d1e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:329 +0x62
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x2c190e0, 0xc4202fa700, 0xc420418a40, 0xc42145c3c0, 0x2c0d800, 0xc4206e07e0, 0x18, 0xe2c110)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:487 +0x35
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x2c0d800, 0xc4206e07e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:502 +0x66
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202fa720, 0xc4206d2000, 0xc4206e0780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45901 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000021)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf1a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef350, 0xc421278bd0, 0xc4203ef340)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45639 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000010)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420f08060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420f08060)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6c90, 0xc420b674d0, 0xc4202c6c80)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45464 [semacquire]:
sync.runtime_notifyListWait(0xc4204e44b0, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4204e44a0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4204e43c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45951 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x2c0d800, 0xc420bcf6e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1182 +0x122
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203efc20, 0xc421278bd0, 0xc4203e9d20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45940 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000038)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf590)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf590)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef9d0, 0xc421278bd0, 0xc4203ef9c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45161 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc420d62000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1276 +0x13c
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:294 +0xdd2

goroutine 45912 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000037)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd50b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd50b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef550, 0xc421278bd0, 0xc4203ef540)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45461 [chan receive]:
testing.(*T).Run(0xc42145c0f0, 0x0, 0x0, 0xc420396050, 0x78eaad)
	/usr/local/go/src/testing/testing.go:790 +0x2fc
github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands(0xc42145c0f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1010 +0xfe
testing.tRunner(0xc42145c0f0, 0x1e41790)
	/usr/local/go/src/testing/testing.go:746 +0xd0
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:789 +0x2de

goroutine 45659 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000002c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770540)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7130, 0xc420b674d0, 0xc4202c7120)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45925 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000013)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd4cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd4cf0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef790, 0xc421278bd0, 0xc4203ef780)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45643 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000001c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770210)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6df0, 0xc420b674d0, 0xc4202c6de0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45910 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000034)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd5050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd5050)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef4f0, 0xc421278bd0, 0xc4203ef4e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45905 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000028)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf2c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef410, 0xc421278bd0, 0xc4203ef3f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45689 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2c0d800, 0xc4212d59b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:365 +0x2a9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7c70, 0xc420b674d0, 0xc4212b4a50)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45159 [IO wait]:
internal/poll.runtime_pollWait(0x7fc0ffe73c70, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420b4e818, 0x72, 0xffffffffffffff00, 0x2bfae80, 0x2bef910)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420b4e818, 0xc421720000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420b4e800, 0xc421720000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420b4e800, 0xc421720000, 0x8000, 0x8000, 0x69c42b, 0x6c5a2b, 0xc420974c28)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42051c0d8, 0xc421720000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420f3c6c0, 0xc4203f70d8, 0x9, 0x9, 0x1e437e8, 0xc420974d60, 0x69d262)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2bf1640, 0xc420f3c6c0, 0xc4203f70d8, 0x9, 0x9, 0x9, 0xc420279340, 0x80000000006, 0x20)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2bf1640, 0xc420f3c6c0, 0xc4203f70d8, 0x9, 0x9, 0x0, 0xc420974e28, 0x6c2d96)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203f70d8, 0x9, 0x9, 0x2bf1640, 0xc420f3c6c0, 0x0, 0xc400000000, 0xc420974e6d, 0xc421417290)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203f70a0, 0x2bfe280, 0xc421417290, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc4212d56e0, 0xc421417290, 0xc421417290, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420d62000)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 45886 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000003)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bced80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bced80)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef140, 0xc421278bd0, 0xc4203ef130)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45921 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc40000000b)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd4c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd4c30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef710, 0xc421278bd0, 0xc4203ef700)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45599 [IO wait]:
internal/poll.runtime_pollWait(0x7fc0ffe73970, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420b4e618, 0x72, 0xc420e15c00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420b4e618, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420b4e600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420b4e600, 0xc42119c378, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42119c360, 0xb50639, 0x6f16c0, 0xc420e15e28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42119c360, 0x1e42ae8, 0xc4204e5400, 0x2c193e0, 0xc42119c378)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4204e5400, 0x2c0be00, 0xc42119c360, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2c0d800, 0xc4212d5650)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204060b0, 0xc4206d2000, 0xc420288c00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45670 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000037)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770780)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c73f0, 0xc420b674d0, 0xc4202c73e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45423 [IO wait]:
internal/poll.runtime_pollWait(0x7fc0ffe73a30, 0x72, 0xffffffffffffffff)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420f12598, 0x72, 0xc42084cc00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420f12598, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc420f12580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2
net.(*netFD).accept(0xc420f12580, 0xc420ff8160, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e148, 0xb50639, 0x6f16c0, 0xc42084ce28)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e148, 0x1e42ae8, 0xc4210c4640, 0x2c193e0, 0xc420ff8160)
	/usr/local/go/src/net/tcpsock.go:247 +0x49
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc4210c4640, 0x2c0be00, 0xc42000e148, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196
github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2c0d800, 0xc42118f950)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042ada0, 0xc4206d2000, 0xc420423f20)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45684 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2c0d800, 0xc420770b40)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1311 +0x3e9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7a20, 0xc420b674d0, 0xc4212b4820)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45891 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000008)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcef30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcef30)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef1e0, 0xc421278bd0, 0xc4203ef1d0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45888 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000005)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcee10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcee10)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef180, 0xc421278bd0, 0xc4203ef170)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45956 [runnable]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203efcc0, 0xc421278bd0, 0xc4208984b0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45672 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000030)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770630)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7450, 0xc420b674d0, 0xc4202c7420)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45644 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc40000001d)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770240)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6e10, 0xc420b674d0, 0xc4202c6e00)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45653 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000026)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770420)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c7000, 0xc420b674d0, 0xc4202c6ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45719 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2c0d800, 0xc420141dd0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042b0d0, 0xc4206d2000, 0xc4203a82c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45908 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc40000002e)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd4ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd4ff0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef4a0, 0xc421278bd0, 0xc4203ef480)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45914 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc40000002c)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf320)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef630, 0xc421278bd0, 0xc4203ef620)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45648 [semacquire]:
sync.runtime_notifyListWait(0xc42117b8d0, 0xc400000021)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc42117b8c0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420b67680, 0x2c0d800, 0xc420770330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420770330)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4202c6ec0, 0xc420b674d0, 0xc4202c6eb0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45717 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc421496a80)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff

goroutine 45470 [IO wait]:
internal/poll.runtime_pollWait(0x7fc0ffe73bb0, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420111818, 0x72, 0xffffffffffffff00, 0x2bfae80, 0x2bef910)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
internal/poll.(*pollDesc).waitRead(0xc420111818, 0xc420ace000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420111800, 0xc420ace000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a
net.(*netFD).Read(0xc420111800, 0xc420ace000, 0x8000, 0x8000, 0xae5fd7, 0xc4202288c0, 0xc420df9cb0)
	/usr/local/go/src/net/fd_unix.go:202 +0x52
net.(*conn).Read(0xc42119c1d8, 0xc420ace000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:176 +0x6d
bufio.(*Reader).Read(0xc420620300, 0xc420228738, 0x9, 0x9, 0xc420df9d50, 0xb3b496, 0xc4209beed0)
	/usr/local/go/src/bufio/bufio.go:213 +0x30b
io.ReadAtLeast(0x2bf1640, 0xc420620300, 0xc420228738, 0x9, 0x9, 0x9, 0x0, 0x0, 0xc420df9e5f)
	/usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x2bf1640, 0xc420620300, 0xc420228738, 0x9, 0x9, 0x510412e401, 0x8800000000000051, 0x51)
	/usr/local/go/src/io/io.go:327 +0x58
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc420228738, 0x9, 0x9, 0x2bf1640, 0xc420620300, 0x0, 0x0, 0xc4204c3f20, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420228700, 0x2bfe180, 0xc420994fc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420994f90, 0xc420994fc0, 0xc420994fc0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc4206b5520)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8

goroutine 45596 [semacquire]:
sync.runtime_notifyListWait(0xc4204e53b0, 0xc400000009)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc4204e53a0)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc4204e52c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:478 +0xa2
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:465 +0x490

goroutine 45326 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc42067c820, 0x2c0d800, 0xc4202322a0, 0xc420e0e000, 0x2c18a80, 0xc420278980, 0xc4206d2000, 0xc4203a6640, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:323 +0x350
github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x2c0d800, 0xc4202322a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:130 +0x3cd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420278920, 0xc4206d2000, 0xc4210c69c0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45896 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000012)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcf080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcf080)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef2b0, 0xc421278bd0, 0xc4203ef2a0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45417 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc42033c100, 0x2c18ae0, 0xc42042a100, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:198 +0x61d
github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x1d21140, 0xc42033c100, 0x2c16620, 0xc421290000, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:211 +0xb2
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc4204e4640, 0x2c17d60, 0xc420b96000, 0xc420bb4b00, 0xc4208e8f00, 0x2bd07e0, 0x0, 0x0, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:891 +0x943
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc4204e4640, 0x2c17d60, 0xc420b96000, 0xc420bb4b00, 0x0)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:985 +0x14c1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc420260050, 0xc4204e4640, 0x2c17d60, 0xc420b96000, 0xc420bb4b00)
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:551 +0x9f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:549 +0xa1

goroutine 45491 [chan receive]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2c0d800, 0xc420f40180)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420396aa0, 0xc4206d2000, 0xc420415060)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45892 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc40000000a)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420bcef90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420bcef90)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4203ef200, 0xc421278bd0, 0xc4203ef1f0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45720 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2c0d800, 0xc420232120)
	/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1136 +0x331
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42042b0f0, 0xc4206d2000, 0xc42042b0e0)
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
	/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad

goroutine 45935 [semacquire]:
sync.runtime_notifyListWait(0xc420654910, 0xc400000023)
	/usr/local/go/src/runtime/sema.go:507 +0x110
sync.(*Cond).Wait(0xc420654900)
	/usr/local/go/src/sync/cond.go:56 +0x80
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc421278c60, 0x2c0d800, 0xc420cd4ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2c0d800, 0xc420cd4ed0)
	/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.f

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 19, 2017
tbg added a commit to tbg/cockroach that referenced this issue Oct 20, 2017
@tbg
storage: de-flake TestRefreshPendingCommands
1b44843 
The test ran a for loop without preemption points. The loop checked a
condition that would only become true after another goroutine had been
scheduled and carried out its job.

If, with only few cores (four in my case) GC kicked in before that other
goroutine got scheduled, that loop would just run hot forever until the
test timed out, and the resulting stack dump looked quite unhelpful.

Add a small sleep so the runtime can preempt the goroutine.

The issue was harder to run into when stressing only the test, since there
was less garbage available at that point. Adding some print statements,
I accidentally made it much more likely.

Previously flaked (got stuck) within <500iters, now ran past 1.5k without
problems.

Fixes cockroachdb#19397.
Fixes cockroachdb#19388.
Touches cockroachdb#19367.
Fixes cockroachdb#18554.
@tbg tbg mentioned this issue Oct 20, 2017
Merged
@tbg
Copy link
Member

tbg commented Oct 20, 2017 •
edited
Loading

The schema change test simply ran out of time (smaller DigitalOcean machine) and everything else has been fixed (or has open PRs).

@tbg tbg closed this as completed Oct 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tbg @cockroach-teamcity