We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The following tests appear to have failed:
#369201:
--- FAIL: test/TestRefreshPendingCommands (0.000s) Test ended in panic. --- FAIL: test/TestRefreshPendingCommands/#00 (0.000s) Test ended in panic. ------- Stdout: ------- I171005 05:45:02.378634 44797 storage/store.go:1209 [s1] [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available W171005 05:45:02.378778 44883 storage/store.go:1296 [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled W171005 05:45:02.378792 44884 storage/store.go:1296 [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled I171005 05:45:02.381396 44797 gossip/gossip.go:331 [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:34313" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > W171005 05:45:02.388125 44797 gossip/gossip.go:1277 [n2] no incoming or outgoing connections I171005 05:45:02.388251 44797 storage/store.go:1209 [s2] [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available I171005 05:45:02.388296 44797 gossip/gossip.go:331 [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:44664" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > I171005 05:45:02.388474 44471 gossip/client.go:129 [n2] started gossip client to 127.0.0.1:34313 W171005 05:45:02.459274 44797 gossip/gossip.go:1277 [n3] no incoming or outgoing connections I171005 05:45:02.459407 44797 storage/store.go:1209 [s3] [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available I171005 05:45:02.459458 44797 gossip/gossip.go:331 [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:50344" > attrs:<> locality:<> ServerVersion:<major_val:0 minor_val:0 patch:0 unstable:0 > I171005 05:45:02.459813 45131 gossip/client.go:129 [n3] started gossip client to 127.0.0.1:34313 I171005 05:45:02.533031 45030 storage/replica_raftstorage.go:526 [s1,r1/1:/M{in-ax}] generated preemptive snapshot dac77424 at index 15 I171005 05:45:02.533605 45030 storage/store.go:3504 [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 5, rate-limit: 2.0 MiB/sec, 0ms I171005 05:45:02.533855 44478 storage/replica_raftstorage.go:732 [s2,r1/?:{-}] applying preemptive snapshot at index 15 (id=dac77424, encoded size=4620, 1 rocksdb batches, 5 log entries) I171005 05:45:02.534149 44478 storage/replica_raftstorage.go:738 [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms] I171005 05:45:02.534460 45030 storage/replica_command.go:3684 [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2] I171005 05:45:02.535355 45035 storage/replica.go:3071 [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3 I171005 05:45:02.535990 45038 storage/replica_raftstorage.go:526 [s1,r1/1:/M{in-ax}] generated preemptive snapshot 32bedb13 at index 17 I171005 05:45:02.536760 45038 storage/store.go:3504 [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 7, rate-limit: 2.0 MiB/sec, 1ms I171005 05:45:02.537003 45189 storage/replica_raftstorage.go:732 [s3,r1/?:{-}] applying preemptive snapshot at index 17 (id=32bedb13, encoded size=5667, 1 rocksdb batches, 7 log entries) I171005 05:45:02.537301 45189 storage/replica_raftstorage.go:738 [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 0ms [clear=0ms batch=0ms entries=0ms commit=0ms] I171005 05:45:02.537638 45038 storage/replica_command.go:3684 [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3] I171005 05:45:02.538426 45198 storage/raft_transport.go:455 raft transport stream to node 1 established I171005 05:45:02.539648 44480 storage/replica.go:3071 [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 W171005 05:45:02.678351 45235 storage/raft_transport.go:281 unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3 W171005 05:45:02.678535 45172 storage/store.go:3242 [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found W171005 05:45:02.678614 45158 storage/raft_transport.go:461 raft transport stream to node 3 failed: store 3 was not found W171005 05:45:02.678684 45235 storage/raft_transport.go:281 unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3 I171005 05:45:02.679015 45159 storage/raft_transport.go:455 raft transport stream to node 3 established W171005 05:45:02.679187 45237 storage/raft_transport.go:281 unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3 W171005 05:45:02.679417 45161 storage/store.go:3242 [s1,r1/1:/M{in-ax}] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found W171005 05:45:02.679429 45237 storage/raft_transport.go:281 unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3 W171005 05:45:02.679645 45207 storage/raft_transport.go:281 unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1 W171005 05:45:02.679801 45200 storage/store.go:3242 [s2,r1/2:/M{in-ax}] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found W171005 05:45:02.679824 45198 storage/raft_transport.go:461 raft transport stream to node 1 failed: store 1 was not found W171005 05:45:02.680273 45312 storage/store.go:1296 [s1,r1/1:/M{in-ax}] could not gossip system config: periodic gossip is disabled I171005 05:45:02.680282 44797 storage/client_test.go:1194 test clock advanced to: 1.800000125,0 W171005 05:45:02.680297 45313 storage/store.go:1296 [s1,r1/1:/M{in-ax}] could not gossip node liveness: periodic gossip is disabled SIGQUIT: quit PC=0x6f6341 m=0 sigcode=0 goroutine 0 [idle]: runtime.futex(0x3035c90, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffc2e25e698, 0x6a789b, ...) /usr/local/go/src/runtime/sys_linux_amd64.s:438 +0x21 runtime.futexsleep(0x3035c90, 0x7ffc00000000, 0xffffffffffffffff) /usr/local/go/src/runtime/os_linux.go:45 +0x62 runtime.notesleep(0x3035c90) /usr/local/go/src/runtime/lock_futex.go:151 +0x9b runtime.stopm() /usr/local/go/src/runtime/proc.go:1670 +0xe5 runtime.findrunnable(0xc420021900, 0x0) /usr/local/go/src/runtime/proc.go:2125 +0x4d2 runtime.schedule() /usr/local/go/src/runtime/proc.go:2245 +0x12c runtime.park_m(0xc420460480) /usr/local/go/src/runtime/proc.go:2308 +0xb6 runtime.mcall(0x7ffc2e25e830) /usr/local/go/src/runtime/asm_amd64.s:286 +0x5b goroutine 1 [chan receive]: testing.(*T).Run(0xc4204d22d0, 0x1d1579a, 0x1a, 0x1dda9e8, 0x725b01) /usr/local/go/src/testing/testing.go:790 +0x2fc testing.runTests.func1(0xc4204d22d0) /usr/local/go/src/testing/testing.go:1004 +0x64 testing.tRunner(0xc4204d22d0, 0xc4206a7b80) /usr/local/go/src/testing/testing.go:746 +0xd0 testing.runTests(0xc42048ba80, 0x2b56ac0, 0x1df, 0x1df, 0xc4206a7c98) /usr/local/go/src/testing/testing.go:1002 +0x2d8 testing.(*M).Run(0xc420605f18, 0xc4206a7d80) /usr/local/go/src/testing/testing.go:921 +0x111 github.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc4206a7f18) /go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:57 +0x1e7 main.main() github.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:1018 +0xdb goroutine 26 [chan receive, 4 minutes]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x3034fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1005 +0x81 created by github.com/cockroachdb/cockroach/pkg/util/log.init.0 /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:581 +0xbf goroutine 34 [syscall, 4 minutes]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:131 +0xa6 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:28 +0x41 goroutine 44433 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b77f20, 0xc425ad9dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42033c350, 0xc420c97440, 0xc420394280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44809 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc4202c4410, 0xc420f70680, 0xc420451490, 0xf, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:419 +0x2bb github.com/cockroachdb/cockroach/pkg/rpc.(*Context).GRPCDial.func1.2.1(0x2b77f20, 0xc420a6c480) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:353 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204c5000, 0xc4201cc360, 0xc424ca01e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45275 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc40000002d) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c0c0, 0xc420c4ae10, 0xc42031c0a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44941 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000000d) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204afdb0, 0xc420c97440, 0xc4204afd80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44925 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2b77f20, 0xc424d2c570) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204af8e0, 0xc4201cc360, 0xc4204953a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44977 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000031) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e5e0, 0xc420c97440, 0xc42048e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44950 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000016) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e1c0, 0xc420c97440, 0xc42048e1b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44966 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000045) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e480, 0xc420c97440, 0xc42048e470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45253 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000005) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc4207a97d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc4207a97d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247c20, 0xc420c4ae10, 0xc420247c10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44958 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000001e) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e2c0, 0xc420c97440, 0xc42048e2b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44938 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000004a) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204afbc0, 0xc420c97440, 0xc4204afba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45274 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc40000002b) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c070, 0xc420c4ae10, 0xc42031c040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45013 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b77f20, 0xc425ad9e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42033c460, 0xc420c97440, 0xc420394320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44971 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000002b) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad92f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad92f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e520, 0xc420c97440, 0xc42048e510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45134 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4248d5ea0, 0xc4202fa030, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:126 +0x300 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*transportReader).Read(0xc420c75410, 0xc4202fa030, 0x5, 0x5, 0x70df4e, 0xc4208cc3c0, 0xc420a71780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:391 +0x55 io.ReadAtLeast(0x2b616a0, 0xc420c75410, 0xc4202fa030, 0x5, 0x5, 0x5, 0xc420d75d40, 0xc4208cc300, 0xc400000005) /usr/local/go/src/io/io.go:309 +0x86 io.ReadFull(0x2b616a0, 0xc420c75410, 0xc4202fa030, 0x5, 0x5, 0xa, 0x0, 0xb) /usr/local/go/src/io/io.go:327 +0x58 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*Stream).Read(0xc4208cc300, 0xc4202fa030, 0x5, 0x5, 0x0, 0xc420022c70, 0xc420022c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/transport.go:375 +0xbf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4202fa020, 0x7fffffff, 0xc420b1e840, 0xc400000001, 0x3078aa0, 0x0, 0x0, 0xc420704d80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:265 +0x65 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc4202fa020, 0x2b77760, 0x3077700, 0xc4208cc300, 0x2b6a1e0, 0x3077700, 0x1c07e80, 0xc420e15630, 0x7fffffff, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:361 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc424b04840, 0x1c07e80, 0xc420e15630, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:398 +0xe6 github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc42035aba0, 0x2b77f20, 0xc420c75470, 0xc420d47c80) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:194 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0x2b82f60, 0xc42035aba0, 0xc424d9f040, 0x2b77f20, 0xc420c75470, 0xc420d47c80, 0xc420dadf70, 0x6c2349) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:311 +0x35 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x2b77f20, 0xc420c75470) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:319 +0xc0 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42035ac60, 0xc4201cc360, 0xc420a717c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44923 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc424a94380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff goroutine 44979 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000033) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e620, 0xc420c97440, 0xc42048e610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45301 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000030) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c4f0, 0xc420c4ae10, 0xc42031c4e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45307 [semacquire]: sync.runtime_Semacquire(0xc420c4af24) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc420c4af18) /usr/local/go/src/sync/waitgroup.go:131 +0x72 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc420c4aea0, 0x2b77f20, 0xc420aee9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:173 +0x31 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm(0x2b77f20, 0xc420aee9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3635 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c5f0, 0xc420c4ae10, 0xc42031c5e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44981 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000035) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad94d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad94d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e660, 0xc420c97440, 0xc42048e650) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45317 [runnable]: github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1.1(0xc420d75380, 0xc4203ea100, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077 created by github.com/cockroachdb/cockroach/pkg/storage_test.TestRefreshPendingCommands.func1 /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_raft_test.go:1077 +0x765 goroutine 45267 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000022) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247ea0, 0xc420c4ae10, 0xc420247e90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45312 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x2b77f20, 0xc4207a9ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1308 +0x3e9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c6e0, 0xc420c4ae10, 0xc424b20320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44947 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000013) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e160, 0xc420c97440, 0xc42048e150) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44939 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000004b) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204afc30, 0xc420c97440, 0xc4204afbd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45254 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000009) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247c40, 0xc420c4ae10, 0xc420247c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44810 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x2b77f20, 0xc420a6c4b0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:384 +0x5a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204c5040, 0xc4201cc360, 0xc4203077e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44917 [IO wait]: internal/poll.runtime_pollWait(0x7f7d3697ad30, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420310698, 0x72, 0xc4208a2c00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc420310698, 0xffffffffffffff00, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc420310680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:334 +0x1e2 net.(*netFD).accept(0xc420310680, 0xc4209ee080, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc420752298, 0xb50589, 0x6f1610, 0xc4208a2e28) /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc420752298, 0x1ddbd28, 0xc420c26f00, 0x2b838c0, 0xc4209ee080) /usr/local/go/src/net/tcpsock.go:247 +0x49 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc420c26f00, 0x2b76560, 0xc420752298, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:444 +0x196 github.com/cockroachdb/cockroach/pkg/util/netutil.ListenAndServeGRPC.func2(0x2b77f20, 0xc425ad86f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:56 +0x3c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204af660, 0xc4201cc360, 0xc420495140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45138 [running]: goroutine running on other thread; stack unavailable created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:549 +0xa1 goroutine 45178 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x2b77f20, 0xc4207a96e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:157 +0x49 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247a00, 0xc420c4ae10, 0xc420337200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44936 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000048) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4204afb50, 0xc420c97440, 0xc4204afb30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44975 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000002f) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad93b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad93b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e5a0, 0xc420c97440, 0xc42048e590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45213 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b77f20, 0xc420a501b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42044a5b0, 0xc420c4ae10, 0xc420264d60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44906 [IO wait]: internal/poll.runtime_pollWait(0x7f7d3697a8b0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc42474a098, 0x72, 0xffffffffffffff00, 0x2b655e0, 0x2b5a210) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc42474a098, 0xc420e7e000, 0x8000, 0x8000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc42474a080, 0xc420e7e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a net.(*netFD).Read(0xc42474a080, 0xc420e7e000, 0x8000, 0x8000, 0x69c24c, 0xc4206c5800, 0x4) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4209ee080, 0xc420e7e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d bufio.(*Reader).Read(0xc420ed01e0, 0xc4203a43b8, 0x9, 0x9, 0xc4208a3a90, 0xc4208a3b18, 0xb39fa7) /usr/local/go/src/bufio/bufio.go:213 +0x30b io.ReadAtLeast(0x2b5bf20, 0xc420ed01e0, 0xc4203a43b8, 0x9, 0x9, 0x9, 0xc4247ab838, 0xc42030d950, 0x4b) /usr/local/go/src/io/io.go:309 +0x86 io.ReadFull(0x2b5bf20, 0xc420ed01e0, 0xc4203a43b8, 0x9, 0x9, 0x4b, 0x4b, 0x0) /usr/local/go/src/io/io.go:327 +0x58 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203a43b8, 0x9, 0x9, 0x2b5bf20, 0xc420ed01e0, 0x0, 0x4b00000000, 0x4b, 0x4b) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203a4380, 0x4b, 0x4b, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc425ad8780, 0xc425ad87b0, 0xc425ad87b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc424b042c0, 0xc425ad8930, 0x1ddbd60) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:377 +0x30d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420c26f00, 0x2b82240, 0xc424b042c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:547 +0x140 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveHTTP2Transport(0xc420c26f00, 0x2b838c0, 0xc4209ee080, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:540 +0x463 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc420c26f00, 0x2b838c0, 0xc4209ee080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:507 +0x497 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:476 +0x5b9 goroutine 44945 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000011) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e0d0, 0xc420c97440, 0xc42048e0c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44991 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000037) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e840, 0xc420c97440, 0xc42048e7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45298 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc40000003b) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc4207a9e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc4207a9e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c490, 0xc420c4ae10, 0xc42031c480) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45049 [IO wait]: internal/poll.runtime_pollWait(0x7f7d3697abb0, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc42474a918, 0x72, 0xffffffffffffff00, 0x2b655e0, 0x2b5a210) /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae internal/poll.(*pollDesc).waitRead(0xc42474a918, 0xc424d6a000, 0x8000, 0x8000) /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc42474a900, 0xc424d6a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:125 +0x18a net.(*netFD).Read(0xc42474a900, 0xc424d6a000, 0x8000, 0x8000, 0x5a, 0x0, 0xc424f93cb0) /usr/local/go/src/net/fd_unix.go:202 +0x52 net.(*conn).Read(0xc4209ee198, 0xc424d6a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:176 +0x6d bufio.(*Reader).Read(0xc4248a5c80, 0xc4203a5298, 0x9, 0x9, 0xc420b4c638, 0xc420c74fc0, 0xc400000007) /usr/local/go/src/bufio/bufio.go:213 +0x30b io.ReadAtLeast(0x2b5bf20, 0xc4248a5c80, 0xc4203a5298, 0x9, 0x9, 0x9, 0xc424f93d00, 0x69834e, 0xc424f93e5f) /usr/local/go/src/io/io.go:309 +0x86 io.ReadFull(0x2b5bf20, 0xc4248a5c80, 0xc4203a5298, 0x9, 0x9, 0x516f2cac01, 0xa600000000000051, 0x0) /usr/local/go/src/io/io.go:327 +0x58 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203a5298, 0x9, 0x9, 0x2b5bf20, 0xc4248a5c80, 0x0, 0x0, 0xc420c74ea0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x7b github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4203a5260, 0x2b688e0, 0xc420c74ea0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa4 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc420c74e70, 0xc420c74ea0, 0xc420c74ea0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http_util.go:592 +0x2f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420b4c4e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1154 +0xc0 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:262 +0xbf8 goroutine 45214 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x2b77f20, 0xc420a501e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:504 +0x1bb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42044a7f0, 0xc420c4ae10, 0xc420264d80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45182 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000004) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc4207a97a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc4207a97a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247b10, 0xc420c4ae10, 0xc420247b00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44985 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000039) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad95c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad95c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e6e0, 0xc420c97440, 0xc42048e6d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45052 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc424e2e1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:927 +0x1de created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:630 +0x6ff goroutine 44969 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000029) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e4e0, 0xc420c97440, 0xc42048e4d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45263 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000019) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247df0, 0xc420c4ae10, 0xc420247dd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45303 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc40000003a) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aeea80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aeea80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c540, 0xc420c4ae10, 0xc42031c520) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45272 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000027) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc4207a9b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc4207a9b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c010, 0xc420c4ae10, 0xc42031c000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44948 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc400000014) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad8e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad8e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e180, 0xc420c97440, 0xc42048e170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44986 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000003a) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad95f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad95f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e710, 0xc420c97440, 0xc42048e700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45055 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x2b77f20, 0xc420c75260) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1136 +0x331 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42039fea0, 0xc4201cc360, 0xc42039fe90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 44921 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc4206d7860) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1227 +0x148 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:292 +0xd2e goroutine 44987 [semacquire]: sync.runtime_notifyListWait(0xc420800ad0, 0xc40000003b) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc420800ac0) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c974d0, 0x2b77f20, 0xc425ad9620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc425ad9620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42048e740, 0xc420c97440, 0xc42048e730) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45315 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x2b77f20, 0xc4207a9fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:366 +0x2a9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c800, 0xc420c4ae10, 0xc424b20410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45261 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000015) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc420aee600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc420aee600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc420247d90, 0xc420c4ae10, 0xc420247d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xad goroutine 45291 [semacquire]: sync.runtime_notifyListWait(0xc424df1a10, 0xc400000036) /usr/local/go/src/runtime/sema.go:507 +0x110 sync.(*Cond).Wait(0xc424df1a00) /usr/local/go/src/sync/cond.go:56 +0x80 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc420c4aea0, 0x2b77f20, 0xc4207a9cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:198 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x2b77f20, 0xc4207a9cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc42031c3b0, 0xc420c4ae10, 0xc42031c3a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0xf3 created by gi
Please assign, take a look and update the issue accordingly.
The text was updated successfully, but these errors were encountered:
#18554
Sorry, something went wrong.
No branches or pull requests
The following tests appear to have failed:
#369201:
Please assign, take a look and update the issue accordingly.
The text was updated successfully, but these errors were encountered: