Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stress: failed test in cockroach/storage/storage.test: TestSplitSnapshotRace_SnapshotWins #8170

Closed
cockroach-teamcity opened this issue Aug 1, 2016 · 7 comments
Closed

stress: failed test in cockroach/storage/storage.test: TestSplitSnapshotRace_SnapshotWins #8170

cockroach-teamcity opened this issue Aug 1, 2016 · 7 comments
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone
Q3

Comments

@cockroach-teamcity
Copy link
Member

Binary: cockroach/static-tests.tar.gz sha: https://github.com/cockroachdb/cockroach/commits/b624f98deda333293ba82b5bf9bf603e6670728d

Stress build found a failed test:

=== RUN   TestSplitSnapshotRace_SnapshotWins
I160801 05:20:24.713904 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160801 05:20:24.714570 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.715668 storage/store.go:2774  system config not yet available
I160801 05:20:24.716234 storage/replica_command.go:1716  store=1:1 ([]=) range=1 [/Min-/Max): new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0 [physicalTime=1970-01-01 00:00:00 +0000 UTC]
I160801 05:20:24.716487 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160801 05:20:24.717119 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.717547 storage/store.go:2774  system config not yet available
I160801 05:20:24.717660 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160801 05:20:24.718042 gossip/client.go:75  starting client to 127.0.0.1:34696
W160801 05:20:24.718335 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.718920 storage/store.go:2774  system config not yet available
I160801 05:20:24.719006 gossip/client.go:75  starting client to 127.0.0.1:34696
I160801 05:20:24.719118 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160801 05:20:24.719868 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.722615 storage/store.go:2774  system config not yet available
I160801 05:20:24.722713 gossip/client.go:75  starting client to 127.0.0.1:34696
I160801 05:20:24.723264 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
W160801 05:20:24.723855 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.725033 storage/store.go:2774  system config not yet available
I160801 05:20:24.725440 storage/engine/rocksdb.go:353  opening in memory rocksdb instance
I160801 05:20:24.725686 gossip/client.go:75  starting client to 127.0.0.1:43660
W160801 05:20:24.725973 gossip/gossip.go:1006  not connected to cluster; use --join to specify a connected node
I160801 05:20:24.726393 storage/store.go:2774  system config not yet available
I160801 05:20:24.726520 gossip/client.go:75  starting client to 127.0.0.1:34696
I160801 05:20:24.726813 gossip/server.go:212  refusing gossip from node 6 (max 3 conns); forwarding to 4 ({tcp 127.0.0.1:36138})
I160801 05:20:24.726903 gossip/server.go:212  refusing gossip from node 6 (max 3 conns); forwarding to 2 ({tcp 127.0.0.1:43660})
I160801 05:20:24.726952 gossip/client.go:99  closing client to node 1 (127.0.0.1:34696): received forward from node 1 to 4 (127.0.0.1:36138)
I160801 05:20:24.727037 gossip/client.go:75  starting client to 127.0.0.1:36138
I160801 05:20:24.728271 storage/replica_command.go:2305  store=1:1 ([]=) range=1 [/Min-/Max): initiating a split of this range at key /System/Max
E160801 05:20:24.729729 storage/queue.go:508  [replicate] (purgatory) on store=1:1 ([]=) range=1 [/Min-/System/Max): 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster
I160801 05:20:24.729978 storage/replica_raftstorage.go:467  generated snapshot for range 2 at index 10 in 110.119µs. encoded size=2445, 25 KV pairs, 0 log entries
E160801 05:20:24.730115 storage/queue.go:508  [replicate] (purgatory) on store=1:1 ([]=) range=2 [/System/Max-/Max): 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster
I160801 05:20:24.730218 storage/replica_command.go:3181  store=1:1 ([]=) range=2 [/System/Max-/Max): change replicas of 2: read existing descriptor range_id:2 start_key:"\005" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 
W160801 05:20:24.730459 storage/stores.go:206  range not contained in one range: [/Meta2/Max,"\x05\x00"), but have [/Min,/System/Max)
I160801 05:20:24.730663 storage/replica_raftstorage.go:588  store=2:2 ([]=) range=2 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 2 at index 10 (encoded size=2445, 25 KV pairs, 0 log entries)
I160801 05:20:24.730874 storage/replica_raftstorage.go:591  store=2:2 ([]=) range=2 [/System/Max-/Max): with replicaID [?], applied preemptive snapshot for range 2 in 178.12µs
I160801 05:20:24.730887 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:24.731613 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
W160801 05:20:24.732190 storage/intent_resolver.go:307  store=1:1 ([]=) range=1 [/Min-/System/Max): failed to push during intent resolution: failed to push "change-replica" id=ac455d6c key=/Local/Range/"\x05"/RangeDescriptor rw=true pri=0.01138914 iso=SERIALIZABLE stat=PENDING epo=0 ts=0.000000000,88 orig=0.000000000,88 max=0.000000000,88 wto=false rop=false
I160801 05:20:24.732254 storage/replica.go:1419  store=1:1 ([]=) range=2 [/System/Max-/Max): proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]
I160801 05:20:24.732453 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.733057 storage/replica_raftstorage.go:467  generated snapshot for range 2 at index 14 in 135.92µs. encoded size=3486, 27 KV pairs, 4 log entries
I160801 05:20:24.733326 storage/replica_command.go:3181  store=1:1 ([]=) range=2 [/System/Max-/Max): change replicas of 2: read existing descriptor range_id:2 start_key:"\005" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > next_replica_id:3 
I160801 05:20:24.733741 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:24.734079 storage/replica_raftstorage.go:588  store=3:3 ([]=) range=2 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 2 at index 14 (encoded size=3486, 27 KV pairs, 4 log entries)
I160801 05:20:24.734492 storage/replica_raftstorage.go:591  store=3:3 ([]=) range=2 [/System/Max-/Max): with replicaID [?], applied preemptive snapshot for range 2 in 375.98µs
I160801 05:20:24.734625 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.734891 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.735639 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.735957 storage/replica.go:1419  store=1:1 ([]=) range=2 [/System/Max-/Max): proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:3} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3}]
I160801 05:20:24.736008 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.736783 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.737074 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.737771 storage/replica_raftstorage.go:467  generated snapshot for range 2 at index 17 in 196.888µs. encoded size=4329, 28 KV pairs, 7 log entries
I160801 05:20:24.738603 storage/replica_raftstorage.go:588  store=4:4 ([]=) range=2 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 2 at index 17 (encoded size=4329, 28 KV pairs, 7 log entries)
I160801 05:20:24.738830 storage/replica_command.go:3181  store=1:1 ([]=) range=2 [/System/Max-/Max): change replicas of 2: read existing descriptor range_id:2 start_key:"\005" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > next_replica_id:4 
I160801 05:20:24.739077 storage/replica_raftstorage.go:591  store=4:4 ([]=) range=2 [/System/Max-/Max): with replicaID [?], applied preemptive snapshot for range 2 in 419.807µs
I160801 05:20:24.739148 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:24.739440 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.739664 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.740172 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.740326 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.740454 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.740510 storage/replica.go:1419  store=1:1 ([]=) range=2 [/System/Max-/Max): proposing ADD_REPLICA {NodeID:4 StoreID:4 ReplicaID:4} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3} {NodeID:4 StoreID:4 ReplicaID:4}]
I160801 05:20:24.741169 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.741466 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.741805 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.741937 storage/replica_command.go:3181  store=1:1 ([]=) range=2 [/System/Max-/Max): change replicas of 2: read existing descriptor range_id:2 start_key:"\005" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > replicas:<node_id:4 store_id:4 replica_id:4 > next_replica_id:5 
I160801 05:20:24.742246 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:24.743595 storage/replica.go:1982  store=4:4 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.743793 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.743825 storage/replica.go:1982  store=4:4 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.744005 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.744180 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.744222 storage/replica.go:1419  store=1:1 ([]=) range=2 [/System/Max-/Max): proposing REMOVE_REPLICA {NodeID:1 StoreID:1 ReplicaID:1} for range 2: [{NodeID:4 StoreID:4 ReplicaID:4} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3}]
I160801 05:20:24.744358 storage/replica.go:1982  store=4:4 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"\x05"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:24.745095 storage/replica.go:1982  store=1:1 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.745229 storage/replica.go:1982  store=2:2 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.745325 storage/replica.go:1982  store=3:3 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:24.745378 storage/replica.go:1982  store=4:4 ([]=) range=2 [/System/Max-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"\x05"/RangeDescriptor,/Min), pErr=<nil>
E160801 05:20:24.745445 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.745471 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.745599 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.745623 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.745744 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.745765 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.745842 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.745867 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.745962 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.745984 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746049 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746064 storage/client_test.go:817  engine 5: missing key "a"
W160801 05:20:24.746086 storage/intent_resolver.go:350  could not GC completed transaction: range 2 was not found
E160801 05:20:24.746157 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746179 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746246 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746266 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746346 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746362 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746422 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746439 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746497 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746511 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746580 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746596 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746653 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746668 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746736 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746751 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.746837 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.746857 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.747003 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.747020 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.747176 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.747192 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.747379 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.747397 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.747715 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.747735 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.748139 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.748166 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.748868 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.748888 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.750103 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.750127 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.752416 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.752436 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.756740 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.756764 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.765373 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.765404 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:24.782423 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.782447 storage/client_test.go:817  engine 5: missing key "a"
W160801 05:20:24.816158 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
E160801 05:20:24.816202 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.816220 storage/client_test.go:817  engine 5: missing key "a"
W160801 05:20:24.816254 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 3} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:24.816309 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
E160801 05:20:24.883535 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:24.883568 storage/client_test.go:817  engine 5: missing key "a"
W160801 05:20:24.916373 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:24.916543 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 3} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:24.916555 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:25.016367 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:25.016380 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
W160801 05:20:25.016401 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 3} closed by the remote: rpc error: code = 2 desc = range 2: discarding message from {NodeID:1 StoreID:1 ReplicaID:1}, older than NextReplicaID 5
E160801 05:20:25.018033 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:25.018070 storage/client_test.go:817  engine 5: missing key "a"
I160801 05:20:25.020718 storage/replica_command.go:1732  range 2: replicaID 2 transfer raft leadership to replicaID 4
I160801 05:20:25.021113 storage/replica_command.go:1716  store=4:4 ([]=) range=2 [/System/Max-/Max): new range lease replica {4 4 4} 1970-01-01 00:00:00.9 +0000 UTC 1.8s following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms [physicalTime=1970-01-01 00:00:01.8 +0000 UTC]
E160801 05:20:25.286685 storage/client_test.go:817  engine 0: missing key "a"
E160801 05:20:25.286742 storage/client_test.go:817  engine 4: missing key "a"
E160801 05:20:25.286759 storage/client_test.go:817  engine 5: missing key "a"
E160801 05:20:25.286782 storage/client_test.go:817  engine 0: missing key "z"
E160801 05:20:25.286815 storage/client_test.go:817  engine 4: missing key "z"
E160801 05:20:25.286829 storage/client_test.go:817  engine 5: missing key "z"
W160801 05:20:25.287493 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.318205 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.519691 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:3 StoreID:3 ReplicaID:3}
W160801 05:20:25.520367 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:3 StoreID:3 ReplicaID:3}
I160801 05:20:25.521727 storage/replica_command.go:1732  range 2: replicaID 3 transfer raft leadership to replicaID 2
I160801 05:20:25.522125 storage/replica_command.go:1716  store=2:2 ([]=) range=2 [/System/Max-/Max): new range lease replica {2 2 2} 1970-01-01 00:00:02.7 +0000 UTC 3.6s following replica {4 4 4} 1970-01-01 00:00:00.9 +0000 UTC 1.8s [physicalTime=1970-01-01 00:00:05.4 +0000 UTC]
W160801 05:20:25.523066 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
I160801 05:20:25.523602 storage/replica_command.go:2305  store=2:2 ([]=) range=2 [/System/Max-/Max): initiating a split of this range at key "m"
W160801 05:20:25.618778 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.619216 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
I160801 05:20:25.620942 storage/replica_raftstorage.go:467  generated snapshot for range 11 at index 12 in 85.294µs. encoded size=2594, 24 KV pairs, 2 log entries
I160801 05:20:25.621361 storage/replica_command.go:3181  store=2:2 ([]=) range=11 ["m"-/Max): change replicas of 11: read existing descriptor range_id:11 start_key:"m" end_key:"\377\377" replicas:<node_id:4 store_id:4 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > next_replica_id:4 
I160801 05:20:25.621865 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:25.622011 storage/replica_raftstorage.go:588  store=5:5 ([]=) range=11 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 11 at index 12 (encoded size=2594, 24 KV pairs, 2 log entries)
I160801 05:20:25.622734 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.623161 storage/replica.go:1419  store=2:2 ([]=) range=11 ["m"-/Max): proposing ADD_REPLICA {NodeID:5 StoreID:5 ReplicaID:4} for range 11: [{NodeID:4 StoreID:4 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3} {NodeID:5 StoreID:5 ReplicaID:4}]
I160801 05:20:25.623267 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.623955 storage/replica_raftstorage.go:591  store=5:5 ([]=) range=11 ["m"-/Max): with replicaID [?], applied preemptive snapshot for range 11 in 1.908036ms
I160801 05:20:25.624064 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.624447 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.625911 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.626079 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.627286 storage/replica_raftstorage.go:467  generated snapshot for range 11 at index 16 in 96.49µs. encoded size=3650, 25 KV pairs, 6 log entries
I160801 05:20:25.627565 storage/replica_command.go:3181  store=2:2 ([]=) range=11 ["m"-/Max): change replicas of 11: read existing descriptor range_id:11 start_key:"m" end_key:"\377\377" replicas:<node_id:4 store_id:4 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > replicas:<node_id:5 store_id:5 replica_id:4 > next_replica_id:5 
I160801 05:20:25.628140 storage/replica_raftstorage.go:588  store=6:6 ([]=) range=11 [/Min-/Min): with replicaID [?], applying preemptive snapshot for range 11 at index 16 (encoded size=3650, 25 KV pairs, 6 log entries)
I160801 05:20:25.628342 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:25.628544 storage/replica_raftstorage.go:591  store=6:6 ([]=) range=11 ["m"-/Max): with replicaID [?], applied preemptive snapshot for range 11 in 364.957µs
I160801 05:20:25.629231 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.629432 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.629601 storage/replica.go:1419  store=2:2 ([]=) range=11 ["m"-/Max): proposing ADD_REPLICA {NodeID:6 StoreID:6 ReplicaID:5} for range 11: [{NodeID:4 StoreID:4 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3} {NodeID:5 StoreID:5 ReplicaID:4} {NodeID:6 StoreID:6 ReplicaID:5}]
I160801 05:20:25.629645 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.630311 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.630622 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.630637 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.631976 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.632217 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.632695 storage/replica_command.go:3181  store=2:2 ([]=) range=11 ["m"-/Max): change replicas of 11: read existing descriptor range_id:11 start_key:"m" end_key:"\377\377" replicas:<node_id:4 store_id:4 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > replicas:<node_id:5 store_id:5 replica_id:4 > replicas:<node_id:6 store_id:6 replica_id:5 > next_replica_id:6 
I160801 05:20:25.633161 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:25.634192 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.634331 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.634403 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.634450 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.634601 storage/replica.go:1419  store=2:2 ([]=) range=11 ["m"-/Max): proposing REMOVE_REPLICA {NodeID:3 StoreID:3 ReplicaID:3} for range 11: [{NodeID:4 StoreID:4 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:6 StoreID:6 ReplicaID:5} {NodeID:5 StoreID:5 ReplicaID:4}]
I160801 05:20:25.635542 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.635886 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.635972 storage/replica.go:1982  store=3:3 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.636065 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
W160801 05:20:25.636474 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:3 StoreID:3 ReplicaID:3}, older than NextReplicaID 6
I160801 05:20:25.638283 storage/replica_command.go:3181  store=2:2 ([]=) range=11 ["m"-/Max): change replicas of 11: read existing descriptor range_id:11 start_key:"m" end_key:"\377\377" replicas:<node_id:4 store_id:4 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:6 store_id:6 replica_id:5 > replicas:<node_id:5 store_id:5 replica_id:4 > next_replica_id:6 
I160801 05:20:25.638798 storage/replica.go:1982  store=1:1 ([]=) range=1 [/Min-/System/Max): applied part of replica change txn: Noop [/Min,/Min), Noop [/Min,/Min), Put [/Meta2/Max,/Min), pErr=<nil>
I160801 05:20:25.639756 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.639975 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.640033 storage/replica.go:1419  store=2:2 ([]=) range=11 ["m"-/Max): proposing REMOVE_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 11: [{NodeID:4 StoreID:4 ReplicaID:1} {NodeID:5 StoreID:5 ReplicaID:4} {NodeID:6 StoreID:6 ReplicaID:5}]
I160801 05:20:25.640286 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: BeginTransaction [/Local/Range/"m"/RangeDescriptor,/Min), ConditionalPut [/Local/Range/"m"/RangeDescriptor,/Min), Noop [/Min,/Min), pErr=<nil>
I160801 05:20:25.640867 storage/replica.go:1982  store=2:2 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.641211 storage/replica.go:1982  store=6:6 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
I160801 05:20:25.641474 storage/replica.go:1982  store=5:5 ([]=) range=11 ["m"-/Max): applied part of replica change txn: EndTransaction [/Local/Range/"m"/RangeDescriptor,/Min), pErr=<nil>
W160801 05:20:25.642453 storage/intent_resolver.go:350  could not GC completed transaction: range 11 was not found
I160801 05:20:25.715777 gossip/client.go:75  starting client to 127.0.0.1:43660
I160801 05:20:25.716260 gossip/client.go:99  closing client to node 2 (127.0.0.1:43660): stopping outgoing client to node 2 (127.0.0.1:43660); already have incoming
I160801 05:20:25.716287 gossip/gossip.go:976  node has connected to cluster via gossip
W160801 05:20:25.718158 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.718183 storage/replica.go:337  range 11: outgoing raft transport stream to {6 6 5} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.718243 storage/replica.go:337  range 11: outgoing raft transport stream to {5 5 4} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.718261 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.818282 storage/replica.go:337  range 11: outgoing raft transport stream to {6 6 5} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.818333 storage/replica.go:337  range 11: outgoing raft transport stream to {5 5 4} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.818347 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.818362 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.918454 storage/replica.go:337  range 11: outgoing raft transport stream to {5 5 4} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.918483 storage/replica.go:337  range 11: outgoing raft transport stream to {6 6 5} closed by the remote: rpc error: code = 2 desc = range 11: discarding message from {NodeID:2 StoreID:2 ReplicaID:2}, older than NextReplicaID 6
W160801 05:20:25.918487 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.918497 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 4} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:4}: no store registered for {NodeID:2 StoreID:2 ReplicaID:2}
W160801 05:20:25.927275 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:6 StoreID:6 ReplicaID:5}
W160801 05:20:25.928052 storage/replica.go:337  range 11: outgoing raft transport stream to {4 4 1} closed by the remote: rpc error: code = 2 desc = unable to accept Raft message from {NodeID:4 StoreID:4 ReplicaID:1}: no store registered for {NodeID:6 StoreID:6 ReplicaID:5}
I160801 05:20:25.929953 storage/replica_command.go:1716  store=5:5 ([]=) range=11 ["m"-/Max): new range lease replica {5 5 4} 1970-01-01 00:00:13.5 +0000 UTC 7.2s following replica {2 2 2} 1970-01-01 00:00:02.7 +0000 UTC 10.8s [physicalTime=1970-01-01 00:00:21.6 +0000 UTC]
E160801 05:20:25.932963 storage/client_test.go:817  engine 0: missing key "z"
E160801 05:20:25.933018 storage/client_test.go:817  engine 2: missing key "z"
E160801 05:20:25.933181 storage/client_test.go:817  engine 0: missing key "z"
E160801 05:20:25.933225 storage/client_test.go:817  engine 2: missing key "z"
W160801 05:20:25.933334 storage/replica.go:1302  store=1:1 ([]=) range=1 [/Min-/System/Max): unable to cancel expired Raft command ResolveIntent [/Meta2/Max,/Min)
E160801 05:20:25.933360 storage/client_test.go:817  engine 0: missing key "z"
E160801 05:20:25.933392 storage/client_test.go:817  engine 1: missing key "z"
E160801 05:20:25.933415 storage/client_test.go:817  engine 2: missing key "z"
I160801 05:20:49.073872 gossip/gossip.go:976  node has connected to cluster via gossip
I160801 05:20:50.256830 gossip/gossip.go:976  node has connected to cluster via gossip
I160801 05:20:55.259668 gossip/gossip.go:976  node has connected to cluster via gossip
I160801 05:20:57.225644 gossip/gossip.go:976  node has connected to cluster via gossip
I160801 05:21:00.859151 gossip/gossip.go:976  node has connected to cluster via gossip
W160801 05:21:24.730436 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 0} closed by the remote: <nil>
W160801 05:21:24.734080 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 0} closed by the remote: <nil>
W160801 05:21:24.738346 storage/replica.go:337  range 2: outgoing raft transport stream to {4 4 0} closed by the remote: <nil>
W160801 05:21:24.745462 storage/replica.go:337  range 2: outgoing raft transport stream to {1 1 1} closed by the remote: <nil>
W160801 05:21:24.745691 storage/replica.go:337  range 2: outgoing raft transport stream to {1 1 1} closed by the remote: <nil>
W160801 05:21:24.746496 storage/replica.go:337  range 2: outgoing raft transport stream to {1 1 1} closed by the remote: <nil>
W160801 05:21:25.223038 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: <nil>
W160801 05:21:25.223075 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 3} closed by the remote: <nil>
W160801 05:21:25.621759 storage/replica.go:337  range 11: outgoing raft transport stream to {5 5 0} closed by the remote: <nil>
W160801 05:21:25.627827 storage/replica.go:337  range 11: outgoing raft transport stream to {6 6 0} closed by the remote: <nil>
W160801 05:21:25.641552 storage/replica.go:337  range 11: outgoing raft transport stream to {2 2 2} closed by the remote: <nil>
W160801 05:21:25.825597 storage/replica.go:337  range 11: outgoing raft transport stream to {2 2 2} closed by the remote: <nil>
W160801 05:21:25.918182 storage/replica.go:337  range 2: outgoing raft transport stream to {3 3 3} closed by the remote: <nil>
W160801 05:21:25.918440 storage/replica.go:337  range 2: outgoing raft transport stream to {2 2 2} closed by the remote: <nil>
W160801 05:21:25.933430 storage/replica.go:337  range 11: outgoing raft transport stream to {5 5 4} closed by the remote: <nil>
W160801 05:21:25.933610 storage/replica.go:337  range 11: outgoing raft transport stream to {6 6 5} closed by the remote: <nil>
SIGABRT: abort
PC=0x64ed51 m=0

goroutine 0 [idle]:
runtime.futex(0x2bc55c8, 0x0, 0x0, 0x0, 0x0, 0x2bc41b0, 0x0, 0x0, 0x5fd4e4, 0x2bc55c8, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:306 +0x21
runtime.futexsleep(0x2bc55c8, 0x0, 0xffffffffffffffff)
    /usr/local/go/src/runtime/os1_linux.go:40 +0x53
runtime.notesleep(0x2bc55c8)
    /usr/local/go/src/runtime/lock_futex.go:145 +0xa4
runtime.stopm()
    /usr/local/go/src/runtime/proc.go:1538 +0x10b
runtime.gcstopm()
    /usr/local/go/src/runtime/proc.go:1742 +0x128
runtime.findrunnable(0xc82001aa00, 0x0)
    /usr/local/go/src/runtime/proc.go:1807 +0x35
runtime.schedule()
    /usr/local/go/src/runtime/proc.go:2075 +0x24f
runtime.park_m(0xc82025aa80)
    /usr/local/go/src/runtime/proc.go:2140 +0x18b
runtime.mcall(0x7fff1e060490)
    /usr/local/go/src/runtime/asm_amd64.s:233 +0x5b

goroutine 1 [chan receive, 8 minutes]:
testing.RunTests(0x205b538, 0x28a8e40, 0x128, 0x128, 0xc820270301)
    /usr/local/go/src/testing/testing.go:583 +0x8d2
testing.(*M).Run(0xc8204dbef8, 0xc820247c98)
    /usr/local/go/src/testing/testing.go:515 +0x81
github.com/cockroachdb/cockroach/storage_test.TestMain(0xc8204dbef8)
    /go/src/github.com/cockroachdb/cockroach/storage/main_test.go:57 +0x260
main.main()
    github.com/cockroachdb/cockroach/storage/_test/_testmain.go:654 +0x114

goroutine 17 [syscall, 9 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [runnable]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x2bc40c0)
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:1015 +0x64
created by github.com/cockroachdb/cockroach/util/log.init.1
    /go/src/github.com/cockroachdb/cockroach/util/log/clog.go:598 +0x8a

goroutine 82159 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827c08270)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82288 [select, 2 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc82e6bf940, 0xc82e6f7530, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc8252f4b40, 0xc82e6f7530, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc8252f4b40, 0xc82e6f7530, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc8252f4b40, 0xc82e6f7530, 0x5, 0x5, 0xae4ffa, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e6f7520, 0x7fffffff, 0xd40000c82e0f1920, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e6f7520, 0x7fd8f681dcb8, 0x2c14d90, 0xc8252f4b40, 0x0, 0x0, 0x1b57840, 0xc82531f7c0, 0x7fffffff, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*clientStream).RecvMsg(0xc82508e900, 0x1b57840, 0xc82531f7c0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xbb
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc827f7f850, 0xc820768160, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x7e
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x7fd8f92e8088, 0xc827f7f850, 0xc8252e4340, 0xc820768160, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x37
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x51
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e0a1110)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82151 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf08)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).Lock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:88 +0xa1
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).stopStore(0xc82518f380, 0x1)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:643 +0x37
github.com/cockroachdb/cockroach/storage_test.setupSplitSnapshotRace(0xc8208806c0, 0xc82518f380, 0xc8201425e8, 0x1, 0x8, 0xc8201425f0, 0x1, 0x8)
    /go/src/github.com/cockroachdb/cockroach/storage/client_split_test.go:977 +0x1518
github.com/cockroachdb/cockroach/storage_test.TestSplitSnapshotRace_SnapshotWins(0xc8208806c0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_split_test.go:1026 +0xaf
testing.tRunner(0xc8208806c0, 0x28aa8b0)
    /usr/local/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:582 +0x892

goroutine 82320 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc820809eec)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc820809ee0)
    /usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2575 +0x50d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1af080)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82302 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:266 +0x262
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279adce0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82322 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc82088abd0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 83198 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContextKVTransport).SendNext(0xc8207851e0, 0xc8252b2a20)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:369 +0xe6
github.com/cockroachdb/cockroach/kv.(*DistSender).sendToReplicas(0xc820880bd0, 0x7fd8f9367658, 0xc82024f770, 0x2540be400, 0xb2d05e00, 0xc8252df080, 0xb, 0xc8203fe640, 0x3, 0x3, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:924 +0x623
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc820880bd0, 0x7fd8f9367658, 0xc82024f770, 0xb, 0xc8203fe640, 0x3, 0x3, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:345 +0x1ee
github.com/cockroachdb/cockroach/kv.(*DistSender).sendSingleRange(0xc820880bd0, 0x7fd8f9367658, 0xc82024f770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:461 +0x1ea
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk.func2(0xc8208abed8, 0xc8208abc10, 0xc8208ac5c8, 0xc820880bd0, 0xc8208ac5b8, 0xd, 0xc820809193)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:720 +0x443
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk(0xc820880bd0, 0x7fd8f9367658, 0xc82024f770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:721 +0x9f5
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc820880bd0, 0x7fd8f9367658, 0xc82024f770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:565 +0x6d4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc8252136e0, 0x7fd8f9367658, 0xc82024f770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:374 +0xaa8
github.com/cockroachdb/cockroach/internal/client.(*DB).send(0xc827d1de50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:486 +0x3ec
github.com/cockroachdb/cockroach/internal/client.(*DB).(github.com/cockroachdb/cockroach/internal/client.send)-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:433 +0x3a
github.com/cockroachdb/cockroach/internal/client.sendAndFill(0xc8208ad488, 0xc827bad680, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:415 +0x1ff
github.com/cockroachdb/cockroach/internal/client.(*DB).Run(0xc827d1de50, 0xc827bad680, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/internal/client/db.go:433 +0x91
github.com/cockroachdb/cockroach/storage.(*intentResolver).maybePushTransactions(0xc82e66d9c0, 0x7fd8f9368080, 0xc82024f710, 0xc820864a20, 0x1, 0x1, 0x35a4e9000, 0x12, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/intent_resolver.go:234 +0x783
github.com/cockroachdb/cockroach/storage.(*intentResolver).processIntentsAsync.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/intent_resolver.go:286 +0x17a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask.func1(0xc820880b40, 0x259d03b, 0x1a, 0x115, 0xc825213800, 0xc82e0216c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:269 +0xb8
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunLimitedAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:270 +0x240

goroutine 82242 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2578 +0xc29
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e1b79a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82426 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc827e40c60)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc827e40c60)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 82156 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1b7260)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82389 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc82e6877a0, 0xecf30d318, 0x2ae3ea77, 0x2bc2f80, 0xc827d874c0, 0xc82e319290, 0x0, 0x1)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd3e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82315 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681e450, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82e687b10, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82e687b10, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82e687ab0, 0x0, 0x7fd8f4d34b88, 0xc82e0f2800)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc824ea23a0, 0xc824c77ea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc824ea23a0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc824e3f4a0, 0x7fd8f681dda8, 0xc824ea23a0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1aee60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82314 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1aee40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82281 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc8252e4340, 0x7fd8f929efd8, 0xc82e6bf500, 0xc820768160, 0x7fd8f4ccee98, 0xc827a581e0, 0xc820880a20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x63f
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x316
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e6bf300)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82277 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f7040)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82944 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc82def3590, 0xc82e314e20, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82778b410)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82357 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc82075cd00, 0x7fd8f92e8190, 0xc827b84d30, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:158 +0xac5
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x1af9fc0, 0xc82075cd00, 0x7fd8f92e8148, 0xc820646ea0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827aaf590, 0xc827b831d0, 0x289b660, 0xc82828a7b0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x489
google.golang.org/grpc.(*Server).handleStream(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827aaf590, 0xc82828a7b0)
    /go/src/google.golang.org/grpc/server.go:770 +0x1151
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc827df09e0, 0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827aaf590)
    /go/src/google.golang.org/grpc/server.go:419 +0xa0
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0x9a

goroutine 82269 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df0a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82260 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc825169650, 0xecf30d318, 0x2aac9fce, 0x2bc2f80, 0xc827d874c0, 0xc820880b40, 0xc827ebf800, 0xc827c79698)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc827ebf800, 0xc827b828a0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:214 +0x78
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc827b82b10, 0xc827b7d880)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:283 +0x246
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:216 +0x21e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6de5c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82354 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df200)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82506 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc8208ccf70, 0x7fd8f929efd8, 0xc827eec940, 0xc8207942c0, 0x7fd8f4ccee98, 0xc820034628, 0xc820880a20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x63f
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x316
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827eec8c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82270 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df0c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82234 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed2420)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82303 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc824f6580c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc824f65800)
    /usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2575 +0x50d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279add20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82347 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e4186d0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82549 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc82e08e6c0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 82273 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc82e686310, 0xecf30d318, 0x2adb0267, 0x2bc2f80, 0xc827d874c0, 0xc82e318fc0, 0x0, 0xc827b67f28)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df120)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82316 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1aef00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82161 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:266 +0x262
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e1b7400)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82271 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df0e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82404 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc82075cd00, 0x7fd8f92e8190, 0xc82e418940, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:158 +0xac5
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x1af9fc0, 0xc82075cd00, 0x7fd8f92e8148, 0xc827b8e000, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827f9a0f0, 0xc827b831d0, 0x289b660, 0xc82e6ba450, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x489
google.golang.org/grpc.(*Server).handleStream(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827f9a0f0, 0xc82e6ba450)
    /go/src/google.golang.org/grpc/server.go:770 +0x1151
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc827df09e0, 0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc827f9a0f0)
    /go/src/google.golang.org/grpc/server.go:419 +0xa0
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0x9a

goroutine 82312 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:290 +0x23f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1ae960)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82371 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc827d7eae0)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82030 [select, 8 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc8251e6800, 0xc82e6df4b0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827aaf590, 0xc82e6df4b0, 0x5, 0x5, 0xc82e0ecb40, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827aaf590, 0xc82e6df4b0, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827aaf590, 0xc82e6df4b0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e6df4a0, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e6df4a0, 0x7fd8f681dcb8, 0x2c14d90, 0xc827aaf590, 0x0, 0x0, 0x1b722e0, 0xc82e0ecc00, 0x400000, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*serverStream).RecvMsg(0xc820646ea0, 0x1b722e0, 0xc82e0ecc00, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xf3
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc827b84d30, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x7e
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc82075cd08, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x40
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc82075cd00, 0xc82026c470, 0xc82e0ec9c0, 0xc82817df50, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:268 +0x917
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x8b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e0eca00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82388 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd3c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82240 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc827befdc0, 0xecf30d318, 0x2b54511c, 0x2bc2f80, 0xc827d874c0, 0xc827a261b0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed24e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82335 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd300)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82579 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc827ec9440)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc827ec9440)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 82430 [select, 2 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc8207d9a80, 0xc82e1b6b70, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc8279682d0, 0xc82e1b6b70, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc8279682d0, 0xc82e1b6b70, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc8279682d0, 0xc82e1b6b70, 0x5, 0x5, 0xae4ffa, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e1b6b60, 0x7fffffff, 0x2f0000c8201b3200, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e1b6b60, 0x7fd8f681dcb8, 0x2c14d90, 0xc8279682d0, 0x0, 0x0, 0x1b57840, 0xc82531f9f0, 0x7fffffff, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*clientStream).RecvMsg(0xc827e40d80, 0x1b57840, 0xc82531f9f0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xbb
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc827e3a4d0, 0xc820794000, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x7e
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x7fd8f92e8088, 0xc827e3a4d0, 0xc8208cc680, 0xc820794000, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x37
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x51
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e581050)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82360 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc827ab8660)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82152 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/rpc.NewContext.func1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:105 +0x57
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e66cf40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82319 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:266 +0x262
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1af020)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82311 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc82e13f680)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82363 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57cc20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82280 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc8251692d0, 0xecf30d318, 0x2aacf596, 0x2bc2f80, 0xc827d874c0, 0xc820880b40, 0xc827ebf800, 0xc827c77698)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc827ebf800, 0xc827b828a0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:214 +0x78
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc827b82960, 0xc827c7ffc0)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:283 +0x246
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:216 +0x21e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f70a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82345 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc824c61240)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82255 [select, 2 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc8207d80c0, 0xc82e1b6090, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827968000, 0xc82e1b6090, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827968000, 0xc82e1b6090, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827968000, 0xc82e1b6090, 0x5, 0x5, 0xae4ffa, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e1b6080, 0x7fffffff, 0x40000c82e0a1cb0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e1b6080, 0x7fd8f681dcb8, 0x2c14d90, 0xc827968000, 0x0, 0x0, 0x1b57840, 0xc82df34820, 0x7fffffff, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*clientStream).RecvMsg(0xc827e40120, 0x1b57840, 0xc82df34820, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xbb
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc827e3b1b0, 0xc820834160, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x7e
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x7fd8f92e8088, 0xc827e3b1b0, 0xc8200ec270, 0xc820834160, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x37
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x51
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e5804b0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82467 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc8206260a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 83000 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc825318120)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc825318120)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 82496 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc825228f00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82355 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc82e686770, 0xecf30d318, 0x2adb33a8, 0x2bc2f80, 0xc827d874c0, 0xc82e318fc0, 0x0, 0xc827b67f28)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df220)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82246 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1288 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e1b7ac0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82376 [IO wait]:
net.runtime_pollWait(0x7fd8f681e810, 0x72, 0xc820912000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827a1cf40, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827a1cf40, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827a1cee0, 0xc820912000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc827d4e008, 0xc820912000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc820011c20)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc820011c20, 0xc827ee0e78, 0x9, 0x9, 0xc81fc1da81, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc820011c20, 0xc827ee0e78, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc820011c20, 0xc827ee0e78, 0x9, 0x9, 0xc827c1de08, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827ee0e78, 0x9, 0x9, 0x7fd8f4db2630, 0xc820011c20, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827ee0e40, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc827f381e0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc820892000)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 83075 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc8208c5100)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82391 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc8208801b0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 82346 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e4186c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82289 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc8252e4680, 0x7fd8f929efd8, 0xc8251e6600, 0xc8207682c0, 0x7fd8f4ccee98, 0xc82026c468, 0xc820880a20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x63f
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x316
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e2be240)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82390 [IO wait]:
net.runtime_pollWait(0x7fd8f681e750, 0x72, 0xc824c9a000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827f14060, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827f14060, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827f14000, 0xc824c9a000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82036a000, 0xc824c9a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8252243c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8252243c0, 0xc8208620f8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc8252243c0, 0xc8208620f8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc8252243c0, 0xc8208620f8, 0x9, 0x9, 0xc820346180, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc8208620f8, 0x9, 0x9, 0x7fd8f4db2630, 0xc8252243c0, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc8208620c0, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc827f74060, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc8208801b0, 0xc827f74180)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc824e3ebe0, 0x7fd8f4db2680, 0xc8208801b0)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc824e3ebe0, 0x7fd8f6862e00, 0xc82036a000, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc824e3ebe0, 0x7fd8f6862e00, 0xc82036a000)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82252 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc82e687a40, 0xecf30d318, 0x2ae41b1d, 0x2bc2f80, 0xc827d874c0, 0xc82e319290, 0x0, 0x1)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1b7e80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82324 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc82075cd00, 0x7fd8f92e8190, 0xc827df0b70, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:158 +0xac5
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x1af9fc0, 0xc82075cd00, 0x7fd8f92e8148, 0xc82088ae10, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc8208dd770, 0xc827b831d0, 0x289b660, 0xc827e37bf0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x489
google.golang.org/grpc.(*Server).handleStream(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc8208dd770, 0xc827e37bf0)
    /go/src/google.golang.org/grpc/server.go:770 +0x1151
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc827df09e0, 0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0, 0xc8208dd770)
    /go/src/google.golang.org/grpc/server.go:419 +0xa0
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0x9a

goroutine 82343 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc824c60780)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82155 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8252136e0)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc8252df280)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82469 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc820626100)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 83034 [IO wait]:
net.runtime_pollWait(0x7fd8f681e690, 0x72, 0xc828204000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82511b790, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82511b790, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82511b730, 0xc828204000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8208167a8, 0xc828204000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e2ea300)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e2ea300, 0xc8203a72f8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e2ea300, 0xc8203a72f8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e2ea300, 0xc8203a72f8, 0x9, 0x9, 0xc82e2dc000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc8203a72f8, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e2ea300, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc8203a72c0, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc8251f4690, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc824d73170, 0xc8251f4720)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc827f6f7c0, 0x7fd8f4db2680, 0xc824d73170)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc827f6f7c0, 0x7fd8f6862e00, 0xc8208167a8, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc827f6f7c0, 0x7fd8f6862e00, 0xc8208167a8)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82153 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc8252121e0)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82295 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:290 +0x23f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279ad7e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 79253 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc827bef030, 0xecf30d318, 0x2b4e3e1f, 0x2bc2f80, 0xc827d874c0, 0xc82009de60, 0x0, 0x2bd1aa0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc8208d01e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82294 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc820642fc0)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82336 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd320)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82369 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc820809fbc)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc820809fb0)
    /usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2575 +0x50d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57ce20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 78908 [IO wait]:
net.runtime_pollWait(0x7fd8f681ebd0, 0x72, 0xc827cd0000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82e09c610, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82e09c610, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82e09c5b0, 0xc827cd0000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc827a58010, 0xc827cd0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8205b65a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8205b65a0, 0xc827ea41b8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc8205b65a0, 0xc827ea41b8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc8205b65a0, 0xc827ea41b8, 0x9, 0x9, 0xc82535d080, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827ea41b8, 0x9, 0x9, 0x7fd8f4db2630, 0xc8205b65a0, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827ea4180, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc82086c540, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc820646000, 0xc82086c6f0)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc827e1afa0, 0x7fd8f4db2680, 0xc820646000)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc827e1afa0, 0x7fd8f6862e00, 0xc827a58010, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc827e1afa0, 0x7fd8f6862e00, 0xc827a58010)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82209 [IO wait]:
net.runtime_pollWait(0x7fd8f681e510, 0x72, 0xc8250ba000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8208b9a30, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8208b9a30, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8208b99d0, 0xc8250ba000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820254258, 0xc8250ba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e6f28a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e6f28a0, 0xc827ea4db8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e6f28a0, 0xc827ea4db8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e6f28a0, 0xc827ea4db8, 0x9, 0x9, 0xc82535d500, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827ea4db8, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e6f28a0, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827ea4d80, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc827e37380, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc82088abd0, 0xc827e37410)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc827f6ea00, 0x7fd8f4db2680, 0xc82088abd0)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc827f6ea00, 0x7fd8f6862e00, 0xc820254258, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc827f6ea00, 0x7fd8f6862e00, 0xc820254258)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82276 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f7020)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 78911 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc8252e4ea0, 0x7fd8f92e8190, 0xc8208e6330, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:158 +0xac5
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x1af9fc0, 0xc8252e4ea0, 0x7fd8f92e8148, 0xc820646090, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc827e1afa0, 0x7fd8f4db2680, 0xc820646000, 0xc8252f40f0, 0xc82e6ba210, 0x289b660, 0xc82086cf30, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x489
google.golang.org/grpc.(*Server).handleStream(0xc827e1afa0, 0x7fd8f4db2680, 0xc820646000, 0xc8252f40f0, 0xc82086cf30)
    /go/src/google.golang.org/grpc/server.go:770 +0x1151
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc8208e61f0, 0xc827e1afa0, 0x7fd8f4db2680, 0xc820646000, 0xc8252f40f0)
    /go/src/google.golang.org/grpc/server.go:419 +0xa0
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0x9a

goroutine 82033 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc820626060)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82262 [select, 2 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc827850100)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82286 [select, 8 minutes]:
google.golang.org/grpc.NewClientStream.func1(0x7fd8f4db2700, 0xc827850100, 0xc8252f4b40, 0xc82508e900)
    /go/src/google.golang.org/grpc/stream.go:187 +0x505
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0x11b5

goroutine 82362 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc8204fede0)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc828157db0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 83074 [IO wait]:
net.runtime_pollWait(0x7fd8f681e390, 0x72, 0xc827e88000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82e2d9100, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82e2d9100, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82e2d90a0, 0xc827e88000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8201a0558, 0xc827e88000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e3024e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e3024e0, 0xc82e14cff8, 0x9, 0x9, 0xc81f8cd6e7, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e3024e0, 0xc82e14cff8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e3024e0, 0xc82e14cff8, 0x9, 0x9, 0xc827c4b238, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc82e14cff8, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e3024e0, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc82e14cfc0, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc82e3ac450, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc8208c5100)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 82508 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc827e41560)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc827e41560)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 82450 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc820809cac)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc820809ca0)
    /usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2575 +0x50d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57dc40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82301 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e64a2a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82325 [select, 8 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc827e27640, 0xc827dfcb30, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc8208dd770, 0xc827dfcb30, 0x5, 0x5, 0xc8280360e0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc8208dd770, 0xc827dfcb30, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc8208dd770, 0xc827dfcb30, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc827dfcb20, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc827dfcb20, 0x7fd8f681dcb8, 0x2c14d90, 0xc8208dd770, 0x0, 0x0, 0x1b722e0, 0xc827d18440, 0x400000, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*serverStream).RecvMsg(0xc82088ae10, 0x1b722e0, 0xc827d18440, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xf3
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc827df0b70, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x7e
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc82075cd08, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x40
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc82075cd00, 0xc820254260, 0xc827e27700, 0xc8205aff50, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:268 +0x917
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x8b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827e27780)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82338 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1194 +0x13d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1af0e0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82368 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:266 +0x262
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57cdc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82514 [select, 8 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc82081a8c0, 0xc827dda8b0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc8252f40f0, 0xc827dda8b0, 0x5, 0x5, 0xc827f427e0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc8252f40f0, 0xc827dda8b0, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc8252f40f0, 0xc827dda8b0, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc827dda8a0, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc827dda8a0, 0x7fd8f681dcb8, 0x2c14d90, 0xc8252f40f0, 0x0, 0x0, 0x1b722e0, 0xc827d37100, 0x400000, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*serverStream).RecvMsg(0xc820646090, 0x1b722e0, 0xc827d37100, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xf3
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc8208e6330, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x7e
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc8252e4ea8, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x40
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc8252e4ea0, 0xc827a58018, 0xc827d36a80, 0xc827937f50, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:268 +0x917
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x8b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827d36bc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82367 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc828156ed0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82275 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f7000)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82386 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd360)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82154 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:290 +0x23f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e66d780)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82317 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e64b1b0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82466 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc820626080)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 79252 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc8208d01c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82297 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8279adb40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82444 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func1()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:51 +0x50
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57da60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82468 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc8206260c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82361 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:290 +0x23f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57c880)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82437 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1288 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57cec0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82446 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57db00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82237 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed2480)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82337 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd340)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82236 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed2460)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 78906 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc827f36070, 0xecf30d318, 0x2b544ec1, 0x2bc2f80, 0xc827d874c0, 0xc827a261b0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827dda380)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82356 [select, 8 minutes]:
google.golang.org/grpc.NewClientStream.func1(0x7fd8f4db2700, 0xc827850100, 0xc827aaf3b0, 0xc827d95200)
    /go/src/google.golang.org/grpc/stream.go:187 +0x505
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0x11b5

goroutine 82453 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57dcc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 78905 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827dda360)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82245 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e1b7aa0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82278 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f7060)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82365 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57ccc0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 83225 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc82821a400, 0x7fd8f9367658, 0xc820217980, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1281 +0x7db
github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc82821a400, 0x7fd8f9367658, 0xc820217980, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:827 +0x23d
github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc827f2b680, 0x7fd8f929efd8, 0xc82e6e2e00, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2207 +0xd05
github.com/cockroachdb/cockroach/storage.(*Stores).Send(0xc827a0ac40, 0x7fd8f929efd8, 0xc82e6e2e00, 0x0, 0x0, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/stores.go:182 +0x3a0
github.com/cockroachdb/cockroach/storage_test.(*multiTestContextKVTransport).SendNext.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:381 +0x20f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc82009de60, 0x259bf06, 0x16, 0x196, 0xc8201bd1c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x8d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0x124

goroutine 82470 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc827beed20, 0xecf30d318, 0x2b4e5e2f, 0x2bc2f80, 0xc827d874c0, 0xc82009de60, 0x0, 0x2bd4fe0)
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:166 +0x6ad
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/scanner.go:219 +0x27c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc820626120)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82328 [select, 2 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc8251e6700, 0xc82e6df330, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827aaf3b0, 0xc82e6df330, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827aaf3b0, 0xc82e6df330, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827aaf3b0, 0xc82e6df330, 0x5, 0x5, 0xae4ffa, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e6df320, 0x7fffffff, 0xb30000c827bd9980, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e6df320, 0x7fd8f681dcb8, 0x2c14d90, 0xc827aaf3b0, 0x0, 0x0, 0x1b57840, 0xc82092a9b0, 0x7fffffff, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*clientStream).RecvMsg(0xc827d95200, 0x1b57840, 0xc82092a9b0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xbb
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc827b84cc0, 0xc8207682c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x7e
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x7fd8f92e8088, 0xc827b84cc0, 0xc8252e4680, 0xc8207682c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x37
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x51
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827c06360)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82364 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681ec90, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827bef100, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827bef100, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc827bef0a0, 0x0, 0x7fd8f4d34b88, 0xc82e5ec480)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc82e6e40b8, 0xc8250ccea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82e6e40b8, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc827f6eaa0, 0x7fd8f681dda8, 0xc82e6e40b8, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57cc40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82436 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57cea0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82259 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6de5a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82366 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc828156ec0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82244 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1194 +0x13d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e1b7a20)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82454 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1288 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57dce0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82580 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc8252e1590, 0xc827e29430, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e6b4c60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82032 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc820626040)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82416 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).Gossip(0xc82e6904e0, 0x7fd8f92e8190, 0xc82e418fe0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:158 +0xac5
github.com/cockroachdb/cockroach/gossip._Gossip_Gossip_Handler(0x1af9fc0, 0xc82e6904e0, 0x7fd8f92e8148, 0xc827b8e240, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:211 +0xd8
google.golang.org/grpc.(*Server).processStreamingRPC(0xc824e3ebe0, 0x7fd8f4db2680, 0xc8208801b0, 0xc827f9a3c0, 0xc827bd8f90, 0x289b660, 0xc82e6bbb00, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:686 +0x489
google.golang.org/grpc.(*Server).handleStream(0xc824e3ebe0, 0x7fd8f4db2680, 0xc8208801b0, 0xc827f9a3c0, 0xc82e6bbb00)
    /go/src/google.golang.org/grpc/server.go:770 +0x1151
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc827f2c080, 0xc824e3ebe0, 0x7fd8f4db2680, 0xc8208801b0, 0xc827f9a3c0)
    /go/src/google.golang.org/grpc/server.go:419 +0xa0
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /go/src/google.golang.org/grpc/server.go:420 +0x9a

goroutine 82427 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc8279681e0, 0xc82018e710, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e580e10)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 78909 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc820646000)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 82405 [select, 8 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc827f7bb40, 0xc824c60490, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827f9a0f0, 0xc824c60490, 0x5, 0x5, 0xc8203fd030, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827f9a0f0, 0xc824c60490, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827f9a0f0, 0xc824c60490, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc824c60480, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc824c60480, 0x7fd8f681dcb8, 0x2c14d90, 0xc827f9a0f0, 0x0, 0x0, 0x1b722e0, 0xc824d4cd80, 0x400000, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*serverStream).RecvMsg(0xc827b8e000, 0x1b722e0, 0xc824d4cd80, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xf3
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc82e418940, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x7e
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc82075cd08, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x40
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc82075cd00, 0xc82083e028, 0xc827f7a300, 0xc8204d9f50, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:268 +0x917
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x8b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827f7a340)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82358 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc8200ec270, 0x7fd8f929efd8, 0xc8207d8040, 0xc820834160, 0x7fd8f4ccee98, 0xc827d4e000, 0xc820880a20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x63f
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x316
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8205abe00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82878 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*idAllocator).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:124 +0xa2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82815ed50)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82238 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed24a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82435 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1194 +0x13d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82009de60, 0xc82e57ce80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82428 [select, 8 minutes]:
google.golang.org/grpc.NewClientStream.func1(0x7fd8f4db2700, 0xc820892000, 0xc8279682d0, 0xc827e40d80)
    /go/src/google.golang.org/grpc/stream.go:187 +0x505
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0x11b5

goroutine 82535 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).maybeAddToPurgatory.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:527 +0xca9
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e115a80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82157 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681e5d0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc825169aa0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc825169aa0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc825169a40, 0x0, 0x7fd8f4d34b88, 0xc827dfc8c0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc820034638, 0xc825230ea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820034638, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc827f6ea00, 0x7fd8f681dda8, 0xc820034638, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1b7280)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82305 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1194 +0x13d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279add80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82340 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1288 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1af120)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82239 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed24c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82387 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc827dfd380)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82318 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e64b1c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 83001 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc827aaf950, 0xc82e3c05c0, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e3744b0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82300 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e64a290)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82307 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1288 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279ade00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82261 [IO wait]:
net.runtime_pollWait(0x7fd8f681e090, 0x72, 0xc827c80000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827f37db0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827f37db0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827f37d50, 0xc827c80000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82026c430, 0xc827c80000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8205fc840)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8205fc840, 0xc827a2a4b8, 0x9, 0x9, 0xc81fc00c41, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc8205fc840, 0xc827a2a4b8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc8205fc840, 0xc827a2a4b8, 0x9, 0x9, 0xc827c49b38, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827a2a4b8, 0x9, 0x9, 0x7fd8f4db2630, 0xc8205fc840, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827a2a480, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc82e0a38c0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc827850100)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 82235 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827ed2440)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82298 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681eb10, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82e686840, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82e686840, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82e6867e0, 0x0, 0x7fd8f4d34b88, 0xc82e6c60e0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc824ea22e0, 0xc82e2e3ea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc824ea22e0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc824e3ebe0, 0x7fd8f681dda8, 0xc824ea22e0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8279adb60)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82306 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc8279adda0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82284 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc8252f4a50, 0xc827b97370, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e0a0de0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82158 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e1b7300)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 81980 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*idAllocator).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:124 +0xa2f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc827d3e0c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82548 [IO wait]:
net.runtime_pollWait(0x7fd8f681e8d0, 0x72, 0xc82dffc000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827d89800, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827d89800, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827d897a0, 0xc82dffc000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200346a0, 0xc82dffc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e13e420)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e13e420, 0xc827ab0638, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e13e420, 0xc827ab0638, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e13e420, 0xc827ab0638, 0x9, 0x9, 0xc82e69b800, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827ab0638, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e13e420, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827ab0600, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc827e2e7b0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc82e08e6c0, 0xc827e2e960)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc824e3f4a0, 0x7fd8f4db2680, 0xc82e08e6c0)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc824e3f4a0, 0x7fd8f6862e00, 0xc8200346a0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc824e3f4a0, 0x7fd8f6862e00, 0xc8200346a0)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82274 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f6fe0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82943 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc824d785a0)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc824d785a0)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 82272 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df100)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82313 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc82e1838c0)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e64ab10)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82417 [select, 8 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc827f7b580, 0xc824c61350, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827f9a3c0, 0xc824c61350, 0x5, 0x5, 0xc827c5f6c0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827f9a3c0, 0xc824c61350, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827f9a3c0, 0xc824c61350, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc824c61340, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc824c61340, 0x7fd8f681dcb8, 0x2c14d90, 0xc827f9a3c0, 0x0, 0x0, 0x1b722e0, 0xc824ca9080, 0x400000, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*serverStream).RecvMsg(0xc827b8e240, 0x1b722e0, 0xc824ca9080, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:482 +0xf3
github.com/cockroachdb/cockroach/gossip.(*gossipGossipServer).Recv(0xc82e418fe0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:230 +0x7e
github.com/cockroachdb/cockroach/gossip.(Gossip_GossipServer).Recv-fm(0xc82e6904e8, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x40
github.com/cockroachdb/cockroach/gossip.(*server).gossipReceiver(0xc82e6904e0, 0xc82083e0e0, 0xc827f7b680, 0xc82054ff50, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:268 +0x917
github.com/cockroachdb/cockroach/gossip.(*server).Gossip.func3.1()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:124 +0x8b
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827f7b6c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82424 [select, 2 minutes]:
github.com/cockroachdb/cockroach/gossip.(*client).gossip(0xc8208cc680, 0x7fd8f929efd8, 0xc8207d9700, 0xc820794000, 0x7fd8f4ccee98, 0xc8200340c0, 0xc820880a20, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:278 +0x63f
github.com/cockroachdb/cockroach/gossip.(*client).start.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:93 +0x316
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8207d94c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82509 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc8250f3ec0, 0xc8279683c0, 0xc827e3b030, 0xf, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:259 +0x611
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1()
    /go/src/github.com/cockroachdb/cockroach/rpc/context.go:173 +0x69
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e581cb0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82377 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc820892000)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82452 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func4()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1194 +0x13d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57dca0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82268 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df040)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82296 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc82e13e540)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc827b979a0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82448 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827f729c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82445 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681ea50, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827f36140, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827f36140, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc827f360e0, 0x0, 0x7fd8f4d34b88, 0xc824d0afc0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc82e6e41a0, 0xc825371ea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82e6e41a0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc827f6f7c0, 0x7fd8f681dda8, 0xc82e6e41a0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc82e57da80)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82251 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1b7e40)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82279 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880b40, 0xc82e6f7080)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82299 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
    /go/src/github.com/cockroachdb/cockroach/gossip/server.go:325 +0x5c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8279adbe0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82512 [select, 6 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc820484b00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82513 [select, 8 minutes]:
google.golang.org/grpc.NewClientStream.func1(0x7fd8f4db2700, 0xc820484b00, 0xc827968960, 0xc827e419e0)
    /go/src/google.golang.org/grpc/stream.go:187 +0x505
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0x11b5

goroutine 82511 [IO wait]:
net.runtime_pollWait(0x7fd8f681e210, 0x72, 0xc824ee6000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827d88d80, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827d88d80, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827d88d20, 0xc824ee6000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc820034668, 0xc824ee6000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc8250f34a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc8250f34a0, 0xc827ab0578, 0x9, 0x9, 0xc81f8cd6fc, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc8250f34a0, 0xc827ab0578, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc8250f34a0, 0xc827ab0578, 0x9, 0x9, 0xc827a6e488, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827ab0578, 0x9, 0x9, 0x7fd8f4db2630, 0xc8250f34a0, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827ab0540, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc82e581dd0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc820484b00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 82267 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/queue.go:402 +0x2aa
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e318fc0, 0xc82e6df020)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82441 [chan receive, 8 minutes]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc820658c00)
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:384 +0x3a
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
    /go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:385 +0xb46

goroutine 82160 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:891 +0x311
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827c08280)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82447 [select, 8 minutes]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:864 +0x238
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc827f729b0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82449 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/reservation.go:266 +0x262
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57dc00)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82253 [select, 8 minutes]:
google.golang.org/grpc.NewClientStream.func1(0x7fd8f4db2700, 0xc827850100, 0xc827968000, 0xc827e40120)
    /go/src/google.golang.org/grpc/stream.go:187 +0x505
created by google.golang.org/grpc.NewClientStream
    /go/src/google.golang.org/grpc/stream.go:207 +0x11b5

goroutine 82443 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc820659f80)
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:203 +0xdc4
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
    /go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:188 +0x20
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc827f72250)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82442 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:290 +0x23f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc827a261b0, 0xc82e57d740)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82859 [IO wait]:
net.runtime_pollWait(0x7fd8f681e150, 0x72, 0xc825272000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82781cbc0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82781cbc0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82781cb60, 0xc825272000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8208023a8, 0xc825272000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e005e00)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e005e00, 0xc8200e5ef8, 0x9, 0x9, 0xc81fc1da3c, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e005e00, 0xc8200e5ef8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e005e00, 0xc8200e5ef8, 0x9, 0x9, 0xc827a6e908, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc8200e5ef8, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e005e00, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc8200e5ec0, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc824f7a660, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc820277800)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 82860 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc820277800)
    /go/src/google.golang.org/grpc/transport/http2_client.go:965 +0x5e0
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:206 +0x14bd

goroutine 82339 [select, 2 minutes]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:1269 +0x3cf
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc82e319290, 0xc82e1af100)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82494 [select, 2 minutes]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc827eecd00, 0xc82e0f2210, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:142 +0x8ab
google.golang.org/grpc/transport.(*Stream).Read(0xc827968960, 0xc82e0f2210, 0x5, 0x5, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/transport.go:315 +0x74
io.ReadAtLeast(0x7fd8f4db29d8, 0xc827968960, 0xc82e0f2210, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db29d8, 0xc827968960, 0xc82e0f2210, 0x5, 0x5, 0xae4ffa, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
google.golang.org/grpc.(*parser).recvMsg(0xc82e0f2200, 0x7fffffff, 0xb80000c827e04db0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/rpc_util.go:231 +0xbc
google.golang.org/grpc.recv(0xc82e0f2200, 0x7fd8f681dcb8, 0x2c14d90, 0xc827968960, 0x0, 0x0, 0x1b57840, 0xc82df34af0, 0x7fffffff, 0x0, ...)
    /go/src/google.golang.org/grpc/rpc_util.go:315 +0x52
google.golang.org/grpc.(*clientStream).RecvMsg(0xc827e419e0, 0x1b57840, 0xc82df34af0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/stream.go:295 +0xbb
github.com/cockroachdb/cockroach/gossip.(*gossipGossipClient).Recv(0xc827e3b420, 0xc8207942c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/gossip.pb.go:194 +0x7e
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2.1(0x7fd8f92e8088, 0xc827e3b420, 0xc8208ccf70, 0xc8207942c0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:266 +0x37
github.com/cockroachdb/cockroach/gossip.(*client).gossip.func2()
    /go/src/github.com/cockroachdb/cockroach/gossip/client.go:274 +0x51
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc8208d8420)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82495 [IO wait]:
net.runtime_pollWait(0x7fd8f681ed50, 0x72, 0xc82e398000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc827be2a00, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc827be2a00, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc827be29a0, 0xc82e398000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc82083e170, 0xc82e398000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc820642540)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc820642540, 0xc827d8c338, 0x9, 0x9, 0xc81fc1da98, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc820642540, 0xc827d8c338, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc820642540, 0xc827d8c338, 0x9, 0x9, 0xc827c49bc8, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc827d8c338, 0x9, 0x9, 0x7fd8f4db2630, 0xc820642540, 0x0, 0xc800000000, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc827d8c300, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc8208d85a0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Client).reader(0xc825228f00)
    /go/src/google.golang.org/grpc/transport/http2_client.go:887 +0x109
created by google.golang.org/grpc/transport.newHTTP2Client
    /go/src/google.golang.org/grpc/transport/http2_client.go:176 +0xc22

goroutine 82344 [IO wait, 8 minutes]:
net.runtime_pollWait(0x7fd8f681e990, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82e0c4370, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82e0c4370, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82e0c4310, 0x0, 0x7fd8f4d34b88, 0xc827dda5a0)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc82083e118, 0xc82e2e7ea8, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc82083e118, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
google.golang.org/grpc.(*Server).Serve(0xc827e1afa0, 0x7fd8f681dda8, 0xc82083e118, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:348 +0x1cf
github.com/cockroachdb/cockroach/util/netutil.ListenAndServeGRPC.func2()
    /go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:58 +0x3f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc820880a20, 0xc824c60800)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:187 +0x7d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:188 +0x62

goroutine 82283 [select, 8 minutes]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc82508e6c0)
    /go/src/google.golang.org/grpc/clientconn.go:646 +0x783
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc82508e6c0)
    /go/src/google.golang.org/grpc/clientconn.go:441 +0x26f
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
    /go/src/google.golang.org/grpc/clientconn.go:442 +0x5e7

goroutine 83368 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).getNodeIDAddress(0xc82518f380, 0xc800000003, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:198 +0x66
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).(github.com/cockroachdb/cockroach/storage_test.getNodeIDAddress)-fm(0x3, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:542 +0x44
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:217 +0x5e
github.com/rubyist/circuitbreaker.(*Breaker).Call(0xc82e3198c0, 0xc82df6d1d0, 0x0, 0x0, 0x0)
    /go/src/github.com/rubyist/circuitbreaker/circuitbreaker.go:338 +0x9d
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn(0xc827b6db80, 0xc800000003, 0xc8277c07e0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:216 +0x2b4
github.com/cockroachdb/cockroach/storage.RaftSender.SendAsync(0xc827b6db80, 0xc82e6ca2c0, 0xc8277c07e0, 0x200000002)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:324 +0xdf
github.com/cockroachdb/cockroach/storage.(*Replica).sendRaftMessage(0xc820804600, 0x8, 0x3, 0x2, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1698 +0x85d
github.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc820804600, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1510 +0x86c
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc824f65800, 0xc82e13ee40, 0xc820804600)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2568 +0x25
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2573 +0x4da

goroutine 83381 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).getNodeIDAddress(0xc82518f380, 0xc800000006, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:198 +0x66
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).(github.com/cockroachdb/cockroach/storage_test.getNodeIDAddress)-fm(0x6, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:542 +0x44
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:217 +0x5e
github.com/rubyist/circuitbreaker.(*Breaker).Call(0xc8278345a0, 0xc8201b9a40, 0x0, 0x0, 0x0)
    /go/src/github.com/rubyist/circuitbreaker/circuitbreaker.go:338 +0x9d
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn(0xc8205aa540, 0x6, 0xc824d5aea0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:216 +0x2b4
github.com/cockroachdb/cockroach/storage.RaftSender.SendAsync(0xc8205aa540, 0xc82e365600, 0xc824d5aea0, 0x500000005)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:324 +0xdf
github.com/cockroachdb/cockroach/storage.(*Replica).sendRaftMessage(0xc82821a400, 0x2, 0x5, 0x4, 0x0, 0x0, 0x0, 0xc82e6e1450, 0x1, 0x1, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1698 +0x85d
github.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc82821a400, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1510 +0x86c
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc820809fb0, 0xc820658180, 0xc82821a400)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2568 +0x25
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2573 +0x4da

goroutine 82982 [IO wait]:
net.runtime_pollWait(0x7fd8f681e2d0, 0x72, 0xc824e12000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82511a5a0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82511a5a0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82511a540, 0xc824e12000, 0x8000, 0x8000, 0x0, 0x7fd8f939d050, 0xc82000c078)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8208166b0, 0xc824e12000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
bufio.(*Reader).fill(0xc82e4451a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc82e4451a0, 0xc8203a70b8, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fd8f4db2630, 0xc82e4451a0, 0xc8203a70b8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd8f4db2630, 0xc82e4451a0, 0xc8203a70b8, 0x9, 0x9, 0xc82e6c8c00, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
golang.org/x/net/http2.readFrameHeader(0xc8203a70b8, 0x9, 0x9, 0x7fd8f4db2630, 0xc82e4451a0, 0x27000000, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:237 +0xa5
golang.org/x/net/http2.(*Framer).ReadFrame(0xc8203a7080, 0x0, 0x0, 0x0, 0x0)
    /go/src/golang.org/x/net/http2/frame.go:464 +0x106
google.golang.org/grpc/transport.(*framer).readFrame(0xc82e3226c0, 0x0, 0x0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/transport/http_util.go:505 +0x3d
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc824d72000, 0xc82e322750)
    /go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x6fa
google.golang.org/grpc.(*Server).serveStreams(0xc827f6eaa0, 0x7fd8f4db2680, 0xc824d72000)
    /go/src/google.golang.org/grpc/server.go:421 +0x159
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc827f6eaa0, 0x7fd8f6862e00, 0xc8208166b0, 0x0, 0x0)
    /go/src/google.golang.org/grpc/server.go:408 +0x49d
google.golang.org/grpc.(*Server).handleRawConn(0xc827f6eaa0, 0x7fd8f6862e00, 0xc8208166b0)
    /go/src/google.golang.org/grpc/server.go:385 +0x4ee
created by google.golang.org/grpc.(*Server).Serve
    /go/src/google.golang.org/grpc/server.go:357 +0x38c

goroutine 82983 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc824d72000)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 83035 [select, 8 minutes]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc824d73170)
    /go/src/google.golang.org/grpc/transport/http2_server.go:673 +0x6ed
created by google.golang.org/grpc/transport.newHTTP2Server
    /go/src/google.golang.org/grpc/transport/http2_server.go:139 +0x84f

goroutine 83380 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).getNodeIDAddress(0xc82518f380, 0xc800000002, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:198 +0x66
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).(github.com/cockroachdb/cockroach/storage_test.getNodeIDAddress)-fm(0x2, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:542 +0x44
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:217 +0x5e
github.com/rubyist/circuitbreaker.(*Breaker).Call(0xc82e319b00, 0xc8201b99b0, 0x0, 0x0, 0x0)
    /go/src/github.com/rubyist/circuitbreaker/circuitbreaker.go:338 +0x9d
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn(0xc82e1c7d80, 0x2, 0xc824d5ac60)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:216 +0x2b4
github.com/cockroachdb/cockroach/storage.RaftSender.SendAsync(0xc82e1c7d80, 0xc827e3bdd0, 0xc824d5ac60, 0x300000003)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:324 +0xdf
github.com/cockroachdb/cockroach/storage.(*Replica).sendRaftMessage(0xc828102800, 0x5, 0x2, 0x3, 0xb, 0xa, 0x23, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1698 +0x85d
github.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc828102800, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1510 +0x86c
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc820809ee0, 0xc824c510e0, 0xc828102800)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2568 +0x25
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2573 +0x4da

goroutine 83281 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).getNodeIDAddress(0xc82518f380, 0xc800000005, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:198 +0x66
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).(github.com/cockroachdb/cockroach/storage_test.getNodeIDAddress)-fm(0x5, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:542 +0x44
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:217 +0x5e
github.com/rubyist/circuitbreaker.(*Breaker).Call(0xc8202a93b0, 0xc8201b9770, 0x0, 0x0, 0x0)
    /go/src/github.com/rubyist/circuitbreaker/circuitbreaker.go:338 +0x9d
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn(0xc827a0af40, 0x5, 0xc824d5a900)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:216 +0x2b4
github.com/cockroachdb/cockroach/storage.RaftSender.SendAsync(0xc827a0af40, 0xc827e08750, 0xc824d5a900, 0x600000006)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:324 +0xdf
github.com/cockroachdb/cockroach/storage.(*Replica).sendRaftMessage(0xc827b30400, 0x3, 0x4, 0x5, 0x7, 0x7, 0x20, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1698 +0x85d
github.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc827b30400, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1510 +0x86c
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc820809ca0, 0xc8208f3e60, 0xc827b30400)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2568 +0x25
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2573 +0x4da

rax    0xca
rbx    0x0
rcx    0x64ed53
rdx    0x0
rdi    0x2bc55c8
rsi    0x0
rbp    0x1
rsp    0x7fff1e0602c0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x2
r13    0x2055248
r14    0x1
r15    0x8
rip    0x64ed51
rflags 0x286
cs     0x33
fs     0x0
gs     0x0


ERROR: exit status 2

Run Details:

0 runs so far, 0 failures, over 5s
0 runs so far, 0 failures, over 10s
0 runs so far, 0 failures, over 15s
0 runs so far, 0 failures, over 20s
0 runs so far, 0 failures, over 25s
0 runs so far, 0 failures, over 30s
0 runs so far, 0 failures, over 35s
0 runs so far, 0 failures, over 40s
0 runs so far, 0 failures, over 45s
0 runs so far, 0 failures, over 50s
0 runs so far, 0 failures, over 55s
0 runs so far, 0 failures, over 1m0s
0 runs so far, 0 failures, over 1m5s
0 runs so far, 0 failures, over 1m10s
0 runs so far, 0 failures, over 1m15s
1 runs so far, 0 failures, over 1m20s
13 runs so far, 0 failures, over 1m25s
15 runs so far, 0 failures, over 1m30s
15 runs so far, 0 failures, over 1m35s
15 runs so far, 0 failures, over 1m40s
15 runs so far, 0 failures, over 1m45s
15 runs so far, 0 failures, over 1m50s
15 runs so far, 0 failures, over 1m55s
15 runs so far, 0 failures, over 2m0s
15 runs so far, 0 failures, over 2m5s
15 runs so far, 0 failures, over 2m10s
15 runs so far, 0 failures, over 2m15s
15 runs so far, 0 failures, over 2m20s
15 runs so far, 0 failures, over 2m25s
15 runs so far, 0 failures, over 2m30s
15 runs so far, 0 failures, over 2m35s
18 runs so far, 0 failures, over 2m40s
29 runs so far, 0 failures, over 2m45s
29 runs so far, 0 failures, over 2m50s
29 runs so far, 0 failures, over 2m55s
29 runs so far, 0 failures, over 3m0s
29 runs so far, 0 failures, over 3m5s
29 runs so far, 0 failures, over 3m10s
29 runs so far, 0 failures, over 3m15s
29 runs so far, 0 failures, over 3m20s
29 runs so far, 0 failures, over 3m25s
29 runs so far, 0 failures, over 3m30s
29 runs so far, 0 failures, over 3m35s
29 runs so far, 0 failures, over 3m40s
29 runs so far, 0 failures, over 3m45s
29 runs so far, 0 failures, over 3m50s
30 runs so far, 0 failures, over 3m55s
36 runs so far, 0 failures, over 4m0s
42 runs so far, 0 failures, over 4m5s
42 runs so far, 0 failures, over 4m10s
42 runs so far, 0 failures, over 4m15s
43 runs so far, 0 failures, over 4m20s
43 runs so far, 0 failures, over 4m25s
43 runs so far, 0 failures, over 4m30s
43 runs so far, 0 failures, over 4m35s
43 runs so far, 0 failures, over 4m40s
43 runs so far, 0 failures, over 4m45s
43 runs so far, 0 failures, over 4m50s
43 runs so far, 0 failures, over 4m55s
43 runs so far, 0 failures, over 5m0s
44 runs so far, 0 failures, over 5m5s
45 runs so far, 0 failures, over 5m10s
48 runs so far, 0 failures, over 5m15s
56 runs so far, 0 failures, over 5m20s
56 runs so far, 0 failures, over 5m25s
57 runs so far, 0 failures, over 5m30s
57 runs so far, 0 failures, over 5m35s
57 runs so far, 0 failures, over 5m40s
57 runs so far, 0 failures, over 5m45s
57 runs so far, 0 failures, over 5m50s
57 runs so far, 0 failures, over 5m55s
57 runs so far, 0 failures, over 6m0s
57 runs so far, 0 failures, over 6m5s
57 runs so far, 0 failures, over 6m10s
58 runs so far, 0 failures, over 6m15s
58 runs so far, 0 failures, over 6m20s
59 runs so far, 0 failures, over 6m25s
62 runs so far, 0 failures, over 6m30s
69 runs so far, 0 failures, over 6m35s
71 runs so far, 0 failures, over 6m40s
71 runs so far, 0 failures, over 6m45s
71 runs so far, 0 failures, over 6m50s
71 runs so far, 0 failures, over 6m55s
71 runs so far, 0 failures, over 7m0s
71 runs so far, 0 failures, over 7m5s
71 runs so far, 0 failures, over 7m10s
71 runs so far, 0 failures, over 7m15s
71 runs so far, 0 failures, over 7m20s
72 runs so far, 0 failures, over 7m25s
72 runs so far, 0 failures, over 7m30s
72 runs so far, 0 failures, over 7m35s
75 runs so far, 0 failures, over 7m40s
76 runs so far, 0 failures, over 7m45s
84 runs so far, 0 failures, over 7m50s
85 runs so far, 0 failures, over 7m55s
85 runs so far, 0 failures, over 8m0s
85 runs so far, 0 failures, over 8m5s
85 runs so far, 0 failures, over 8m10s
85 runs so far, 0 failures, over 8m15s
85 runs so far, 0 failures, over 8m20s
85 runs so far, 0 failures, over 8m25s
85 runs so far, 0 failures, over 8m30s
86 runs so far, 0 failures, over 8m35s
86 runs so far, 0 failures, over 8m40s
86 runs so far, 0 failures, over 8m45s
87 runs so far, 0 failures, over 8m50s
89 runs so far, 0 failures, over 8m55s
91 runs so far, 0 failures, over 9m0s
98 runs so far, 0 failures, over 9m5s
98 runs so far, 0 failures, over 9m10s
98 runs so far, 0 failures, over 9m15s
98 runs so far, 0 failures, over 9m20s
98 runs so far, 0 failures, over 9m25s
98 runs so far, 0 failures, over 9m30s
99 runs so far, 0 failures, over 9m35s
99 runs so far, 0 failures, over 9m40s
99 runs so far, 0 failures, over 9m45s
99 runs so far, 0 failures, over 9m50s
99 runs so far, 0 failures, over 9m55s
99 runs so far, 0 failures, over 10m0s
100 runs completed, 1 failures, over 10m0s
FAIL

Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added this to the Q3 milestone Aug 1, 2016
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Aug 1, 2016
@tamird
Copy link
Contributor

tamird commented Aug 3, 2016

Looks like a deadlock on multiTestContext.mu. @bdarnell?

@tamird
Copy link
Contributor

tamird commented Aug 3, 2016

Root cause looks same as #8149.

@tamird tamird closed this as completed Aug 3, 2016
@tamird tamird mentioned this issue Aug 3, 2016
Closed
@tbg
Copy link
Member

tbg commented Aug 3, 2016

I don't know how this ties into the lock, but definitely looks like one processRaft goroutine is stuck as well. That's likely fallout, but I don't immediately see where the replica lock is held.

@tbg
Copy link
Member

tbg commented Aug 3, 2016

Ah, here's the held mtc mutex:

goroutine 83225 [select, 8 minutes]:
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc82821a400, 0x7fd8f9367658, 0xc820217980, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1281 +0x7db
github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc82821a400, 0x7fd8f9367658, 0xc820217980, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:827 +0x23d
github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc827f2b680, 0x7fd8f929efd8, 0xc82e6e2e00, 0x50775d800, 0x5, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2207 +0xd05
github.com/cockroachdb/cockroach/storage.(*Stores).Send(0xc827a0ac40, 0x7fd8f929efd8, 0xc82e6e2e00, 0x0, 0x0, 0x500000005, 0x4, 0xb, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/stores.go:182 +0x3a0
github.com/cockroachdb/cockroach/storage_test.(*multiTestContextKVTransport).SendNext.func1()
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:381 +0x20f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc82009de60, 0x259bf06, 0x16, 0x196, 0xc8201bd1c0)
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:231 +0x8d
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
    /go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:232 +0x124

So one Raft command isn't coming back.

@tbg
Copy link
Member

tbg commented Aug 3, 2016

Yep, and it isn't coming back because the Raft transport also wants in on the action:

goroutine 83380 [semacquire, 8 minutes]:
sync.runtime_Semacquire(0xc82e66cf0c)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*RWMutex).RLock(0xc82e66cf00)
    /usr/local/go/src/sync/rwmutex.go:37 +0x58
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).getNodeIDAddress(0xc82518f380, 0xc800000002, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:198 +0x66
github.com/cockroachdb/cockroach/storage_test.(*multiTestContext).(github.com/cockroachdb/cockroach/storage_test.getNodeIDAddress)-fm(0x2, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/client_test.go:542 +0x44
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn.func1(0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:217 +0x5e
github.com/rubyist/circuitbreaker.(*Breaker).Call(0xc82e319b00, 0xc8201b99b0, 0x0, 0x0, 0x0)
    /go/src/github.com/rubyist/circuitbreaker/circuitbreaker.go:338 +0x9d
github.com/cockroachdb/cockroach/storage.(*RaftTransport).getNodeConn(0xc82e1c7d80, 0x2, 0xc824d5ac60)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:216 +0x2b4
github.com/cockroachdb/cockroach/storage.RaftSender.SendAsync(0xc82e1c7d80, 0xc827e3bdd0, 0xc824d5ac60, 0x300000003)
    /go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:324 +0xdf
github.com/cockroachdb/cockroach/storage.(*Replica).sendRaftMessage(0xc828102800, 0x5, 0x2, 0x3, 0xb, 0xa, 0x23, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1698 +0x85d
github.com/cockroachdb/cockroach/storage.(*Replica).handleRaftReady(0xc828102800, 0x0, 0x0)
    /go/src/github.com/cockroachdb/cockroach/storage/replica.go:1510 +0x86c
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc820809ee0, 0xc824c510e0, 0xc828102800)
    /go/src/github.com/cockroachdb/cockroach/storage/store.go:2568 +0x25

@tbg
Copy link
Member

tbg commented Aug 3, 2016

Essentially we're holding the RLock during SendNext, then in another thread we try to stop a store which tries to get a write lock, blocking all future RLocks. But SendNext won't come back until Raft has had a chance to get another read lock at a later point. Frankly I think this pretty beyond repair.

The only "easy" fix I can think of is this is acquiring an RLock only in the first stage of stopStore. Will try.

tbg added a commit to tbg/cockroach that referenced this issue Aug 3, 2016
@tbg
storage: fix deadlock in multiTestContext
f1eedf2 
See cockroachdb#7488 and cockroachdb#8170. Attempting to acquire a write lock early in `stopStore`
could lead to situations in which an outstanding Raft proposal never returned
(due to address resolution calling back into `multiTestContext` with a RLock),
but at the same time that write lock being stuck on a read lock held in
`SendNext` which in turn waited on Raft:

SendNext[hold RLock] -> Raft[want RLock]
            ʌ               /
              \            v
            stopStore[want Lock]

The solution (which I wasn't able to test, for the flakiness doesn't easily
reproduce on my laptop and that's all I have available at the moment) is to
acquire first a read lock to quiesce the stopper, which should tell everything
downstream to let go of what they're trying to accomplish before the
opportunity for deadlock presents itself.
I'm sure there will be another one, though.
@tbg tbg mentioned this issue Aug 3, 2016
Merged
@bdarnell bdarnell mentioned this issue Aug 3, 2016
Closed
@bdarnell
Copy link
Contributor

bdarnell commented Aug 3, 2016

Frankly I think this pretty beyond repair.

There's no longer much need to do all this manual mocking; we should move these tests to TestCluster. Filed #8299

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
Q3
Development

No branches or pull requests
4 participants
@bdarnell @tamird @tbg @cockroach-teamcity