: TestStoreRangeRebalance failed under stress #10156

cockroach-teamcity · 2016-10-23T06:19:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/7071c024c8695bd9963af293e311626ebc675961

Stress build found a failed test:

I161023 06:19:13.857865 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:13.866801 15360 gossip/gossip.go:236  initial resolvers: []
W161023 06:19:13.867032 15360 gossip/gossip.go:1063  no resolvers found; use --join to specify a connected node
I161023 06:19:13.867297 15360 gossip/gossip.go:271  NodeID set to 1
I161023 06:19:13.887198 15360 gossip/gossip.go:271  NodeID set to 1
I161023 06:19:13.895068 15360 storage/store.go:1125  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I161023 06:19:13.895381 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:50882" > attrs:<> locality:<> 
I161023 06:19:13.897317 15344 storage/replica_proposal.go:292  [s1,r1/1:/M{in-ax}] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=1970-01-01 00:00:00 +0000 UTC]
I161023 06:19:13.901462 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:13.902556 15360 gossip/gossip.go:236  initial resolvers: [127.0.0.1:50882]
W161023 06:19:13.902668 15360 gossip/gossip.go:1065  no incoming or outgoing connections
I161023 06:19:13.902895 15360 gossip/gossip.go:271  NodeID set to 2
I161023 06:19:13.907546 15360 gossip/gossip.go:271  NodeID set to 2
I161023 06:19:13.918629 15360 storage/store.go:1125  [n2,s2]: failed initial metrics computation: [n2,s2]: system config not yet available
I161023 06:19:13.919053 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:41300" > attrs:<> locality:<> 
I161023 06:19:13.920868 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:13.922421 15360 gossip/gossip.go:236  initial resolvers: [127.0.0.1:50882]
I161023 06:19:13.922517 15453 gossip/client.go:126  node 2: started gossip client to 127.0.0.1:50882
W161023 06:19:13.922574 15360 gossip/gossip.go:1065  no incoming or outgoing connections
I161023 06:19:13.922955 15360 gossip/gossip.go:271  NodeID set to 3
I161023 06:19:13.931846 15360 gossip/gossip.go:271  NodeID set to 3
I161023 06:19:13.940300 15360 storage/store.go:1125  [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I161023 06:19:13.940565 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:56363" > attrs:<> locality:<> 
I161023 06:19:13.953209 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:13.954988 15360 gossip/gossip.go:236  initial resolvers: [127.0.0.1:50882]
W161023 06:19:13.956271 15360 gossip/gossip.go:1065  no incoming or outgoing connections
I161023 06:19:13.957483 15360 gossip/gossip.go:271  NodeID set to 4
I161023 06:19:13.961455 15496 gossip/client.go:126  node 3: started gossip client to 127.0.0.1:50882
I161023 06:19:14.011599 15360 gossip/gossip.go:271  NodeID set to 4
I161023 06:19:14.016492 15360 storage/store.go:1125  [n4,s4]: failed initial metrics computation: [n4,s4]: system config not yet available
I161023 06:19:14.019972 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:4 address:<network_field:"tcp" address_field:"127.0.0.1:44141" > attrs:<> locality:<> 
I161023 06:19:14.026819 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:14.030126 15592 gossip/client.go:126  node 4: started gossip client to 127.0.0.1:50882
I161023 06:19:14.032731 15360 gossip/gossip.go:236  initial resolvers: [127.0.0.1:50882]
W161023 06:19:14.037207 15360 gossip/gossip.go:1065  no incoming or outgoing connections
I161023 06:19:14.037471 15360 gossip/gossip.go:271  NodeID set to 5
I161023 06:19:14.061042 15360 gossip/gossip.go:271  NodeID set to 5
I161023 06:19:14.065484 15610 gossip/client.go:126  node 5: started gossip client to 127.0.0.1:50882
I161023 06:19:14.067642 15585 gossip/server.go:260  node 1: refusing gossip from node 5 (max 3 conns); forwarding to 4 ({tcp 127.0.0.1:44141})
I161023 06:19:14.072670 15610 gossip/client.go:131  node 5: closing client to node 1 (127.0.0.1:50882): received forward from node 1 to 4 (127.0.0.1:44141)
I161023 06:19:14.073897 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:5 address:<network_field:"tcp" address_field:"127.0.0.1:52812" > attrs:<> locality:<> 
I161023 06:19:14.075768 15684 gossip/client.go:126  node 5: started gossip client to 127.0.0.1:44141
I161023 06:19:14.096030 15360 storage/engine/rocksdb.go:349  opening in memory rocksdb instance
I161023 06:19:14.097682 15360 gossip/gossip.go:236  initial resolvers: [127.0.0.1:50882]
W161023 06:19:14.097789 15360 gossip/gossip.go:1065  no incoming or outgoing connections
I161023 06:19:14.098067 15360 gossip/gossip.go:271  NodeID set to 6
I161023 06:19:14.116158 15360 gossip/gossip.go:271  NodeID set to 6
I161023 06:19:14.145724 15360 storage/store.go:1125  [n6,s6]: failed initial metrics computation: [n6,s6]: system config not yet available
I161023 06:19:14.146076 15360 gossip/gossip.go:286  NodeDescriptor set to node_id:6 address:<network_field:"tcp" address_field:"127.0.0.1:41724" > attrs:<> locality:<> 
I161023 06:19:14.181932 15767 gossip/client.go:126  node 6: started gossip client to 127.0.0.1:50882
I161023 06:19:14.191821 15772 gossip/server.go:260  node 1: refusing gossip from node 6 (max 3 conns); forwarding to 3 ({tcp 127.0.0.1:56363})
I161023 06:19:14.202020 15772 gossip/server.go:260  node 1: refusing gossip from node 6 (max 3 conns); forwarding to 4 ({tcp 127.0.0.1:44141})
I161023 06:19:14.207933 15767 gossip/client.go:131  node 6: closing client to node 1 (127.0.0.1:50882): received forward from node 1 to 3 (127.0.0.1:56363)
I161023 06:19:14.209202 15778 gossip/client.go:126  node 6: started gossip client to 127.0.0.1:56363
I161023 06:19:14.238212 15360 storage/replica_command.go:2354  [s1,r1/1:/M{in-ax}] initiating a split of this range at key "split" [r2]
E161023 06:19:14.253075 15368 storage/queue.go:569  [replicate] (purgatory) on [n1,s1,r1/1:{/Min-"split"}]: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster
E161023 06:19:14.255698 15368 storage/queue.go:569  [replicate] (purgatory) on [n1,s1,r2/1:{"split"-/Max}]: 0 of 0 stores with an attribute matching []; likely not enough nodes in cluster
I161023 06:19:14.261036 15360 storage/replica_raftstorage.go:446  [s1,r1/1:{/Min-"split"}] generated snapshot d485eaa5 for range 1 at index 24 in 188.263µs.
I161023 06:19:14.263986 15360 storage/store.go:2965  streamed snapshot: kv pairs: 26, log entries: 14
I161023 06:19:14.271843 15790 storage/replica_raftstorage.go:577  [s2] [n2,s2,r1/?:{-}]: with replicaID [?], applying preemptive snapshot at index 24 (id=d485eaa5, encoded size=16, 1 rocksdb batches, 14 log entries)
I161023 06:19:14.273862 15790 storage/replica_raftstorage.go:580  [s2] [n2,s2,r1/?:{/Min-"split"}]: with replicaID [?], applied preemptive snapshot in 0.002s
I161023 06:19:14.280487 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:1 start_key:"" end_key:"split" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 
I161023 06:19:14.289606 15846 storage/replica.go:1765  [s1,r1/1:{/Min-"split"}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]
I161023 06:19:14.304204 15360 storage/replica_raftstorage.go:446  [s1,r1/1:{/Min-"split"}] generated snapshot 0082999d for range 1 at index 29 in 204.774µs.
I161023 06:19:14.307447 15360 storage/store.go:2965  streamed snapshot: kv pairs: 32, log entries: 19
I161023 06:19:14.308324 15866 storage/replica_raftstorage.go:577  [s3] [n3,s3,r1/?:{-}]: with replicaID [?], applying preemptive snapshot at index 29 (id=0082999d, encoded size=16, 1 rocksdb batches, 19 log entries)
I161023 06:19:14.309930 15866 storage/replica_raftstorage.go:580  [s3] [n3,s3,r1/?:{/Min-"split"}]: with replicaID [?], applied preemptive snapshot in 0.001s
I161023 06:19:14.311051 15877 storage/raft_transport.go:423  raft transport stream to node 1 established
I161023 06:19:14.313367 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:1 start_key:"" end_key:"split" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > next_replica_id:3 
I161023 06:19:14.386243 15857 storage/replica.go:1765  [s1,r1/1:{/Min-"split"}] proposing ADD_REPLICA {NodeID:3 StoreID:3 ReplicaID:3} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3}]
I161023 06:19:14.416898 15360 storage/replica_raftstorage.go:446  [s1,r2/1:{"split"-/Max}] generated snapshot be5e80e9 for range 2 at index 11 in 4.624959ms.
I161023 06:19:14.420285 15360 storage/store.go:2965  streamed snapshot: kv pairs: 28, log entries: 1
I161023 06:19:14.421708 15901 storage/replica_raftstorage.go:577  [s2] [n2,s2,r2/?:{-}]: with replicaID [?], applying preemptive snapshot at index 11 (id=be5e80e9, encoded size=16, 1 rocksdb batches, 1 log entries)
I161023 06:19:14.433086 15901 storage/replica_raftstorage.go:580  [s2] [n2,s2,r2/?:{"split"-/Max}]: with replicaID [?], applied preemptive snapshot in 0.011s
I161023 06:19:14.434061 15949 storage/raft_transport.go:423  raft transport stream to node 1 established
I161023 06:19:14.452022 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:2 start_key:"split" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > next_replica_id:2 
I161023 06:19:14.518583 15959 storage/replica.go:1765  [s1,r2/1:{"split"-/Max}] proposing ADD_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2}]
I161023 06:19:14.605168 15360 storage/replica_raftstorage.go:446  [s1,r2/1:{"split"-/Max}] generated snapshot 745f1171 for range 2 at index 14 in 22.565994ms.
I161023 06:19:14.655996 15360 storage/store.go:2965  streamed snapshot: kv pairs: 30, log entries: 4
I161023 06:19:14.657992 15984 storage/replica_raftstorage.go:577  [s4] [n4,s4,r2/?:{-}]: with replicaID [?], applying preemptive snapshot at index 14 (id=745f1171, encoded size=16, 1 rocksdb batches, 4 log entries)
I161023 06:19:14.676652 15984 storage/replica_raftstorage.go:580  [s4] [n4,s4,r2/?:{"split"-/Max}]: with replicaID [?], applied preemptive snapshot in 0.019s
I161023 06:19:14.744208 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:2 start_key:"split" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > next_replica_id:3 
I161023 06:19:14.925842 16084 storage/replica.go:1765  [s1,r2/1:{"split"-/Max}] proposing ADD_REPLICA {NodeID:4 StoreID:4 ReplicaID:3} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:4 StoreID:4 ReplicaID:3}]
I161023 06:19:14.972218 15360 storage/replica_raftstorage.go:446  [s1,r2/1:{"split"-/Max}] generated snapshot 630b2aa5 for range 2 at index 18 in 234.012µs.
I161023 06:19:14.976430 15360 storage/store.go:2965  streamed snapshot: kv pairs: 31, log entries: 8
I161023 06:19:14.977936 16091 storage/replica_raftstorage.go:577  [s5] [n5,s5,r2/?:{-}]: with replicaID [?], applying preemptive snapshot at index 18 (id=630b2aa5, encoded size=16, 1 rocksdb batches, 8 log entries)
I161023 06:19:14.994682 16091 storage/replica_raftstorage.go:580  [s5] [n5,s5,r2/?:{"split"-/Max}]: with replicaID [?], applied preemptive snapshot in 0.017s
I161023 06:19:15.007098 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:2 start_key:"split" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:4 store_id:4 replica_id:3 > next_replica_id:4 
I161023 06:19:15.022044 16094 storage/raft_transport.go:423  raft transport stream to node 1 established
I161023 06:19:15.132002 16133 storage/replica.go:1765  [s1,r2/1:{"split"-/Max}] proposing ADD_REPLICA {NodeID:5 StoreID:5 ReplicaID:4} for range 2: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:4 StoreID:4 ReplicaID:3} {NodeID:5 StoreID:5 ReplicaID:4}]
I161023 06:19:15.206766 15360 storage/replica_command.go:3236  change replicas: read existing descriptor range_id:2 start_key:"split" end_key:"\377\377" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:4 store_id:4 replica_id:3 > replicas:<node_id:5 store_id:5 replica_id:4 > next_replica_id:5 
I161023 06:19:15.210587 16189 storage/raft_transport.go:423  raft transport stream to node 1 established
I161023 06:19:15.380624 16144 storage/replica.go:1765  [s1,r2/1:{"split"-/Max}] proposing REMOVE_REPLICA {NodeID:1 StoreID:1 ReplicaID:1} for range 2: [{NodeID:5 StoreID:5 ReplicaID:4} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:4 StoreID:4 ReplicaID:3}]
I161023 06:19:15.415348 16186 storage/store.go:2826  [s1] [n1,s1,r2/1:{"split"-/Max}]: added to replica GC queue (peer suggestion)
I161023 06:19:15.629360 16324 storage/raft_transport.go:423  raft transport stream to node 2 established
I161023 06:19:15.630270 16325 storage/raft_transport.go:423  raft transport stream to node 5 established
I161023 06:19:15.634646 16331 storage/raft_transport.go:423  raft transport stream to node 4 established
I161023 06:19:15.643526 16332 storage/raft_transport.go:423  raft transport stream to node 5 established
I161023 06:19:15.786016 16391 storage/raft_transport.go:423  raft transport stream to node 2 established
I161023 06:19:15.788162 16392 storage/raft_transport.go:423  raft transport stream to node 4 established
W161023 06:19:15.892106 15398 raft/raft.go:696  [s1,r2/1:{"split"-/Max}] 1 stepped down to follower since quorum is not active
I161023 06:19:16.104613 15393 storage/replica_proposal.go:292  [s2,r2/2:{"split"-/Max}] new range lease replica {2 2 2} 1970-01-01 00:00:00.9 +0000 UTC 1.8s following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900ms [physicalTime=1970-01-01 00:00:01.8 +0000 UTC]
I161023 06:19:16.443973 15826 storage/queue.go:620  [replicate] purgatory is now empty
I161023 06:19:16.553466 15360 storage/client_test.go:416  gossip network initialized
I161023 06:19:16.559602 15360 storage/replica_raftstorage.go:446  [replicate,s1,r1/1:{/Min-"split"}] generated snapshot 4094125d for range 1 at index 123 in 381.015µs.
I161023 06:19:16.613472 15360 storage/store.go:2965  [replicate,s1,r1/1:{/Min-"split"}] streamed snapshot: kv pairs: 111, log entries: 113
I161023 06:19:16.625356 16619 storage/replica_raftstorage.go:577  [s6] [n6,s6,r1/?:{-}]: with replicaID [?], applying preemptive snapshot at index 123 (id=4094125d, encoded size=16, 1 rocksdb batches, 113 log entries)
I161023 06:19:16.678112 16619 storage/replica_raftstorage.go:580  [s6] [n6,s6,r1/?:{/Min-"split"}]: with replicaID [?], applied preemptive snapshot in 0.053s
I161023 06:19:16.697578 15360 storage/replica_command.go:3236  [replicate,s1,r1/1:{/Min-"split"}] change replicas: read existing descriptor range_id:1 start_key:"" end_key:"split" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > next_replica_id:4 
I161023 06:19:16.849319 16698 storage/replica.go:1765  [s1,r1/1:{/Min-"split"}] proposing ADD_REPLICA {NodeID:6 StoreID:6 ReplicaID:4} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:2 StoreID:2 ReplicaID:2} {NodeID:3 StoreID:3 ReplicaID:3} {NodeID:6 StoreID:6 ReplicaID:4}]
I161023 06:19:16.947191 16757 storage/raft_transport.go:423  raft transport stream to node 1 established
I161023 06:19:16.959578 15344 storage/replica_raftstorage.go:446  [s1,r1/1:{/Min-"split"}] generated snapshot b9394c77 for range 1 at index 140 in 737.029µs.
I161023 06:19:16.962281 15360 storage/replica_command.go:3236  [replicate,s1,r1/1:{/Min-"split"}] change replicas: read existing descriptor range_id:1 start_key:"" end_key:"split" replicas:<node_id:1 store_id:1 replica_id:1 > replicas:<node_id:2 store_id:2 replica_id:2 > replicas:<node_id:3 store_id:3 replica_id:3 > replicas:<node_id:6 store_id:6 replica_id:4 > next_replica_id:5 
I161023 06:19:16.975984 16762 storage/store.go:2965  [s1,r1/1:{/Min-"split"}] streamed snapshot: kv pairs: 128, log entries: 10
I161023 06:19:16.976469 16765 storage/replica_raftstorage.go:577  [s6,r1/?:{/Min-"split"}] [n6,s6,r1/?:{/Min-"split"}]: with replicaID 4, applying Raft snapshot at index 140 (id=b9394c77, encoded size=16, 1 rocksdb batches, 10 log entries)
I161023 06:19:16.993232 16765 storage/replica_raftstorage.go:580  [s6,r1/4:{/Min-"split"}] [n6,s6,r1/4:{/Min-"split"}]: with replicaID 4, applied Raft snapshot in 0.017s
I161023 06:19:17.109256 16750 storage/replica.go:1765  [s1,r1/1:{/Min-"split"}] proposing REMOVE_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:6 StoreID:6 ReplicaID:4} {NodeID:3 StoreID:3 ReplicaID:3}]
I161023 06:19:17.384790 16918 util/stop/stopper.go:425  quiescing; tasks left:
3      storage/client_test.go:503
I161023 06:19:17.388449 16920 util/stop/stopper.go:425  quiescing; tasks left:
1      storage/client_test.go:503
I161023 06:19:17.389113 16918 util/stop/stopper.go:425  quiescing; tasks left:
2      storage/client_test.go:503
E161023 06:19:17.393321 15377 storage/node_liveness.go:141  [hb] failed liveness heartbeat: failed to send RPC: sending to all 3 replicas failed; last error: range 1: replica {3 3 3} not lease holder; node_id:1 store_id:1 replica_id:1  is
E161023 06:19:17.398148 15435 storage/node_liveness.go:141  [hb] failed liveness heartbeat: failed to send RPC: sending to all 3 replicas failed; last error: range 1: replica {3 3 3} not lease holder; node_id:1 store_id:1 replica_id:1  is
E161023 06:19:17.400104 15423 storage/node_liveness.go:141  [hb] failed liveness heartbeat: node unavailable; try another peer
I161023 06:19:17.437606 16918 util/stop/stopper.go:425  quiescing; tasks left:
1      storage/client_test.go:503
E161023 06:19:17.438962 15766 storage/node_liveness.go:141  [hb] failed liveness heartbeat: node unavailable; try another peer
I161023 06:19:17.445080 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.448072 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.450491 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
E161023 06:19:17.474891 15599 storage/node_liveness.go:141  [hb] failed liveness heartbeat: node unavailable; try another peer
I161023 06:19:17.481371 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.483825 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.510605 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.513754 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
W161023 06:19:17.528119 16332 storage/raft_transport.go:428  raft transport stream to node 5 failed: EOF
W161023 06:19:17.528571 16324 storage/raft_transport.go:428  raft transport stream to node 2 failed: rpc error: code = 13 desc = transport is closing
I161023 06:19:17.530349 15460 http2_server.go:276  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:56363->127.0.0.1:33197: use of closed network connection
I161023 06:19:17.534833 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.535528 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.536311 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.536746 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.537146 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
I161023 06:19:17.537540 16917 util/stop/stopper.go:353  stop has been called, stopping or quiescing all running tasks
    client_raft_test.go:1933: expected 0 normal snapshots, but found 1

The text was updated successfully, but these errors were encountered:

tbg · 2016-11-04T10:38:49Z

Fails instantly: make stressrace PKG=./storage TESTS=TestStoreRangeRebalance

tbg · 2016-11-04T10:46:45Z

Another failure mode: make stressrace PKG=./storage TESTS=TestStoreRangeRebalance STRESSFLAGS='-maxfails 1 -stderr -ignore "expected 0 normal snapshots"'

This is on #10327, I wasn't able to repro this on master (though I suspect timing changes).

Sure seems like we should look into this test more proactively. I might try after I get my PR merged.

I161104 10:43:17.842292 1602 storage/replica.go:2000  [s1,r1/1:{/Min-"split"}] proposing REMOVE_REPLICA {NodeID:2 StoreID:2 ReplicaID:2} for range 1: [{NodeID:1 StoreID:1 ReplicaID:1} {NodeID:6 StoreID:6 ReplicaID:4} {NodeID:3 StoreID:3 ReplicaID:3}]
I161104 10:43:17.886931 914 storage/store.go:2960  [s2] [n2,s2,r1/2:{/Min-"split"}]: added to replica GC queue (peer suggestion)
F161104 10:43:17.912125 1619 storage/replica_command.go:1121  [s2,r1/2:{/Min-"split"}] range lookup of meta key /Meta2/System/NodeLiveness/3 found only non-matching ranges: []
goroutine 1619 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x398b000, 0x365dee0d, 0x2fc7da0, 0x0)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:849 +0xc2
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x2fc9440, 0xc400000004, 0x296bdf8, 0x1a, 0x461, 0xc4215c1a80, 0x71)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:710 +0x9ce
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x7fa69f7c0838, 0xc421865080, 0x4, 0x2, 0x1f56fc3, 0x3f, 0xc42057c568, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:140 +0x331
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x7fa69f7c0838, 0xc421865080, 0x1, 0x4, 0x1f56fc3, 0x3f, 0xc42057c568, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:88 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x7fa69f7c0838, 0xc421865080, 0x1f56fc3, 0x3f, 0xc42057c568, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:172 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).RangeLookup(0xc420afa700, 0x7fa69f7c0838, 0xc421865080, 0x2c81da0, 0xc420571090, 0x6b49d27d, 0x298, 0x200000002, 0x2, 0x1, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1121 +0x160d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeCmd(0xc420afa700, 0x7fa69f7c0838, 0xc421865080, 0x0, 0x0, 0x0, 0x2c81da0, 0xc420571090, 0x0, 0x6b49d27d, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:135 +0xfed
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatch(0xc420afa700, 0x7fa69f7c0838, 0xc421865080, 0x0, 0x0, 0x2c81da0, 0xc420571090, 0x0, 0x6b49d27d, 0x298, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3613 +0x4d4
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).addReadOnlyCmd(0xc420afa700, 0x7fa69f7c0838, 0xc421865080, 0x6b49d27d, 0x298, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1553 +0x30f
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420afa700, 0x7fa69f7c0838, 0xc421865080, 0x6b49d27d, 0x298, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1110 +0x57e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc4201aee00, 0x7fa69f7c0838, 0xc421865020, 0x6b49d27d, 0x298, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2354 +0x8f8
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc42066e2a0, 0x7fa6a1ed1728, 0xc420eb6a40, 0x0, 0x0, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24b
github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext.func1(0x7fa6a1ed1728, 0xc420eb6a40)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:471 +0x24d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc420600630, 0x296aabd, 0x16, 0x1f5, 0x0, 0x0, 0xc4207337e0, 0x7fa6a1ed1728, 0xc420eb6a40)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xed
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x2b0

tbg · 2016-11-04T10:53:38Z

Nevermind, on master too:

E161104 10:50:20.911665 91 storage/queue.go:568  [replicate] on [n1,s1,r2/1:{"split"-/Max}]: [n1,s1,r2/1:{"split"-/Max}]: could not obtain lease: range 2 was not found
F161104 10:50:20.921625 1650 storage/replica_command.go:1122  [s2,r1/2:{/Min-"split"}] range lookup of meta key /Meta2/System/NodeLiveness/4 found only non-matching ranges: []
goroutine 1650 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x397de00, 0x36eee4aa, 0x2fbac60, 0x0)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:849 +0xc2
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x2fbc300, 0xc400000004, 0x295ed1a, 0x1a, 0x462, 0xc420e83880, 0x71)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:710 +0x9ce
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x7f1473d552b8, 0xc4210ab710, 0x4, 0x2, 0x1f4ee9e, 0x3f, 0xc4213ec9e0, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:140 +0x331
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x7f1473d552b8, 0xc4210ab710, 0x1, 0x4, 0x1f4ee9e, 0x3f, 0xc4213ec9e0, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:88 +0x9a
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(0x7f1473d552b8, 0xc4210ab710, 0x1f4ee9e, 0x3f, 0xc4213ec9e0, 0x2, 0x2)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:172 +0x90
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).RangeLookup(0xc420cf4000, 0x7f1473d552b8, 0xc4210ab710, 0x2c74d20, 0xc42054c910, 0x6b49d27d, 0x1eb, 0x200000002, 0x2, 0x1, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1122 +0x160d
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeCmd(0xc420cf4000, 0x7f1473d552b8, 0xc4210ab710, 0x0, 0x0, 0x0, 0x2c74d20, 0xc42054c910, 0x0, 0x6b49d27d, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:135 +0xfed
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeBatch(0xc420cf4000, 0x7f1473d552b8, 0xc4210ab710, 0x0, 0x0, 0x2c74d20, 0xc42054c910, 0x0, 0x6b49d27d, 0x1eb, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:3481 +0x537
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).addReadOnlyCmd(0xc420cf4000, 0x7f1473d552b8, 0xc4210ab710, 0x6b49d27d, 0x1eb, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1538 +0x30f
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc420cf4000, 0x7f1473d552b8, 0xc4210ab710, 0x6b49d27d, 0x1eb, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1107 +0x57e
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc42017ee00, 0x7f1473d552b8, 0xc4210ab6b0, 0x6b49d27d, 0x1eb, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2353 +0x8f8
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc4206b8660, 0x7f1473d55240, 0xc420a39200, 0x0, 0x0, 0x200000002, 0x2, 0x1, 0x0, 0x0, ...)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24b
github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext.func1(0x7f1473d55240, 0xc420a39200)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:471 +0x24d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc4205562d0, 0x295d9df, 0x16, 0x1f5, 0x0, 0x0, 0xc420a37140, 0x7f1473d55240, 0xc420a39200)
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xed
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x2b0


ERROR: exit status 255

957 runs completed, 1 failures, over 3m19s
FAIL
Makefile:128: recipe for target 'stress' failed
make[1]: *** [stress] Error 1

@tamird I think you mentioned said error to me recently.

tbg · 2016-11-04T11:04:31Z

Another failure mode (seen in exp-prop-kv without exp-prop-kv enabled):

make stressrace PKG=./storage TESTS=TestStoreRangeRebalance STRESSFLAGS='-maxfails 1 -stderr -ignore "expected 0 normal snapshots|found only non-matching"'
--- FAIL: TestStoreRangeRebalance (4.32s)
        client_raft_test.go:2135: expected 7 preemptive snapshots, but found 6

tbg · 2016-11-04T11:15:28Z

And a data race, this time from a CI run on proposer-evaluated KV. Wasn't able to repro this either where it happened nor on master.

The offending line is storage/store.go:2279:

        if pErr != nil {
            pErr.Now = now // <--
        } else {

The race happens right after the "new range lease" message and before a purgatory is now empty message. One of those two operations likely triggered an error to be returned to two people at the same time. Something like

pErr := make_error()
client1Chan <- pErr
client2Chan <- pErr

I161103 22:12:31.888992 19275 storage/replica_proposal.go:334  [s2,r2/2:{"split"-/Max}] new range lease replica {2 2 2} 1970-01-01 00:00:00.900000124 +0000 UTC 1.800000002s following replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 900.000124ms [physicalTime=1970-01-01 00:00:01.800000125 +0000 UTC]
==================
WARNING: DATA RACE
Write at 0x00c42147fd48 by goroutine 437:
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2279 +0x325
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2423 +0xd84
  github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24a
  github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:471 +0x24c
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xec

Previous write at 0x00c42147fd48 by goroutine 258:
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2279 +0x325
  github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:2423 +0xd84
  github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:187 +0x24a
  github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:471 +0x24c
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:264 +0xec

Goroutine 437 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x2af
  github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:501 +0x481
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1082 +0x36c
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:363 +0x354
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:432 +0x184
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:873 +0x365
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:755 +0x574
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:576 +0x32e
  github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:414 +0x7b4
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:511 +0x14c
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:445 +0x8e
  github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:416 +0x1ab
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:447 +0x125
  github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).processIntentsAsync.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:378 +0x8b6
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:326 +0x119

Goroutine 258 (finished) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:265 +0x2af
  github.com/cockroachdb/cockroach/pkg/storage_test.(*multiTestContextKVTransport).SendNext()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/client_test.go:501 +0x481
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1082 +0x36c
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:363 +0x354
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:432 +0x184
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:873 +0x365
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:755 +0x574
  github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:576 +0x32e
  github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send()
      /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:414 +0x7b4
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:511 +0x14c
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:445 +0x8e
  github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:416 +0x1ab
  github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run()
      /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:447 +0x125
  github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).maybePushTransactions()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:234 +0xd2d
  github.com/cockroachdb/cockroach/pkg/storage.(*intentResolver).processIntentsAsync.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/storage/intent_resolver.go:287 +0x1dc
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func1()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:326 +0x119
==================

Rather than the somewhat complicated rebalancing scenario, use a simple scenario that we perform up-replication of range 1 from 1 to 3 nodes. We check that this up-replication is performed using preemptive snapshots. The more complicated scenario was very fragile, frequently being broken by innocuous changes. Fixes cockroachdb#10497 Fixes cockroachdb#10193 Fixes cockroachdb#10156 Fixes cockroachdb#9395

Rather than the somewhat complicated rebalancing scenario, use a simple scenario that we perform up-replication of range 1 from 1 to 3 nodes. We check that this up-replication is performed using preemptive snapshots. The more complicated scenario was very fragile, frequently being broken by innocuous changes. Fixes cockroachdb#10193 Fixes cockroachdb#10156 Fixes cockroachdb#9395

Add the check that preemptive snapshots are being used to TestStoreRangeUpReplicate. Add TestReplicateQueueRebalance for testing that basic rebalancing is working. Fixes cockroachdb#10193 Fixes cockroachdb#10156 Fixes cockroachdb#9395

cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Oct 23, 2016

tamird assigned BramGruneir Oct 23, 2016

This was referenced Oct 26, 2016

: TestStoreRangeRebalance failed under stress #10193

Closed

: TestStoreRangeRebalance failed under stress #10315

Closed

teamcity: failed tests on master: testrace/TestStoreRangeRebalance #9980

Closed

tbg mentioned this issue Nov 4, 2016

: TestStoreRangeRebalance failed under stress #10452

Closed

tamird mentioned this issue Nov 7, 2016

storage: range lookup of meta key found only non-matching ranges #10501

Closed

petermattis mentioned this issue Nov 7, 2016

storage: rewrite TestStoreRangeRebalance #10515

Merged

petermattis closed this as completed in #10515 Nov 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

: TestStoreRangeRebalance failed under stress #10156

: TestStoreRangeRebalance failed under stress #10156

cockroach-teamcity commented Oct 23, 2016 •

edited by tamird

Loading

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016 •

edited

Loading

tbg commented Nov 4, 2016

: TestStoreRangeRebalance failed under stress #10156

: TestStoreRangeRebalance failed under stress #10156

Comments

cockroach-teamcity commented Oct 23, 2016 • edited by tamird Loading

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016

tbg commented Nov 4, 2016 • edited Loading

tbg commented Nov 4, 2016

cockroach-teamcity commented Oct 23, 2016 •

edited by tamird

Loading

tbg commented Nov 4, 2016 •

edited

Loading