Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teamcity: failed tests on master: testrace/TestSnapshotAfterTruncation/sameTerm, testrace/TestSnapshotAfterTruncation, testrace/TestSnapshotAfterTruncation/differentTerm #17448

Closed
cockroach-teamcity opened this issue Aug 4, 2017 · 12 comments
Closed

teamcity: failed tests on master: testrace/TestSnapshotAfterTruncation/sameTerm, testrace/TestSnapshotAfterTruncation, testrace/TestSnapshotAfterTruncation/differentTerm #17448

cockroach-teamcity opened this issue Aug 4, 2017 · 12 comments
Assignees
@spencerkimball @a-robinson @irfansharif
Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.

Comments

@cockroach-teamcity
Copy link
Member

The following tests appear to have failed:

#315166:

--- FAIL: testrace/TestSnapshotAfterTruncation/sameTerm (0.000s)
Test ended in panic.

------- Stdout: -------
I170804 17:05:28.217569 36299 storage/store.go:1275  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I170804 17:05:28.217839 36299 gossip/gossip.go:295  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:42547" > attrs:<> locality:<> 
W170804 17:05:28.261514 36299 gossip/gossip.go:1204  [n2] no incoming or outgoing connections
I170804 17:05:28.266415 36175 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:42547
I170804 17:05:28.292221 36299 gossip/gossip.go:295  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:47133" > attrs:<> locality:<> 
W170804 17:05:28.316488 36299 gossip/gossip.go:1204  [n3] no incoming or outgoing connections
I170804 17:05:28.325732 36503 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:42547
I170804 17:05:28.330424 36299 storage/store.go:1275  [n3,s3]: failed initial metrics computation: [n3,s3]: system config not yet available
I170804 17:05:28.330670 36299 gossip/gossip.go:295  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:46901" > attrs:<> locality:<> 
I170804 17:05:28.363722 36705 storage/replica_raftstorage.go:509  [s1,r1/1:/M{in-ax}] generated preemptive snapshot e6ac5f1d at index 16
I170804 17:05:28.368476 36705 storage/store.go:3519  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 6, rate-limit: 2.0 MiB/sec, 4ms
I170804 17:05:28.369684 36666 storage/replica_raftstorage.go:718  [s2,r1/?:{-}] applying preemptive snapshot at index 16 (id=e6ac5f1d, encoded size=6047, 1 rocksdb batches, 6 log entries)
I170804 17:05:28.370950 36666 storage/replica_raftstorage.go:726  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170804 17:05:28.373220 36705 storage/replica_command.go:3600  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I170804 17:05:28.378027 36353 storage/replica.go:3007  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I170804 17:05:28.391777 36727 storage/replica_raftstorage.go:509  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 5a4bd708 at index 18
I170804 17:05:28.428666 36727 storage/store.go:3519  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 8, rate-limit: 2.0 MiB/sec, 33ms
I170804 17:05:28.430061 36709 storage/replica_raftstorage.go:718  [s3,r1/?:{-}] applying preemptive snapshot at index 18 (id=5a4bd708, encoded size=7594, 1 rocksdb batches, 8 log entries)
I170804 17:05:28.431555 36709 storage/replica_raftstorage.go:726  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=1ms commit=0ms]
I170804 17:05:28.433480 36727 storage/replica_command.go:3600  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I170804 17:05:28.436880 36715 storage/raft_transport.go:451  raft transport stream to node 1 established
I170804 17:05:28.443686 36719 storage/replica.go:3007  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W170804 17:05:28.734895 36730 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:28.736729 36712 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:28.736908 36710 storage/raft_transport.go:457  raft transport stream to node 2 failed: store 2 was not found
I170804 17:05:28.739361 36678 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:28.740655 36744 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:28.741341 36680 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:28.743822 36745 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:28.745073 36682 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:28.745940 36770 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:28.746712 36682 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
I170804 17:05:28.785501 36762 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:28.807865 36345 storage/replica_raftstorage.go:509  [raftsnapshot,s1,r1/1:/M{in-ax}] generated Raft snapshot 0c1fdf41 at index 25
I170804 17:05:28.814842 36345 storage/store.go:3519  [raftsnapshot,s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):2: kv pairs: 44, log entries: 3, rate-limit: 8.0 MiB/sec, 6ms
I170804 17:05:28.815325 36823 storage/replica_raftstorage.go:718  [s2,r1/2:/M{in-ax}] applying Raft snapshot at index 25 (id=0c1fdf41, encoded size=5682, 1 rocksdb batches, 3 log entries)
I170804 17:05:28.817086 36874 util/stop/stopper.go:471  quiescing; tasks left:
1      storage.raftsnapshot: processing replica
I170804 17:05:28.817514 36823 storage/replica_raftstorage.go:726  [s2,r1/2:/M{in-ax}] applied Raft snapshot in 2ms [clear=0ms batch=0ms entries=0ms commit=0ms]
W170804 17:05:28.825525 36670 storage/raft_transport.go:278  unable to accept Raft message from (n2,s2):2: no handler registered for (n1,s1):1
W170804 17:05:28.829167 36717 storage/store.go:3272  [s2] raft error: node 1 claims to not contain store 1 for replica (n1,s1):1: store 1 was not found
W170804 17:05:28.829389 36715 storage/raft_transport.go:457  raft transport stream to node 1 failed: store 1 was not found
W170804 17:05:28.830137 36670 storage/raft_transport.go:278  unable to accept Raft message from (n3,s3):?: no handler registered for (n1,s1):?
I170804 17:05:28.847694 36303 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:47133->127.0.0.1:55766: use of closed network connection
W170804 17:05:28.848475 36732 storage/raft_transport.go:457  raft transport stream to node 3 failed: EOF
I170804 17:05:28.848615 36311 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:42547->127.0.0.1:43233: use of closed network connection
I170804 17:05:28.848972 36502 vendor/google.golang.org/grpc/grpclog/grpclog.go:75  transport: http2Server.HandleStreams failed to read frame: read tcp 127.0.0.1:46901->127.0.0.1:48531: use of closed network connection
--- FAIL: testrace/TestSnapshotAfterTruncation (0.000s)
Test ended in panic.

--- FAIL: testrace/TestSnapshotAfterTruncation/differentTerm (0.000s)
Test ended in panic.

------- Stdout: -------
I170804 17:05:28.915335 36764 storage/store.go:1275  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available
I170804 17:05:28.915633 36764 gossip/gossip.go:295  [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:45635" > attrs:<> locality:<> 
W170804 17:05:28.950370 36764 gossip/gossip.go:1204  [n2] no incoming or outgoing connections
I170804 17:05:28.961060 37001 gossip/client.go:129  [n2] started gossip client to 127.0.0.1:45635
I170804 17:05:28.970510 36764 gossip/gossip.go:295  [n2] NodeDescriptor set to node_id:2 address:<network_field:"tcp" address_field:"127.0.0.1:39120" > attrs:<> locality:<> 
W170804 17:05:29.003082 36764 gossip/gossip.go:1204  [n3] no incoming or outgoing connections
I170804 17:05:29.019576 36748 gossip/client.go:129  [n3] started gossip client to 127.0.0.1:45635
I170804 17:05:29.042447 36764 gossip/gossip.go:295  [n3] NodeDescriptor set to node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:58297" > attrs:<> locality:<> 
I170804 17:05:29.070213 37148 storage/replica_raftstorage.go:509  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 5b60b7bd at index 16
I170804 17:05:29.075399 37224 storage/replica_raftstorage.go:718  [s2,r1/?:{-}] applying preemptive snapshot at index 16 (id=5b60b7bd, encoded size=6047, 1 rocksdb batches, 6 log entries)
I170804 17:05:29.076784 37224 storage/replica_raftstorage.go:726  [s2,r1/?:/M{in-ax}] applied preemptive snapshot in 1ms [clear=0ms batch=0ms entries=0ms commit=0ms]
I170804 17:05:29.077466 37148 storage/store.go:3519  [s1,r1/1:/M{in-ax}] streamed snapshot to (n2,s2):?: kv pairs: 34, log entries: 6, rate-limit: 2.0 MiB/sec, 6ms
I170804 17:05:29.078861 37148 storage/replica_command.go:3600  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n2,s2):2): read existing descriptor r1:/M{in-ax} [(n1,s1):1, next=2]
I170804 17:05:29.083146 37266 storage/replica.go:3007  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n2,s2):2): updated=[(n1,s1):1 (n2,s2):2] next=3
I170804 17:05:29.091986 37272 storage/replica_raftstorage.go:509  [s1,r1/1:/M{in-ax}] generated preemptive snapshot 71a94e95 at index 18
I170804 17:05:29.094881 37272 storage/store.go:3519  [s1,r1/1:/M{in-ax}] streamed snapshot to (n3,s3):?: kv pairs: 37, log entries: 8, rate-limit: 2.0 MiB/sec, 2ms
I170804 17:05:29.096493 37275 storage/replica_raftstorage.go:718  [s3,r1/?:{-}] applying preemptive snapshot at index 18 (id=71a94e95, encoded size=7594, 1 rocksdb batches, 8 log entries)
I170804 17:05:29.098179 37275 storage/replica_raftstorage.go:726  [s3,r1/?:/M{in-ax}] applied preemptive snapshot in 2ms [clear=0ms batch=0ms entries=1ms commit=0ms]
I170804 17:05:29.100241 37272 storage/replica_command.go:3600  [s1,r1/1:/M{in-ax}] change replicas (ADD_REPLICA (n3,s3):3): read existing descriptor r1:/M{in-ax} [(n1,s1):1, (n2,s2):2, next=3]
I170804 17:05:29.105665 37232 storage/raft_transport.go:451  raft transport stream to node 1 established
I170804 17:05:29.113491 37302 storage/replica.go:3007  [s1,r1/1:/M{in-ax}] proposing ADD_REPLICA((n3,s3):3): updated=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4
W170804 17:05:29.416814 37231 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:29.417652 37229 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:29.417816 37227 storage/raft_transport.go:457  raft transport stream to node 2 failed: store 2 was not found
I170804 17:05:29.419422 37315 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:29.423676 37331 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:29.425610 37331 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:29.425903 37317 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:29.428090 37244 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:29.429316 37280 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:29.430150 37304 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:29.876762 37443 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:29.877574 37241 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:29.878430 37254 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:29.878618 37252 storage/raft_transport.go:457  raft transport stream to node 3 failed: store 3 was not found
W170804 17:05:29.882102 37428 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:29.883040 37306 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:30.275763 37444 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:30.276956 37445 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:30.280684 37430 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:30.285551 37308 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:30.287864 37447 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:30.289551 37310 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:30.677531 37246 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:30.679277 37449 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
I170804 17:05:30.679912 37245 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:30.680104 37423 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:30.681082 37421 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:30.681909 37248 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:31.176400 37424 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:31.177488 37319 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
I170804 17:05:31.177890 37425 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:31.178984 37451 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:31.179745 37459 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:31.180494 37474 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:31.475780 37256 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:31.476638 37255 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:31.477946 37432 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:31.479005 37461 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:31.479902 37258 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:31.481134 37463 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:31.776425 37433 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:31.778122 37321 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:31.778808 37320 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:31.779530 37434 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:31.780722 37438 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:31.781714 37436 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:32.075727 37323 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:32.076707 37324 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:32.079228 37260 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:32.080099 37470 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:32.080865 37468 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:32.082063 37326 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:32.477007 37327 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:32.478170 37491 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:32.478907 37441 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:32.482379 37328 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:32.483461 37506 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:32.484151 37472 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:32.975864 37508 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:32.977057 37495 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:32.977928 37493 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:32.979159 37507 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:32.980087 37522 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:32.980899 37510 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:33.376031 37511 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:33.376768 37512 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:33.378136 37528 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:33.378961 37526 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:33.386135 37524 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:33.386965 37514 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:33.875900 37515 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:33.876945 37516 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:33.878080 37538 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:33.879109 37540 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:33.879795 37518 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:33.880450 37520 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:34.375721 37542 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:34.376207 37541 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:34.382635 37532 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:34.383804 37262 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:34.390291 37530 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:34.393313 37544 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:34.676115 37546 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:34.676908 37545 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:34.677998 37265 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:34.678843 37534 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:34.680345 37264 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:34.681105 37548 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:35.176051 37312 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:35.176175 37313 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:35.177313 37481 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:35.177998 37479 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:35.178941 37554 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:35.179836 37536 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:35.575576 37549 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:35.576494 37550 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:35.577630 37572 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:35.578686 37574 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:35.579378 37552 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:35.580015 37570 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:36.076107 37586 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:36.076504 37587 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:36.077969 37578 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:36.078859 37483 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:36.079894 37576 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:36.080592 37589 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:36.475772 37590 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:36.476729 37591 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:36.477971 37597 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:36.478901 37599 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:36.479633 37593 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:36.479967 37595 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:36.976363 37484 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:36.978065 37485 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:36.979688 37603 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:36.980420 37556 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:36.981230 37601 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:36.982035 37487 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:37.276265 37605 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:37.277094 37094 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
I170804 17:05:37.277238 37604 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:37.278354 37607 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:37.279180 37558 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:37.281633 37096 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:37.576693 37097 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:37.576866 37098 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:37.577753 37609 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:37.577832 37101 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:37.578514 37560 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:37.578635 37102 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:37.876732 37103 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:37.877724 37104 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:37.880120 37615 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:37.880891 37613 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:37.882070 37611 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:37.882750 37618 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:38.275541 37634 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:38.276336 37489 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:38.277560 37640 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:38.278459 37642 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:38.287779 37636 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:38.288672 37638 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:38.576087 37643 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:38.576790 37644 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:38.577070 37562 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:38.577947 37646 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:38.578128 37566 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:38.579037 37564 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:38.975719 37647 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:38.975816 37648 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:38.977220 37666 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:38.978204 37652 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:38.979227 37650 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:38.979927 37568 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:39.277327 37683 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:39.278171 37682 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:39.278645 37687 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:39.279215 37656 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:39.279358 37685 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:39.280054 37654 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:39.776252 37667 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:39.777125 37668 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:39.778353 37497 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:39.779362 37499 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:39.780129 37670 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:39.780749 37672 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:40.075777 37500 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:40.076715 37501 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:40.077883 37699 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:40.079808 37701 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:40.080592 37503 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:40.081241 37505 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:40.475625 37703 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:40.476458 37702 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:40.477700 37709 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:40.478742 37711 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:40.479427 37705 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:40.480051 37707 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:40.975621 37579 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:40.976554 37580 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:40.977911 37714 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:40.978994 37716 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:40.979684 37582 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:40.980327 37584 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:41.275606 37660 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:41.276442 37659 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:41.277777 37731 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:41.278639 37732 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:41.279344 37662 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:41.280099 37664 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:41.778813 37718 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:41.779849 37717 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:41.781444 37724 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:41.782395 37726 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:41.783113 37720 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:41.783815 37722 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:42.175715 37689 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:42.176758 37690 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:42.178249 37696 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:42.181333 37746 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:42.189403 37692 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:42.191102 37694 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:42.575857 37728 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:42.576309 37727 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:42.577244 37748 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:42.577415 37750 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:42.577933 37674 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:42.578088 37762 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:42.878591 37675 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:42.879566 37676 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:42.880702 37778 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:42.881603 37780 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:42.882274 37678 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:42.882930 37680 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:43.375677 37764 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:43.376489 37763 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:43.377197 37752 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:43.377924 37766 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:43.379996 37754 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:43.380736 37768 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:43.776285 37755 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:43.776393 37756 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:43.778411 37760 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:43.778620 37734 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:43.779929 37782 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:43.781043 37758 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:44.176543 37794 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:44.177507 37770 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
I170804 17:05:44.177588 37795 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:44.178208 37797 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:44.178711 37736 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:44.179354 37772 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:44.675664 37617 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:44.676523 37810 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:44.677644 37816 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:44.678499 37818 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:44.679163 37812 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:44.679761 37814 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:45.175545 37783 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:45.176474 37784 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:45.177534 37790 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:45.178535 37792 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:45.179516 37786 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:45.179596 37788 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:45.475615 37819 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:45.476278 37820 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:45.476415 37826 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:45.477154 37822 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:45.477395 37824 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:45.478057 37828 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:45.775918 37774 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:45.776992 37773 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:45.778135 37844 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:45.779087 37846 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:45.779796 37776 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:45.780475 37842 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:46.075547 37847 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:46.076318 37848 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:46.077595 37854 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:46.078587 37856 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:46.079255 37850 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:46.080334 37852 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:46.375547 37798 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:46.376370 37799 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:46.377636 37830 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:46.378581 37832 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:46.379364 37801 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:46.379433 37803 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:46.675431 37857 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:46.675536 37858 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:46.676528 37875 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:46.677261 37874 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:46.678234 37805 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:46.688474 37860 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:46.976081 37863 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:46.976565 37862 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:46.977236 37890 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:46.978215 37891 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:46.979502 37834 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:46.980261 37866 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:47.276099 37894 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:47.277521 37808 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
I170804 17:05:47.278475 37893 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:47.279508 37908 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:47.280349 37906 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:47.281190 37896 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:47.576132 37898 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:47.577357 37897 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:47.577582 37739 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:47.578274 37868 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:47.578436 37910 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:47.578998 37900 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:48.075963 37902 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:48.076709 37901 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:48.076939 37741 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:48.077703 37922 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:48.077823 37904 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:48.078571 37870 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:48.376533 37876 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:48.376793 37877 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:48.377947 37881 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:48.378300 37912 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:48.379639 37872 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:48.380425 37879 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:48.875778 37835 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:48.876266 37836 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:48.877901 37916 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:48.878402 37914 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:48.879041 37883 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:48.879181 37838 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:49.175670 37924 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:49.176175 37925 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:49.177171 37938 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:49.177321 37929 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:49.178049 37743 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:49.178246 37939 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:49.675776 37744 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:49.676609 37745 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:49.677774 37841 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:49.678674 37971 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:49.679344 37955 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:49.679400 37957 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:49.988661 37941 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:49.990711 37940 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:49.991846 37959 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:49.992114 37918 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:49.995781 37944 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:49.996366 37973 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:50.375969 37947 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:50.376679 37946 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:50.377219 37932 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:50.377524 37951 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:50.378111 37949 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:50.378409 37934 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:50.876826 37974 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:50.877104 37975 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:50.878142 37920 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:50.878369 37936 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:50.880149 37977 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
W170804 17:05:50.880223 37961 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
I170804 17:05:51.277211 37952 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:51.277384 37953 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:51.278499 38002 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:51.278874 37989 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:51.279975 37979 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:51.280422 37987 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:51.775579 38003 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:51.776935 38004 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:51.778376 38010 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:51.779361 38012 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:51.780029 38006 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:51.780641 38008 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:51.862942 36901 gossip/gossip.go:1218  [n1] node has connected to cluster via gossip
I170804 17:05:52.180700 37980 storage/raft_transport.go:451  raft transport stream to node 2 established
I170804 17:05:52.180847 37981 storage/raft_transport.go:451  raft transport stream to node 3 established
W170804 17:05:52.182362 38014 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:52.182445 37985 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3
W170804 17:05:52.183613 37963 storage/store.go:3272  [s1] raft error: node 2 claims to not contain store 2 for replica (n2,s2):2: store 2 was not found
W170804 17:05:52.183775 37983 storage/store.go:3272  [s1] raft error: node 3 claims to not contain store 3 for replica (n3,s3):3: store 3 was not found
I170804 17:05:52.478645 38018 storage/raft_transport.go:451  raft transport stream to node 3 established
I170804 17:05:52.479633 37937 storage/raft_transport.go:451  raft transport stream to node 2 established
W170804 17:05:52.480708 38024 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n2,s2):2
W170804 17:05:52.481582 38026 storage/raft_transport.go:278  unable to accept Raft message from (n1,s1):1: no handler registered for (n3,s3):3\n```
Please assign, take a look and update the issue accordingly.
@cockroach-teamcity cockroach-teamcity added O-robot Originated from a bot. C-test-failure Broken test (automatically or manually discovered). labels Aug 4, 2017
@tamird
Copy link
Contributor

tamird commented Aug 4, 2017 •
edited
Loading

Earlier incarnations of this failure were caused by 5837fbe - could this be caused by c2dc44b?

@jordanlewis jordanlewis mentioned this issue Aug 7, 2017
Closed
This was referenced Aug 8, 2017
Closed
Closed
Closed
@a-robinson
Copy link
Contributor

Well, it's a little stupid, but git bisect actually points all the way back to 9a68241. And sure enough, reverting that one-character change on top of head fixes the flakes (although of course we don't actually want to revert that for real clusters).

I think the test and/or the multiTestContext are making invalid assumptions around bringing down and restarting nodes, which is made significantly more complicated by all the hackiness involved in the multiTestContext (sharing a transport, using a manual clock, etc). I'm fairly confident it's not an issue with non-test code.

@a-robinson
Copy link
Contributor

To make my previous comment more reproducible, all tests I ran were run on a 24 vCPU gceworker with the command:

./build/builder.sh make stressrace PKG=./pkg/storage TESTS=TestSnapshotAfterTruncation STRESSFLAGS='-stderr -maxfails 1 -p 32'

This was referenced Aug 10, 2017
Closed
Closed
@a-robinson
Copy link
Contributor

Hm, I retried that change because I was considering putting it in as a temporary fix to stop the flakiness and found that it only actually fixes things if I also revert e100e6d and c2dc44b (but notably reverting those two without changing storeSchedulerConcurrency is still flaky).

My testing the other day about patching just the storeSchedulerConcurrency change on top of head must have been flawed. I'm confused, but this isn't worth my time right now, so I think we're probably best off just reverting those commits to be safe for 1.1. Are you cool with that, @tamird?

In short,

@tamird
Copy link
Contributor

tamird commented Aug 10, 2017

I mean, I guess? Seems to me that we've had cases in the past where someone has asserted that things are likely mere test issues which were later discovered to be not test-only, so I'm a bit uncomfortable with that assertion.

Stopping the bleeding seems good, but we should certainly prioritize getting to the bottom of this. Perhaps @irfansharif's newly acquired familiarity with this area can be of use here.

a-robinson added a commit to a-robinson/cockroach that referenced this issue Aug 10, 2017
@a-robinson
storage: Temporary de-flake for TestSnapshotAfterTruncation
405b855 
This isn't a great fix, but it'll stop the flakes we've been seeing
(cockroachdb#17448). We should follow up and dig deeper when we're not in a rush to
polish off a release.
@a-robinson
Copy link
Contributor

I agree our experience has shown that test flakiness is often due to a real issue. I just have other real issues that are definitely real that I need to look into first.

Also, it seems that c2dc44b was definitely associated with a spike in flakiness, so getting rid of it seems good until we're able to look into it. If @irfansharif has cycles to look into this stuff sooner, I'm fine with not reverting.

@petermattis
Copy link
Collaborator

I thought c2dc44b wasn't supposed to change any of the lease renewal configuration, just simplify the calculations. Did we miss something?

@a-robinson
Copy link
Contributor

a-robinson commented Aug 10, 2017 •
edited
Loading

It changed any tests that relied on StoreConfig.SetDefaults() to fill in StoreConfig.RangeLeaseActiveDuration and StoreConfig.RangeLeaseRenewalDuration. Our non-test code fills those in manually, but most of our tests don't. For them, the change of rangeLeaseRenewalFraction from 0.8 to 0.5 was relevant. If anyone wants to pull on this thread, that would probably be the place to start.

@irfansharif
Copy link
Contributor

irfansharif commented Aug 10, 2017 •
edited
Loading

Would still prefer reverting c2dc44b in the interim, it's not likely (not for me at least) that a fix for this will make into 1.1 anyway. Assigning myself here but not my top priority at the moment.

+cc #16164.

@irfansharif irfansharif self-assigned this Aug 10, 2017
This was referenced Aug 15, 2017
Closed
Closed
@nvanbenschoten
Copy link
Member

I've been trying to re-enable this test so that I can use similar infrastructure to test a theory for #17524. It seems to be nice and easy to reproduce the flake if TestSnapshotAfterTruncation/differentTerm. Just add a 1-second sleep between restarting live stores and restarting the stopped store (for instance, on line 722):

cockroach/pkg/storage/client_raft_test.go

Lines 715 to 725 in 8ef84ea

mtc.restartStore(i)
// Disable the snapshot queue on the live stores so that
// stoppedStore won't get a snapshot as soon as it starts
// back up.
mtc.stores[i].SetRaftSnapshotQueueActive(false)
}
}
// Restart the stopped store and wait for raft
// election/heartbeat traffic to settle down. Specifically, we
// need stoppedStore to know about the new term number before

I'm a little confused how this ever was supposed to work, so I'm probably misunderstanding something. multiTestContext.restartStore blocks until a store is able to perform a liveness heartbeat. This means that when we restart the stopped store, we wait for this liveness heartbeat to succeed (which is where we're getting stuck). But if we wait until the other two stores change terms and then refuse to snapshot the stopped node at the lower term, will it ever be able to perform a liveness heartbeat?

My theory about the flake is that most of the time we weren't waiting for the next leader election to complete after restarting the other two stores before restarting the third. When we add the sleep, we allow this leader election to finish with much higher probability, so it's much more likely the third store will get stuck performing a liveness update. DISCLAIMER: I'm not very confident about this theory because I still think I have something confused.

Also, if we're not going to fix this immediately, I'll at least re-enable TestSnapshotAfterTruncation/sameTerm.

@tbg
Copy link
Member

tbg commented Sep 7, 2017

I think that makes sense. Most of the time the blocked node comes up fast enough to still be considered live. If it isn't though, it won't be able to heartbeat.

I'd add restartStoreWithoutHeartbeat and use that -- you don't need it to be live in this test since you have your own raft-checking SucceedsSoon after you restart.

@nvanbenschoten
Copy link
Member

👍 thanks for the confirmation. The changes you suggested sound good.

nvanbenschoten added a commit to nvanbenschoten/cockroach that referenced this issue Sep 7, 2017
@nvanbenschoten
storage: deflake and re-enable TestSnapshotAfterTruncation
15170da 
Fixes cockroachdb#17448.

This change adds a `restartStoreWithoutHeartbeat` method to `multiTestContext`,
which is used in cases like `TestSnapshotAfterTruncation/differentTerm` where
a successful heartbeat on store restart is not required.
@nvanbenschoten nvanbenschoten mentioned this issue Sep 7, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spencerkimball spencerkimball

@a-robinson a-robinson

@irfansharif irfansharif

Labels
C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@tamird @tbg @nvanbenschoten @spencerkimball @petermattis @a-robinson @irfansharif @cockroach-teamcity