recover lessor before recovering mvcc store and transactionally revoke leases

[xiang90]

1. always recover lessor first. When we recover the mvcc.KV it will reattach keys to its leases. If we recover mvcc.KV first, it will attach the keys to the wrong lessor before it recovers.
2. any touch to mvcc.KV will increase the consistent index. Increasing consistent index will avoid the applier to re-apply the same entry. For the same entry, we must ensure all kv ops in the entry executed in a txn.

[heyitsanthony]

lgtm following CI fixups

[xiang90]

@heyitsanthony There is one additional fix I am working on.

[xiang90]

@heyitsanthony PTAL. We def need to add tests around leases for recovery, snapshot path.

[heyitsanthony]

if le.rd == nil { return nil } to pull back the indent for the interesting path?

[xiang90]

ok.

[heyitsanthony]

lgtm

maybe worth failpointing all the Recover()s for general testing

[xiang90]

@heyitsanthony We should do it for sure. This is too easy to get wrong. The original design was simple that only lessor knows about KV. After we decided to put leaseID into the key, lessor and KV are coupled together, causing a few issues.

[xiang90]

@heyitsanthony

Test failed with

--- FAIL: TestKVPutFailGetRetry (18.43s)
    kv_test.go:622: timed out waiting for get
    kv_test.go:609: grpc: the client connection is closing

Do you think it is related to this change?

[heyitsanthony]

@xiang90 that test resets an etcdserver; I wouldn't be surprised if it's affected by recovery path changes

[xiang90]

@heyitsanthony I will take a closer look.

[xiang90]

@heyitsanthony I checked the log. etcd restarting is fine. gRPC call blocks for more than 5 seconds.

comparing to a successful case, there is one additional reconnecting happened which should not after the node restarted.

2016-08-05 03:51:32.583520 I | v3rpc/grpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial unix localhost:node670786626.sock.bridge: connect: no such file or directory"; Reconnecting to {"localhost:node670786626.sock.bridge" <nil>}
2016-08-05 03:51:48.157978 I | v3rpc/grpc: grpc: addrConn.transportMonitor exits due to: grpc: the connection is closing
2016-08-05 03:51:48.158019 I | integration: terminating node670786626 (unix://localhost:node670786626.sock.bridge)

[heyitsanthony]

@xiang90 OK, I suspect this is related to the grpc backoff strategy / reconnect logic. Given the failing test took about ~13s before reaching the select timeout, I can see the retries getting out of hand.

[xiang90]

@heyitsanthony I reran the tests for a few times. No failure occured. So shall we assume it is unrelated and get this merged? I will open a new issue for the failed test.

[heyitsanthony]

@xiang90 yeah, it's fine. The reconnection code has diverged in master since the last grpc vendoring anyway.