Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

functional-tester: fail with "found data inconsistency with peers" #9022

Closed
gyuho opened this issue Dec 15, 2017 · 0 comments
Closed

functional-tester: fail with "found data inconsistency with peers" #9022

gyuho opened this issue Dec 15, 2017 · 0 comments
Assignees
@gyuho
Labels
area/functional-testing type/bug

Comments

@gyuho
Copy link
Contributor

gyuho commented Dec 15, 2017 •
edited
Loading

tester

Dec 15 19:07:33 ft-normal-tester etcd-tester[19621]: [round#9 case#1] recovering failure "kill majority of the cluster"
Dec 15 19:07:48 ft-normal-tester etcd-tester[19621]: INFO: 2017/12/15 19:07:48 ccBalancerWrapper: updating state and picker called by balancer: TRANSIENT_FAILURE, 0xc4207612c0
Dec 15 19:07:48 ft-normal-tester etcd-tester[19621]: WARNING: 2017/12/15 19:07:48 Failed to dial 10.138.0.2:2379: grpc: the connection is closing; please retry.
Dec 15 19:07:48 ft-normal-tester etcd-tester[19621]: WARNING: 2017/12/15 19:07:48 Failed to dial 10.138.0.2:2379: grpc: the connection is closing; please retry.
Dec 15 19:07:48 ft-normal-tester etcd-tester[19621]: #2 setHealthKey error (rpc error: code = Unavailable desc = transport is closing (http://10.138.0.2:2379))

agent3

2017-12-15 19:07:48.332071 W | etcdserver: 6316ae9b81b47490 hash-kv error "context deadline exceeded" on peer "http://10.138.0.3:2380" with revision 75148
2017-12-15 19:07:48.348264 E | etcdserver: 6316ae9b81b47490's hash 3707811689 != c3c2727e16a88e06's hash 1324347234 (revision 75148, peer revision 75286, compact revision 60219)
2017-12-15 19:07:48.348303 E | etcdserver: 6316ae9b81b47490's hash 3707811689 != e5bf145d1e8e2597's hash 1324347234 (revision 75148, peer revision 75286, compact revision 60219)
2017-12-15 19:07:48.348653 C | etcdmain: 6316ae9b81b47490 found data inconsistency with peers

Since #8922
@gyuho gyuho self-assigned this Dec 15, 2017
gyuho added a commit to gyuho/etcd that referenced this issue Dec 18, 2017
@gyuho
mvcc: fetch revisions with current revision, not 0
31910b9 
It was getting revisions with "atRev==0", which makes
"available" from "keep" method always empty since
"walk" on "keyIndex" only returns true.

"available" should be populated with all revisions to be
kept if the compaction happens with the given revision.
But, "available" was being empty when "kvindex.Keep(0)"
since it's always the case that "rev.main > atRev==0".

Fix etcd-io#9022.

Signed-off-by: Gyuho Lee <[email protected]>
@gyuho gyuho mentioned this issue Dec 18, 2017
Merged
gyuho added a commit to gyuho/etcd that referenced this issue Dec 18, 2017
@gyuho
mvcc: fetch revisions with current revision, not 0, in HashByRev
df8533b 
It was getting revisions with "atRev==0", which makes
"available" from "keep" method always empty since
"walk" on "keyIndex" only returns true.

"available" should be populated with all revisions to be
kept if the compaction happens with the given revision.
But, "available" was being empty when "kvindex.Keep(0)"
since it's always the case that "rev.main > atRev==0".

Fix etcd-io#9022.

Signed-off-by: Gyuho Lee <[email protected]>
gyuho added a commit to gyuho/etcd that referenced this issue Dec 18, 2017
@gyuho
mvcc: fetch revisions with current revision, not 0, in HashByRev
2e95ace 
It was getting revisions with "atRev==0", which makes
"available" from "keep" method always empty since
"walk" on "keyIndex" only returns true.

"available" should be populated with all revisions to be
kept if the compaction happens with the given revision.
But, "available" was being empty when "kvindex.Keep(0)"
since it's always the case that "rev.main > atRev==0".

Fix etcd-io#9022.

Signed-off-by: Gyuho Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gyuho gyuho

Labels
area/functional-testing type/bug
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xiang90 @gyuho