-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow joining node to trigger term bump #53338
Allow joining node to trigger term bump #53338
Conversation
In rare circumstances it is possible for an isolated node to have a greater term than the currently-elected leader. Today such a node will attempt to join the cluster but will not offer a vote to the leader and will reject its cluster state publications due to their stale term. This situation persists since there is no mechanism for the joining node to inform the leader that its term is stale and a new election is required. This commit adds the current term of the joining node to the join request. Once the join has been validated, the leader will perform another election to increase its term far enough to allow the isolated node to join properly. Fixes elastic#53271
Pinging @elastic/es-distributed (:Distributed/Cluster Coordination) |
Note to reviewers: I contemplated a few alternative mechanisms for telling the master that it needs to bump its term in these circumstances. I chose this one because it's a small and reasonably clear change, but it does have the disadvantage that the join attempt that triggers the term bump will fail and require a retry. IMO this is rare enough that that's ok. I also considered
Liveness is hard. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
In rare circumstances it is possible for an isolated node to have a greater term than the currently-elected leader. Today such a node will attempt to join the cluster but will not offer a vote to the leader and will reject its cluster state publications due to their stale term. This situation persists since there is no mechanism for the joining node to inform the leader that its term is stale and a new election is required. This commit adds the current term of the joining node to the join request. Once the join has been validated, the leader will perform another election to increase its term far enough to allow the isolated node to join properly. Fixes #53271
In rare circumstances it is possible for an isolated node to have a greater
term than the currently-elected leader. Today such a node will attempt to join
the cluster but will not offer a vote to the leader and will reject its cluster
state publications due to their stale term. This situation persists since there
is no mechanism for the joining node to inform the leader that its term is
stale and a new election is required.
This commit adds the current term of the joining node to the join request. Once
the join has been validated, the leader will perform another election to
increase its term far enough to allow the isolated node to join properly.
Fixes #53271