Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not update leader last-contact when granting a pre-vote request #609

Merged
merged 2 commits into from
Aug 28, 2024

Conversation

dhiaayachi
Copy link
Contributor

Description

This fix a theoretical bug with pre-vote. The issue could happen when a former leader continuously send pre-vote requests to former followers before they get the chance to transition to Candidates without necessarily winning the pre-vote election (because they haven't got enough pre-vote granted for example). This could happen in a scenario when a node is misconfigured and have a low election timeout with a node configured differently from the others.

In that scenario, the pre-vote request will set the leader last contact in the node in follower state which prevent it from going to Candidate state and could render the cluster unstable for undetermined time.

Testing

I added an integration test that recreate the condition artificially. The test execute the following steps:

  • Create a 3 nodes cluster with a deterministic leader
  • Start a routine that continuously send pre-vote requests to all the followers
  • Deactivate heartbeat from the leader to the followers
  • Check that a new leader is elected out of the followers

I executed the test with and without the fix and it consistently pass/fail based on it.

@dhiaayachi dhiaayachi requested review from a team as code owners August 22, 2024 20:24
@dhiaayachi dhiaayachi requested review from skpratt and banks and removed request for a team August 22, 2024 20:24
Copy link
Member

@banks banks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job Dhia! Thanks!

@dhiaayachi dhiaayachi merged commit 42d3446 into main Aug 28, 2024
13 checks passed
@dhiaayachi dhiaayachi deleted the dhiaayachi/prevote-no-lastcontact-update branch August 28, 2024 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants