Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix apply outdate membership change #4107

Merged
merged 3 commits into from
Apr 6, 2022
Merged

Conversation

critical27
Copy link
Contributor

@critical27 critical27 commented Apr 1, 2022

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

Introduced in #3846, my bad. The key point is that, whenever we start-up a partition, we use the peers in meta, which is a snapshot of state machine by applying all previous membership changes. So after we start-up, we should not apply any membership change before we started.

Here is a example:

  1. We have done lots of membership change, so there are plenty logs like (removePeer, addLearner, addPeer, etc)
  2. We start a new part, with 5 peers (from meta) besides it self
  3. If we apply the membership change which happens before we start-up (those in step 1), our peers maybe total wrong. For example, after applying them, we have only 2 peers...
  4. At this point, anything weird could happen...

However, this is not a silver bullet, the membership change need to be re-considered totally after this version.

BTW:
Fix crash when trying to transfer leader to a empty host ("":0)

How do you solve it?

Do not apply any membership change before a part started.

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.)
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:
Introduced in the same version, not related

@critical27 critical27 added the ready-for-testing PR: ready for the CI test label Apr 1, 2022
@Sophie-Xie Sophie-Xie added this to the v3.1.0 milestone Apr 1, 2022
@Sophie-Xie Sophie-Xie added the cherry-pick-v3.1 PR: need cherry-pick to this version label Apr 1, 2022
liwenhui-soul
liwenhui-soul previously approved these changes Apr 1, 2022
Copy link
Contributor

@pengweisong pengweisong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Sophie-Xie Sophie-Xie merged commit 6b20961 into vesoft-inc:master Apr 6, 2022
Sophie-Xie added a commit that referenced this pull request Apr 7, 2022
* fix apply outdate membership change

* fix transfer leader to '':0 cause crash

Co-authored-by: Sophie <[email protected]>
yixinglu pushed a commit that referenced this pull request Apr 7, 2022
* fix apply outdate membership change (#4107)

* fix apply outdate membership change

* fix transfer leader to '':0 cause crash

Co-authored-by: Sophie <[email protected]>

* add code for part peers backward compatible (#4101)

* add code for part peers backward compatible

* add balance keys

* fix bug: pass tests

* add gflag && ignore

* change int to int64_t

* chage to emplace

Co-authored-by: panda-sheep <[email protected]>
Co-authored-by: Sophie <[email protected]>

* fixed (#4116)

Co-authored-by: Sophie <[email protected]>

* move KW_CLEAR to unresolved keyword (#4118)

* Limits for add and recover balance data or zone balance jobs (#4104)

* If there are failed or stopped data balance or zone balance job, must firstly recover it

* address wenhaocs's comments

* address comments

* add more comments

* recover job

* add more ut

* adjust comment format

* adjust comment format

* adjust comment format

Co-authored-by: Doodle <[email protected]>
Co-authored-by: pengwei.song <[email protected]>
Co-authored-by: panda-sheep <[email protected]>
Co-authored-by: haifei.zhao <[email protected]>
@critical27 critical27 deleted the fix branch April 12, 2022 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick-v3.1 PR: need cherry-pick to this version ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants