Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wait leaders transfer back after a tikv upgrade #4863

Merged
merged 12 commits into from
Feb 12, 2023

Conversation

KanShiori
Copy link
Collaborator

@KanShiori KanShiori commented Feb 8, 2023

What problem does this PR solve?

  • record leader count in the status.tikv.store[].leaderCountBeforeUpgrade before a tikv eviction begin.
  • record the time when a tikv leader eviction end in the annotation tidb.pingcap.com/tikv-evict-leader-end-at of the pod.
  • Before a tikv leader eviction begins, wait for leaders of the previous tikv to transfer back.
    The following cases indicate that the transfer is done:
    1. miss information to decide the result, like: leaderCountBeforeUpgrade or tidb.pingcap.com/tikv-evict-leader-end-at.
    2. leaderCountBeforeUpgrade is less than 200.
    3. timeout, default to 400s.
    4. current leader count is greater than leaderCountBeforeUpgrade * 2/3

What is changed and how does it work?

Code changes

  • Has Go code change
  • Has CI related scripts change

Tests

  • Unit test
  • E2E test
  • Manual test
  1. Deploy a TiDB cluster.
  2. Upgrade the TiKV.
  3. Check that wait for leaders to transfer back before upgrading the next tikv.
  • No code

Side effects

  • Breaking backward compatibility
  • Other side effects:

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.


@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 8, 2023

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • csuzhangxc
  • july2993

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@codecov-commenter
Copy link

codecov-commenter commented Feb 8, 2023

Codecov Report

Merging #4863 (3ea0114) into master (933f7ed) will increase coverage by 5.55%.
The diff coverage is 80.61%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4863      +/-   ##
==========================================
+ Coverage   59.39%   64.94%   +5.55%     
==========================================
  Files         226      230       +4     
  Lines       25628    28776    +3148     
==========================================
+ Hits        15221    18688    +3467     
+ Misses       8959     8569     -390     
- Partials     1448     1519      +71     
Flag Coverage Δ
e2e 42.85% <53.06%> (?)
unittest 59.43% <77.41%> (+0.04%) ⬆️

Comment on lines 310 to 314
leaderCountBefore := int(*store.LeaderCountBeforeUpgrade)
if leaderCountBefore < 200 {
klog.Infof("%s: leader count is %d and less than 200, so skip waiting leaders for transfer back", logPrefix, leaderCountBefore)
return true
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be moved to the head of isLeaderTransferBackOrTimeout.

The parse of evictLeaderEndTime is before this check but the usage of evictLeaderEndTime is after this check.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 3ea0114

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind pull-e2e-kind-basic pull-e2e-kind-serial

@KanShiori
Copy link
Collaborator Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 3ea0114

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind-serial

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind pull-e2e-kind-serial

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind

3 similar comments
@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind

@KanShiori
Copy link
Collaborator Author

/test pull-e2e-kind

@ti-chi-bot ti-chi-bot merged commit 0ff338f into pingcap:master Feb 12, 2023
@KanShiori KanShiori deleted the shiori/wait-leaders-transfer-back branch February 12, 2023 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants