Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix duplicated ID DTFile that cause inconsistent query result (#2770) #2775

Merged
merged 10 commits into from
Sep 1, 2021
Merged

Fix duplicated ID DTFile that cause inconsistent query result (#2770) #2775

merged 10 commits into from
Sep 1, 2021

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #2770

What problem does this PR solve?

Issue Number: close #2756

Problem Summary: When deployed on multi-disks, we may happen to generate duplicated ID DTFile among different disks.

  1. A DTFile created by delta merge or apply snapshot, it acquires an ID (100 for example) and generate a dmf_100 on /data2
  2. TiFlash crashes / stops before persistenting dmf_100 to storage_pool.data
  3. TiFlash restart and storage_pool.data restored with max_id=99
  4. Delta merge or apply snapshot acquire an ID for new DTFile, it will get the same ID again and this time, the dmf_100 is created on /data1
  5. Next time when TiFlash restarts, we may restore the dmf_100 from /data2 instead of /data1, which make we read wrong data and make inconsistent query result comparing to TiKV

What is changed and how it works?

  • When acquiring an ID for DTFile, we need to check whether the ID exists on disk or not (by checking StableDiskDelegator)
  • Throw an exception when try to add duplicated ID for DTFile

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch:

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

Release note 

Fix the potential issue of data inconsistency after crashes when deployed on multi-disks

@ti-chi-bot ti-chi-bot mentioned this pull request Aug 25, 2021
Merged
@ti-chi-bot
Copy link
Member Author

ti-chi-bot commented Aug 25, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • JaySon-Huang

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/cherry-pick-not-approved size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. type/bugfix This PR fixes a bug. type/cherry-pick-for-release-5.0 labels Aug 25, 2021
@ti-chi-bot ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 25, 2021
@flowbehappy flowbehappy self-requested a review August 26, 2021 06:17
@JaySon-Huang JaySon-Huang added this to the v5.0.4 milestone Aug 26, 2021
@JaySon-Huang
Resolve conflict
c60530b 
Signed-off-by: JaySon-Huang <[email protected]>
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 26, 2021
@JaySon-Huang
Resolve conflict
8de64c0 
Signed-off-by: JaySon-Huang <[email protected]>
@JaySon-Huang
Copy link
Contributor

/run-all-tests

@tisonkun
empty
5f4b884 
Signed-off-by: tison <[email protected]>
@JaySon-Huang
Copy link
Contributor

/run-all-tests

@JaySon-Huang
Copy link
Contributor

Should we merge this fix into v5.0.4 first? @zanmato1984

@JaySon-Huang
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member Author

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member Author

@JaySon-Huang: /merge in this pull request requires 1 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 1, 2021
@zhouqiang-cl zhouqiang-cl added the cherry-pick-approved Cherry pick PR approved by release team. label Sep 1, 2021
@ti-chi-bot ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/cherry-pick-not-approved labels Sep 1, 2021
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 1, 2021
@JaySon-Huang
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member Author

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member Author

This pull request has been accepted and is ready to merge.

Commit hash: a00e775

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 1, 2021
@JaySon-Huang
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member Author

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@JaySon-Huang
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member Author

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 9d95db0 into pingcap:release-5.0 Sep 1, 2021
@JaySon-Huang JaySon-Huang deleted the cherry-pick-2770-to-release-5.0 branch September 1, 2021 13:13
@windtalker windtalker mentioned this pull request Sep 6, 2021
Merged
windtalker added a commit that referenced this pull request Sep 6, 2021
@windtalker @ti-chi-bot
Fix duplicated ID DTFile that cause inconsistent query result (#2770) (…
348cea5 
…#2775) (#3014)

Co-authored-by: Ti Chi Robot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JaySon-Huang JaySon-Huang JaySon-Huang approved these changes

@flowbehappy flowbehappy Awaiting requested review from flowbehappy

Assignees

@JaySon-Huang JaySon-Huang

Labels
cherry-pick-approved Cherry pick PR approved by release team. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug. type/cherry-pick-for-release-5.0
Projects
None yet
Milestone
v5.0.4
Development

Successfully merging this pull request may close these issues.
4 participants
@ti-chi-bot @JaySon-Huang @zhouqiang-cl @tisonkun