Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage: Fix page_id being mis-reuse when upgrade from cluster < 6.5 (#9041) #9067

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #9041

What problem does this PR solve?

Issue Number: close #9039

Problem Summary:

#9039 (comment)

The root cause is:

  1. When the cluster upgrade from version lower than v6.5.0 to higher or equal to v6.5.0, TiFlash will try to migrate the data from PageStorage V2 to PageStorage V3.
  2. When the table has no data in the delta layer, it will run under PageStorage V3 mode
  3. But we get a wrong max_id for this situation, we must use max_data_page_id = std::max(data_storage_v2->getMaxId(), data_storage_v3->getMaxId()); to avoid the page_id from being mis-reused. Otherwise it will cause data corruption and make tiflash fails to restart in the next restart

max_log_page_id = log_storage_v3->getMaxId();
max_data_page_id = data_storage_v3->getMaxId();
max_meta_page_id = meta_storage_v3->getMaxId();

What is changed and how it works?

  • In StoragePool::restore, after "meta" and "data" transformed into PSV3, update the max_id by the maximum of that id in PSV2 and PSV3
  • Review other codes that call raiseXXXPageIdLowerBound()
  • Add unit test case

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    1. Deploy a cluster with tiflash replica and load some data, the cluster version < 6.5.0
    2. Before upgrade, alter table ... compact tiflash replica to make the table has no data in the delta layer (this raises the possibility of the bug happening)
    3. Upgrade the cluster to a version >= 6.5.0 and write more rows into table, then alter table ... compact tiflash replica
    4. Restart the tiflash instance
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the issue that metadata may be corrupt and make tiflash panic after upgrading from a cluster less than v6.5.0

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR. labels May 22, 2024
Copy link
Contributor

ti-chi-bot bot commented May 22, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jinhelin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot added the cherry-pick-approved Cherry pick PR approved by release team. label May 22, 2024
@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/cherry-pick-not-approved size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 22, 2024
@JaySon-Huang
Copy link
Contributor

replaced by #9056

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick-approved Cherry pick PR approved by release team. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-6.5 This PR is cherry-picked to release-6.5 from a source PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants