Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storage: Fix page_id being mis-reuse when upgrade from cluster < 6.5 (#9041) #9053

Conversation

JaySon-Huang
Copy link
Contributor

This is an automated cherry-pick of #9041

What problem does this PR solve?

Issue Number: close #9039

Problem Summary:

#9039 (comment)

The root cause is:

  1. When the cluster upgrade from version lower than v6.5.0 to higher or equal to v6.5.0, TiFlash will try to migrate the data from PageStorage V2 to PageStorage V3.
  2. When the table has no data in the delta layer, it will run under PageStorage V3 mode
  3. But we get a wrong max_id for this situation, we must use max_data_page_id = std::max(data_storage_v2->getMaxId(), data_storage_v3->getMaxId()); to avoid the page_id from being mis-reused. Otherwise it will cause data corruption and make tiflash fails to restart in the next restart

max_log_page_id = log_storage_v3->getMaxId();
max_data_page_id = data_storage_v3->getMaxId();
max_meta_page_id = meta_storage_v3->getMaxId();

What is changed and how it works?

  • In StoragePool::restore, after "meta" and "data" transformed into PSV3, update the max_id by the maximum of that id in PSV2 and PSV3
  • Review other codes that call raiseXXXPageIdLowerBound()
  • Add unit test case

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    1. Deploy a cluster with tiflash replica and load some data, the cluster version < 6.5.0
    2. Before upgrade, alter table ... compact tiflash replica to make the table has no data in the delta layer (this raises the possibility of the bug happening)
    3. Upgrade the cluster to a version >= 6.5.0 and write more rows into table, then alter table ... compact tiflash replica
    4. Restart the tiflash instance
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the issue that metadata may be corrupt and make tiflash panic after upgrading from a cluster less than v6.5.0

@ti-chi-bot ti-chi-bot bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 16, 2024
Copy link
Contributor

ti-chi-bot bot commented May 16, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jayson-huang, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 16, 2024
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

The integration-test to release-7.1 PR is passed: #9048

@JaySon-Huang JaySon-Huang merged commit 2f1d64f into pingcap:release-7.1-20240516-v7.1.5 May 16, 2024
3 of 5 checks passed
@JaySon-Huang JaySon-Huang deleted the fix_max_id_after_upgrade_71_hotfix branch May 16, 2024 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant