Fix restore problems in PageStorage. #4350

jiaqizho · 2022-03-18T13:35:18Z

Signed-off-by: jiaqizho [email protected]

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

Begin of restore, we should let mvcc call gcInMemEntries, Then it won't ask BlobStore to cleanup invalid data.
The getEntry method in mvcc will return a null entry that is being refed but has already accepted del op(happened in restore method). Then BlobStore won't apply for this entry, so this entry will lose.

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: jiaqizho <[email protected]>

ti-chi-bot · 2022-03-18T13:35:20Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

CalvinNeo
JaySon-Huang

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

CalvinNeo

LGTM

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

JaySon-Huang · 2022-03-21T03:17:46Z

dbms/src/Storages/Page/V3/PageStorageImpl.cpp

@@ -44,6 +44,7 @@ PageStorageImpl::~PageStorageImpl() = default;

 void PageStorageImpl::restore()
 {
+    // TODO: clean up blobstore.


What thing do you plan to optimize by adding this "TODO" comment? Can you explain more?

There is such an extreme example.
Before shutdown:

blob 1 : valid entries 10 blob 2 : valid entries 100 blob 3 : valid entries 100

after shutdown, Then we do the restore:

blob 1 : all of entries is invalid blob 2 : valid entries less than 100, but still exist valid entry blob 3 : valid entries less than 100, but still exist valid entry

Then blob 1 won't restored, because it has not any valid entries.
But at this time, we also won't remove it in the disk. Because it is not created on memory. So we need add a scan after we restored pagestorage. Then removed blob 1.

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

Co-authored-by: JaySon <[email protected]>

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

JaySon-Huang · 2022-03-21T07:51:33Z

/hold

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

find incorrect changes

…-restore-error

JaySon-Huang · 2022-03-21T08:24:12Z

/run-unit-test

sre-bot · 2022-03-21T08:42:45Z

Coverage for changed files

Filename                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PageDirectory.cpp                    456                64    85.96%          31                 2    93.55%        1009               183    81.86%         374                83    77.81%
PageDirectory.h                       25                 6    76.00%          25                 6    76.00%         104                21    79.81%           0                 0         -
PageDirectoryFactory.cpp              36                 6    83.33%           4                 0   100.00%         106                16    84.91%          42                 6    85.71%
PageStorageImpl.cpp                   80                22    72.50%          21                 7    66.67%         194                62    68.04%          44                23    47.73%
tests/gtest_page_storage.cpp        2689               985    63.37%          32                 7    78.12%         740               154    79.19%         858               511    40.44%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                               3286              1083    67.04%         113                22    80.53%        2153               436    79.75%        1318               623    52.73%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17022      9571             43.77%    191752  97023        49.40%

full coverage report (for internal network access only)

JaySon-Huang

LGTM

JaySon-Huang · 2022-03-21T08:45:47Z

/merge

ti-chi-bot · 2022-03-21T08:45:48Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-21T08:45:54Z

This pull request has been accepted and is ready to merge.

Commit hash: da2a63a

sre-bot · 2022-03-21T08:54:23Z

Coverage for changed files

Filename                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PageDirectory.cpp                    456                64    85.96%          31                 2    93.55%        1009               183    81.86%         374                83    77.81%
PageDirectory.h                       25                 6    76.00%          25                 6    76.00%         104                21    79.81%           0                 0         -
PageDirectoryFactory.cpp              36                 6    83.33%           4                 0   100.00%         106                16    84.91%          42                 6    85.71%
PageStorageImpl.cpp                   80                22    72.50%          21                 7    66.67%         194                62    68.04%          44                23    47.73%
tests/gtest_page_storage.cpp        2689               985    63.37%          32                 7    78.12%         740               154    79.19%         858               511    40.44%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                               3286              1083    67.04%         113                22    80.53%        2153               436    79.75%        1318               623    52.73%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17022      9571             43.77%    191752  97020        49.40%

full coverage report (for internal network access only)

JaySon-Huang · 2022-03-21T08:57:16Z

/merge

ti-chi-bot · 2022-03-21T08:57:17Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

JaySon-Huang · 2022-03-21T08:57:29Z

/unhold

sre-bot · 2022-03-21T09:11:52Z

Coverage for changed files

Filename                         Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PageDirectory.cpp                    456                64    85.96%          31                 2    93.55%        1009               183    81.86%         374                83    77.81%
PageDirectory.h                       25                 6    76.00%          25                 6    76.00%         104                21    79.81%           0                 0         -
PageDirectoryFactory.cpp              36                 6    83.33%           4                 0   100.00%         106                16    84.91%          42                 6    85.71%
PageStorageImpl.cpp                   80                22    72.50%          21                 7    66.67%         194                62    68.04%          44                23    47.73%
tests/gtest_page_storage.cpp        2689               985    63.37%          32                 7    78.12%         740               154    79.19%         858               511    40.44%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                               3286              1083    67.04%         113                22    80.53%        2153               436    79.75%        1318               623    52.73%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17023      9572             43.77%    191754  97017        49.41%

full coverage report (for internal network access only)

Signed-off-by: JaySon-Huang <[email protected]>

JaySon-Huang · 2022-03-21T09:21:15Z

/merge

ti-chi-bot · 2022-03-21T09:21:16Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-21T09:21:22Z

This pull request has been accepted and is ready to merge.

Commit hash: a251e92

sre-bot · 2022-03-21T09:37:32Z

Coverage for changed files

Filename                           Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PageDirectory.cpp                      456                64    85.96%          31                 2    93.55%        1009               183    81.86%         374                83    77.81%
PageDirectory.h                         25                 6    76.00%          25                 6    76.00%         104                21    79.81%           0                 0         -
PageDirectoryFactory.cpp                39                 5    87.18%           4                 0   100.00%         117                15    87.18%          46                 5    89.13%
PageDirectoryFactory.h                   2                 0   100.00%           2                 0   100.00%           8                 0   100.00%           0                 0         -
PageStorageImpl.cpp                     80                22    72.50%          21                 7    66.67%         194                62    68.04%          44                23    47.73%
tests/gtest_page_directory.cpp        8833              1575    82.17%          50                 0   100.00%        1633                 7    99.57%        2774              1405    49.35%
tests/gtest_page_storage.cpp          2689               985    63.37%          32                 7    78.12%         740               154    79.19%         858               511    40.44%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                12124              2657    78.08%         165                22    86.67%        3805               442    88.38%        4096              2027    50.51%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
17027      9572             43.78%    191821  97032        49.42%

full coverage report (for internal network access only)

Fix pagestorage restore may loss data.

1139f24

Signed-off-by: jiaqizho <[email protected]>

jiaqizho requested a review from JaySon-Huang March 18, 2022 13:35

ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 18, 2022

jiaqizho changed the title ~~Fix restore in PageStorage V3 may loss some data.~~ Fix restore problems in PageStorage. Mar 18, 2022

jiaqizho requested a review from CalvinNeo March 18, 2022 13:36

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 20, 2022

CalvinNeo approved these changes Mar 21, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 21, 2022

JaySon-Huang reviewed Mar 21, 2022

View reviewed changes

jiaqizho and others added 3 commits March 21, 2022 12:01

Update dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

312dac0

Co-authored-by: JaySon <[email protected]>

update

378f7cb

Merge branch 'master' into fix-restore-error

d5a30a8

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 21, 2022

JaySon-Huang reviewed Mar 21, 2022

View reviewed changes

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp Outdated Show resolved Hide resolved

Update dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp

5539552

ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 21, 2022

JaySon-Huang previously approved these changes Mar 21, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 21, 2022

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 21, 2022

JaySon-Huang reviewed Mar 21, 2022

View reviewed changes

dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp Outdated Show resolved Hide resolved

jiaqizho added 2 commits March 21, 2022 16:13

fix

348cd5e

Merge branch 'fix-restore-error' of github.com:jiaqizho/tics into fix…

da2a63a

…-restore-error

JaySon-Huang approved these changes Mar 21, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2022

Merge branch 'master' into fix-restore-error

a2a9a66

ti-chi-bot removed the do-not-merge/needs-linked-issue label Mar 21, 2022

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 21, 2022

Fix for createFromEdit

a251e92

Signed-off-by: JaySon-Huang <[email protected]>

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed status/can-merge Indicates a PR has been approved by a committer. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 21, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 21, 2022

ti-chi-bot merged commit 48c78f9 into pingcap:master Mar 21, 2022

JaySon-Huang deleted the fix-restore-error branch March 21, 2022 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix restore problems in PageStorage. #4350

Fix restore problems in PageStorage. #4350

jiaqizho commented Mar 18, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Mar 18, 2022 •

edited

Loading

CalvinNeo left a comment

JaySon-Huang Mar 21, 2022

jiaqizho Mar 21, 2022

JaySon-Huang Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

sre-bot commented Mar 21, 2022

JaySon-Huang left a comment

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

sre-bot commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

sre-bot commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

sre-bot commented Mar 21, 2022

Fix restore problems in PageStorage. #4350

Fix restore problems in PageStorage. #4350

Conversation

jiaqizho commented Mar 18, 2022 • edited by JaySon-Huang Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 18, 2022 • edited Loading

CalvinNeo left a comment

Choose a reason for hiding this comment

JaySon-Huang Mar 21, 2022

Choose a reason for hiding this comment

jiaqizho Mar 21, 2022

Choose a reason for hiding this comment

JaySon-Huang Mar 21, 2022

Choose a reason for hiding this comment

JaySon-Huang commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

sre-bot commented Mar 21, 2022

Coverage for changed files

Coverage summary

JaySon-Huang left a comment

Choose a reason for hiding this comment

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

sre-bot commented Mar 21, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

JaySon-Huang commented Mar 21, 2022

sre-bot commented Mar 21, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

ti-chi-bot commented Mar 21, 2022

sre-bot commented Mar 21, 2022

Coverage for changed files

Coverage summary

jiaqizho commented Mar 18, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Mar 18, 2022 •

edited

Loading