Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: GC segments contained by the pack #6010

Merged
merged 23 commits into from
Sep 22, 2022
Merged

storage: GC segments contained by the pack #6010

merged 23 commits into from
Sep 22, 2022

Conversation

breezewish
Copy link
Member

@breezewish breezewish commented Sep 22, 2022

What problem does this PR solve?

Issue Number: close #5975

What is changed and how it works?

Previously, we count segment rows by using the sum rows of overlapped packs. When the segment rows are < 0.7 rows of DTFiles, the DTFile will be regarded as "reclaimable" and the stable will be re-generated.

In this PR, we count segment rows by using the sum rows of fully contained packs. This means, the calculated segment rows will be smaller than before, and segments will more likely being GCed.

Here is an illustration of the new contained rule:

image

Consider the case where there is only 1 pack in the DTFile. When segment is fully or partially contained by this pack, the segment will be GCed, no matter how many actual rows are there in the segment. This leads to extra GCs in some cases, but the negative impact is small, as the cost of rewriting 1 pack should be small.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 22, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • flowbehappy
  • lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-6.3 release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 22, 2022
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 22, 2022
Signed-off-by: Wish <[email protected]>
Copy link
Contributor

@lidezhu lidezhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 22, 2022
Copy link
Contributor

@JaySon-Huang JaySon-Huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@breezewish
Copy link
Member Author

/run-all-tests

breezewish added a commit to breezewish/tiflash that referenced this pull request Sep 22, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Sep 22, 2022

Coverage for changed files

Filename                                                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FailPoint.cpp                                                590               213    63.90%           8                 2    75.00%          84                36    57.14%         200               104    48.00%
Storages/DeltaMerge/DeltaMergeStore.cpp                             897               156    82.61%          46                 2    95.65%        1312               229    82.55%         470               153    67.45%
Storages/DeltaMerge/DeltaMergeStore_InternalBg.cpp                  510               180    64.71%          16                 1    93.75%         630               152    75.87%         228                87    61.84%
Storages/DeltaMerge/Segment.cpp                                    1141               276    75.81%          55                 1    98.18%        1348               115    91.47%         358               103    71.23%
Storages/DeltaMerge/StableValueSpace.cpp                            136                23    83.09%          18                 1    94.44%         332                48    85.54%          90                25    72.22%
Storages/DeltaMerge/StableValueSpace.h                               19                 5    73.68%          16                 2    87.50%          32                18    43.75%           2                 2     0.00%
Storages/DeltaMerge/tests/gtest_dm_segment.cpp                      402               131    67.41%          36                 0   100.00%        1134                 6    99.47%         164                77    53.05%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp         238                36    84.87%          20                 1    95.00%         277                12    95.67%          66                29    56.06%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.h             2                 0   100.00%           2                 0   100.00%           6                 0   100.00%           0                 0         -
Storages/DeltaMerge/tests/gtest_dm_store_background.cpp             235                85    63.83%          30                 0   100.00%         328                 4    98.78%          66                50    24.24%
Storages/DeltaMerge/tests/gtest_segment.cpp                         836               280    66.51%          46                 0   100.00%         877                23    97.38%         236               178    24.58%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                              5006              1385    72.33%         293                10    96.59%        6360               643    89.89%        1880               808    57.02%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18305      7509             58.98%    214081  77146        63.96%

full coverage report (for internal network access only)

@breezewish breezewish added the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2022
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2022
@breezewish
Copy link
Member Author

/merge

@ti-chi-bot
Copy link
Member

@breezewish: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: ca9529d

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Sep 22, 2022
JaySon-Huang pushed a commit that referenced this pull request Sep 22, 2022
@breezewish
Copy link
Member Author

/run-unit-test

@sre-bot
Copy link
Collaborator

sre-bot commented Sep 22, 2022

Coverage for changed files

Filename                                                        Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/FailPoint.cpp                                                590               213    63.90%           8                 2    75.00%          84                36    57.14%         200               104    48.00%
Storages/DeltaMerge/DeltaMergeStore.cpp                             897               156    82.61%          46                 2    95.65%        1312               229    82.55%         470               150    68.09%
Storages/DeltaMerge/DeltaMergeStore_InternalBg.cpp                  510               180    64.71%          16                 1    93.75%         630               152    75.87%         228                87    61.84%
Storages/DeltaMerge/Segment.cpp                                    1141               276    75.81%          55                 1    98.18%        1348               115    91.47%         358               102    71.51%
Storages/DeltaMerge/StableValueSpace.cpp                            136                23    83.09%          18                 1    94.44%         332                48    85.54%          90                25    72.22%
Storages/DeltaMerge/StableValueSpace.h                               19                 5    73.68%          16                 2    87.50%          32                18    43.75%           2                 2     0.00%
Storages/DeltaMerge/tests/gtest_dm_segment.cpp                      402               131    67.41%          36                 0   100.00%        1134                 6    99.47%         164                77    53.05%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp         238                36    84.87%          20                 1    95.00%         277                12    95.67%          66                29    56.06%
Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.h             2                 0   100.00%           2                 0   100.00%           6                 0   100.00%           0                 0         -
Storages/DeltaMerge/tests/gtest_dm_store_background.cpp             235                85    63.83%          30                 0   100.00%         328                 4    98.78%          66                50    24.24%
Storages/DeltaMerge/tests/gtest_segment.cpp                         836               280    66.51%          46                 0   100.00%         877                23    97.38%         236               178    24.58%
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                              5006              1385    72.33%         293                10    96.59%        6360               643    89.89%        1880               804    57.23%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
18305      7507             58.99%    214081  77094        63.99%

full coverage report (for internal network access only)

@breezewish breezewish merged commit 43571c0 into master Sep 22, 2022
@breezewish breezewish deleted the wenxuan/gc branch September 22, 2022 16:37
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: failed to apply #6010 on top of branch "release-6.3":

failed to git commit: exit status 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-6.3 release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tiflash can't gc completely after set tiflash replica 0
6 participants