WALStore support multi path #4085

jiaqizho · 2022-02-21T09:13:50Z

Signed-off-by: jiaqizho [email protected]

What problem does this PR solve?

Issue Number: ref #3594

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: jiaqizho <[email protected]>

ti-chi-bot · 2022-02-21T09:13:51Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JaySon-Huang
hehechen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

dbms/src/Storages/Page/V3/WALStore.h

dbms/src/Storages/Page/V3/WALStore.cpp

JaySon-Huang · 2022-03-15T05:50:11Z

@jiaqizho PTAL https://github.com/jiaqizho/tics/pull/8

Walstore support multi path

dbms/src/Storages/Page/V3/WALStore.h

Signed-off-by: jiaqizho <[email protected]>

jiaqizho · 2022-03-15T07:40:40Z

/run-all-tests

jiaqizho · 2022-03-15T08:33:37Z

/rebuild

JaySon-Huang · 2022-03-15T17:24:03Z

/merge

ti-chi-bot · 2022-03-15T17:24:04Z

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2022-03-15T17:24:10Z

This pull request has been accepted and is ready to merge.

Commit hash: 89501cd

sre-bot · 2022-03-15T17:44:17Z

Coverage for changed files

too many lines from llvm-cov, please refer to full report instead

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16924      9502             43.85%    190713  96475        49.41%

full coverage report (for internal network access only)

JaySon-Huang · 2022-03-16T03:54:35Z

/run-integration-test

jiaqizho · 2022-03-16T04:01:17Z

/merge

ti-chi-bot · 2022-03-16T04:01:18Z

@jiaqizho: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-03-16T04:11:18Z

Coverage for changed files

Filename                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Page/V3/WAL/WALReader.cpp                  72                10    86.11%           9                 0   100.00%         176                15    91.48%          54                14    74.07%
Page/V3/WALStore.cpp                       74                39    47.30%           8                 1    87.50%         134                42    68.66%          36                19    47.22%
Page/V3/WALStore.h                          1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/gtest_wal_store.cpp        1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
tests/TiFlashStorageTestBasic.h            27                 5    81.48%           6                 0   100.00%          51                12    76.47%          18                 9    50.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                    2122               314    85.20%          36                 1    97.22%         829                82    90.11%         754               356    52.79%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16924      9502             43.85%    190713  96475        49.41%

full coverage report (for internal network access only)

JaySon-Huang · 2022-03-16T04:45:33Z

/rebuild

JaySon-Huang · 2022-03-16T04:45:46Z

/run-integration-test

sre-bot · 2022-03-16T05:03:32Z

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Flash/Coprocessor/DAGUtils.cpp                     290               233    19.66%          36                24    33.33%         505               404    20.00%         324               230    29.01%
Functions/FunctionsDateTime.cpp                     27                19    29.63%           4                 2    50.00%          99                25    74.75%          16                13    18.75%
Functions/FunctionsDateTime.h                     1073               638    40.54%         322               209    35.09%        2018              1226    39.25%         534               340    36.33%
Functions/tests/gtest_dayofweekyear.cpp            258                74    71.32%           2                 0   100.00%         135                 0   100.00%          80                42    47.50%
Storages/Page/V3/WAL/WALReader.cpp                  72                10    86.11%           9                 0   100.00%         176                15    91.48%          54                14    74.07%
Storages/Page/V3/WALStore.cpp                       74                39    47.30%           8                 1    87.50%         134                42    68.66%          36                19    47.22%
Storages/Page/V3/WALStore.h                          1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Storages/Page/V3/tests/gtest_wal_store.cpp        1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
Storages/tests/TiFlashStorageTestBasic.h            27                 5    81.48%           6                 0   100.00%          51                12    76.47%          18                 9    50.00%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             3770              1278    66.10%         400               236    41.00%        3586              1737    51.56%        1708               981    42.56%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16930      9504             43.86%    190864  96491        49.45%

full coverage report (for internal network access only)

JaySon-Huang · 2022-03-16T05:50:34Z

/run-integration-test

JaySon-Huang · 2022-03-16T06:33:39Z

/run-integration-test

ti-chi-bot · 2022-03-16T06:53:05Z

@jiaqizho: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

sre-bot · 2022-03-16T07:10:20Z

Coverage for changed files

Filename                                       Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/TiFlashMetrics.h                             18                 0   100.00%          11                 0   100.00%          47                 1    97.87%           8                 0   100.00%
Flash/Coprocessor/DAGUtils.cpp                     290               233    19.66%          36                24    33.33%         505               404    20.00%         324               230    29.01%
Flash/EstablishCall.h                                2                 2     0.00%           2                 2     0.00%           2                 2     0.00%           0                 0         -
Flash/Mpp/MPPTunnel.cpp                            185               185     0.00%          16                16     0.00%         241               241     0.00%          94                94     0.00%
Flash/Mpp/MPPTunnel.h                                9                 9     0.00%           9                 9     0.00%          18                18     0.00%           0                 0         -
Functions/FunctionsDateTime.cpp                     27                19    29.63%           4                 2    50.00%          99                25    74.75%          16                13    18.75%
Functions/FunctionsDateTime.h                     1073               638    40.54%         322               209    35.09%        2018              1226    39.25%         534               340    36.33%
Functions/tests/gtest_dayofweekyear.cpp            258                74    71.32%           2                 0   100.00%         135                 0   100.00%          80                42    47.50%
Storages/Page/V3/WAL/WALReader.cpp                  72                10    86.11%           9                 0   100.00%         176                15    91.48%          54                14    74.07%
Storages/Page/V3/WALStore.cpp                       74                39    47.30%           8                 1    87.50%         134                42    68.66%          36                19    47.22%
Storages/Page/V3/WALStore.h                          1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Storages/Page/V3/tests/gtest_wal_store.cpp        1948               260    86.65%          12                 0   100.00%         465                13    97.20%         646               314    51.39%
Storages/tests/TiFlashStorageTestBasic.h            27                 5    81.48%           6                 0   100.00%          51                12    76.47%          18                 9    50.00%
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                             3984              1474    63.00%         438               263    39.95%        3894              1999    48.66%        1810              1075    40.61%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16931      9505             43.86%    190876  96504        49.44%

full coverage report (for internal network access only)

JaySon-Huang · 2022-03-16T07:51:06Z

/run-integration-test

ref pingcap#3594

WALStore support multi path

773bace

Signed-off-by: jiaqizho <[email protected]>

jiaqizho requested a review from JaySon-Huang February 21, 2022 09:13

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 21, 2022

JaySon-Huang reviewed Feb 22, 2022

View reviewed changes

dbms/src/Storages/Page/V3/WALStore.h Outdated Show resolved Hide resolved

update

01c13f3

ti-chi-bot added do-not-merge/needs-linked-issue and removed do-not-merge/needs-linked-issue labels Feb 23, 2022

JaySon-Huang reviewed Feb 24, 2022

View reviewed changes

dbms/src/Storages/Page/V3/WALStore.cpp Show resolved Hide resolved

jiaqizho mentioned this pull request Mar 3, 2022

Introduce PageStorage V3 to reduce CPU usage and write amplification #3594

Closed

76 tasks

Merge branch 'master' into walstore-support-multi-path

59de2cc

ti-chi-bot added the do-not-merge/needs-linked-issue label Mar 15, 2022

JaySon-Huang added 2 commits March 15, 2022 13:23

Fix concurrent bug

ba1f297

Make ut run for multi-disk

709016f

Merge pull request #8 from JaySon-Huang/walstore-support-multi-path

42e4066

Walstore support multi path

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 15, 2022

JaySon-Huang approved these changes Mar 15, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 15, 2022

jiaqizho requested review from CalvinNeo and hehechen March 15, 2022 07:18

hehechen reviewed Mar 15, 2022

View reviewed changes

dbms/src/Storages/Page/V3/WALStore.h Show resolved Hide resolved

hehechen approved these changes Mar 15, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 15, 2022

update

9397882

Signed-off-by: jiaqizho <[email protected]>

update

89501cd

ti-chi-bot added status/can-merge Indicates a PR has been approved by a committer. and removed do-not-merge/needs-linked-issue labels Mar 15, 2022

Merge branch 'master' into walstore-support-multi-path

0ab82d0

Merge branch 'master' into walstore-support-multi-path

6ae1859

Merge branch 'master' into walstore-support-multi-path

e51d7ac

ti-chi-bot merged commit fa1e9eb into pingcap:master Mar 16, 2022

JaySon-Huang deleted the walstore-support-multi-path branch March 16, 2022 08:44

JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this pull request Mar 17, 2022

WALStore support multi path (pingcap#4085)

989bbb5

ref pingcap#3594

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WALStore support multi path #4085

WALStore support multi path #4085

jiaqizho commented Feb 21, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Feb 21, 2022 •

edited

Loading

JaySon-Huang commented Mar 15, 2022

jiaqizho commented Mar 15, 2022

jiaqizho commented Mar 15, 2022

JaySon-Huang commented Mar 15, 2022

ti-chi-bot commented Mar 15, 2022

ti-chi-bot commented Mar 15, 2022

sre-bot commented Mar 15, 2022

JaySon-Huang commented Mar 16, 2022

jiaqizho commented Mar 16, 2022

ti-chi-bot commented Mar 16, 2022

sre-bot commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

sre-bot commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

ti-chi-bot commented Mar 16, 2022

sre-bot commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

WALStore support multi path #4085

WALStore support multi path #4085

Conversation

jiaqizho commented Feb 21, 2022 • edited by JaySon-Huang Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Feb 21, 2022 • edited Loading

JaySon-Huang commented Mar 15, 2022

jiaqizho commented Mar 15, 2022

jiaqizho commented Mar 15, 2022

JaySon-Huang commented Mar 15, 2022

ti-chi-bot commented Mar 15, 2022

ti-chi-bot commented Mar 15, 2022

sre-bot commented Mar 15, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 16, 2022

jiaqizho commented Mar 16, 2022

ti-chi-bot commented Mar 16, 2022

sre-bot commented Mar 16, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

sre-bot commented Mar 16, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 16, 2022

JaySon-Huang commented Mar 16, 2022

ti-chi-bot commented Mar 16, 2022

sre-bot commented Mar 16, 2022

Coverage for changed files

Coverage summary

JaySon-Huang commented Mar 16, 2022

jiaqizho commented Feb 21, 2022 •

edited by JaySon-Huang

Loading

ti-chi-bot commented Feb 21, 2022 •

edited

Loading