Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix invalid storage dir configurations lead to unexpected behavior #4105

Merged
merged 5 commits into from
Feb 24, 2022

Conversation

JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Feb 23, 2022

What problem does this PR solve?

Issue Number: close #4093

Problem Summary:
If the user set the config file with

[storage]
[storage.main]
dir = "/data0/db"
# Should be: dir = ["/data0/db"]
[storage.raft]
dir = "/data0/db"
# Should be: dir = ["/data0/db"]

If storage.main.dir,storage.latest.dir,storage.raft.dir is a string instead of string-array, TiFlash now considers it is an empty array without any warning/exception, which leads to unexpected behavior

What is changed and how it works?

Check whether storage.main.dir, storage.latest.dir, storage.raft.dir is string array or not. If not, throw an exception and stop tiflash.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Fix the bug that invalid storage dir configurations lead to unexpected behavior

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Feb 23, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • flowbehappy

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-triage-completed release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 23, 2022
@JaySon-Huang JaySon-Huang changed the title Enhance the verification of storage dir(s) check Fix invalid storage dir configurations lead to unexpected behavior Feb 23, 2022
@ti-chi-bot ti-chi-bot added needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 needs-cherry-pick-release-5.2 PR which needs to be cherry-picked to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. and removed do-not-merge/needs-triage-completed labels Feb 23, 2022
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Feb 23, 2022

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/tests/gtest_cpptoml.cpp                294                45    84.69%           3                 0   100.00%         104                 0   100.00%         104                48    53.85%
Server/StorageConfigParser.cpp                398               145    63.57%          22                 1    95.45%         400                71    82.25%         234               105    55.13%
Server/tests/gtest_storage_config.cpp        2900              1022    64.76%          15                 2    86.67%         590                34    94.24%         884               535    39.48%
Storages/PathCapacityMetrics.cpp              147                62    57.82%           9                 0   100.00%         190                24    87.37%          88                35    60.23%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        3739              1274    65.93%          49                 3    93.88%        1284               129    89.95%        1310               723    44.81%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16588      9494             42.77%    183771  95923        47.80%

full coverage report (for internal network access only)

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Feb 23, 2022

Going to add more test by the coverage report More tests added

image

@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Feb 23, 2022

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/tests/gtest_cpptoml.cpp                294                45    84.69%           3                 0   100.00%         104                 0   100.00%         104                48    53.85%
Server/StorageConfigParser.cpp                402                88    78.11%          22                 1    95.45%         400                20    95.00%         234                84    64.10%
Server/tests/gtest_storage_config.cpp        3067              1055    65.60%          16                 2    87.50%         649                34    94.76%         936               562    39.96%
Storages/PathCapacityMetrics.cpp              147                62    57.82%           9                 0   100.00%         190                24    87.37%          88                35    60.23%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        3910              1250    68.03%          50                 3    94.00%        1343                78    94.19%        1362               729    46.48%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16589      9494             42.77%    183830  95857        47.86%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 23, 2022
@JaySon-Huang JaySon-Huang self-assigned this Feb 24, 2022
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: cef12ba

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 24, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Feb 24, 2022

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/tests/gtest_cpptoml.cpp                294                45    84.69%           3                 0   100.00%         104                 0   100.00%         104                48    53.85%
Server/StorageConfigParser.cpp                402                88    78.11%          22                 1    95.45%         400                20    95.00%         234                84    64.10%
Server/tests/gtest_storage_config.cpp        3067              1055    65.60%          16                 2    87.50%         649                34    94.76%         936               562    39.96%
Storages/PathCapacityMetrics.cpp              147                62    57.82%           9                 0   100.00%         190                24    87.37%          88                35    60.23%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        3910              1250    68.03%          50                 3    94.00%        1343                78    94.19%        1362               729    46.48%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16589      9494             42.77%    183830  95852        47.86%

full coverage report (for internal network access only)

@sre-bot
Copy link
Collaborator

sre-bot commented Feb 24, 2022

Coverage for changed files

Filename                                  Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Common/tests/gtest_cpptoml.cpp                294                45    84.69%           3                 0   100.00%         104                 0   100.00%         104                48    53.85%
Server/StorageConfigParser.cpp                402                88    78.11%          22                 1    95.45%         400                20    95.00%         234                84    64.10%
Server/tests/gtest_storage_config.cpp        3067              1055    65.60%          16                 2    87.50%         649                34    94.76%         936               562    39.96%
Storages/PathCapacityMetrics.cpp              147                62    57.82%           9                 0   100.00%         190                24    87.37%          88                35    60.23%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        3910              1250    68.03%          50                 3    94.00%        1343                78    94.19%        1362               729    46.48%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16682      9491             43.11%    184707  95796        48.14%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit e50c06c into pingcap:master Feb 24, 2022
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4130.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4131.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4132.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4133.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4134.

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 24, 2022
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #4135.

@JaySon-Huang JaySon-Huang deleted the enhance_config_verify branch February 24, 2022 04:55
@JaySon-Huang JaySon-Huang added the type/bugfix This PR fixes a bug. label Apr 20, 2022
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Jun 16, 2022
JaySon-Huang added a commit to ti-chi-bot/tiflash that referenced this pull request Jun 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 needs-cherry-pick-release-5.1 PR which needs to be cherry-picked to release-5.1 needs-cherry-pick-release-5.2 PR which needs to be cherry-picked to release-5.2 needs-cherry-pick-release-5.3 Type: Need cherry pick to release-5.3 needs-cherry-pick-release-5.4 Should cherry pick this PR to release-5.4 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/bugfix This PR fixes a bug.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Should check whether the dirs under "storage" configuration is valid string array
4 participants