Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDC Server support data-dir #1879

Merged
merged 80 commits into from
Jun 16, 2021
Merged

CDC Server support data-dir #1879

merged 80 commits into from
Jun 16, 2021

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented May 29, 2021 •
edited
Loading

What problem does this PR solve?

This PR makes TiCDC support --data-dir at the server level.

What is changed and how it works?

  • --data-dir is supported at the server level, default to "", if not set, it will be set by conditions:
    • if there is no other changefeed, /tmp/cdc_data will be set as default data-dir
    • if there is any changefeed with the --sort-dir set, choose the valid one with the largest available space as data-dir.
      • if there are no candidates who meet the requirement, server start failed
  • --sort-dir set at both server level and changefeed level won't take effect, and default to /tmp/sorter, this means it can not be changed by both command flag or config file
    • sort dir of the server and changefeed will be set as /{data-dir}/tmp/sorter
  • during server running, if the available space is lower than 10% of the disk, serve running will fail, due to trying to consume disk space (create new sort temporary file)
  • warning user when set data dir and space unavailable.

Check List

Tests

  • Unit test

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to update key monitor metrics in both TiCDC document and official document

Release note

@ti-chi-bot ti-chi-bot requested review from amyangfei and zier-one May 29, 2021 15:09
@ti-chi-bot ti-chi-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label May 29, 2021
@3AceShowHand
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 31, 2021
@3AceShowHand
Copy link
Contributor Author

/run-integration-tests

@codecov-commenter
Copy link

codecov-commenter commented May 31, 2021 •
edited
Loading

Codecov Report

Merging #1879 (1965d88) into master (0138edc) will increase coverage by 16.3682%.
The diff coverage is 61.3058%.

@@                Coverage Diff                @@
##             master      #1879         +/-   ##
=================================================
+ Coverage   37.8006%   54.1688%   +16.3682%     
=================================================
  Files           110        164         +54     
  Lines         11394      17523       +6129     
=================================================
+ Hits           4307       9492       +5185     
- Misses         6662       7002        +340     
- Partials        425       1029        +604

@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 3, 2021
@ti-chi-bot ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 3, 2021
@amyangfei amyangfei added needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. release-blocker This issue blocks a release. Please solve it ASAP. labels Jun 16, 2021
overvenus
overvenus approved these changes Jun 16, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 16, 2021
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 16, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b8fdbe8

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 16, 2021
@3AceShowHand
Copy link
Contributor Author

/run-integration-tests

@ti-chi-bot ti-chi-bot merged commit 9135351 into pingcap:master Jun 16, 2021
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 16, 2021
@ti-chi-bot
This is an automated cherry-pick of pingcap#1879
c47682e 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 16, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2069.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 16, 2021
@ti-chi-bot
This is an automated cherry-pick of pingcap#1879
9bc2310 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 16, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2070.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jun 16, 2021
@ti-chi-bot
CDC Server support data-dir (pingcap#1879)
bc1ad0e
@ti-chi-bot ti-chi-bot mentioned this pull request Jun 16, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2071.

@amyangfei amyangfei mentioned this pull request Jun 17, 2021
Closed
ti-chi-bot added a commit that referenced this pull request Jun 17, 2021
@ti-chi-bot
CDC Server support data-dir (#1879) (#2071)
a91a10b
3AceShowHand added a commit that referenced this pull request Jun 23, 2021
@ti-chi-bot @3AceShowHand @amyangfei
CDC Server support data-dir (#1879) (#2070)
6dfdbbf 
* This is an automated cherry-pick of #1879

Signed-off-by: ti-chi-bot <[email protected]>

* fix conflicts.

* add etcdClient to server, try to fix lack of field.

* fix server config marshal test.

* make check, fix go mod.

* fix config test, make ci happy.

* fix config conflict in marshal config.

* try to fix tests.

* fix backend_sorter_test.

* Update pkg/config/config.go

* revert changes in unified sorter.

* use owner etcd client to get changefeed related info.

Co-authored-by: Ling Jin <[email protected]>
Co-authored-by: JinlingChristopher <[email protected]>
Co-authored-by: amyangfei <[email protected]>
Co-authored-by: amyangfei <[email protected]>
ti-chi-bot added a commit that referenced this pull request Jul 9, 2021
@ti-chi-bot
CDC Server support data-dir (#1879) (#2069)
14965f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus approved these changes

@amyangfei amyangfei amyangfei approved these changes

@zier-one zier-one Awaiting requested review from zier-one

@liuzix liuzix Awaiting requested review from liuzix

@asddongmen asddongmen Awaiting requested review from asddongmen

@ben1009 ben1009 Awaiting requested review from ben1009

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. release-blocker This issue blocks a release. Please solve it ASAP. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
v5.1.0
Development

Successfully merging this pull request may close these issues.
6 participants
@3AceShowHand @codecov-commenter @ti-chi-bot @amyangfei @overvenus @liuzix