Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd: Make changefeed sort dir visible #2222

Merged
merged 12 commits into from
Jul 7, 2021
Merged

cmd: Make changefeed sort dir visible #2222

merged 12 commits into from
Jul 7, 2021

Conversation

3AceShowHand
Copy link
Contributor

@3AceShowHand 3AceShowHand commented Jul 5, 2021 •
edited
Loading

What problem does this PR solve?

This PR try to revert part of the work in #1784, also fix a problem when try to set data-dir by existed changefeed information.

  • sort-dir can be read but cannot be written.
  • create dir temporarily when try to find best data-dir by existed changefeed information.

What is changed and how it works?

revert changefeed info struct and related read and write.

Check List

Tests

  • Unit test

Code changes

Side effects

Related changes

Release note

  • No release note

@ti-chi-bot ti-chi-bot requested review from amyangfei and zier-one July 5, 2021 07:29
@ti-chi-bot ti-chi-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jul 5, 2021
@amyangfei amyangfei added needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. labels Jul 5, 2021
@amyangfei
Copy link
Contributor

/run-all-tests

@codecov-commenter
Copy link

codecov-commenter commented Jul 5, 2021 •
edited
Loading

Codecov Report

Merging #2222 (6dfe200) into master (0138edc) will increase coverage by 14.8406%.
The diff coverage is 60.9573%.

@@                Coverage Diff                @@
##             master      #2222         +/-   ##
=================================================
+ Coverage   37.8006%   52.6412%   +14.8405%     
=================================================
  Files           110        171         +61     
  Lines         11394      18609       +7215     
=================================================
+ Hits           4307       9796       +5489     
- Misses         6662       7760       +1098     
- Partials        425       1053        +628

@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 5, 2021
amyangfei
amyangfei reviewed Jul 6, 2021
View reviewed changes
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix ci

@ben1009
Copy link
Contributor

ben1009 commented Jul 6, 2021 

[2021-07-05T12:10:10.861Z] FAIL: fileutil_test.go:68: fileUtilSuite.TestIsDirReadWritable

[2021-07-05T12:10:10.861Z] 

[2021-07-05T12:10:10.861Z] fileutil_test.go:77:

[2021-07-05T12:10:10.861Z]     c.Assert(err, check.ErrorMatches, ".*no such file or directory")

[2021-07-05T12:10:10.861Z] ... value = nil

[2021-07-05T12:10:10.861Z] ... regex string = ".*no such file or directory"

[2021-07-05T12:10:10.861Z] ... Error value is nil

seems we have done several fix on fix about sort-dir related issues lately, do we need have a discussion or it is already fixed, thanks

@3AceShowHand 3AceShowHand force-pushed the make-changefeed-sort-dir-visible branch from 17341c9 to 88c92c0 Compare July 6, 2021 14:16
@ti-chi-bot ti-chi-bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 6, 2021
@ti-chi-bot ti-chi-bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 6, 2021
@3AceShowHand
Copy link
Contributor Author

/run-all-tests

overvenus
overvenus reviewed Jul 7, 2021
View reviewed changes
cdc/server.go Outdated Show resolved Hide resolved
@overvenus overvenus added the type/bugfix This PR fixes a bug. label Jul 7, 2021
@3AceShowHand 3AceShowHand requested a review from overvenus July 7, 2021 06:09
@3AceShowHand
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 7, 2021
@amyangfei
Copy link
Contributor

By the way, have we tested the sort-dir configured in changefeed can work as expected @3AceShowHand

@3AceShowHand
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 7, 2021
@3AceShowHand
Copy link
Contributor Author

3AceShowHand commented Jul 7, 2021 •
edited
Loading

By the way, have we tested the sort-dir configured in changefeed can work as expected @3AceShowHand

yes, it's tested.

  • data-dir can be set by existed ch3angefeed info, which means we can also query it
  • when creating changefeed, it will fail.

@3AceShowHand
Copy link
Contributor Author

/run-all-tests

overvenus
overvenus approved these changes Jul 7, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 7, 2021
@overvenus
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 198b3b0

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 7, 2021
@ti-chi-bot ti-chi-bot merged commit 3ee9744 into pingcap:master Jul 7, 2021
@ti-chi-bot ti-chi-bot mentioned this pull request Jul 7, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2239.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jul 7, 2021
@3AceShowHand @ti-chi-bot
This is an automated cherry-pick of pingcap#2222
a386cda 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Jul 7, 2021
Closed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2240.

@ti-chi-bot ti-chi-bot mentioned this pull request Jul 7, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2241.

3AceShowHand added a commit that referenced this pull request Jul 8, 2021
@ti-chi-bot @3AceShowHand
cmd: Make changefeed sort dir visible (#2222) (#2239)
bf28daa 
* make sort-dir field visible in changefeed info struct.

* fix related tests.

* add a simple test for sort-dir field.

* add dir checker .

* do not remove new created directory.

* update log.

* print log to make debug for data-dir easier.

Co-authored-by: Ling Jin <[email protected]>
3AceShowHand added a commit that referenced this pull request Jul 9, 2021
@ti-chi-bot @3AceShowHand
cmd: Make changefeed sort dir visible (#2222) (#2241)
9f014ba 
* make sort-dir field visible in changefeed info struct.

* fix related tests.

* add a simple test for sort-dir field.

* add dir checker .

* do not remove new created directory.

* update log.

* print log to make debug for data-dir easier.

Co-authored-by: Ling Jin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus approved these changes

@amyangfei amyangfei amyangfei approved these changes

@zier-one zier-one Awaiting requested review from zier-one

@liuzix liuzix Awaiting requested review from liuzix

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/bugfix This PR fixes a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@3AceShowHand @amyangfei @codecov-commenter @ben1009 @ti-chi-bot @overvenus