Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cluster/executor: implement native SCP download instead of cat #1382

Merged
merged 3 commits into from
May 25, 2021

Conversation

AstroProfundis
Copy link
Contributor

What problem does this PR solve?

Use native SCP to download file from remote, instead of cat the file content.

What is changed and how it works?

Protocol doc: https://web.archive.org/web/20170215184048/https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works
The implementation is partially inspired by https://github.com/dtylman/scp

Check List

Tests

  • Integration test
  • Manual test (add detailed scripts or steps below)

Code changes

  • Has exported function/method change

Release notes:

cluster: adjust file transferring with remote server to use native SCP

@AstroProfundis AstroProfundis added type/enhancement Categorizes issue or PR as related to an enhancement. component/cluster Issues about the tiup-cluster component labels May 24, 2021
@AstroProfundis AstroProfundis added this to the v1.5.0 milestone May 24, 2021
@AstroProfundis AstroProfundis requested a review from lucklove May 24, 2021 08:30
@AstroProfundis AstroProfundis self-assigned this May 24, 2021
@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 24, 2021
@codecov-commenter
Copy link

codecov-commenter commented May 24, 2021

Codecov Report

Merging #1382 (bb5d8ff) into master (fdadd94) will increase coverage by 28.79%.
The diff coverage is 36.97%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #1382       +/-   ##
===========================================
+ Coverage   25.34%   54.14%   +28.79%     
===========================================
  Files         265      291       +26     
  Lines       20517    22466     +1949     
===========================================
+ Hits         5201    12165     +6964     
+ Misses      14517     8370     -6147     
- Partials      799     1931     +1132     
Flag Coverage Δ
cluster 41.83% <35.96%> (?)
dm 24.55% <5.93%> (?)
integrate 48.70% <36.13%> (+33.10%) ⬆️
playground 13.74% <0.00%> (?)
tiup 15.50% <0.00%> (-0.10%) ⬇️
unittest 21.57% <3.38%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
components/dm/spec/logic.go 78.65% <0.00%> (+78.04%) ⬆️
pkg/cluster/ansible/config.go 0.00% <0.00%> (ø)
pkg/cluster/ctxt/context.go 89.09% <ø> (+89.09%) ⬆️
pkg/cluster/executor/local.go 55.71% <ø> (ø)
pkg/cluster/operation/telemetry.go 0.00% <0.00%> (ø)
pkg/cluster/spec/alertmanager.go 66.12% <0.00%> (+58.06%) ⬆️
pkg/cluster/spec/cdc.go 75.75% <0.00%> (+72.72%) ⬆️
pkg/cluster/spec/drainer.go 58.76% <0.00%> (+56.70%) ⬆️
pkg/cluster/spec/pd.go 68.26% <0.00%> (+53.29%) ⬆️
pkg/cluster/spec/pump.go 69.14% <0.00%> (+48.93%) ⬆️
... and 239 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fdadd94...bb5d8ff. Read the comment docs.

@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 25, 2021
@@ -46,6 +46,7 @@ var (
checkpoint.Field("src", reflect.DeepEqual),
checkpoint.Field("dst", reflect.DeepEqual),
checkpoint.Field("download", reflect.DeepEqual),
checkpoint.Field("limit", reflect.DeepEqual),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not needed.

@lucklove
Copy link
Member

/lgtm

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lucklove

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label May 25, 2021
@lucklove
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: bb5d8ff

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 25, 2021
@ti-chi-bot ti-chi-bot merged commit 876377b into pingcap:master May 25, 2021
@AstroProfundis AstroProfundis deleted the scp branch May 26, 2021 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/cluster Issues about the tiup-cluster component size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT1 Indicates that a PR has LGTM 1. type/enhancement Categorizes issue or PR as related to an enhancement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants