Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exp: add initial ssh executor #6833

Merged
merged 13 commits into from
Oct 26, 2021
Merged

exp: add initial ssh executor #6833

merged 13 commits into from
Oct 26, 2021

Conversation

pmrowla
Copy link
Contributor

@pmrowla pmrowla commented Oct 19, 2021 •
edited
Loading

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Will need rebase after #6797

Related: #6265, #6266

  • separates Git and DVC initialization in exp executors (previously it was all handled in a unified __init__ method)
  • adds initial SSHExecutor implementation
    • currently only supports executor (remote git+dvc) initialization
    • git and dvc (push) initialization is handled directly over ssh

@pmrowla pmrowla force-pushed the ssh-executor-init-v2 branch from e3e08fb to 828e43f Compare October 21, 2021 05:44
pmrowla
pmrowla commented Oct 21, 2021
View reviewed changes
dvc/repo/experiments/executor/ssh.py
)
with self.sshfs() as fs:
odb = get_odb(fs, fs.PATH_CLS(cache_url), **fs.config)
push(dvc, revs=[rev], run_cache=run_cache, odb=odb)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

push is called this way instead of dvc.push because mypy crashes otherwise (it can't type function definitions imported into a class the way we do it in Repo)

@pmrowla pmrowla force-pushed the ssh-executor-init-v2 branch from f02b19e to d553e9e Compare October 21, 2021 08:26
@pmrowla pmrowla self-assigned this Oct 21, 2021
@pmrowla pmrowla added the A: executors Related to the executors feature label Oct 21, 2021
@pmrowla
Copy link
Contributor Author

pmrowla commented Oct 21, 2021 •
edited
Loading

updated remote execution POC on top of this PR:

asciicast

@pmrowla pmrowla marked this pull request as ready for review October 21, 2021 09:10
@pmrowla pmrowla requested a review from a team as a code owner October 21, 2021 09:10
@pmrowla pmrowla requested a review from karajan1001 October 21, 2021 09:10
@pmrowla pmrowla changed the title [WIP] exp: add initial ssh executor exp: add initial ssh executor Oct 21, 2021
@pmrowla
Copy link
Contributor Author

pmrowla commented Oct 25, 2021

another POC demo including remote execution (dvc exp run) and git collection

asciicast

@pmrowla pmrowla merged commit 0d895e2 into iterative:master Oct 26, 2021
@pmrowla pmrowla deleted the ssh-executor-init-v2 branch October 26, 2021 07:08
@skshetry skshetry added the skip-changelog Skips changelog label Nov 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karajan1001 karajan1001 karajan1001 approved these changes

Assignees

@pmrowla pmrowla

Labels
A: executors Related to the executors feature skip-changelog Skips changelog
Projects
No open projects
DVC
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pmrowla @karajan1001 @skshetry