Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

external_repo: use RepoTree for fetch/import/get #3841

Merged
merged 22 commits into from
May 22, 2020

Conversation

pmrowla
Copy link
Contributor

@pmrowla pmrowla commented May 21, 2020

  • ❗ I have followed the Contributing to DVC checklist.

  • πŸ“– If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here. If the CLI API is changed, I have updated tab completion scripts.

  • ❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)

Thank you for the contribution - we'll try to review it as soon as possible. πŸ™

Related to #3811.
Will fix #3611.

Changes:

  • erepo: use GitTree instead of git checkout for cloned repos (unless dvcx for_write=True)
  • erepo: use RepoTree/cache.save() for fetching external DVC+ git files
  • erepo: remove pull_to
    • RepoDependency: use erepo.fetch_external() -> cache.checkout with host repo cache
    • get: use erepo.fetch_external() -> cache.checkout() with erepo tmpdir cache
  • ls: use RepoTree

@pmrowla pmrowla force-pushed the erepo-repo-tree branch from 211db8c to 95ea7e2 Compare May 22, 2020 04:50
@pmrowla pmrowla force-pushed the erepo-repo-tree branch from e1d94de to aea0346 Compare May 22, 2020 07:09
* otherwise md5 checksums will not match when comparing git version to
  cached filesystem version on windows
@pmrowla pmrowla force-pushed the erepo-repo-tree branch from aea0346 to f6bf5bf Compare May 22, 2020 07:24
@pmrowla pmrowla changed the title [WIP] external_repo: use RepoTree for fetch/import/get external_repo: use RepoTree for fetch/import/get May 22, 2020
@pmrowla pmrowla requested review from efiop and skshetry May 22, 2020 07:42
@pmrowla pmrowla requested a review from pared May 22, 2020 07:42
Copy link
Contributor

@efiop efiop left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great!

For the record: discussed next steps to possibly unify and abstract trees by, possibly, splitting BaseRemote into one Remote + S3Tree(and so on for each type) + Cache.

@efiop efiop merged commit b7cd785 into iterative:master May 22, 2020
Copy link
Contributor

@pared pared left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pmrowla pmrowla deleted the erepo-repo-tree branch May 22, 2020 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

external_repo doesn't respect parent cache and progress bar
3 participants