Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvc exp run: Checks out deps and outs of all stages even if not required by stage to run #5472

Closed
Tracked by #5367
schuhschuh opened this issue Feb 16, 2021 · 0 comments · Fixed by #5519
Closed
Tracked by #5367
Assignees
Labels
enhancement Enhances DVC p2-medium Medium priority, should be done, but less important

Comments

@schuhschuh
Copy link

Bug Report

Description

When executing dvc exp run (with or without --temp, also when specifying --single-item), it checks out all cached deps (and outs?) of stages listed in the dvc.yaml file of the stage specified as target, and also all stages in other dvc.yaml files of other pipelines in the same DVC project.

Reproduce

Expected

Command dvc run exp target should only checkout the data files specified as deps of the target to run, plus maybe all previous stages in the pipeline that produce these if necessary and when --single-item is not specified. Especially when using the --temp option (i.e., --queue + --run-all), this results in a long wait for dvc to check out many data files from unrelated pipelines / stages.

Environment information

Output of dvc version:

$ dvc version
DVC version: 2.0.0a0+273b02
---------------------------------
Platform: Python 3.7.9 on Linux-3.10.0-1127.8.2.el7.x86_64-x86_64-with-centos-7.8.2003-Core
Supports: http, https, s3

Additional Information (if any):

@pared pared added enhancement Enhances DVC p2-medium Medium priority, should be done, but less important labels Feb 16, 2021
@pmrowla pmrowla self-assigned this Feb 23, 2021
@dberenbaum dberenbaum mentioned this issue Feb 25, 2021
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhances DVC p2-medium Medium priority, should be done, but less important
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants