Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make dvc repro STAGE running all parametrized stages #4958

Closed
johnnychen94 opened this issue Nov 24, 2020 · 3 comments · Fixed by #4976
Closed

make dvc repro STAGE running all parametrized stages #4958

johnnychen94 opened this issue Nov 24, 2020 · 3 comments · Fixed by #4976
Assignees
Labels
A: templating Related to the templating feature feature request Requesting a new feature

Comments

@johnnychen94
Copy link
Contributor

johnnychen94 commented Nov 24, 2020

stages:
  build:
    foreach:
      - 15
      - 25
      - 35
    do:
      cmd: echo ${item}
jc@mathai3:~/Downloads/dvc_test$ dvc repro build@15
Running callback stage 'build@15' with command:
	echo 15
15
Use `dvc push` to send your updates to remote storage.

The following version doesn't work, even though build is defined in stages section.

jc@mathai3:~/Downloads/dvc_test$ dvc repro build
ERROR: "Stage 'build' not found inside 'dvc.yaml' file"

I wish we could make it equivalent to

for item in 15 25 35; do
    dvc repro build@$item
done

Furthermore, depends on whether there's a lock, multiple stages can be run in parallel.

@skshetry
Copy link
Member

Hi, @johnnychen94. Glad to see an issue for the parameterization (and, seeing someone using it 😄). I am working on this feature and will be implemented around the pre-release (which might be backwards-incompatible).

@skshetry skshetry added feature request Requesting a new feature A: templating Related to the templating feature labels Nov 24, 2020
@skshetry skshetry self-assigned this Nov 24, 2020
skshetry added a commit to skshetry/dvc that referenced this issue Nov 26, 2020
@skshetry
Copy link
Member

@johnnychen94, I have created #4976 to implement this. Your feedback would be greatly appreciated.

@johnnychen94
Copy link
Contributor Author

johnnychen94 commented Nov 26, 2020

It's not closed related so I put my thoughts in #4979 on what I'm expecting.


I'm impressed with how fast you DVC guys respond and the schedule you've made. Definitely something I should introduce to the Julia community 😆

skshetry added a commit that referenced this issue Nov 27, 2020
* repro: support regex/foreach-group to run at once

Fixes #4912
Fixes #4886
Fixes #4958

* Use `tree.isdir` rather than `os.path.isdir`

* Use glob rather than regex

* Update dvc/command/repro.py

* s/regex/glob

* disable glob on `collect_granular`

There's no need for a glob here

* add tests for `collect` and `collect_granular`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: templating Related to the templating feature feature request Requesting a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants