Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvc exp run --queue needs absolute path #8747

Open
walternat1ve opened this issue Dec 29, 2022 · 7 comments
Open

dvc exp run --queue needs absolute path #8747

walternat1ve opened this issue Dec 29, 2022 · 7 comments
Labels
A: experiments Related to dvc exp

Comments

@walternat1ve
Copy link

walternat1ve commented Dec 29, 2022

dvc exp run --queue ... needs absolute path for the file in the command, otherwise queued experiments will fail because they cannot find the file from the command (if relative).
while
dvc exp run works also with relative path

maybe i missed it but this was nowhere documented imo and tbh it's a bit inconsistent.

@pmrowla
Copy link
Contributor

pmrowla commented Dec 30, 2022

Are you referring to file arguments passed to commands in your dvc.yaml stages?

Using exp run --queue or exp run --temp runs your experiment in a temporary directory separate from your main repo workspace. If the files are in your repo and are tracked by git or DVC, relative paths will still work, since those files are all included in the temporary workspace. But as you noted, if your command takes a path to a file that is not tracked at all in your repo, you will need to use the absolute path to that file.

@pmrowla pmrowla added awaiting response we are waiting for your reply, please respond! :) A: experiments Related to dvc exp labels Dec 30, 2022
@walternat1ve
Copy link
Author

you are right, the file was not tracked. now it works. thx for clearing that up. however, when creating a queue of 3 experiments with varying parameters, they all had the same name. thus resulted in only 1 exp when using "dvc exp show". i thought each one would get a unique name automatically.

@karajan1001
Copy link
Contributor

karajan1001 commented Jan 18, 2023

@walternat1ve Could you please elaborate it? You dvc.yaml file? And the dvc exp run command.

@walternat1ve
Copy link
Author

all good. the problem was that i changed the params for each queued experiment but didnt provide them as a dependency in dvc.yaml, therefore they were not treated as new experiments but rather i got the cached result and thus the same name. now it makes sense to me. thx for coming back to this issue.

@pmrowla pmrowla removed the awaiting response we are waiting for your reply, please respond! :) label Jan 19, 2023
@walternat1ve
Copy link
Author

walternat1ve commented Oct 17, 2023

what can i do when a script or file is in a submodule which is tracked by git? it doesn't copy the content to tmp unfortunately

@walternat1ve walternat1ve reopened this Oct 17, 2023
@pmrowla
Copy link
Contributor

pmrowla commented Oct 17, 2023

submodules arent currently supported (#7186, #5855) but you should be able to use -C/--copy-paths to copy the specify files you need into the tempdir

@walternat1ve
Copy link
Author

walternat1ve commented Oct 17, 2023

thx for the quick reply, i will try this workaround.

edit: @pmrowla i was not able to apply the workaround. i tried adding the full directory of submodule but it said its already there (as far as i can tell the directory was there but not the content). it's probably not feasible to specify each single file. what do you think is the best way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: experiments Related to dvc exp
Projects
None yet
Development

No branches or pull requests

3 participants