dvc queue start: doesn't start exp, but raises JSONDecodeError
#9358
Labels
A: experiments
Related to dvc exp
Comments
vitalwarley
changed the title
dvc queue start: doesn't start exp, but raisesJSONDecodeErrordvc queue start: doesn't start exp, but raises JSONDecodeError
|
The error persists after updating DVC. It also shows with |
|
Have you tried |
|
@dberenbaum, I didn't, but after trying it the error continues Is there any other info you need to help us better diagnose the problem? |
|
Is it important to preserve what's in the queue, or would you be okay to regenerate the queue if you can get back to a working state? You could try to remove everything in |
|
I regenerated the queue after removing I only get failed experiments after some time. I also can't get the logs to inspect what happened |
|
I managed to get the logs from one experiment
In the benchmark-ocr:
foreach:
- test
- validation
do:
cmd: >-
python src/python/cvt/tools/main.py benchmark
...
deps:
- src/python/cvt/tools |
|
I removed the dep and tried again This path is also listed as a dependency deps:
- ${base.dataset-dir}/${base.scope}/${benchmark.cd-dir}/${item}/images
- ${base.dataset-dir}/${base.scope}/${benchmark.cr-dir}/${item}/labels
- ${base.models-dir}/${base.scope}/${benchmark.weights-cd}
- ${base.models-dir}/${base.scope}/${benchmark.weights-cr} |
|
@vitalwarley submodule dependencies are not supported in queue runs, see #7186 |
@pmrowla Would there be any unexpected side effect if |
|
@daavoo I think that should be OK as a workaround for now. Ideally we would handle submodules correctly when we generate the temp git workspace, but #7186 hasn't been addressed yet because it needs some more investigation/research. Basically I'm not currently sure what the correct way to handle unstaged or uncommitted changes in the submodule is, but technically I think we should be trying to include them in the experiment (and not just doing But copying the entire submodule with |
Ah, I see. Thanks for the info.
I reverted the submodule dependency removal and tried this parameter. It seems to solve the first issue (submodule listed as dep there), but the other (submodule removed as dep there) one happened again. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Description
I set up multiple experiments with Hydra range sweep, but I can't start them.
Reproduce
dvc exp run -s benchmark-ocr@validation -S 'benchmark.conf-thresh-cd=range(0.1, 0.9, 0.1)' -S 'benchmark.conf-thresh-cr=range(0.1, 0.9, 0.1)' --queuedvc queue startExpected
Running experiments.
Environment information
Output of
dvc doctor:$ dvc doctorAdditional Information (if any):
exp run output
queue start verbose output
The text was updated successfully, but these errors were encountered: