DVC pipeline parallel steps

[rahulguptajss]

Does DVC supports parallel steps in its pipeline. I am unable to create parallel steps using dvc command.

[efiop]

No, dvc doesn't support parallel execution of stages as of today. We do have plans to support it in the future.

[rahulguptajss]

Can you please provide any timeline for this.

Thanks.

[efiop]

0.9.8 most certainly. So less then a month or so.

[rahulguptajss]

Thanks. It will be helpful when we have multiple models to be run on same data with separate py files.

[efiop]

Wait a minute, did you mean 'parallel execution' or just 'parallel steps'? We do not support parallel execution as of now, only executing stage-by-stage. However, parallel stages should work:

echo foo > foo
dvc add foo
dvc run -d foo -o out1 cp foo out1 # stage1
dvc run -d foo -o out2 cp foo out2 # stage2

or, with your model:

dvc add data
dvc run -d data -o out1 python model1.py
dvc run -d data -o out2 python model2.py

[rahulguptajss]

yes, actually for this thread , I was looking for parallel steps. I'll try this and update.

[efiop]

Closing as stale. Please feel free to reopen.

As for parallel execution, there is a #755 planned for 0.11.0.