Support for circular dependencies - eg have model as both input and ouput of train stage

[davesteps]

It quite common when training deep learning models not to start training from scratch every time but rather use a previous model as initial starting weights. For example you may train an basic model then make some adjustments to your training data or your loss function or adjust some other parameter and continue training.

I am currently using a solution suggested on the forum of using import to import a previous version of my model. However, it was also suggested that a better solution could be supported where the model object could be both an input and output of your train stage.

The original question for more context:
https://discuss.dvc.org/t/use-previous-version-of-model-as-initial-weights/525/2

[karajan1001]

I have the same problem. Currently, I didn't put model into the dependency of the train stage and just maintain it manually.

[efiop]

@davesteps @karajan1001 We have an upcoming experimental feature for experiment checkpoints #4498 , maybe it would be useful in your scenario as well?

[davesteps]

@efiop It sounds like the checkpoints feature is very close to what I would like to achieve. I'm certainly willing to give it a try when it is available. I also wasn't aware of the --outs-persist option which could be useful. Thanks.

[efiop]

@davesteps There is no official documentation yet and it is still an experimental feature, but we have a wiki for it https://github.com/iterative/dvc/wiki/Experiments#checkpoint-experiments . Feel free to give it a try 🙂

[efiop]

@davesteps Checkpoints have been released in 2.0, did you have a chance to try them?

[efiop]

Closing as stale for now.