dvc.yaml: parametrize imports

[tillbier]

When importing dependencies via the dvc import feature, I want to be able to set a revision for a stage in the dvc.yaml, from which the dependency should be pulled, so that i can change the revision with the parametrisation feature.

This could look like the following:

dvc.yaml

vars:
  - params.yaml

stages:
  train:
    cmd: ./trainModel.sh
    deps:
    - images:
        revision: ${training.images.version}
    outs:
    - model.hdf5:

[efiop]

We are thinking about moving imports to dvc.yaml too #4841 , but so far we are thinking about a format that looks something like:

vars:
  - params.yaml

imports:
- images:
    repo: https://github.com/iterative/dvc
    path: data
    rev: ${training.images.version}

stages:
  train:
    cmd: ./trainModel.sh
    deps:
    - images
    outs:
    - model.hdf5

or similar, the main idea is that imports are separated from stages. Would something like that work for you?

[tillbier]

This looks like it would serve our needs pretty well. Thank you!

[efiop]

CC @skshetry WDYT?

[efiop]

For the record: had a discussion with @skshetry earlier and decided to wait with this likely after 2.0 release, there will be some preparations leading to that in the near future.

[karajan1001]

We are thinking about moving imports to dvc.yaml too #4841 , but so far we are thinking about a format that looks something like:

vars:
  - params.yaml

imports:
- images:
    repo: https://github.com/iterative/dvc
    path: data
    rev: ${training.images.version}

stages:
  train:
    cmd: ./trainModel.sh
    deps:
    - images
    outs:
    - model.hdf5

or similar, the main idea is that imports are separated from stages. Would something like that work for you?

@efiop
Excuse me, what about the stages from dvc add cmd?

[skshetry]

We haven't yet decided on that.

@karajan1001, what do you think? Would it to be good to get rid of .dvc files?

[karajan1001]

@skshetry @efiop

My opinion in #4278 (comment)

[karajan1001]

In tensorlfow we save a checkpoint into three files.

meta file: describes the saved graph structure,
data file: saves the values of all variables.
index file: ia string-string immutable table describes the metadata of a tensor: which of the "data" files contains the content of a tensor, the offset into that file, checksum, some auxiliary data, etc.

Here in DVC's DAG we have

dvc.yaml: describes the graph structure
dvc/cache: saves the values of nodes.
index file: two ways, distributed in .dvc files or centralized in some file like dvc.lock

centralization are more manageable and distribution are readable when no data existing on current workspace (For example in a Git repo). I don't know which is more important.