Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvc.yaml: introduce set keyword #4757

Merged
merged 1 commit into from
Nov 2, 2020
Merged

Conversation

skshetry
Copy link
Member

@skshetry skshetry commented Oct 21, 2020

Thank you for the contribution - we'll try to review it as soon as possible. πŸ™

On top of #4734
Here's a wiki documenting the feature: https://github.com/iterative/dvc/wiki/Parametrization

This PR introduces set keyword, which can be used to alias a variable.
Some examples:

  1. Setting variable inside for a stage to be DRY
stages:
  build:
    set:
      item: "value"
    cmd: python script.py --thresh ${item}
    always_changed: ${item}
  1. set can set list to a variable. But, it is not allowed to contain nested list or nested dictionary inside of it.
stages:
  build:
    set:
      item:
      - foo
      - bar
      - baz
      thresh: 10
    cmd: python script.py --thresh ${thresh}
    # note: DVC does not allow to change schema like this right now
    # just an example of how you could potentially use this
    outs: ${item}
  1. set can set dict to a variable. But, it is not allowed to contain nested list or nested dictionary inside of it.
stages:
  build:
    set:
      item:
        foo: foo
        bar: bar
      thresh: 10
    cmd: python script.py --thresh ${thresh}
    outs: ${item}
  1. set to create an alias
use: params.json
stages:
  build:
    set:  # optional, loop-wide effect
      data: ${models}
    foreach: ${data}
    in:
      set:  # optional, affects only one instance/iteration
        thresh: ${item.thresh}
      cmd: command --value ${thresh}

NOTE: set is always evaluated first, before foreach, so it's available for iteration on foreach. Similarly, cmd and friends are also evaluated later after set and foreach( if it exists).

Also, note that joining strings is not supported with "set":
eg:

set: 
  # not supported
   url: ${base_url}/${filename}
   # also, not supported
   name: My name is ${name}

@skshetry skshetry added feature is a feature skip-changelog Skips changelog labels Oct 21, 2020
@skshetry skshetry self-assigned this Oct 21, 2020
@skshetry skshetry merged commit 6a9ab9c into iterative:master Nov 2, 2020
@skshetry skshetry deleted the set-dvcyaml branch November 2, 2020 14:47
@skshetry skshetry added the A: templating Related to the templating feature label Dec 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: templating Related to the templating feature feature is a feature skip-changelog Skips changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant