Skip to content

dvc.yaml: introduce set keyword#4757

Merged
skshetry merged 1 commit into
treeverse:masterfrom
skshetry:set-dvcyaml
Nov 2, 2020
Merged

dvc.yaml: introduce set keyword#4757
skshetry merged 1 commit into
treeverse:masterfrom
skshetry:set-dvcyaml

Conversation

@skshetry
Copy link
Copy Markdown
Collaborator

@skshetry skshetry commented Oct 21, 2020

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

On top of #4734
Here's a wiki documenting the feature: https://github.com/iterative/dvc/wiki/Parametrization

This PR introduces set keyword, which can be used to alias a variable.
Some examples:

  1. Setting variable inside for a stage to be DRY
stages:
  build:
    set:
      item: "value"
    cmd: python script.py --thresh ${item}
    always_changed: ${item}
  1. set can set list to a variable. But, it is not allowed to contain nested list or nested dictionary inside of it.
stages:
  build:
    set:
      item:
      - foo
      - bar
      - baz
      thresh: 10
    cmd: python script.py --thresh ${thresh}
    # note: DVC does not allow to change schema like this right now
    # just an example of how you could potentially use this
    outs: ${item}
  1. set can set dict to a variable. But, it is not allowed to contain nested list or nested dictionary inside of it.
stages:
  build:
    set:
      item:
        foo: foo
        bar: bar
      thresh: 10
    cmd: python script.py --thresh ${thresh}
    outs: ${item}
  1. set to create an alias
use: params.json
stages:
  build:
    set:  # optional, loop-wide effect
      data: ${models}
    foreach: ${data}
    in:
      set:  # optional, affects only one instance/iteration
        thresh: ${item.thresh}
      cmd: command --value ${thresh}

NOTE: set is always evaluated first, before foreach, so it's available for iteration on foreach. Similarly, cmd and friends are also evaluated later after set and foreach( if it exists).

Also, note that joining strings is not supported with "set":
eg:

set: 
  # not supported
   url: ${base_url}/${filename}
   # also, not supported
   name: My name is ${name}

@skshetry skshetry added feature is a feature skip-changelog Skips changelog labels Oct 21, 2020
@skshetry skshetry self-assigned this Oct 21, 2020
@skshetry skshetry merged commit 6a9ab9c into treeverse:master Nov 2, 2020
@skshetry skshetry deleted the set-dvcyaml branch November 2, 2020 14:47
@skshetry skshetry added the A: templating Related to the templating feature label Dec 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A: templating Related to the templating feature feature is a feature skip-changelog Skips changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants