Pipeline variables from params file

[dmpetrov]

With the introduction of the new multiple-stage pipeline, we will need to find a way of defining variables in the pipeline. For example, the intermediate file name cleansed.csv is used from two stages in the following pipeline and it needs to be defined into a variable:

stages:
    process:
        cmd: "./process.bin --input data --output cleansed.csv"
        deps:
             - path: data/
        outs:
             - path: cleansed.csv

    train:
        cmd: "python train.py"
        deps:
             - path: cleansed.csv
             - path: train.py
             - path: params.yaml
                params:
                     lr: 0.042
                     layers: 8
                     classes: 4
        outs:
             - path: model.pkl
             - path: log.csv
                cache: true
             - path: summary.json

We need to solve two problems here:

1. Define a variable in one place and reuse it from multiple places/stages.
2. Often users prefer to read file names from config files (like in the train stage), not from the command line (like in the process stage).

We can solve both of the problems using a single abstraction - parameters file variable:

stages:
    process:
        cmd: ./process.bin
        outs:
             - path: "params.yaml:cleansed_file_name"
        ....
    train:
        cmd: "python train.py"
        deps:
             - path: "params.yaml:cleansed_file_name"

This feature is useful in the current DVC design as well. It is convenient to read file names from params file and still define dependency properly like dvc run -d params.yaml:input_file -o params.yaml:model.pkl

[karajan1001]

In my case I stored these global variables (vocabulary file, cleaned data file, label name file) in a configuration file as default. And the processing cli can accept all these variables to override the defualt one. But my configuration file parsing and merging is coded inside the program, and had to be re-coded in every program.

[tall-josh]

Before I knew about multiple-stage pipelines I used the params functionality and hacked together a script to build a richer dvc run command. I used the same dot.notation you can use in the params command line args ie: dvc run -p params.yaml:data.output,train ... In the end my params.yaml looks like:

data:
    url: s3://mydata.npz
    out_path: data/mydata.npz
data/-o:  # Each element in this list will be
          # added as a  -o <arg>
    # For access strings within the current stage
    # you can also use '.out_path'
    - data.out_path
data/script:
    - python src/cli/data_download.py
    - --data-size
    - .url  # equivalent to data.url
    - --out-path
    - .out_path # equivalent to data.out_path

train:
  lr: 0.001
  ckpts: arteftacts/ckpts
train/-d:
  - data.out_path
train/-o:
  - artefacts/tflogs
train/script:
  - python train.py
  - --data
  - data.out_path
  - --ckpts
  - .ckpts
  - --lr
  - .lr

I'm not suggesting the multi-stage pipeline changes. I guess all I'm saying is:

• I like that dot.notation and,
• I like the idea of self-referencing inside the same yaml file.

After Thought

After sleeping on it I thought, 'I can use jinja2 for this':

stages:
  data:
    cmd: python src/cli/data_download.py --data-size {{ args.data.size }} --out-path {{ args.data.out_path }}
    wdir: ..
    outs:
    - {{ args.data.out_path }}
  train:
    cmd: python src/cli/train.py -d {{ args.data.out_path }} --params {{ args.train.params }} --log-dir {{ args.train.lodir }} --metrics-path {{ args.train.metrics }} --ckpts {{ args.train.ckpt_path }}
    wdir: ..
    deps:
    - {{ args.data.out_path }}
    - {{ args.train.params }}
    metrics:
    - {{ args.train.metrics }}
    outs:
    - {{ args.train.logdir }}
    - {{ args.train.ckpt_dir }}

I then load my args.yaml file and convert it into a namedtuple (so I can use the dot.notation). Bingo bango, we have our filled in .dvc file

[dsuess]

Unfortunately, YAML anchors can only solve part of this: You can use them to fill in repeating dependencies like this:

stages:
    process:
        outs:
             - path: &cleansed cleansed.csv
       ...

    train:
        deps:
             - path: *cleansed
        ...

However, there doesn't seem to be an easy way to also "paste" them into a full command, unless we write some custom YAML parsing.

[dmpetrov]

@dsuess can jinja2-like approach from @tall-josh potentially solve this issue?

[dsuess]

@dmpetrov I like the idea, but I see two drawbacks:

1. the necessary information is split across two files
2. allowing full jinja-templating on the configuration file might be too much

To alleviate the first, maybe we can add a special section variables to the Dvcfile like this:

variables:
  data:
    size: 10
    out_path: ./data/
  ...
  
stages:
  data:
    cmd: python src/cli/data_download.py --data-size {{ args.data.size }} --out-path {{ args.data.out_path }}
    wdir: ..
    outs:
    - {{ args.data.out_path }}
  train:
     ...

and then only template the stages section with those variables. No templating would be allowed for that first section.

For the second one, we could either look into not allowing certain functionality (like conditionals or loops) in jinja2. Alternatively, we need to implement our own simple templating engine that only supports a very limited sets of features. So far, I only see variable substitution to be necessary.

Alternatively, we could implement our own PyYaml tag to support pasting like this

[karajan1001]

@dsuess, good idea, but there is a questionvariables need to be shared among DVCfiles, many of them ( for me embedding size, vocabulary file, sequence length and etc.) must stay the same in all stages. Maybe we should make the DVCfile inherit parameters from its ancestors.

[tall-josh]

There are definitely use cases for both in-file variables and external variable files. I agree with @dsuess in that full templating is probably not the way to go as a proper solution.
Maybe it's just a terminology thing but @karajan1001, if your thinking full blown inheritance then I think things could get messy. A simple way of importing from 1 or more files cloud be nice.

vars:
  IMPORT:
    somevars: some/vars.yaml
    morevars: more/vars.yaml

  data:
    size: 10
    out_path: ./data/
  ...
  
stages:
  data:
    cmd: python src/cli/data_download.py --data-size {{ vars.data.size }} --out-path {{ vars.data.out_path }}
    wdir: ..
    outs:
    - {{ args.data.out_path }}
  train:
     ...
    outs:
    - {{ somevars.something.x }}
    - {{ morevars.someotherthing.y }}

[dmpetrov]

@dsuess the variables section makes total sense. I see that as a simple form of variables with no param file associated. We definitely need it and this part has a higher priority.

It looks like all of us on the same page regarding Jinja - full support is too much.

Also, I like @tall-josh's idea of importing which unifies the explicit vars definition:

vars:
  IMPORT:
    somevars: some/vars.yaml
    morevars: more/vars.yaml

It looks like the high-level design is ready :) and we need to agree on the syntax and then a templating engine.

On the syntax side, I see these options:

1. Special patter in strings like "$vars.somevar"
2. jinja2-style variables: {{ vars.somevar }}
3. PyYaml tag: !var [*vars.somevar]
4. other options

What are your thoughts on this?

(3) seems like a more formalized way of doing (1)

PS: dvc run can use a bit different syntax or we can decide not to support variables on the command level - let's keep this discussion separately.

[tall-josh]

Hi @dmpetrov
Yep, we're on the same page. I like the first syntax simply because it's concise.

[dsuess]

@karajan1001 Agreed. I think this change make most sense with #3584

@dmpetrov If we want to share variables between files, I think @tall-josh solution is better than just implicitly passing variables through from the dependencies. That would also cover the use case of having a single variables-file that's shared throughout the repo. However, I also think that not passing them between files wouldn't be a problem, if the multi-stage files from #3584 are the new standard. This way, files would scope variables nicely.

Regarding the syntax: I don't have any preference as long as the files are still valid YAML. The first one will interfere with standard shell variable substitution in the command, which might be a good thing as using non-tracked variables in the command will break reproducibility. It might also be a bad thing if there's a valid use case for having shell-variables in the command.

[tall-josh]

@dsuess I'm not sure multi-stage is quite at the point of being the new standard, though I think it will get there soon. Someone correct me if I'm wrong, but as of a few days ago multi-stage ( #3584 ) did not support --params :-( . Something I'm been meaning to bring up their thread. I'll do that now I guess.

[dmpetrov]

@dsuess agree, defining variables from command line is not the high priority feature. We should have a great solution in the pipeline file. Also, a good point about env variables.

@tall-josh it would be great to hear your params ideas! You can post it on the multi-stage issue or just here.