2.0 Pipelines (parametrization)

[jorgeorpinel]

Based on https://github.com/iterative/dvc/wiki/Parametrization

• DVC Files guide - dvc.yaml section

  • General vars info
  • Parameters files
  • Global/Local vars
  • Multi-stages (foreach "loops")
  • Specific/multiple/partial params files in vars
  • Other notes e.g. that "parametrized variables are also displayed by params diff" and about wdir.

• params cmd ref.

• term "parametrize" -> "parametErize" + intro "templat(e|ing)"

• Follow this up with guide: make proper DVC File guide + separate reference? #2059 immediately after.

• Pending: terminology research (see 2.0 Pipelines (parametrization) #2052 (review)).

[jorgeorpinel]

Note: This example has no changes (which would be out of scope) but I moved the one about Python param files to the bottom so it looks like this is added.

It's the next example the one I added.

[shcheklein]

just to check, how correct the substitution expression term is?

[shcheklein]

it also feels that we should define this way before we jumped into different options for vars

[jorgeorpinel]

how correct the substitution expression term is?

Will research ⌛

should define this way before

We do intro the term early on before using ${} in any example. Do you mean putting "substitution expression supports these forms..." before dvc.yaml examples?

[jorgeorpinel]

OK @shcheklein, on the question of the "substitution expression" I here some the alternatives:

• We call it "${} expression" and use "interpolation" to explain it in the original wiki page, but that latter term seemed too mathematical to me.
• Helm (K8s YAML) calls it a "directive" (e.g. {{ .Something }}) and uses words "value", "generated", and "render/inject/insert" for the explanations.

  BTW apparently there are some strong opinions ([1], [2]) out there against templating YAML 😆

• Kustomize (YAML) uses a different approach (template-free) with terms like "generate", "variants" and "overlays" — not directly comparable but listing here because it's an interesting alternative, maybe we can consider adopting something like that in the future.

  See also yq

• Handlebars (HTML/JS) calls them "handlebars expressions" (e.g. {{prop}}); uses "generate/render", "property/value", "replace"
• Twig/Django (HTML/PHP): "tags/filters" (e.g. {% var %}, {{ function }}); "render", "generate", "values"
• Mustache (Ruby, JS, etc.): "tags" (e.g. {{key}}); "render", "replace", "values".
• Go (text): "actions" (e.g. {{ "string" }}); "values", "evaluation", "generate"
• Bash calls $(command) a "command substitution"

Conclusions:

• "Substitution" is not very used out there, I guess the $ in our syntax reminded me more of Bash than of modern templating engines.
• For ${}, just "expression" is OK. Most templating formats have a special term though, usually "tag" for value injections.
• For related terms, I think in our case "replace" and "generate" are the most appropriate.

[shcheklein]

how about foreach/do - using loops to generate stages

[jorgeorpinel]

I'm not sure we want to introduce the concept of "loops" (even when its implied by the foreach field name); According to treeverse/dvc#5181 (comment), the items given to foreach are not considered ordered so it doesn't seem like a normal loop to me.

[jorgeorpinel]

(This title is Generating multiple stages now BTW.)

[shcheklein]

Okay, it's impossible to iterate on this PR further (GH breaks at this stage, it's impossible to follow all the comments, etc). Let's merge and work separately on the next version. The biggest concern is still structure:

All these titles do not look good to me. They are fine for Basic Concepts I guess, but not as a top level section in the User Guide. They should higher level.

After that we define the expected structure we can start iterating on the actual content.

• advanced-dvc.yaml - is too heavy and a mix of a reference (precise spec) and guide (titles are high level, no structure - story like)
• do we need two seprate dvc.yaml sections? why do we consider basic templating and advanced feature?
• .dvc file - is clearly a reference thing. Where do we put it?

[shcheklein]

@skshetry I agree that structure is not ideal, we'll iterate on it next thing. Let's merge it and do it in the next PR, this is one is not manageable.

Re the split/non-split - to the large extent if feels that it's a matter of preferences and we won't find a solution that fits all. I think the current approach is to keep pages reasonably short, to a single point (e.g. it's bad to have .dvc and the whole dvc.yaml - they are too different concepts. Can dvc.yaml fit a single page - good question-let's see in the next PR. I really hope so, I'm though worried that about some points I describe below, e.g. complexity of the right hand bar).

Usual reasons to split (besides personal preferences, which should not serve as an argument unless we do a focus group testing):

• have a clear link to separate page that describes one thing (good for SEO, since term becomes part of the URL, page has tailored meta section, etc)
• when you have too much on a single page- you won't have any navigation at all (due to multiple levels of nesting). Or we'll have to complicate right hand side bar.

Why is it neglected so much?

Nothing is neglected (e.g. we changed Get started when we had time from 10+ pages to 4). But I find traversing through those pages difficult is not an argument that we can use as a guideline to my mind. If you have some good structure or guideline in mind, please suggest it.

[jorgeorpinel]

Merged and moved discussion to #2059 (comment).

[jorgeorpinel]

It feels like we are splitting pages arbitrarily (just look at installation instructions).

@skshetry I don't see how Install pages are spit arbitrarily. But if you'd like to elaborate or have better examples feel free to open a separate issue. Thanks