pipelines: clarify whether it's possible to have more than one _pipeline file_ for a DVC project in the docs

[iesahin]

There are some places like content/docs/user-guide/project-structure/pipelines-files.md and content/docs/user-guide/project-structure/index.md that talks about pipeline files while describing dvc.yaml. e.g.

dvc.yaml pipelines files define stages that form the pipeline(s) of a
project. All stage-based features such as dvc params, dvc metrics, and
dvc plots are specified here.

AFAICT there is a single file with multiple pipelines and commands don't receive a pipeline file argument to replace the default dvc.yaml. In these descriptions we may need to clarify that there is a single file where we describe all pipelines. WDYT @jorgeorpinel?

[jorgeorpinel]

like content/docs/user-guide/project-structure/pipelines-files.md

We already start the dvc.yaml guide with a pretty clear indication about this:

and content/docs/user-guide/project-structure/index.md

"pipelines files define" is plural as well.

It would be good to review most mentions of dvc.yaml throughout docs to double check that at least we don't speak about it in singular terms (except in specific examples where it is just one) but otherwise I'm not sure if there's need to further clarify this. At some point we may specifically explain a project structure pattern with multiple dvc.yaml files (perhaps in a future Best Practices section, see #72).

commands don't receive a pipeline file argument to replace the default dvc.yaml

Actually commands that work with stages accept targets which can be a path to another dvc.yaml file. See for example https://dvc.org/doc/command-reference/repro#options but that's the only ref. where we treat the cmd argument (targets) as an "Option" and explain it in detail. Maybe we should focus on apply that to other refs, instead for this issue? (please change the title and desc here if you agree).

[iesahin]

It would be good to review most mentions of dvc.yaml throughout docs to double check that at least we don't speak about it in singular terms (except in specific examples where it is just one) but otherwise I'm not sure if there's need to further clarify this.

I think we need to change dvc.yaml mentions to */dvc.yaml to show that we are talking about possibly multiple files in different dirs where necessary. When a document mentions a particular filename I tend to think it's about a single file for the whole repository, e.g. .dvc is a single dir for the repository and even if you say pipelines files my mind sees a typo there, instead of multiple files in different dirs. 🤦🏼 😄

I'll be more careful about this. Thanks.

[iesahin]

For example in https://raw.githubusercontent.com/iterative/dvc.org/master/content/blog/2020-06-22-dvc-1-0-release.md

In DVC 1.0, the DVC metafile format was changed in three big ways. First,
instead of multiple DVC "stage files" (*.dvc), each project has a single
dvc.yaml file. By default, all stages go in this single YAML file.

makes me think there is a single global dvc.yaml file for each project and since project is the directory where .dvc/ is present, I think this is a valid interpretation.

[iesahin]

Evolution of my understanding can be here :) a039900?branch=a039900ed35d31380e9bd6f7a5ef3738ce24d595&diff=split

💡

[shcheklein]

@iesahin looks good to me! let's make a PR for this?

[jorgeorpinel]

change dvc.yaml mentions to */dvc.yaml

I'm not sure. That language is too technical and terminal-like. dvc.yaml is a schema, not a specific file.

I think it's OK to speak about it in singular like "change field xyz in your dvc.yaml" — it doesn't imply there's one only ever. I don't remember many users having this confusion based on support cases either. Same with *.dvc although that one I like a bit more.

For now someone could definitely try to review all dvc.yaml mentions (there's a lot though) to make sure the context doesn't imply that constraint.

.dvc is a single dir for the repository

Good point but that's rarely mentioned and people used to Git and many other tools probably assume there's just one such hidden dir per project. And in its docs we clearly state there's just one.

if you say pipelines files my mind sees a typo

I agree it's a strange term. Maybe it should be "pipeline file(s)" but each dvc.yaml can contain multiple pipelines so that plural is correct. We discussed this back in the day and couldn't come up with a better answer, open to suggestions.

[iesahin]

There are over 900 references to dvc.yaml and I read maybe the first few hundred. There is specifically an answered question for this in the blog. I think I can close this as no major reference to a single dvc.yaml seems to be in the docs. Thank you @jorgeorpinel

[nxorable]

perhaps in a future Best Practices section

I would have greatly benefited from examples of project structure with multiple dvc.yaml files in a best practices section. This thread was what I had to find to get a clue I was going in the right direction, and then I eventually figured out something that worked through trial and error.