store whole DAG in one DVC-file

[Casyfill]

I understand the merits of having multiple .dvc files for complex processes,
but it would be just great to have the option to store the whole DAG in one Dvcfile!

I feel it might help the overall readability of the structure

I might be overestimating this, but it looks to me that we would need to do a huge refactor to support this, or at least not treating Dvcfile as a stage and create a separate class to deal with the logic of parsing such file to come up with a pipeline.

I can see the analogy with make on this one, but from my perspective, dvc is a CLI tool to write, read and execute (stage) files, whereas make is reduced to read and control de execution of (make) files.

I do appreciate the aesthetics of having everything on a single file, tho.

[Casyfill]

I see.
as a last resort, perhaps it is at least possible to add paths to the previous steps as comments in the file?

[shcheklein]

@Casyfill I think we will release a new version very soon (next week? @efiop can confirm) that will be preserving comments in the DVC files. I'm not sure we should be generating these comments on the DVC side though - it's hard to keep them up to date. What editor are using? May be it's a good feature to support on the editor level - navigate up/down the pipeline cc @prihoda

[shcheklein]

Just for the context - we got a request on the ods.ai #tool_dvc channel to support this:

(translation is mine)

Hi! Is there a way (feature/third-party tool/script/...) to describe the whole dvc-pipeline as a single file (as it is happening in build tools - make, bazel, maven, etc) instead of having a file per stage

[salotz]

I can see the analogy with make on this one, but from my perspective, dvc is a CLI tool to write, read and execute (stage) files, whereas make is reduced to read and control de execution of (make) files.

The single-file approach could be built upon the CLI approach as an extra optional layer. I suggested something like tup as another syntax option to make, which I happen to like. It inverts the dependency graph to be a flow graph and they look like unix pipes. It also can be faster due to the way it checks for updates.

I do appreciate the aesthetics of having everything on a single file, tho.

It is more than just about aesthetics (although great aesthetics in a build tool would be nice). Its a complex problem and waterbed theory is real.
If you only want to have dvc be analogous to gcc I suppose that is fine, but as projects get more complex you will want some sort of build tool like make, scons, waf, redo and so on. Eventually people will make the same thing in some ad hoc way bad way. So I see it as an inevitability that some sort of build automation tool will come about. I will probably be integrating them into my invoke tasks so that I can take care of pipeline things that aren't directly related to ML pipelines and wouldn't require dvc run. You could consider integrating or pairing with scons or waf which are both python projects.

Since it is a python project there is always the option of making pipelines in python, even though the syntax suffers a little bit. The way that prefect does their pipelines isn't so bad using a context manager.

Not really a solution since not everyone uses emacs and you can't emulate the awesome UI features it has on all platforms but magit is really the only way I interact with git and makes it very easy. A graphical interface could also solve some of the issues without sacrificing the versionability of generated files and not introducing new files.

My two cents 😄 and also this essay is awesome reading for exactly this kind of stuff https://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/