pipeline: show: doesn't show all DAGs in the projects when ran without arguments

[drorata]

As per discussion with @pared we realized that the behavior of dvc pipeline show --ascii contradicts the documentation. Current behavior looks for Dvcfile and fails if it is not found.

The expected behavior of dvc pipeline show --ascii is that it will find all .dvc files in the workspace and plot all pipelines. It does make sense IMHO that if a .dvc is provided then the yielded graph is showing the steps up to the file's corresponding stage.

[pared]

Some context
During update of help message for targets argument we introduced same help message for dvc pipeliens show as for other commands (add, checkout, commit). The problem is that it does not work similiar way as the other commands. As of today (0.54.1) if target has not been provided dvc pipelines show will look for Dvcfile. We should reconsider the behaviour, as Dvcfile is created in specific circumstances.

We need to either:

• modify behaviour to look for last possible stage (as it is intuitive behaviour to show whole pipeline) and modify help message accordingly
• roll back help message for pipelines show

[pared]

In case of first approach we need to consider what to do in case like this:

             get_data.dvc
           /               \
   test_data.dvc      train_data.dvc

[drorata]

I think that the most natural behavior is:

• if a target foo.dvc is provide then all steps preceding it will be shown
• if no target is provided then all .dvc files will be discovered and the corresponding pipeline(s) will be plotted.

If this approach will be picked, then the case mentioned by @pared should yield:

             get_data.dvc
           /              
   test_data.dvc

assuming the target test_data.dvc was provided.

[pared]

@drorata showing all paipelines seems reasonable in case of no target

[drorata]

Maybe I'm confused, but, consider the following example. Assume we have a data file data.txt:

foo
bar
hello
world

and we build the following steps:

dvc run -f split.dvc -d data.txt -o first_2.txt -o last_2.txt "head -n 2 data.txt > first_2.txt && tail -n 2 data.txt > last_2.txt"
dvc run -f add_ts.dvc -d first_2.txt -o first_2_w_ts.txt "cp first_2.txt first_2_w_ts.txt && echo $(date) | tee -a first_2_w_ts.txt"
dvc run -f add_sig.dvc -d last_2.txt -o last_2_w_sig.txt "cp last_2.txt last_2_w_sig.txt && echo 'this is my signature' >> last_2_w_sig.txt"
dvc run -f combine.dvc -d first_2_w_ts.txt -d last_2_w_sig.txt -o combined.txt "cat first_2_w_ts.txt last_2_w_sig.txt > combined.txt"
dvc run -f lines_count.dvc -d combined.txt -m lines_count "wc -l combined.txt > lines_count"

Now the result of

dvc pipeline show --ascii lines_count.dvc | less
dvc pipeline show --ascii combine.dvc | less
dvc pipeline show --ascii split.dvc | less

is always the same --- showing the whole pipeline. This is yet another behavior --- independently of the specified .dvc file the result is the overall pipeline.

[efiop]

@drorata dvc pipeline show --ascii stage.dvc is showing the full pipeline that stage.dvc is a part of. Our docs are not right. About the Dvcfile, pipeline show needs to know the target stage, so it could isolate a DAG that it belongs to and show it. Otherwise you might get a few disconnected DAGs, that are not trivial to visualize in any form. E.g. pipeline show(no flags) shows the order in which the pipeline will be ran, and so for multiple DAGs in your project we'll have to probably print it one after another with some newline between them to separate them. Same with --ascii, the navigation won't be trivial.

[shcheklein]

Related #2391

[shcheklein]

May be I'm confused as well, but I think it's reasonable to change the behavior to show only the part of the pipeline up to the target. What are the problems we can expect here, @efiop ?

[efiop]

@shcheklein yep, we could do that by default and have --full or something to behave like we do right now.