Support multiline command?

[skshetry]

Something like done in Github actions with run and script?

stages:
  stage1:
    cmd: |
      cmd1
      cmd2

Or, perhaps:

stages:
  stage1:
    cmd:
      - cmd1
      - cmd2

Or, maybe both?

Need to double check if we will be able to keep the dump in this shape or form.

Why?

Better expression

Current Workaround:

stages:
  stage1:
    cmd: >-
      cmd1
      && cmd2
      && cmd3

[efiop]

For the record:

stages:
  stage1:
    cmd: |
      cmd1
      cmd2

this is already supported and will be passed as is to the shell (we use shell=True in Popen). I would keep it as is.

stages:
  stage1:
    cmd:
      - cmd1
      - cmd2

This looks great and I think it should behave like cmd1 && cmd2 && cmd3, meaning that if one command fails then the rest is not executed and results in a stage failure.

[ClementWalter]

I have started to look at the code. @efiop is it ok to add a self.cmd = " && ".join(cmd) if isintance(cmd, list) else cmd in Stage.__init__ or should the Stage be able to truly handle a list of commands?

[efiop]

Hi @ClementWalter !

Thanks for looking into it! What advantages/disadvantages do you see for doing a join like that?

[efiop]

Keeping open for the question about multiline cmd handling raised by @skshetry

[jorgeorpinel]

Hi. I'm curious what's the use case for this. Is there a good example of multiple commands that share deps and outs? (to improve the docs)

[skshetry]

Hi. I'm curious what's the use case for this.

This is more of a convenience/ergonomical feature. So, anything you can write in a single line can be written in multiple lines or vice versa but might be easier than just cramming it in one with &&.

Is there a good example of multiple commands that share deps and outs? (to improve the docs)

You can use Github search for this (about usage of &&). But, why does it have to share deps/outs at all?

Here's a list of what I could find:

• building a model and testing it with a different script
• building each model in a separate virtual environment and evaluating them
• building each model in a separate directory
• cd ../directory && python script.py - looks like they are keeping dvc repo and code separately.
• fetching a pre-trained model and training them on local datasets (wget, etc).
• cleaning up randomly generated files after building/training/evaluating.
• setting some env vars, PythonPath, etc.
• cd directory && python test.py - maybe cd is more convenient than wdir as cd only affects during runtime.

[jorgeorpinel]

Thanks! I'm aware of POSIX &&. My question is about a real user need that this issue initially tried to cover. I'm getting the idea that this is basically for nice formatting (alt. to && basically).

Also thanks for the specific example ideas! I think that most of them may be better served by other forms thouogh:

Details

• building a model and testing it with a different script
• fetching a pre-trained model and training them on local datasets (wget, etc).
• cleaning up randomly generated files after building/training/evaluating.

Multiple stages

• building each model in a separate virtual environment and evaluating them
• building each model in a separate directory

Stage groups (foreach syntax in DVC 2.0)

• cd ../directory && python script.py
• cd directory && python test.py

wdir (or just keeping &&)

setting several env vars seems like the only instance where this is really convenient, to avoid an ugly string full of &&.

why does it have to share deps/outs at all?

Initially I was assuming that would be the point of having several commands in a stage, since having each command have separate deps/outs and listing those in a single stage doesn't sound very clean (again, better to use 2 stages instead?) Although I suppose if the stages are parallel, it could be a nice use of this feature, along with a desc or # comment about that perhaps.

Anyway, my Q is answered. Will check with Ivan whether it's worth improving the docs for this. Thanks again!

[skshetry]

Multiple stages

You are thinking of an ideal situation. I won't be creating a separate stage just to clean the logfile the script creates. It's a convenience feature.

• building each model in a separate virtual environment and evaluating them
• building each model in a separate directory

Stage groups (foreach syntax in DVC 2.0)

How do you create a virtual environment and run the script with foreach? It's a different thing.

• cd ../directory && python script.py
• cd directory && python test.py

wdir (or just keeping &&)

Like I said, cd definitely has its place. wdir applies to all outs/deps whereas cd is a runtime component.