From 278dab90d36f178a1e46390e15d3d5b8a4b1c830 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cl=C3=A9ment=20Walter?= Date: Sun, 29 Nov 2020 13:11:50 +0100 Subject: [PATCH 01/19] Update dvc-files-and-directories.md Update the dvc.yaml documentation with the list possibility for the `cmd` entry. --- content/docs/user-guide/dvc-files-and-directories.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index a4e1a54afe..092dc26216 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -151,7 +151,10 @@ stages: by the user with the `--name` (`-n`) option of `dvc run`. Each stage can contain the possible following fields: -- `cmd` (always present): Executable command defined in this stage +- `cmd` (always present): Executable commands defined in this stage. This could be + either a single command or a list of commands that should be executed sequentially. + The failure of one of these commandes will result in the failure of the stage. + Note that each command is passed as is to the shell. - `wdir`: Working directory for the stage command to run in (relative to the file's location). If this field is not present explicitly, it defaults to `.` (the file's location). From 7b796f534f4605eb9e3e2aa76434891f00107833 Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Wed, 2 Dec 2020 20:06:31 +0100 Subject: [PATCH 02/19] Fix styling and add more details to new cmd behavior --- content/docs/user-guide/dvc-files-and-directories.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index 092dc26216..96321d4830 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -151,10 +151,12 @@ stages: by the user with the `--name` (`-n`) option of `dvc run`. Each stage can contain the possible following fields: -- `cmd` (always present): Executable commands defined in this stage. This could be - either a single command or a list of commands that should be executed sequentially. - The failure of one of these commandes will result in the failure of the stage. - Note that each command is passed as is to the shell. +- `cmd` (always present): Executable commands defined in this stage. Each + command is passed as is to the shell. The command can be either a single + command or a list of commands. In that latter case: + - the commands are run one after the other + - the failure of one given command will stop the run of the stage and raises + an error. Thus that all the commands before failure are well executed. - `wdir`: Working directory for the stage command to run in (relative to the file's location). If this field is not present explicitly, it defaults to `.` (the file's location). From 6c83768ebc76f234ac5f278a0c2734357cbe4e39 Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Wed, 2 Dec 2020 20:13:56 +0100 Subject: [PATCH 03/19] Fix singular command form in repro doc --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 733d5c2a90..bf81342c62 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -170,7 +170,7 @@ up-to-date and only execute the final stage. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if all stages are up to date or if all stages are successfully executed, otherwise - exit with 1. The command defined in the stage is free to write output + exit with 1. The commands defined in the stage are free to write output regardless of this flag. - `-v`, `--verbose` - displays detailed tracing information. From e08043205c4dfc613feda1e5646acf786715796f Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Wed, 2 Dec 2020 20:24:10 +0100 Subject: [PATCH 04/19] Update command output in repro doc --- content/docs/command-reference/repro.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index bf81342c62..118a71c02f 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -253,8 +253,8 @@ If we now run `dvc repro`, we should see this: ```dvc $ dvc repro Stage 'filter' didn't change, skipping -Running stage 'count' with command: - python process.py numbers.txt > count.txt +Running stage 'count': +$ python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From b01aba4f3ea6debad0847c1c0ce8d5669e0b4b5c Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Wed, 2 Dec 2020 20:26:39 +0100 Subject: [PATCH 05/19] Fix typo --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 118a71c02f..eb030ac477 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -315,5 +315,5 @@ $ dvc dag ``` > Note that using `dvc repro` without `--downstream` in the above example -> results in the execution of the target (`count`), and the preceeding stages +> results in the execution of the target (`count`), and the preceding stages > (only 'filter' in this case). From 18495884eebe5c685b77831d2ef2a4f9de35e331 Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Wed, 2 Dec 2020 20:55:11 +0100 Subject: [PATCH 06/19] Add an example of a usage of a list of cmd --- content/docs/user-guide/dvc-files-and-directories.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index 96321d4830..5ea451bcbc 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -132,7 +132,9 @@ stages: - performance.json training: desc: train your model - cmd: python train.py + cmd: + - test -z "$(git status -s)" + - python train.py deps: - train.py - features @@ -157,6 +159,8 @@ the possible following fields: - the commands are run one after the other - the failure of one given command will stop the run of the stage and raises an error. Thus that all the commands before failure are well executed. + > in the `dvc.yaml` example above, the `python train.py` command will be + > executed only if the command before succeeds, _i.e_ if git is clear - `wdir`: Working directory for the stage command to run in (relative to the file's location). If this field is not present explicitly, it defaults to `.` (the file's location). From 2f937b8326c153b6f7c1c9f7dbf8e796306f07aa Mon Sep 17 00:00:00 2001 From: Clement Walter Date: Thu, 3 Dec 2020 21:28:43 +0100 Subject: [PATCH 07/19] Shorten cmd description and ref to repro --- content/docs/user-guide/dvc-files-and-directories.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index 5ea451bcbc..d2d70a8809 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -153,14 +153,10 @@ stages: by the user with the `--name` (`-n`) option of `dvc run`. Each stage can contain the possible following fields: -- `cmd` (always present): Executable commands defined in this stage. Each - command is passed as is to the shell. The command can be either a single - command or a list of commands. In that latter case: - - the commands are run one after the other - - the failure of one given command will stop the run of the stage and raises - an error. Thus that all the commands before failure are well executed. - > in the `dvc.yaml` example above, the `python train.py` command will be - > executed only if the command before succeeds, _i.e_ if git is clear +- `cmd` (always present): One or more commands executed by the stage (may + contain either a single value, or a list). Commands are executed sequentially + until all are finished or until one of them fails (see + [`dvc repro`](/doc/command-reference/repro)) - `wdir`: Working directory for the stage command to run in (relative to the file's location). If this field is not present explicitly, it defaults to `.` (the file's location). From 2384372be286edf4f80223e3f62b254f7f21934a Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Sun, 6 Dec 2020 22:35:58 -0600 Subject: [PATCH 08/19] Update content/docs/user-guide/dvc-files-and-directories.md --- content/docs/user-guide/dvc-files-and-directories.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index d2d70a8809..e2a8b2497f 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -133,7 +133,7 @@ stages: training: desc: train your model cmd: - - test -z "$(git status -s)" + - echo "Train my model!" - python train.py deps: - train.py From 1df6550fdf97bc1b1f293a490344685b2b58e9dd Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Sun, 6 Dec 2020 22:39:55 -0600 Subject: [PATCH 09/19] Update content/docs/user-guide/dvc-files-and-directories.md --- content/docs/user-guide/dvc-files-and-directories.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index e2a8b2497f..07ec5ef438 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -156,7 +156,7 @@ the possible following fields: - `cmd` (always present): One or more commands executed by the stage (may contain either a single value, or a list). Commands are executed sequentially until all are finished or until one of them fails (see - [`dvc repro`](/doc/command-reference/repro)) + [`dvc repro`](/doc/command-reference/repro) for details). - `wdir`: Working directory for the stage command to run in (relative to the file's location). If this field is not present explicitly, it defaults to `.` (the file's location). From abb79ab8c9b61c51a59a3c20652d6215eda049cb Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Sun, 6 Dec 2020 23:17:37 -0600 Subject: [PATCH 10/19] cmd: update repro sample output per https://github.com/iterative/dvc.org/pull/1980#pullrequestreview-544324549 --- content/docs/command-reference/repro.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index eb030ac477..757c081b53 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -253,8 +253,8 @@ If we now run `dvc repro`, we should see this: ```dvc $ dvc repro Stage 'filter' didn't change, skipping -Running stage 'count': -$ python process.py numbers.txt > count.txt +Running stage 'count' with command: + python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From 12d730c5cc41d0166a33a7ab1cc6d807ae7ada99 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Sun, 6 Dec 2020 23:17:37 -0600 Subject: [PATCH 11/19] cmd: update repro sample outputs per https://github.com/iterative/dvc.org/pull/1980#pullrequestreview-544324549 --- content/docs/command-reference/repro.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index eb030ac477..152a103502 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -253,8 +253,8 @@ If we now run `dvc repro`, we should see this: ```dvc $ dvc repro Stage 'filter' didn't change, skipping -Running stage 'count': -$ python process.py numbers.txt > count.txt +Running stage 'count' with command: + python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` @@ -291,7 +291,7 @@ of only the target (`count`) and following stages (none in this case): ```dvc $ dvc repro --downstream count Running stage 'count' with command: - python process.py numbers.txt > count.txt + python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From 362d9e9f9dba027bf3887efb7a58b588297e0e13 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Sun, 6 Dec 2020 23:24:23 -0600 Subject: [PATCH 12/19] Update content/docs/user-guide/dvc-files-and-directories.md Co-authored-by: Saugat Pachhai --- content/docs/user-guide/dvc-files-and-directories.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/user-guide/dvc-files-and-directories.md b/content/docs/user-guide/dvc-files-and-directories.md index 07ec5ef438..737b400d42 100644 --- a/content/docs/user-guide/dvc-files-and-directories.md +++ b/content/docs/user-guide/dvc-files-and-directories.md @@ -133,7 +133,7 @@ stages: training: desc: train your model cmd: - - echo "Train my model!" + - pip install -r requirements.txt - python train.py deps: - train.py From 56252e34333bd03181ebc53f80e61cb75ede5e8b Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Mon, 7 Dec 2020 22:27:25 -0600 Subject: [PATCH 13/19] cmd: say how multiple cmd's are reproed per https://github.com/iterative/dvc.org/pull/1980#pullrequestreview-545826292 --- content/docs/command-reference/repro.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 152a103502..1092e1d1d2 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -2,7 +2,7 @@ Reproduce complete or partial [pipelines](/doc/command-reference/dag) by executing commands defined in their [stages](/doc/command-reference/run) in the -correct order. The commands to be executed are determined by recursively +correct order. The stage commands to be executed are determined by recursively analyzing dependencies and outputs of the target stages. ## Synopsis @@ -44,8 +44,13 @@ files, intermediate or final results (except if the `--pull` option is used). By default, this command checks all [pipeline](/doc/command-reference/dag) stages to determine which ones have changed. Then it executes the corresponding -commands. [Stage](/doc/command-reference/run) outputs are deleted from the -workspace before executing the stage commands that produce them. +commands (`cmd` field of `dvc.yaml`). [Stage](/doc/command-reference/run) +outputs are deleted from the workspace before executing the stage +commands that produce them. + +For stages with multiple commands (having a list in the `cmd` field), commands +are run one after the other in the order they are defined. The failure of any +command will halt the remaining stage execution, and raises an error. There are a few ways to restrict what will be regenerated by this command: by specifying stages as `targets`, or by using the `--single-item`, `--cwd`, among From c4aa493d119fe3747c73f74ce48f848c21c1a865 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 8 Dec 2020 13:38:31 -0600 Subject: [PATCH 14/19] cmd: remove repro --cwd (again) --- content/docs/command-reference/repro.md | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 1092e1d1d2..c49816ef0d 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -53,8 +53,8 @@ are run one after the other in the order they are defined. The failure of any command will halt the remaining stage execution, and raises an error. There are a few ways to restrict what will be regenerated by this command: by -specifying stages as `targets`, or by using the `--single-item`, `--cwd`, among -other options. +specifying stages as `targets`, or by using `--single-item`, among other +options. > Note that stages without dependencies are considered _always changed_, so > `dvc repro` always executes them. @@ -106,15 +106,6 @@ up-to-date and only execute the final stage. recursive search for changed dependencies. Multiple stages are executed (non-recursively) if multiple stage names are given as `targets`. -- `-c `, `--cwd ` - (**deprecated**) this is now supported for any - command with the global [`dvc --cd `](/doc/command-reference) option. - - Directory within the project to reproduce from. `targets` will be searched - relative to this path. Instead of using `--cwd`, one can alternately specify a - target in a subdirectory as `path/to/target.dvc`. This option can be useful - for example with subdirectories containing a separate pipeline that can be - reproduced independently. - - `-R`, `--recursive` - determines the stages to reproduce by searching each target directory and its subdirectories for stages (in `dvc.yaml`) to inspect. If there are no directories among the targets, this option is ignored. From 4ee75b011c844840cecb8d549c3823948ef668eb Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 11 Dec 2020 18:17:40 -0600 Subject: [PATCH 15/19] Update content/docs/command-reference/repro.md --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 8f99204be2..86d732bb2e 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -2,7 +2,7 @@ Reproduce complete or partial [pipelines](/doc/command-reference/dag) by executing commands defined in their [stages](/doc/command-reference/run) in the -correct order. The stage commands to be executed are determined by recursively +correct order. The commands to be executed are determined by recursively analyzing dependencies and outputs of the target stages. ## Synopsis From ad2dc39887b1a0e3dcbcb4c2ecb46c94c516248c Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 11 Dec 2020 18:18:49 -0600 Subject: [PATCH 16/19] Update content/docs/command-reference/repro.md --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 86d732bb2e..6e81445810 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -249,7 +249,7 @@ If we now run `dvc repro`, we should see this: ```dvc $ dvc repro Stage 'filter' didn't change, skipping -Running stage 'count' with command: +Running stage 'count': python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From 1ef4203e07c38c6cf9a8c04f2b7978a53761f541 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 11 Dec 2020 18:19:43 -0600 Subject: [PATCH 17/19] Update content/docs/command-reference/repro.md --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 6e81445810..107d563f85 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -286,7 +286,7 @@ of only the target (`count`) and following stages (none in this case): ```dvc $ dvc repro --downstream count -Running stage 'count' with command: +Running stage 'count': python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From 0c89f68af31c40e93338a77eef960600b64825d2 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 11 Dec 2020 18:21:03 -0600 Subject: [PATCH 18/19] Update content/docs/command-reference/repro.md --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 107d563f85..7a58122742 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -250,7 +250,7 @@ If we now run `dvc repro`, we should see this: $ dvc repro Stage 'filter' didn't change, skipping Running stage 'count': - python process.py numbers.txt > count.txt +> python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ``` From 49faf55e87c8e5acea765aa30885d67be3c5f393 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Fri, 11 Dec 2020 18:21:26 -0600 Subject: [PATCH 19/19] Update content/docs/command-reference/repro.md --- content/docs/command-reference/repro.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/repro.md b/content/docs/command-reference/repro.md index 7a58122742..60181d434d 100644 --- a/content/docs/command-reference/repro.md +++ b/content/docs/command-reference/repro.md @@ -287,7 +287,7 @@ of only the target (`count`) and following stages (none in this case): ```dvc $ dvc repro --downstream count Running stage 'count': - python process.py numbers.txt > count.txt +> python process.py numbers.txt > count.txt Updating lock file 'dvc.lock' ```