From dc10cc4bada7bad9f52b97ad08ccf77c04d37c3a Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Wed, 15 Apr 2020 03:12:26 -0700 Subject: [PATCH 1/3] Params: remove stage deps and --all option --- content/docs/command-reference/params/diff.md | 45 ++++++++++++------- .../docs/command-reference/params/index.md | 16 ++++--- 2 files changed, 37 insertions(+), 24 deletions(-) diff --git a/content/docs/command-reference/params/diff.md b/content/docs/command-reference/params/diff.md index 925a945d51..152f55b75a 100644 --- a/content/docs/command-reference/params/diff.md +++ b/content/docs/command-reference/params/diff.md @@ -7,7 +7,7 @@ commits in the DVC repository, or between a commit and the ## Synopsis ```usage -usage: dvc params diff [-h] [-q | -v] [--show-json] [a_rev] [b_rev] +usage: dvc params diff [-h] [-q | -v] [--all] [--show-json] [a_rev] [b_rev] positional arguments: a_rev Old Git commit to compare (defaults to HEAD) @@ -17,24 +17,24 @@ positional arguments: ## Description -This command provides a quick way to compare parameter values from your previous -experiments with the current one(s) in your workspace. +This command provides a quick way to compare parameter values from your +experiments in the repository history. > Parameter dependencies are defined with the `-p` option in `dvc run`. See also > `dvc params`. Run without arguments, this command compares parameters currently present in the -workspace (including uncommitted changes) with the latest committed -version. - -❗ It only shows parameters used in any of the currently present -[stage files](/doc/command-reference/run) (DVC-files). +workspace (uncommitted changes) with the latest committed version. Supported parameter _value_ types are: string, integer, float, and arrays. DVC itself does not ascribe any specific meaning for these values. +❗ By default it only shows parameters that were changed. + ## Options +- `--all` - prints all parameters including not changed. + - `--show-json` - prints the command's output in easily parsable JSON format, instead of a human-readable table. @@ -57,9 +57,9 @@ train: epochs: 70 layers: 9 -processing: - threshold: 0.98 - bow_size: 15000 +process: + thresh: 0.98 + bow: 15000 ``` Define a pipeline [stage](/doc/command-reference/run) with parameter @@ -78,20 +78,19 @@ Let's now print parameter values that we are tracking in this $ dvc params diff Path Param Old New params.yaml lr None 0.0041 -params.yaml train.layers None 9 +params.yaml process.bow None 15000 +params.yaml process.thresh None 0.98 params.yaml train.epochs None 70 +params.yaml train.layers None 9 ``` The command above shows the difference in parameters between the workspace and the last committed version of the params file `params.yaml`. Since it did not exist before, all `Old` values are `None`. -❗ Note that not all the parameter were printed. `dvc params diff` prints only -changed parameters that were used in one of the stages and ignores parameters -from the group `processing` that were not used. - In a project with parameters file history (params present in various Git -commits), you will see both `Old` and `New` values: +commits), you will see both `Old` and `New` values. However, the parameters +won't be shown if there are no changes: ```dvc $ dvc params diff @@ -101,6 +100,18 @@ params.yaml train.layers 9 7 params.yaml train.epochs 70 110 ``` +Specify `--all` option to see all the parameters including not changed ones: + +```dvc +$ dvc params diff --all + Path Param Old New +params.yaml lr 0.0041 0.0043 +params.yaml process.bow 15000 15000 +params.yaml process.thresh 0.98 0.98 +params.yaml train.layers 9 7 +params.yaml train.epochs 70 110 +``` + To compare parameters with a specific commit, a tag or any [revision](https://git-scm.com/docs/revisions) should be specified as an additional command line parameter: diff --git a/content/docs/command-reference/params/index.md b/content/docs/command-reference/params/index.md index 6aea5a7206..1ff6297e20 100644 --- a/content/docs/command-reference/params/index.md +++ b/content/docs/command-reference/params/index.md @@ -45,9 +45,9 @@ parameters will be required for the stage invalidation (see `dvc status` and stage. Using parameter dependencies prevents situations where several -[pipeline](/doc/command-reference/pipeline) stages depend on the same file, and -any change in the file invalidates causes the reproduction all those stages -unnecessarily. +[pipeline](/doc/command-reference/pipeline) stages depend on the same +configuration file, and any change in the file invalidates causes the +reproduction all those stages unnecessarily. You should manually write or generate the YAML or JSON parameters files needed for the project, which can be versioned directly with Git. You can then use @@ -81,9 +81,9 @@ train: epochs: 70 layers: 9 -processing: - threshold: 0.98 - bow_size: 15000 +process: + thresh: 0.98 + bow: 15000 ``` Define a [stage](/doc/command-reference/run) that depends on params `lr`, @@ -138,12 +138,14 @@ $ dvc run -d logs/ -o users.csv \ ## Examples: Print all parameter values in the workspace Following the previous example, we can use `dvc params diff` to list all of the -available param values associated to DVC-files in the workspace: +available param values: ```dvc $ dvc params diff Path Param Old New params.yaml lr None 0.0041 +params.yaml process.bow None 15000 +params.yaml process.thresh None 0.98 params.yaml train.layers None 9 params.yaml train.epochs None 70 ``` From f782eaeecec7a1168d947082232d29335337cb7d Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 15 Apr 2020 15:41:35 -0500 Subject: [PATCH 2/3] Update content/docs/command-reference/params/diff.md --- content/docs/command-reference/params/diff.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/command-reference/params/diff.md b/content/docs/command-reference/params/diff.md index 152f55b75a..4fc66516d1 100644 --- a/content/docs/command-reference/params/diff.md +++ b/content/docs/command-reference/params/diff.md @@ -7,7 +7,8 @@ commits in the DVC repository, or between a commit and the ## Synopsis ```usage -usage: dvc params diff [-h] [-q | -v] [--all] [--show-json] [a_rev] [b_rev] +usage: dvc params diff [-h] [-q | -v] [--all] [--show-json] + [a_rev] [b_rev] positional arguments: a_rev Old Git commit to compare (defaults to HEAD) From 4c9a2399809832d5a7200ca8b667c54687d15e0c Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Thu, 16 Apr 2020 11:16:16 -0700 Subject: [PATCH 3/3] New params: after review fixes --- content/docs/command-reference/params/diff.md | 8 +++++--- content/docs/command-reference/params/index.md | 6 +++--- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/content/docs/command-reference/params/diff.md b/content/docs/command-reference/params/diff.md index 152f55b75a..7544a9c417 100644 --- a/content/docs/command-reference/params/diff.md +++ b/content/docs/command-reference/params/diff.md @@ -7,7 +7,8 @@ commits in the DVC repository, or between a commit and the ## Synopsis ```usage -usage: dvc params diff [-h] [-q | -v] [--all] [--show-json] [a_rev] [b_rev] +usage: dvc params diff [-h] [-q | -v] [--all] [--show-json] + [a_rev] [b_rev] positional arguments: a_rev Old Git commit to compare (defaults to HEAD) @@ -17,8 +18,9 @@ positional arguments: ## Description -This command provides a quick way to compare parameter values from your -experiments in the repository history. +This command provides a quick way to compare parameter values among experiments +in the repository history. Requires that Git is being used to version the +project params. > Parameter dependencies are defined with the `-p` option in `dvc run`. See also > `dvc params`. diff --git a/content/docs/command-reference/params/index.md b/content/docs/command-reference/params/index.md index 1ff6297e20..d8419526c5 100644 --- a/content/docs/command-reference/params/index.md +++ b/content/docs/command-reference/params/index.md @@ -45,9 +45,9 @@ parameters will be required for the stage invalidation (see `dvc status` and stage. Using parameter dependencies prevents situations where several -[pipeline](/doc/command-reference/pipeline) stages depend on the same -configuration file, and any change in the file invalidates causes the -reproduction all those stages unnecessarily. +[pipeline](/doc/command-reference/pipeline) stages share a (configuration) file +as a common dependency, and any change in this dependency invalidates all the +stages and causes the reproduction those stages unnecessarily. You should manually write or generate the YAML or JSON parameters files needed for the project, which can be versioned directly with Git. You can then use