From f6f86080359c0e8e97547298d1ace8758acd8c4a Mon Sep 17 00:00:00 2001 From: dberenbaum Date: Thu, 26 May 2022 09:47:45 -0400 Subject: [PATCH 01/10] guide: update how to share many exps --- content/docs/sidebar.json | 3 +- .../cleaning-experiments.md | 60 ++++++++++--------- .../comparing-experiments.md | 20 ++++--- .../sharing-experiments.md | 13 +++- .../how-to/share-many-experiments.md | 19 ------ 5 files changed, 53 insertions(+), 62 deletions(-) delete mode 100644 content/docs/user-guide/how-to/share-many-experiments.md diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 46aecf3f0f..3af0842e8f 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -164,8 +164,7 @@ "update-tracked-data", "add-deps-or-outs-to-a-stage", "merge-conflicts", - "share-a-dvc-cache", - "share-many-experiments" + "share-a-dvc-cache" ] }, "setup-google-drive-remote", diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 5064ec7bee..29aa8007a4 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -18,13 +18,28 @@ main: cnn-64 cnn-128 $ dvc exp remove cnn-32 cnn-64 -$ dvc exp list -main: - cnn-128 +Removed experiments: cnn-32,cnn-64 ``` ## Removing multiple experiments +You may wish to remove multiple experiments at once. For example, if you are +finished experimenting on the current commit and want to discard all experiments +based on that commit, use `--rev HEAD`. + +``` +$ dvc exp list +refs/tags/baseline-experiment: + cnn-128 + cnn-32 + cnn-64 + cnn-96 +$ dvc exp remove --rev HEAD +Removed experiments: cnn-128,cnn-32,cnn-64,cnn-96 +``` + +## Cleaning up experiments + After you've completed a set of experiments, it may be easier to decide which of these to keep rather than which of these to remove. You can use `dvc exp gc` to select a set of experiments to keep and the rest of them are _garbage @@ -63,6 +78,9 @@ branches in this example. ```dvc $ dvc exp gc --workspace +WARNING: This will remove all experiments except those derived from the workspace of the current repo. Run queued experiments will be removed. +Are you sure you want to proceed? [y/n]: y +Removed 5 experiments. To remove unused cache files use 'dvc gc'. $ dvc exp list --all main: exp-abc000 @@ -147,6 +165,9 @@ $ dvc exp show --all-tags ```dvc $ dvc exp gc --all-tags +WARNING: This will remove all experiments except those derived from the workspace of the current repo. Run queued experiments will be removed. +Are you sure you want to proceed? [y/n]: y +Removed 2 experiments. To remove unused cache files use 'dvc gc'. $ dvc exp show --all-tags ``` @@ -171,8 +192,8 @@ workspace that are not committed to the history. ### Deleting Experiment-Related Objects in DVC Cache -Note that `dvc exp gc` and `dvc exp remove` doesn't delete any objects in the -DVC cache. In order to remove the cache objects, e.g. model files, +Note that `dvc exp gc` and `dvc exp remove` don't delete any objects in the DVC +cache. In order to remove the cache objects, e.g. model files, intermediate artifacts, etc. related with the experiments, you can use `dvc gc` command. @@ -183,32 +204,13 @@ well. ## Removing experiments from remotes -As you push the experiments with `dvc exp push`, remotes may be become cluttered -with experiment references. - -DVC doesn't provide a shortcut for cleaning up the experiments in remotes but -you can use Git plumbing commands to remove experiment references from remotes. +As you push the experiments with `dvc exp push`, Git remotes may be become +cluttered with experiment references. To remove experiments from a Git remote, +use `dvc exp remove -g`. -First get the list of experiments with their hash values. - -```dvc -$ git ls-remote origin 'refs/exps/*' -98b237f8d8da0964c9fa60c6b27f1dd4a214dabf refs/exps/17/2b1b9c885f10a73c76bd457a04878bee0e6d6f/exp-7424d -794854926931e84ebc90e829dbc09b3085391659 refs/exps/19/0e697aa566482ebdb8bdb65401382e76b1bce5/exp-ec039 ``` - -Then we can use `git push -d` as any other Git reference: - -```dvc -$ git push -d origin refs/exps/17/2b1b9c885f10a73c76bd457a04878bee0e6d6f/exp-7424d -``` - -If you want to delete **all** experiments in a remote, you can use a loop: - -```dvc -$ git ls-remote origin 'refs/exps/*' | cut -f 2 | while read exppath ; do - git push -d origin "${exppath}" -done +dvc exp remove -g origin exp-ab780 +Removed experiments: exp-bb09c ``` ## Removing queued experiments diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index 1f107ee34e..6e191d3d58 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -17,10 +17,10 @@ refs/tags/baseline-experiment: ``` If you want to list all the experiments in the repo regardless of their parent -commit, use the `--all` flag. +commit, use the `--all-commits`/`-A` flag. ```dvc -$ dvc exp list --all +$ dvc exp list -A refs/tags/baseline-experiment: cnn-64 cnn-128 @@ -42,8 +42,8 @@ refs/tags/baseline-experiment: ``` This command lists remote experiments based on that repo's `HEAD`. You can use -`--all` to list all experiments, or add any other supported option to the remote -`dvc exp list` command. +`--all-commits`/`-A` to list all experiments, or add any other supported option +to the remote `dvc exp list` command. [shared]: /doc/user-guide/experiment-management/sharing-experiments @@ -51,12 +51,13 @@ This command lists remote experiments based on that repo's `HEAD`. You can use `dvc exp list` may be printing too much information when it comes to feed its output to other commands. You can get only the names of the experiments via the -`--name-only` flag. For example, to get all the experiment names from a remote -(`origin`): +`--name-only` flag. For example, to download the model files from all +experiments in a remote: ```dvc -$ for experiment in $(dvc exp list origin --name-only --all) ; do - dvc exp pull "${experiment}" +$ for name in $(dvc exp list origin --name-only -A); do + dvc get --rev ${name} git@github.com:iterative/example-dvc-experiments.git \ + models/model.h5 -o exps/${name}.h5 done ``` @@ -108,7 +109,8 @@ $ dvc exp show ``` `dvc exp show` only tabulates experiments in the workspace and in `HEAD`. You -can use `--all` flag to show all the experiments in the project instead. +can use `--all-commits`/`-A` flag to show all the experiments in the project +instead. Note that [queued experiments] will be marked with an asterisk `*`. diff --git a/content/docs/user-guide/experiment-management/sharing-experiments.md b/content/docs/user-guide/experiment-management/sharing-experiments.md index 3aa2ad00ca..5cd1eeb12a 100644 --- a/content/docs/user-guide/experiment-management/sharing-experiments.md +++ b/content/docs/user-guide/experiment-management/sharing-experiments.md @@ -73,11 +73,8 @@ $ dvc exp push origin exp-abc123 Once pushed, you can easily [list remote experiments] (with `dvc exp list`). -> See also [How to Share Many Experiments][share many]. - [list remote experiments]: /doc/user-guide/experiment-management/comparing-experiments#list-experiments-saved-remotely -[share many]: /doc/user-guide/how-to/share-many-experiments ## Downloading experiments @@ -94,3 +91,13 @@ both of these configured (see this [earlier section](#preparation)). If an experiment being pulled already exists in the local project, DVC won't overwrite it unless you supply `--force`. + +## Sharing many experiments + +Use the flags in `dvc exp push` and `dvc exp pull` to share many experiments at +once. For example, to upload all experiments based on the latest commit of the +current branch (Git `HEAD`), use `--rev HEAD`: + +``` +$ dvc exp push --rev HEAD origin +``` diff --git a/content/docs/user-guide/how-to/share-many-experiments.md b/content/docs/user-guide/how-to/share-many-experiments.md deleted file mode 100644 index 550c8260a7..0000000000 --- a/content/docs/user-guide/how-to/share-many-experiments.md +++ /dev/null @@ -1,19 +0,0 @@ -# How to Share Many Experiments - -`dvc exp push` and `dvc exp pull` allow us to [share experiments] between -repositories via existing DVC and Git remotes. These however work on individual -experiments. - -Here's a simple shell loop to push or pull all experiments (Linux): - -```dvc -$ dvc exp list --all --name-only | while read -r expname ; do \ - dvc exp pull origin ${expname} \ -done -``` - -> 📖 See [Listing Experiments] for more info on `dvc exp list`. - -[share experiments]: /doc/user-guide/experiment-management/sharing-experiments -[listing experiments]: - /doc/user-guide/experiment-management/comparing-experiments#list-experiments-in-the-project From 0b65c5220b1ef82abc6e8b21e90130bc858cf362 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Fri, 27 May 2022 14:30:42 -0400 Subject: [PATCH 02/10] Update content/docs/user-guide/experiment-management/cleaning-experiments.md --- .../user-guide/experiment-management/cleaning-experiments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 29aa8007a4..295863b4d9 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -165,7 +165,7 @@ $ dvc exp show --all-tags ```dvc $ dvc exp gc --all-tags -WARNING: This will remove all experiments except those derived from the workspace of the current repo. Run queued experiments will be removed. +WARNING: This will remove all experiments except those derived from any tags of the current repo. Run queued experiments will be removed. Are you sure you want to proceed? [y/n]: y Removed 2 experiments. To remove unused cache files use 'dvc gc'. $ dvc exp show --all-tags From 2fc6bfcd7b9af955f7e06d6d3065ebb08557b7be Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 31 May 2022 16:05:45 -0500 Subject: [PATCH 03/10] Apply suggestions from code review --- .../experiment-management/cleaning-experiments.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 295863b4d9..b4cc7ad3f4 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -192,8 +192,8 @@ workspace that are not committed to the history. ### Deleting Experiment-Related Objects in DVC Cache -Note that `dvc exp gc` and `dvc exp remove` don't delete any objects in the DVC -cache. In order to remove the cache objects, e.g. model files, +Note that `dvc exp gc` and `dvc exp remove` do not delete any objects in the +DVC cache. In order to remove the cache objects, e.g. model files, intermediate artifacts, etc. related with the experiments, you can use `dvc gc` command. @@ -208,8 +208,8 @@ As you push the experiments with `dvc exp push`, Git remotes may be become cluttered with experiment references. To remove experiments from a Git remote, use `dvc exp remove -g`. -``` -dvc exp remove -g origin exp-ab780 +```dvc +$ dvc exp remove -g origin exp-ab780 Removed experiments: exp-bb09c ``` From cfb96c68b81241c165e38416570fa7839d3590bb Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 31 May 2022 16:11:00 -0500 Subject: [PATCH 04/10] Apply suggestions from code review --- .../experiment-management/comparing-experiments.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index 6e191d3d58..e4069f8ac6 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -17,7 +17,7 @@ refs/tags/baseline-experiment: ``` If you want to list all the experiments in the repo regardless of their parent -commit, use the `--all-commits`/`-A` flag. +commit, use the `--all-commits` (`-A`) flag. ```dvc $ dvc exp list -A @@ -42,7 +42,7 @@ refs/tags/baseline-experiment: ``` This command lists remote experiments based on that repo's `HEAD`. You can use -`--all-commits`/`-A` to list all experiments, or add any other supported option +`--all-commits` (`-A`) to list all experiments, or add any other supported option to the remote `dvc exp list` command. [shared]: /doc/user-guide/experiment-management/sharing-experiments @@ -109,7 +109,7 @@ $ dvc exp show ``` `dvc exp show` only tabulates experiments in the workspace and in `HEAD`. You -can use `--all-commits`/`-A` flag to show all the experiments in the project +can use `--all-commits` (`-A`) flag to show all the experiments in the project instead. Note that [queued experiments] will be marked with an asterisk `*`. From e629e8e92265045cacb4743f23ee5d25162ea608 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 31 May 2022 16:12:46 -0500 Subject: [PATCH 05/10] Update content/docs/user-guide/experiment-management/sharing-experiments.md --- .../user-guide/experiment-management/sharing-experiments.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/docs/user-guide/experiment-management/sharing-experiments.md b/content/docs/user-guide/experiment-management/sharing-experiments.md index 5cd1eeb12a..be0a115170 100644 --- a/content/docs/user-guide/experiment-management/sharing-experiments.md +++ b/content/docs/user-guide/experiment-management/sharing-experiments.md @@ -94,9 +94,9 @@ overwrite it unless you supply `--force`. ## Sharing many experiments -Use the flags in `dvc exp push` and `dvc exp pull` to share many experiments at -once. For example, to upload all experiments based on the latest commit of the -current branch (Git `HEAD`), use `--rev HEAD`: +Use the`--rev` option of `dvc exp push` and `dvc exp pull` to share many +experiments at once. For example, to upload all experiments based on the +latest commit of the current branch (Git `HEAD`), use `--rev HEAD`: ``` $ dvc exp push --rev HEAD origin From 35ccf41aad75ab60b6d7e166fae1e9ffc7c6e2b7 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Tue, 31 May 2022 17:12:57 -0400 Subject: [PATCH 06/10] Update content/docs/user-guide/experiment-management/cleaning-experiments.md Co-authored-by: Jorge Orpinel --- .../experiment-management/cleaning-experiments.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index b4cc7ad3f4..0c31e1c24b 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -21,13 +21,13 @@ $ dvc exp remove cnn-32 cnn-64 Removed experiments: cnn-32,cnn-64 ``` -## Removing multiple experiments +## Removing multiple experiments (based on parent commit) You may wish to remove multiple experiments at once. For example, if you are -finished experimenting on the current commit and want to discard all experiments -based on that commit, use `--rev HEAD`. +finished experimenting on the current Git commit and want to discard all +experiments derived from it, use `dvc exp remove --rev` with `HEAD`. -``` +```dvc $ dvc exp list refs/tags/baseline-experiment: cnn-128 From b23470c311f8968ea49bbb942c34efa0b1f77ab2 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Tue, 31 May 2022 17:13:54 -0400 Subject: [PATCH 07/10] Update content/docs/user-guide/experiment-management/comparing-experiments.md Co-authored-by: Jorge Orpinel --- .../experiment-management/comparing-experiments.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index e4069f8ac6..ee871c6b73 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -56,8 +56,9 @@ experiments in a remote: ```dvc $ for name in $(dvc exp list origin --name-only -A); do - dvc get --rev ${name} git@github.com:iterative/example-dvc-experiments.git \ - models/model.h5 -o exps/${name}.h5 + dvc get --rev ${name} \ + git@github.com:iterative/example-dvc-experiments.git \ + models/model.h5 -o exps/${name}.h5 done ``` From 74fcb897ed8b07df0f934b704765a0142be8d087 Mon Sep 17 00:00:00 2001 From: "restyled-io[bot]" <32688539+restyled-io[bot]@users.noreply.github.com> Date: Tue, 31 May 2022 16:15:53 -0500 Subject: [PATCH 08/10] Restyled by prettier (#3605) Co-authored-by: Restyled.io --- .../user-guide/experiment-management/cleaning-experiments.md | 4 ++-- .../user-guide/experiment-management/comparing-experiments.md | 4 ++-- .../user-guide/experiment-management/sharing-experiments.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 0c31e1c24b..66ab698711 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -192,8 +192,8 @@ workspace that are not committed to the history. ### Deleting Experiment-Related Objects in DVC Cache -Note that `dvc exp gc` and `dvc exp remove` do not delete any objects in the -DVC cache. In order to remove the cache objects, e.g. model files, +Note that `dvc exp gc` and `dvc exp remove` do not delete any objects in the DVC +cache. In order to remove the cache objects, e.g. model files, intermediate artifacts, etc. related with the experiments, you can use `dvc gc` command. diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index ee871c6b73..c92ac41513 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -42,8 +42,8 @@ refs/tags/baseline-experiment: ``` This command lists remote experiments based on that repo's `HEAD`. You can use -`--all-commits` (`-A`) to list all experiments, or add any other supported option -to the remote `dvc exp list` command. +`--all-commits` (`-A`) to list all experiments, or add any other supported +option to the remote `dvc exp list` command. [shared]: /doc/user-guide/experiment-management/sharing-experiments diff --git a/content/docs/user-guide/experiment-management/sharing-experiments.md b/content/docs/user-guide/experiment-management/sharing-experiments.md index be0a115170..679179f9f2 100644 --- a/content/docs/user-guide/experiment-management/sharing-experiments.md +++ b/content/docs/user-guide/experiment-management/sharing-experiments.md @@ -95,8 +95,8 @@ overwrite it unless you supply `--force`. ## Sharing many experiments Use the`--rev` option of `dvc exp push` and `dvc exp pull` to share many -experiments at once. For example, to upload all experiments based on the -latest commit of the current branch (Git `HEAD`), use `--rev HEAD`: +experiments at once. For example, to upload all experiments based on the latest +commit of the current branch (Git `HEAD`), use `--rev HEAD`: ``` $ dvc exp push --rev HEAD origin From c3e08e949a7998db9d10d4052a55d7d4807b6619 Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Tue, 31 May 2022 17:22:47 -0400 Subject: [PATCH 09/10] Update content/docs/user-guide/experiment-management/cleaning-experiments.md Co-authored-by: Jorge Orpinel --- .../user-guide/experiment-management/cleaning-experiments.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 66ab698711..951d64ff9a 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -165,9 +165,7 @@ $ dvc exp show --all-tags ```dvc $ dvc exp gc --all-tags -WARNING: This will remove all experiments except those derived from any tags of the current repo. Run queued experiments will be removed. -Are you sure you want to proceed? [y/n]: y -Removed 2 experiments. To remove unused cache files use 'dvc gc'. + $ dvc exp show --all-tags ``` From 53e9bbb7b2915926b170533a8e7aad89a9bb29ac Mon Sep 17 00:00:00 2001 From: Dave Berenbaum Date: Tue, 31 May 2022 17:24:22 -0400 Subject: [PATCH 10/10] Update content/docs/user-guide/experiment-management/cleaning-experiments.md --- .../user-guide/experiment-management/cleaning-experiments.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 951d64ff9a..82dcc19c0f 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -78,9 +78,6 @@ branches in this example. ```dvc $ dvc exp gc --workspace -WARNING: This will remove all experiments except those derived from the workspace of the current repo. Run queued experiments will be removed. -Are you sure you want to proceed? [y/n]: y -Removed 5 experiments. To remove unused cache files use 'dvc gc'. $ dvc exp list --all main: exp-abc000