From 040b646f143b16814db7c16a108fd66900c1ecb4 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 16:13:08 +0900 Subject: [PATCH 01/28] initial index --- .../command-reference/experiments/index.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 content/docs/command-reference/experiments/index.md diff --git a/content/docs/command-reference/experiments/index.md b/content/docs/command-reference/experiments/index.md new file mode 100644 index 0000000000..74ba0dcbc2 --- /dev/null +++ b/content/docs/command-reference/experiments/index.md @@ -0,0 +1,49 @@ +# exp[eriments] + +A set of commands to generate and compare local _experiments_: +[show](/doc/command-reference/experiments/show), +[checkout](/doc/command-reference/experiments/show), +[diff](/doc/command-reference/experiments/diff), +[run](/doc/command-reference/experiments/run), +[resume](/doc/command-reference/experiments/resume), and +[gc](/doc/command-reference/experiments/gc). + +## Synopsis + +```usage +usage: dvc exp[eriments] [-h] [-q | -v] {show,checkout,diff,run,resume,res,gc} ... + +Commands to display and compare experiments. +Documentation: + +positional arguments: + {show,checkout,diff,run,resume,res,gc} + Use `dvc experiments CMD --help` to display command-specific help. + show Print experiments. + checkout Checkout experiments. + diff Show changes between experiments in the DVC repository. + run Reproduce complete or partial experiment pipelines. + resume (res) Resume checkpoint experiments. + gc Garbage collect unneeded experiments. +``` + +## Types of experiments + +DVC supports two types of experiments: + +1. Standalone experiments +2. Checkpoints + +## Description + +## Options + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Example: Standalone experiment + +## Example: Checkpoint experiment From 8a3d245fe7c25cf4550443cc6674703598271c7c Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 16:13:20 +0900 Subject: [PATCH 02/28] initial show --- .../command-reference/experiments/show.md | 250 ++++++++++++++++++ 1 file changed, 250 insertions(+) create mode 100644 content/docs/command-reference/experiments/show.md diff --git a/content/docs/command-reference/experiments/show.md b/content/docs/command-reference/experiments/show.md new file mode 100644 index 0000000000..c486a09fd6 --- /dev/null +++ b/content/docs/command-reference/experiments/show.md @@ -0,0 +1,250 @@ +# exp[eriments] show + +Print [experiments](/doc/command-reference/experiments) table, with optional +formatting. + +## Synopsis + +```usage +usage: dvc exp show [-h] [-q | -v] [-a] [-T] [--all-commits] [--no-pager] + [--include-metrics ] + [--exclude-metrics ] + [--include-params ] + [--exclude-params ] + [--sort-by ] [--sort-order {asc,desc}] + [--no-timestamp] [--sha] [--show-json] + [--precision ] + +optional arguments: + -a, --all-branches Show metrics for all branches. + -T, --all-tags Show metrics for all tags. + --all-commits Show metrics for all commits. + --no-pager Do not pipe output into a pager. + --include-metrics + Include the specified metrics in output table. + --exclude-metrics + Exclude the specified metrics from output table. + --include-params + Include the specified params in output table. + --exclude-params + Exclude the specified params from output table. + --sort-by + Sort related experiments by the specified metric or param. + --sort-order {asc,desc} + Sort order to use with --sort-by. + --no-timestamp Do not show experiment timestamps. + --sha Always show git commit SHAs instead of branch/tag names. + --show-json Print output in JSON format instead of a human-readable table. + --precision Round metrics/params to `n` digits precision after the decimal point. Rounds to 5 digits by default. +``` + +## Description + +Finds and prints local experiments in the project. + +By default, all experiments derived from the current workspace will be shown. +The `-a`, `-T`, and `--all-commits` options can be used to display more +experiments. + +## Options + +- `-a`, `--all-branches` - print experiments derived from the tip of all Git + branches instead of just those derived from the current workspace. Note that + this can be combined with `-T` below, for example using the `-aT` flag. + +- `-T`, `--all-tags` - same as `-a` above, but applies to Git tags as well as + the workspace. Note that both options can be combined, for example using the + `-aT` flag. + +- `--all-commits` - same as `-a` or `-T` above, but applies to _all_ Git commits + as well as the workspace. This prints all experiments derived from the entire + commit history of the project. + +- `--no-pager` - do not pipe output into a pager. + +- `--include-metrics ` - include the specified metrics in output + table. When this option is used, only the specified metrics columns will be + shown. + +- `--exclude-metrics ` - exclude the specified metrics from output + table. When this option is used, all metric columns except for the specified + columns will be shown. + +- `--include-params ` - include the specified params in output + table. When this option is used, only the specified params columns will be + shown. + +- `--exclude-params ` - exclude the specified params from output + table. When this option is used, all params columns except for the specified + columns will be shown. + +- `--sort-by ` - sort related experiments by the specified metric + or param. Experiments in the printed table will be sorted by the specified + metric or param column. By default, experiments will be sorted by timestamp. + Only one sort column (either metric or param) can be specified. + +- `--sort-order {asc,desc}` - sort order to use with `--sort-by`. Defaults to + descending. + +- `--no-timestamp` - do not show experiment timestamps. + +- `--sha` - always show Git commit SHAs instead of branch/tag names. By default, + if commit is the tip of a Git branch or a Git tag, the branch or tag name will + be shown instead of the Git SHA. + +- `--show-json` - prints the command's output in easily parsable JSON format, + instead of a human-readable table. + +- `--precision ` - round metrics and params to `n` digits precision after the + decimal point. Rounds to 5 digits by default. + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Filtering the output + +By default, the printed experiments table will include columns for all metrics +and params from the entire project. The `--include-metrics`, +`--exclude-metrics`, `--include-params`, and `--exclude-params` options can be +used to limit the columns to be displayed. Each option accepts a comma-separated +list of metric or param names. + +When an `--include-...` option is used, only the specified metric/param columns +will be displayed. When an `--exclude-...` option is used, all columns except +for those specified will be displayed. + +`--include-...` and `--exclude-...` will match nested metrics and params. So for +a stage `featurize` with params `max_features` and `ngrams`, +`--include-params=featurize` would include columns for both +`featurize.max_features` and `featurize.ngrams`. + +Metric and param columns are handled separately. So for the command +`dvc exp show --include-metrics=foo,bar`, `foo` and `bar` would be the only +metrics displayed in the output table, and all possible params columns would +also be displayed in the table. + +## Sorting the output + +By default, experiment rows will be sorted by timestamp in descending order. The +`--sort-by` and `--sort-order` options can be used to sort related experiment +rows on any single metric or param column. + +Note that when sorting experiments, related experiments will remain grouped +together. This means that for a given Git commit `abc123`, all experiments +derived from that commit will be sorted and grouped together. Experiments +derived from a different Git commit `def456` would be sorted in their own group. + +## Paging the output + +This command's output is automatically piped to +[Less](), if available in the +terminal. (The exact command used is `less --chop-long-lines --clear-screen`.) +If `less` is not available (e.g. on Windows), the output is simply printed out. + +> It's also possible to +> [enable Less paging on Windows](/doc/user-guide/running-dvc-on-windows#enabling-paging-with-less). + +### Providing a custom pager + +It's possible to override the default pager via the `DVC_PAGER` environment +variable. For example, the following command will replace the default pager with +[`more`](), for a single run: + +```dvc +$ DVC_PAGER=more dvc dag +``` + +For a persistent change, define `DVC_PAGER` in the shell configuration. For +example in Bash, we could add the following line to `~/.bashrc`: + +```bash +export DVC_PAGER=more +``` + +## Example: Tabular data + +> This example is based on the `evaluate.dvc` stage file of our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +The basic use case shows the values in the current workspace: + +```dvc +$ dvc exp show +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_fea… ┃ featurize.ngrams ┃ prepare.seed ┃ prepare.split ┃ train.n_estimators ┃ train.seed ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +└───────────────────────┴──────────────┴─────────┴────────────────────┴──────────────────┴──────────────┴───────────────┴────────────────────┴────────────┘ +``` + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +To sort experiments by the `auc` metric in ascending order: + +```dvc +$ dvc exp show --include-params=featurize --sort-by=auc --sort-order=asc +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +To see all experiments in the workspace and down the Git history: + +```dvc +$ dvc exp show --all-commits --include-params=featurize --sort-by=auc --sort-order=asc +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ 10-bigrams-model │ Jun 20, 2020 │ 0.54175 │ 1500 │ 2 │ +│ └── 069d9cc │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ +│ 9-evaluation │ Jun 20, 2020 │ 0.54175 │ 500 │ 1 │ +│ 8-ml-pipeline │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 6-prep-stage │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 5-source-code │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 4-import-data │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 2-track-data │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 3-config-remote │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 1-dvc-init │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 0-git-init │ Jun 20, 2020 │ - │ 1500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +Note that in the final example, the top level Git commits remain in their +original order. The experiment sorting only applies to experiments grouped +according to each top level Git commit. + +The +[Compare Experiments](/doc/tutorials/get-started/experiments#compare-experiments) +chapter of our _Get Started_ covers the `-a` option to collect and print a +metrics file value across all Git branches. From be44cf079119e8fa68b7167d979dae76f2cb1a84 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 16:35:31 +0900 Subject: [PATCH 03/28] squash --- content/docs/command-reference/experiments/show.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/experiments/show.md b/content/docs/command-reference/experiments/show.md index c486a09fd6..c38cc19502 100644 --- a/content/docs/command-reference/experiments/show.md +++ b/content/docs/command-reference/experiments/show.md @@ -167,7 +167,7 @@ export DVC_PAGER=more ## Example: Tabular data -> This example is based on the `evaluate.dvc` stage file of our +> This example is based on our > [Get Started](/doc/tutorials/get-started/experiments), where you can find the > actual source code. From 206bcfb1d4b6155b83dbf9bf34ba39f9cff1f0c8 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 16:36:46 +0900 Subject: [PATCH 04/28] initial checkout --- .../command-reference/experiments/checkout.md | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 content/docs/command-reference/experiments/checkout.md diff --git a/content/docs/command-reference/experiments/checkout.md b/content/docs/command-reference/experiments/checkout.md new file mode 100644 index 0000000000..3e6f88e87f --- /dev/null +++ b/content/docs/command-reference/experiments/checkout.md @@ -0,0 +1,138 @@ +# exp[eriments] checkout + +Checkout changes from a previously run experiment into the workspace. + +## Synopsis + +```usage +usage: dvc exp checkout [-h] [-q | -v] experiment + +positional arguments: + experiment Checkout this experiment. +``` + +## Description + +This command will apply the changes from an experiment to the workspace. It is +usually run when you are ready to Git commit the results of an experiment into +the project. + +This command will fail if the specified experiment was not derived from the Git +commit which is currently (Git) checkout-ed in the workspace. + +## Options + +- `experiment` - revision SHA for the experiment to be checked out. + +- `-h`, `--help` - shows the help message and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information from executing the + `dvc pull` command. + +## Example: Checkout and promote an experiment + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have run 3 experiments in our project workspace: + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +We now wish to commit the results of experiment `1dad0d2` into our project. + +```dvc +$ dvc exp checkout 1dad0d2 +Changes for experiment '1dad0d2' have been applied to your current workspace. +``` + +We can inspect these changes with Git: + +```dvc +$ git status +On branch master +Your branch is up to date with 'origin/master'. + +Changes not staged for commit: + modified: dvc.lock + modified: params.yaml + modified: prc.json + modified: scores.json + +$ git diff params.yaml scores.json +diff --git a/params.yaml b/params.yaml +index 4c4d898..faf781a 100644 +--- a/params.yaml ++++ b/params.yaml +@@ -3,7 +3,7 @@ prepare: + seed: 20170428 + + featurize: +- max_features: 1500 ++ max_features: 2000 + ngrams: 2 + + train: +diff --git a/scores.json b/scores.json +index c995f24..c640c4e 100644 +--- a/scores.json ++++ b/scores.json +@@ -1 +1 @@ +-{"auc": 0.6131382960762474} +\ No newline at end of file ++{"auc": 0.5775633054725381} +\ No newline at end of file +``` + +and with DVC: + +``` +$ dvc status +Data and pipelines are up to date. + +$ dvc diff +Modified: + data/features/ + data/features/test.pkl + data/features/train.pkl + model.pkl + prc.json + scores.json + +files summary: 0 added, 0 deleted, 5 modified, 0 not in cache +``` + +To promote this experiment we simply `git add` and `git commit` the changes: + +```dvc +$ git add . +$ git commit -m "promote experiment 1dad0d2" +[master 0412386] promote experiment 1dad0d2 +``` + +Finally, we can now see that the promoted experiment is the new tip of our +master branch: + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.57756 │ 2000 │ 2 │ +│ master │ 04:31 PM │ 0.57756 │ 2000 │ 2 │ +└────────────┴──────────┴─────────┴────────────────────────┴──────────────────┘ +``` From ee41c0346e8380999b10a147df7246858a9905c1 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 16:52:27 +0900 Subject: [PATCH 05/28] initial diff --- .../command-reference/experiments/diff.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 content/docs/command-reference/experiments/diff.md diff --git a/content/docs/command-reference/experiments/diff.md b/content/docs/command-reference/experiments/diff.md new file mode 100644 index 0000000000..68f69b32e2 --- /dev/null +++ b/content/docs/command-reference/experiments/diff.md @@ -0,0 +1,105 @@ +# exp[eriments] diff + +Show changes in [metrics](/doc/command-reference/metrics) and +[parameters](/doc/command-reference/params) between experiments. + +## Synopsis + +```usage +usage: dvc exp diff [-h] [-q | -v] [--all] [--show-json] [--show-md] + [--old] [--no-path] [--precision ] [a_rev] [b_rev] + +positional arguments: + a_rev Old experiment to compare (defaults to HEAD) + b_rev New experiment to compare (defaults to the current workspace) +``` + +## Description + +This command simply combines the output of both `dvc metrics diff` and +`dvc params diff` for quickly comparing two experiments. + +`a_rev` and `b_rev` can be experiment hashes, Git commit hashes, tag, or branch +names. If only `a_rev` is provided and `a_rev` is an experiment hash, it will be +compared to the baseline commit from which the experiment was derived. + +## Options + +- `--all` - prints all parameters including not changed. + +- `--show-json` - prints the command's output in easily parsable JSON format, + instead of a human-readable table. + +- `--show-md` - prints the command's output in the Markdown table format. + +- `--old` - show old metric value in addition to the new value. + +- `--no-path` - don't show metric or param path in the result table. This option + is useful when only one metrics or params file is in use or there is no + intersection between the metric or param names. + +- `--precision ` - + [round](https://docs.python.org/3/library/functions.html#round) metrics to `n` + digits precision after the decimal point. Rounds to 5 digits by default. + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Example + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have run 3 experiments in our project workspace: + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +To compare a experiment `1dad0d2` to its baseline (`11-bigrams-experiment`): + +```dvc +$ dvc exp diff 1dad0d2 +Path Metric Value Change +scores.json auc 0.61314 0.035575 + +Path Param Value Change +params.yaml featurize.max_features 1500 -500 +``` + +To compare two experiments: + +```dvc +$ dvc exp diff 1dad0d2 1df77f7 +Path Metric Value Change +scores.json auc 0.51676 -0.060799 + +Path Param Value Change +params.yaml featurize.max_features 500 -1500 +``` + +To compare an experiment to a Git tag: + +```dvc +$ dvc exp diff 1dad0d2 8-ml-pipeline +Path Metric Value Change +scores.json auc None diff not supported + +Path Param Value Change +params.yaml featurize.max_features 500 -1500 +params.yaml featurize.ngrams 1 -1 +``` From 2a2daf84769004f884da20936a9b5f90ec991e59 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 17:27:49 +0900 Subject: [PATCH 06/28] initial run/resume --- .../command-reference/experiments/resume.md | 36 ++++++++ .../docs/command-reference/experiments/run.md | 89 +++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 content/docs/command-reference/experiments/resume.md create mode 100644 content/docs/command-reference/experiments/run.md diff --git a/content/docs/command-reference/experiments/resume.md b/content/docs/command-reference/experiments/resume.md new file mode 100644 index 0000000000..b49b7d64e5 --- /dev/null +++ b/content/docs/command-reference/experiments/resume.md @@ -0,0 +1,36 @@ +# exp[eriments] res[ume] + +Resume (continue) existing checkpoint experiments. + +## Synopsis + +```usage +usage: dvc exp res [-h] [-q | -v] [-r ] + [--params [:]] +``` + +## Description + +`dvc exp res` can be used to resume execution of a checkpoint experiment which +was previously started via `dvc exp run`. + +## Options + +- `-r ` - if this option is provided, the specified checkpoint + experiment will be resumed. Otherwise, the most recently run checkpoint + experiment will be resumed. + +- `--params [:]]` - if this option is provided, a new + checkpoint branch will be created (starting from the checkpoint provided via + `-r`) using the specified parameter changes. + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if all + stages are up to date or if all stages are successfully executed, otherwise + exit with 1. The command defined in the stage is free to write output + regardless of this flag. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Examples diff --git a/content/docs/command-reference/experiments/run.md b/content/docs/command-reference/experiments/run.md new file mode 100644 index 0000000000..ae6c092ec4 --- /dev/null +++ b/content/docs/command-reference/experiments/run.md @@ -0,0 +1,89 @@ +# exp[eriments] run + +Reproduce complete or partial experiment [pipelines](/doc/command-reference/dag) +by executing commands defined in their [stages](/doc/command-reference/run) in +the correct order. Pipelines will be reproduced within local experiments + +## Synopsis + +```usage +usage: dvc exp run [-h] [-q | -v] [-f] + [ ...] + [--params [:]] + [--queue] [--run-all] [-j ] + [targets [targets ...]] + +positional arguments: + targets Stages to reproduce. 'dvc.yaml' by default. +``` + +## Description + +`dvc exp run` provides a way to easily generate experiment results without +needing to pollute your project with large numbers of experiment Git branches or +commits. + +`dvc exp run` is equivalent to `dvc repro` for experiments. The state of the +current workspace (including code, params, and dependency changes) will be +reproduced as a standalone or checkpoint experiment. The `--params` option can +be used to generate experiments with parameter values specified on the +command-line, without needing to manually edit a params file in the workspace. + +To get hands-on experience with data science and machine learning pipelines, see +[Get Started: Data Pipelines](/doc/start/data-pipelines). + +`dvc exp run` differs from `dvc repro` in that data files and intermediate or +final results will always be saved into the DVC cache. + +Experiment results will be +[checked out](/doc/command-reference/experiments/checkout) into the workspace +upon completion. + +For checkpoint experiments, `dvc exp resume` should be used in conjunction with +`dvc exp run`. + +## Options + +`dvc exp run` accepts the same options as `dvc repro`, with the exception that +`--no-commit` has no effect in `dvc exp run`. See `dvc repro` for a detailed +list of available options. + +- `-f`, `--force` - reproduce an experiment pipeline, regenerating its results, + even if no changes were found. For `checkpoint` experiments, `dvc exp run -f` + will force regeneration of an existing checkpoint experiment branch from + scratch. + +- `--params [:]]` - reproduce an experiment using the + specified [parameter](/doc/command-reference/params) values. Accepts a + comma-separated list of key-value pairs in the form + `[filename:]key1=val1,key2=val2...`. Any specified values will override those + from the contents of the relevant params file. + +- `--queue` - queue an experiment for future execution, but do not actually run + the pipeline. + +- `--run-all` - run all experiments which have been staged for execution via + `--queue`. Can be used in conjunction with `-j` to run experiments in + parallel. + +- `-j ` - run the specified number of experiments at a time in parallel, + only applicable when used in conjunction with `--run-all`. + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if all + stages are up to date or if all stages are successfully executed, otherwise + exit with 1. The command defined in the stage is free to write output + regardless of this flag. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Parallel experiment execution + +Note: parallel execution is currently considered experimental. DVC makes no +attempt to intelligently schedule or prioritize execution of stages within +pipelines. When using `-j` to run experiments in parallel, take caution to only +specify a number of jobs which can be handled given your local machine's +avaiable resources. + +## Examples From fe159d3dbb3de5b3de94e5092eb6d37a5acd0da1 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 17:46:09 +0900 Subject: [PATCH 07/28] initial gc --- .../docs/command-reference/experiments/gc.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 content/docs/command-reference/experiments/gc.md diff --git a/content/docs/command-reference/experiments/gc.md b/content/docs/command-reference/experiments/gc.md new file mode 100644 index 0000000000..6be7b17aad --- /dev/null +++ b/content/docs/command-reference/experiments/gc.md @@ -0,0 +1,144 @@ +# exp[eriments] gc + +Remove unneeded experiments from the project. + +## Synopsis + +```usage +usage: dvc exp gc [-h] [-q | -v] [-w] [-a] [-T] [--all-commits] [--queued] [-f] + +optional arguments: + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose. + -w, --workspace Keep experiments derived from the current workspace. + -a, --all-branches Keep experiments derived from the tips of all Git branches. + -T, --all-tags Keep experiments derived from all Git tags. + --all-commits Keep experiments derived from all Git commits. + --queued Keep queued experiments (experiments run queue will be cleared by default). + -f, --force Force garbage collection - automatically agree to all prompts. +``` + +## Description + +This command deletes (garbage collects) experiments that exist in the project +but are no longer needed. + +After experiments are removed with `dvc exp gc`, files and directories that are +no longer needed may remain in the DVC cache. `dvc gc` should be used as needed +to remove unneeded DVC cache data. + +One of the scope options (`--workspace`, `--all-branches`, `--all-tags`, +`--all-commits`) or a combination of them must be provided. Each of them +corresponds to keeping the experiments derived from the current workspace, and +for a certain set of commits. See the [Options](#options) section for more +details. + +## Options + +- `-w`, `--workspace` - keep experiments _only_ derived from the current + workspace This option is enabled automatically if `--all-tags`, + `--all-branches`, or `--all-commits` are used. + +- `-a`, `--all-branches` - keep experiments derived from the tips of all Git + branches as well as the workspace (implies `-w`). Note that this can be + combined with `-T` below, for example using the `-aT` flag. + +- `-T`, `--all-tags` - same as `-a` above, but applies to Git tags as well as + the workspace (implies `-w`). Note that both options can be combined, for + example using the `-aT` flag. + +- `--all-commits` - same as `-a` or `-T` above, but applies to _all_ Git commits + as well as the workspace (implies `-w`). This is mainly only needed when + clearing the experiments run queue. + +- `--queued` - keep experiments staged for execution via `dvc exp run --queue`. + The experiments run queue will be cleared by default if this option is + omitted. + +- `-f`, `--force` - force garbage collection. Skip confirmation prompt. + +- `-h`, `--help` - prints the usage/help message, and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information. + +## Examples + +Let's say we have the following project, have +[checked out](/docs/command-reference/experiments/checkout) and promoted an +experiment to a full commit: + +```dvc +$ dvc exp show --all-commits --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.57756 │ 2000 │ 2 │ +│ master │ 05:39 PM │ 0.57756 │ 2000 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ 10-bigrams-model │ Jun 20, 2020 │ 0.54175 │ 1500 │ 2 │ +│ └── 069d9cc │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ +│ 9-evaluation │ Jun 20, 2020 │ 0.54175 │ 500 │ 1 │ +│ 8-ml-pipeline │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 6-prep-stage │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 5-source-code │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 4-import-data │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 2-track-data │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 3-config-remote │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 1-dvc-init │ Jun 20, 2020 │ - │ 1500 │ 2 │ +│ 0-git-init │ Jun 20, 2020 │ - │ 1500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +Now wish to remove all old (and now unnecessary) experiments: + +```dvc +$ dvc exp gc -w +WARNING: This will remove all experiments except those derived from the workspace of the current repo. +Are you sure you want to proceed? [y/n] y +Removed 4 experiments. To remove unused cache files use 'dvc gc'. +``` + +We can confirm that the old experiments are now removed: + +```dvc +$ dvc exp show --all-commits --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.57756 │ 2000 │ 2 │ +│ master │ 05:39 PM │ 0.57756 │ 2000 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ 10-bigrams-model │ Jun 20, 2020 │ 0.54175 │ 1500 │ 2 │ +│ 9-evaluation │ Jun 20, 2020 │ 0.54175 │ 500 │ 1 │ +│ 8-ml-pipeline │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 6-prep-stage │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 5-source-code │ Jun 20, 2020 │ - │ 500 │ 1 │ +│ 4-import-data │ Jun 20, 2020 │ - │ 2000 │ 2 │ +│ 2-track-data │ Jun 20, 2020 │ - │ 2000 │ 2 │ +│ 3-config-remote │ Jun 20, 2020 │ - │ 2000 │ 2 │ +│ 1-dvc-init │ Jun 20, 2020 │ - │ 2000 │ 2 │ +│ 0-git-init │ Jun 20, 2020 │ - │ 2000 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +To remove any DVC cache files which are no longer needed, we can use `dvc gc` +with the appropriate options: + +```dvc +$ dvc dvc gc --all-commits +WARNING: This will remove all cache except items used in the workspace and all git commits of the current repo. +Are you sure you want to proceed? [y/n] y +WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files: +name: data/features, md5: a50924d27b5ff0f023344d9b33d1d7bf.dir +Missing cache for directory 'data/features'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y +``` + +Note the use of `--all-commits` to ensure that we do not garbage collect cache +files which are still referenced by top level commits in our project. From 3850f6dc4f3eca31acb30bc835a79053eb03b716 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Wed, 11 Nov 2020 17:56:33 +0900 Subject: [PATCH 08/28] adjust slugs --- .../{experiments => exp}/checkout.md | 2 +- .../{experiments => exp}/diff.md | 2 +- .../{experiments => exp}/gc.md | 6 ++-- .../{experiments => exp}/index.md | 11 +++---- .../{experiments => exp}/resume.md | 8 ++--- .../{experiments => exp}/run.md | 7 ++--- .../{experiments => exp}/show.md | 5 ++- content/docs/sidebar.json | 31 +++++++++++++++++++ 8 files changed, 50 insertions(+), 22 deletions(-) rename content/docs/command-reference/{experiments => exp}/checkout.md (99%) rename content/docs/command-reference/{experiments => exp}/diff.md (99%) rename content/docs/command-reference/{experiments => exp}/gc.md (98%) rename content/docs/command-reference/{experiments => exp}/index.md (79%) rename content/docs/command-reference/{experiments => exp}/resume.md (82%) rename content/docs/command-reference/{experiments => exp}/run.md (96%) rename content/docs/command-reference/{experiments => exp}/show.md (99%) diff --git a/content/docs/command-reference/experiments/checkout.md b/content/docs/command-reference/exp/checkout.md similarity index 99% rename from content/docs/command-reference/experiments/checkout.md rename to content/docs/command-reference/exp/checkout.md index 3e6f88e87f..33e235d82e 100644 --- a/content/docs/command-reference/experiments/checkout.md +++ b/content/docs/command-reference/exp/checkout.md @@ -1,4 +1,4 @@ -# exp[eriments] checkout +# exp checkout Checkout changes from a previously run experiment into the workspace. diff --git a/content/docs/command-reference/experiments/diff.md b/content/docs/command-reference/exp/diff.md similarity index 99% rename from content/docs/command-reference/experiments/diff.md rename to content/docs/command-reference/exp/diff.md index 68f69b32e2..ac8d700f1e 100644 --- a/content/docs/command-reference/experiments/diff.md +++ b/content/docs/command-reference/exp/diff.md @@ -1,4 +1,4 @@ -# exp[eriments] diff +# exp diff Show changes in [metrics](/doc/command-reference/metrics) and [parameters](/doc/command-reference/params) between experiments. diff --git a/content/docs/command-reference/experiments/gc.md b/content/docs/command-reference/exp/gc.md similarity index 98% rename from content/docs/command-reference/experiments/gc.md rename to content/docs/command-reference/exp/gc.md index 6be7b17aad..1289ff8383 100644 --- a/content/docs/command-reference/experiments/gc.md +++ b/content/docs/command-reference/exp/gc.md @@ -1,4 +1,4 @@ -# exp[eriments] gc +# exp gc Remove unneeded experiments from the project. @@ -68,8 +68,8 @@ details. ## Examples Let's say we have the following project, have -[checked out](/docs/command-reference/experiments/checkout) and promoted an -experiment to a full commit: +[checked out](/docs/command-reference/exp/checkout) and promoted an experiment +to a full commit: ```dvc $ dvc exp show --all-commits --include-params=featurize diff --git a/content/docs/command-reference/experiments/index.md b/content/docs/command-reference/exp/index.md similarity index 79% rename from content/docs/command-reference/experiments/index.md rename to content/docs/command-reference/exp/index.md index 74ba0dcbc2..3935a412e3 100644 --- a/content/docs/command-reference/experiments/index.md +++ b/content/docs/command-reference/exp/index.md @@ -1,12 +1,11 @@ # exp[eriments] A set of commands to generate and compare local _experiments_: -[show](/doc/command-reference/experiments/show), -[checkout](/doc/command-reference/experiments/show), -[diff](/doc/command-reference/experiments/diff), -[run](/doc/command-reference/experiments/run), -[resume](/doc/command-reference/experiments/resume), and -[gc](/doc/command-reference/experiments/gc). +[show](/doc/command-reference/exp/show), +[checkout](/doc/command-reference/exp/show), +[diff](/doc/command-reference/exp/diff), [run](/doc/command-reference/exp/run), +[resume](/doc/command-reference/exp/resume), and +[gc](/doc/command-reference/exp/gc). ## Synopsis diff --git a/content/docs/command-reference/experiments/resume.md b/content/docs/command-reference/exp/resume.md similarity index 82% rename from content/docs/command-reference/experiments/resume.md rename to content/docs/command-reference/exp/resume.md index b49b7d64e5..805e235a5b 100644 --- a/content/docs/command-reference/experiments/resume.md +++ b/content/docs/command-reference/exp/resume.md @@ -1,18 +1,18 @@ -# exp[eriments] res[ume] +# exp res[ume] Resume (continue) existing checkpoint experiments. ## Synopsis ```usage -usage: dvc exp res [-h] [-q | -v] [-r ] +usage: dvc exp res[ume] [-h] [-q | -v] [-r ] [--params [:]] ``` ## Description -`dvc exp res` can be used to resume execution of a checkpoint experiment which -was previously started via `dvc exp run`. +`dvc exp resume` can be used to resume execution of a checkpoint experiment +which was previously started via `dvc exp run`. ## Options diff --git a/content/docs/command-reference/experiments/run.md b/content/docs/command-reference/exp/run.md similarity index 96% rename from content/docs/command-reference/experiments/run.md rename to content/docs/command-reference/exp/run.md index ae6c092ec4..d3fac440d6 100644 --- a/content/docs/command-reference/experiments/run.md +++ b/content/docs/command-reference/exp/run.md @@ -1,4 +1,4 @@ -# exp[eriments] run +# exp run Reproduce complete or partial experiment [pipelines](/doc/command-reference/dag) by executing commands defined in their [stages](/doc/command-reference/run) in @@ -35,9 +35,8 @@ To get hands-on experience with data science and machine learning pipelines, see `dvc exp run` differs from `dvc repro` in that data files and intermediate or final results will always be saved into the DVC cache. -Experiment results will be -[checked out](/doc/command-reference/experiments/checkout) into the workspace -upon completion. +Experiment results will be [checked out](/doc/command-reference/exp/checkout) +into the workspace upon completion. For checkpoint experiments, `dvc exp resume` should be used in conjunction with `dvc exp run`. diff --git a/content/docs/command-reference/experiments/show.md b/content/docs/command-reference/exp/show.md similarity index 99% rename from content/docs/command-reference/experiments/show.md rename to content/docs/command-reference/exp/show.md index c38cc19502..63fd46bf1f 100644 --- a/content/docs/command-reference/experiments/show.md +++ b/content/docs/command-reference/exp/show.md @@ -1,7 +1,6 @@ -# exp[eriments] show +# exp show -Print [experiments](/doc/command-reference/experiments) table, with optional -formatting. +Print [experiments](/doc/command-reference/exp) table, with optional formatting. ## Synopsis diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 68b5ca0b26..1718b6fc0b 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -196,6 +196,37 @@ "label": "doctor", "slug": "doctor" }, + { + "label": "experiments", + "slug": "exp", + "source": "exp/index.md", + "children": [ + { + "label": "exp checkout", + "slug": "checkout" + }, + { + "label": "exp diff", + "slug": "diff" + }, + { + "label": "exp gc", + "slug": "gc" + }, + { + "label": "exp resume", + "slug": "resume" + }, + { + "label": "exp run", + "slug": "run" + }, + { + "label": "exp show", + "slug": "show" + } + ] + }, { "label": "fetch", "slug": "fetch", From d7ed11400d8c143f231bf267ef293201b14b7fae Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 12 Nov 2020 01:32:25 -0600 Subject: [PATCH 09/28] Update content/docs/command-reference/exp/index.md --- content/docs/command-reference/exp/index.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 3935a412e3..6719455bda 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -12,9 +12,6 @@ A set of commands to generate and compare local _experiments_: ```usage usage: dvc exp[eriments] [-h] [-q | -v] {show,checkout,diff,run,resume,res,gc} ... -Commands to display and compare experiments. -Documentation: - positional arguments: {show,checkout,diff,run,resume,res,gc} Use `dvc experiments CMD --help` to display command-specific help. From a9f6289848aafe8e0ed837e1411c279c182441fe Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 12 Nov 2020 01:33:30 -0600 Subject: [PATCH 10/28] Update content/docs/command-reference/exp/index.md --- content/docs/command-reference/exp/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 6719455bda..963904410f 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -13,7 +13,7 @@ A set of commands to generate and compare local _experiments_: usage: dvc exp[eriments] [-h] [-q | -v] {show,checkout,diff,run,resume,res,gc} ... positional arguments: - {show,checkout,diff,run,resume,res,gc} + COMMAND Use `dvc experiments CMD --help` to display command-specific help. show Print experiments. checkout Checkout experiments. From 783b48b50b7a6361b420cb0b03d1a258bb90f3de Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 12 Nov 2020 01:33:51 -0600 Subject: [PATCH 11/28] Update content/docs/command-reference/exp/index.md --- content/docs/command-reference/exp/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 963904410f..6432732ef6 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -14,7 +14,6 @@ usage: dvc exp[eriments] [-h] [-q | -v] {show,checkout,diff,run,resume,res,gc} . positional arguments: COMMAND - Use `dvc experiments CMD --help` to display command-specific help. show Print experiments. checkout Checkout experiments. diff Show changes between experiments in the DVC repository. From 09a9256d646f39e60f8163f58c5395dacd1e174e Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 20:37:53 +0900 Subject: [PATCH 12/28] rename exp checkout to exp apply --- .../exp/{checkout.md => apply.md} | 26 +++++++++---------- content/docs/command-reference/exp/gc.md | 2 +- content/docs/command-reference/exp/index.md | 16 ++++++++---- content/docs/sidebar.json | 4 +-- 4 files changed, 27 insertions(+), 21 deletions(-) rename content/docs/command-reference/exp/{checkout.md => apply.md} (87%) diff --git a/content/docs/command-reference/exp/checkout.md b/content/docs/command-reference/exp/apply.md similarity index 87% rename from content/docs/command-reference/exp/checkout.md rename to content/docs/command-reference/exp/apply.md index 33e235d82e..f4ec5537e2 100644 --- a/content/docs/command-reference/exp/checkout.md +++ b/content/docs/command-reference/exp/apply.md @@ -1,14 +1,14 @@ -# exp checkout +# exp apply -Checkout changes from a previously run experiment into the workspace. +Apply the changes from a previously run experiment into the workspace. ## Synopsis ```usage -usage: dvc exp checkout [-h] [-q | -v] experiment +usage: dvc exp apply [-h] [-q | -v] experiment positional arguments: - experiment Checkout this experiment. + experiment Experiment to be applied. ``` ## Description @@ -22,7 +22,7 @@ commit which is currently (Git) checkout-ed in the workspace. ## Options -- `experiment` - revision SHA for the experiment to be checked out. +- `experiment` - name or revision of the experiment to be checked out. - `-h`, `--help` - shows the help message and exit. @@ -32,7 +32,7 @@ commit which is currently (Git) checkout-ed in the workspace. - `-v`, `--verbose` - displays detailed tracing information from executing the `dvc pull` command. -## Example: Checkout and promote an experiment +## Example: Apply and promote an experiment > This example is based on our > [Get Started](/doc/tutorials/get-started/experiments), where you can find the @@ -47,17 +47,17 @@ $ dvc exp show --include-params=featurize ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ └───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ ``` We now wish to commit the results of experiment `1dad0d2` into our project. ```dvc -$ dvc exp checkout 1dad0d2 -Changes for experiment '1dad0d2' have been applied to your current workspace. +$ dvc exp apply exp-1dad0 +Changes for experiment 'exp-1dad0' have been applied to your current workspace. ``` We can inspect these changes with Git: @@ -120,8 +120,8 @@ To promote this experiment we simply `git add` and `git commit` the changes: ```dvc $ git add . -$ git commit -m "promote experiment 1dad0d2" -[master 0412386] promote experiment 1dad0d2 +$ git commit -m "promote experiment exp-1dad0" +[master 0412386] promote experiment exp-1dad0 ``` Finally, we can now see that the promoted experiment is the new tip of our diff --git a/content/docs/command-reference/exp/gc.md b/content/docs/command-reference/exp/gc.md index 1289ff8383..d3df1cf89b 100644 --- a/content/docs/command-reference/exp/gc.md +++ b/content/docs/command-reference/exp/gc.md @@ -68,7 +68,7 @@ details. ## Examples Let's say we have the following project, have -[checked out](/docs/command-reference/exp/checkout) and promoted an experiment +[applied](/docs/command-reference/exp/apply) and promoted an experiment to a full commit: ```dvc diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 6432732ef6..0c8e36366c 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -1,25 +1,31 @@ -# exp[eriments] +# experiments -A set of commands to generate and compare local _experiments_: +A set of commands to generate and compare local experiments: +[apply](/doc/command-reference/exp/apply), [show](/doc/command-reference/exp/show), -[checkout](/doc/command-reference/exp/show), [diff](/doc/command-reference/exp/diff), [run](/doc/command-reference/exp/run), [resume](/doc/command-reference/exp/resume), and [gc](/doc/command-reference/exp/gc). +Note that `experiments` is aliased to `exp`. + ## Synopsis ```usage -usage: dvc exp[eriments] [-h] [-q | -v] {show,checkout,diff,run,resume,res,gc} ... +usage: dvc experiments [-h] [-q | -v] {show,apply,diff,run,resume,res,gc,branch,list,push,pull} ... positional arguments: COMMAND show Print experiments. - checkout Checkout experiments. + apply Apply the changes from an experiment to your workspace. diff Show changes between experiments in the DVC repository. run Reproduce complete or partial experiment pipelines. resume (res) Resume checkpoint experiments. gc Garbage collect unneeded experiments. + branch Promote an experiment to a Git branch. + list List local and remote experiments. + push Push a local experiment to a Git remote. + pull Pull an experiment from a Git remote. ``` ## Types of experiments diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 1718b6fc0b..0dc28d1e56 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -202,8 +202,8 @@ "source": "exp/index.md", "children": [ { - "label": "exp checkout", - "slug": "checkout" + "label": "exp apply", + "slug": "apply" }, { "label": "exp diff", From f37dc399ebf886d6b8d92092e3c1cd51ca5887e2 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 20:39:01 +0900 Subject: [PATCH 13/28] update sidebar and index --- content/docs/command-reference/exp/gc.md | 4 ++-- content/docs/command-reference/exp/index.md | 16 ++++++++++------ content/docs/sidebar.json | 16 ++++++++++++++++ 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/content/docs/command-reference/exp/gc.md b/content/docs/command-reference/exp/gc.md index d3df1cf89b..c6a297c2ed 100644 --- a/content/docs/command-reference/exp/gc.md +++ b/content/docs/command-reference/exp/gc.md @@ -68,8 +68,8 @@ details. ## Examples Let's say we have the following project, have -[applied](/docs/command-reference/exp/apply) and promoted an experiment -to a full commit: +[applied](/docs/command-reference/exp/apply) and promoted an experiment to a +full commit: ```dvc $ dvc exp show --all-commits --include-params=featurize diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 0c8e36366c..3e1e89a1df 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -2,12 +2,16 @@ A set of commands to generate and compare local experiments: [apply](/doc/command-reference/exp/apply), -[show](/doc/command-reference/exp/show), -[diff](/doc/command-reference/exp/diff), [run](/doc/command-reference/exp/run), -[resume](/doc/command-reference/exp/resume), and -[gc](/doc/command-reference/exp/gc). - -Note that `experiments` is aliased to `exp`. +[branch](/doc/command-reference/exp/branch), +[diff](/doc/command-reference/exp/diff), [gc](/doc/command-reference/exp/gc), +[list](/doc/command-reference/exp/list), +[pull](/doc/command-reference/exp/pull), +[push](/doc/command-reference/exp/list), +[resume](/doc/command-reference/exp/resume), +[run](/doc/command-reference/exp/run), and +[show](/doc/command-reference/exp/show). + +Note that `experiments` is also aliased to `exp`. ## Synopsis diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 0dc28d1e56..578b0c95b1 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -205,6 +205,10 @@ "label": "exp apply", "slug": "apply" }, + { + "label": "exp branch", + "slug": "branch" + }, { "label": "exp diff", "slug": "diff" @@ -213,6 +217,18 @@ "label": "exp gc", "slug": "gc" }, + { + "label": "exp list", + "slug": "list" + }, + { + "label": "exp pull", + "slug": "pull" + }, + { + "label": "exp push", + "slug": "pull" + }, { "label": "exp resume", "slug": "resume" From dfc3fef35735b3ec07762268611e184d88690703 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 20:46:39 +0900 Subject: [PATCH 14/28] update diff --- content/docs/command-reference/exp/diff.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/content/docs/command-reference/exp/diff.md b/content/docs/command-reference/exp/diff.md index ac8d700f1e..1ded0465dd 100644 --- a/content/docs/command-reference/exp/diff.md +++ b/content/docs/command-reference/exp/diff.md @@ -64,16 +64,16 @@ $ dvc exp show --include-params=featurize ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ └───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ ``` -To compare a experiment `1dad0d2` to its baseline (`11-bigrams-experiment`): +To compare a experiment `exp-1dad0` to its baseline (`11-bigrams-experiment`): ```dvc -$ dvc exp diff 1dad0d2 +$ dvc exp diff exp-1dad0 Path Metric Value Change scores.json auc 0.61314 0.035575 @@ -84,7 +84,7 @@ params.yaml featurize.max_features 1500 -500 To compare two experiments: ```dvc -$ dvc exp diff 1dad0d2 1df77f7 +$ dvc exp diff exp-1dad0 exp-1df77 Path Metric Value Change scores.json auc 0.51676 -0.060799 @@ -95,7 +95,7 @@ params.yaml featurize.max_features 500 -1500 To compare an experiment to a Git tag: ```dvc -$ dvc exp diff 1dad0d2 8-ml-pipeline +$ dvc exp diff exp-1dad0 8-ml-pipeline Path Metric Value Change scores.json auc None diff not supported From 41d7bf30d649df87915ebb4e7a2f4bab245040a4 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 20:52:51 +0900 Subject: [PATCH 15/28] update gc --- content/docs/command-reference/exp/gc.md | 8 ++++---- content/docs/command-reference/gc.md | 13 +++++++++---- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/content/docs/command-reference/exp/gc.md b/content/docs/command-reference/exp/gc.md index c6a297c2ed..aee0fd2dfb 100644 --- a/content/docs/command-reference/exp/gc.md +++ b/content/docs/command-reference/exp/gc.md @@ -79,11 +79,11 @@ $ dvc exp show --all-commits --include-params=featurize │ workspace │ - │ 0.57756 │ 2000 │ 2 │ │ master │ 05:39 PM │ 0.57756 │ 2000 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ │ 10-bigrams-model │ Jun 20, 2020 │ 0.54175 │ 1500 │ 2 │ -│ └── 069d9cc │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ +│ └── exp-069d9 │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ │ 9-evaluation │ Jun 20, 2020 │ 0.54175 │ 500 │ 1 │ │ 8-ml-pipeline │ Jun 20, 2020 │ - │ 500 │ 1 │ │ 6-prep-stage │ Jun 20, 2020 │ - │ 500 │ 1 │ diff --git a/content/docs/command-reference/gc.md b/content/docs/command-reference/gc.md index 013f96bb4d..13a5f40d24 100644 --- a/content/docs/command-reference/gc.md +++ b/content/docs/command-reference/gc.md @@ -7,7 +7,8 @@ Remove unused files and directories from cache or ```usage usage: dvc gc [-h] [-q | -v] - [-w] [-a] [-T] [--all-commits] [-c] [-r ] + [-w] [-a] [-T] [--all-commits] [--all-experiments] + [-c] [-r ] [-f] [-j ] [-p [ [ ...]]] ``` @@ -23,9 +24,9 @@ explicitly provide the right set of options to specify what data is still needed (so that DVC can figure out what files can be safely deleted). One of the scope options (`--workspace`, `--all-branches`, `--all-tags`, -`--all-commits`) or a combination of them must be provided. Each of them -corresponds to keeping the data for the current workspace, and for a certain set -of commits (determined by reading the DVC-files in them). See the +`--all-commits`, `--all-experiments`) or a combination of them must be provided. +Each of them corresponds to keeping the data for the current workspace, and for +a certain set of commits (determined by reading the DVC-files in them). See the [Options](#options) section for more details. > Note that `dvc gc` tries to fetch any missing @@ -75,6 +76,10 @@ The default remote is cleaned (see `dvc config core.remote`) unless the that is never referenced from the workspace or from any Git commit can still be stored in the project's cache). +- `--all-experiments` same as `a`, `T`, or `--all-commits`, but applies to + [experiments](/doc/command-reference/exp). This keeps all the data used in all + local experiments (including intermediate checkpoints) in the project. + - `-p `, `--projects ` - if a single remote or a single cache is shared among different projects (e.g. a configuration like the one described [here](/doc/use-cases/shared-development-server)), this option can be used to From a76736a962f54fe68789889414875f8d18a5601f Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 20:59:39 +0900 Subject: [PATCH 16/28] update resume --- content/docs/command-reference/exp/resume.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/content/docs/command-reference/exp/resume.md b/content/docs/command-reference/exp/resume.md index 805e235a5b..20df461c5a 100644 --- a/content/docs/command-reference/exp/resume.md +++ b/content/docs/command-reference/exp/resume.md @@ -1,4 +1,4 @@ -# exp res[ume] +# exp resume Resume (continue) existing checkpoint experiments. @@ -7,6 +7,7 @@ Resume (continue) existing checkpoint experiments. ```usage usage: dvc exp res[ume] [-h] [-q | -v] [-r ] [--params [:]] + [-n ] ``` ## Description @@ -14,15 +15,26 @@ usage: dvc exp res[ume] [-h] [-q | -v] [-r ] `dvc exp resume` can be used to resume execution of a checkpoint experiment which was previously started via `dvc exp run`. +Note that `exp resume` is also aliased to `exp res`. + ## Options +In addition to the following options, `dvc exp res` also accepts the same +options as as `dvc repro`. + - `-r ` - if this option is provided, the specified checkpoint experiment will be resumed. Otherwise, the most recently run checkpoint - experiment will be resumed. + experiment will be resumed. `experiment_rev` can be either the name of a + checkpoint experiment or the Git SHA for an intermediate checkpoint experiment + commit. - `--params [:]]` - if this option is provided, a new - checkpoint branch will be created (starting from the checkpoint provided via - `-r`) using the specified parameter changes. + checkpoint experiment will be created (starting from the checkpoint provided + via `-r`) using the specified parameter changes. + +- `-n `, `--name ` - if this option is provided in conjunction with + `--params`, the resulting checkpoint experiment will be named `name`. If + `--params` is not specified, the original experiment name will be preserved. - `-h`, `--help` - prints the usage/help message, and exit. From 7b5bb1fa2d8a4009e46c8e83aee90c5843f114c7 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:08:12 +0900 Subject: [PATCH 17/28] update run --- content/docs/command-reference/exp/run.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/content/docs/command-reference/exp/run.md b/content/docs/command-reference/exp/run.md index d3fac440d6..359366fb49 100644 --- a/content/docs/command-reference/exp/run.md +++ b/content/docs/command-reference/exp/run.md @@ -10,7 +10,7 @@ the correct order. Pipelines will be reproduced within local experiments usage: dvc exp run [-h] [-q | -v] [-f] [ ...] [--params [:]] - [--queue] [--run-all] [-j ] + [-n ] [--queue] [--run-all] [-j ] [targets [targets ...]] positional arguments: @@ -43,9 +43,9 @@ For checkpoint experiments, `dvc exp resume` should be used in conjunction with ## Options -`dvc exp run` accepts the same options as `dvc repro`, with the exception that -`--no-commit` has no effect in `dvc exp run`. See `dvc repro` for a detailed -list of available options. +In addition to the following options, `dvc exp run` also accepts the same +options as `dvc repro`, with the exception that `--no-commit` has no effect in +`dvc exp run`. See `dvc repro` for a detailed list of available options. - `-f`, `--force` - reproduce an experiment pipeline, regenerating its results, even if no changes were found. For `checkpoint` experiments, `dvc exp run -f` @@ -58,6 +58,9 @@ list of available options. `[filename:]key1=val1,key2=val2...`. Any specified values will override those from the contents of the relevant params file. +- `-n `, `--name ` - use the specified name for this experiment. If + `--name` is not provided, a default name will automatically be generated. + - `--queue` - queue an experiment for future execution, but do not actually run the pipeline. @@ -65,8 +68,8 @@ list of available options. `--queue`. Can be used in conjunction with `-j` to run experiments in parallel. -- `-j ` - run the specified number of experiments at a time in parallel, - only applicable when used in conjunction with `--run-all`. +- `-j `, `--jobs ` - run the specified number of experiments at + a time in parallel, only applicable when used in conjunction with `--run-all`. - `-h`, `--help` - prints the usage/help message, and exit. From e4b9c7b82a3601cec86c91b3c17e734362c4dc32 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:11:39 +0900 Subject: [PATCH 18/28] update show --- content/docs/command-reference/exp/show.md | 32 +++++++++++----------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/content/docs/command-reference/exp/show.md b/content/docs/command-reference/exp/show.md index 63fd46bf1f..009e3aaba9 100644 --- a/content/docs/command-reference/exp/show.md +++ b/content/docs/command-reference/exp/show.md @@ -15,9 +15,9 @@ usage: dvc exp show [-h] [-q | -v] [-a] [-T] [--all-commits] [--no-pager] [--precision ] optional arguments: - -a, --all-branches Show metrics for all branches. - -T, --all-tags Show metrics for all tags. - --all-commits Show metrics for all commits. + -a, --all-branches Show experiments derived from the tip of all Git branches. + -T, --all-tags Show experiments derived from all Git tags. + --all-commits Show experiments derived from for all Git commits in the repository. --no-pager Do not pipe output into a pager. --include-metrics Include the specified metrics in output table. @@ -179,9 +179,9 @@ $ dvc exp show ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ -│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ -│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ 20170428 │ 0.2 │ 50 │ 20170428 │ └───────────────────────┴──────────────┴─────────┴────────────────────┴──────────────────┴──────────────┴───────────────┴────────────────────┴────────────┘ ``` @@ -192,9 +192,9 @@ $ dvc exp show --include-params=featurize ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ └───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ ``` @@ -207,9 +207,9 @@ $ dvc exp show --include-params=featurize --sort-by=auc --sort-order=asc ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ └───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ ``` @@ -222,11 +222,11 @@ $ dvc exp show --all-commits --include-params=featurize --sort-by=auc --sort-ord ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ workspace │ - │ 0.61314 │ 1500 │ 2 │ │ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ -│ ├── 1df77f7 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ -│ ├── 1dad0d2 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ -│ └── e6c974b │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ │ 10-bigrams-model │ Jun 20, 2020 │ 0.54175 │ 1500 │ 2 │ -│ └── 069d9cc │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ +│ └── exp-069d9 │ Sep 24, 2020 │ 0.51076 │ 2500 │ 2 │ │ 9-evaluation │ Jun 20, 2020 │ 0.54175 │ 500 │ 1 │ │ 8-ml-pipeline │ Jun 20, 2020 │ - │ 500 │ 1 │ │ 6-prep-stage │ Jun 20, 2020 │ - │ 500 │ 1 │ From a8d7685b88316542e3e2aa42344ceb893e92f48f Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:20:12 +0900 Subject: [PATCH 19/28] document `exp branch` --- content/docs/command-reference/exp/branch.md | 74 ++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 content/docs/command-reference/exp/branch.md diff --git a/content/docs/command-reference/exp/branch.md b/content/docs/command-reference/exp/branch.md new file mode 100644 index 0000000000..50a59bde85 --- /dev/null +++ b/content/docs/command-reference/exp/branch.md @@ -0,0 +1,74 @@ +# exp branch + +Apply the changes from a previously run experiment into the workspace. + +## Synopsis + +```usage +usage: dvc exp branch [-h] [-q | -v] experiment branch + +positional arguments: + experiment Experiment to be promoted. + branch Git branch name to use. +``` + +## Description + +This command will directly promote an experiment to a Git branch. + +## Options + +- `experiment` - name of the experiment to be promoted. + +- `branch` - Git branch name to use for the promoted experiment. + +- `-h`, `--help` - shows the help message and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information from executing the + `dvc pull` command. + +## Example: Promote an experiment + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have run 3 experiments in our project workspace: + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +We now wish to promote the results of experiment `1dad0d2` to a new Git branch +`my-branch`. + +```dvc +$ dvc exp branch exp-1dad0 my-branch ⏎ +Git branch 'my-branch' has been created from experiment 'exp-70841'. +To switch to the new branch run: + + git checkout my-branch +``` + +We can inspect the result with Git: + +```dvc +$ git --no-pager branch -l +* master + my-branch +``` + +The resulting branch can now be merged, rebased, pushed, etc. like any other Git +branch. From b47444cb21be1d1ade410370f76d0d87dcaff697 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:30:14 +0900 Subject: [PATCH 20/28] document `dvc exp list` --- content/docs/command-reference/exp/list.md | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 content/docs/command-reference/exp/list.md diff --git a/content/docs/command-reference/exp/list.md b/content/docs/command-reference/exp/list.md new file mode 100644 index 0000000000..d107158f5a --- /dev/null +++ b/content/docs/command-reference/exp/list.md @@ -0,0 +1,85 @@ +# exp list + +List the available experiments in a local or remote Git repository. + +## Synopsis + +```usage +usage: dvc experiments list [-h] [-q | -v] [--rev ] [--all] [[]] + +positional arguments: + [] Optional Git remote name or Git URL. If provided, experiments from the specified Git repository will be listed instead of local experiments. + +optional arguments: + --rev List experiments derived from the specified revision. Defaults to HEAD if neither `--rev` nor `--all` are specified. + --all List all experiments. +``` + +## Description + +This command can be used to list experiments found in the current local +repository, or experiments which are available in a remote Git repository. + +## Options + +- `[]` - optional name of the remote Git repository. `git_remote` + can either be a Git remote name (i.e. `origin`) or a full Git repository URL + (i.e. `https://github.com/iterative/example-get-started.git`). + +- `--rev ` - list experiments derived from the specified Git revision, + defaults to `HEAD`. + +- `--all` - list all experiments in the repository (overrides `--rev`). + +- `-h`, `--help` - shows the help message and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information from executing the + `dvc pull` command. + +## Example: Listing experiments + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have run 3 experiments in our project workspace: + +```dvc +$ dvc exp show --include-params=featurize +┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ +┃ Experiment ┃ Created ┃ auc ┃ featurize.max_features ┃ featurize.ngrams ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ +│ workspace │ - │ 0.61314 │ 1500 │ 2 │ +│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-e6c97 │ Oct 21, 2020 │ 0.61314 │ 1500 │ 2 │ +│ ├── exp-1dad0 │ Oct 09, 2020 │ 0.57756 │ 2000 │ 2 │ +│ └── exp-1df77 │ Oct 09, 2020 │ 0.51676 │ 500 │ 2 │ +└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘ +``` + +`dvc exp list` can be used to quickly list the available experiments: + +```dvc +$ dvc exp list --all +11-bigrams-experiment: + exp-e6c97 + exp-1dad0 + exp-1df77 +``` + +`dvc exp list` can also list experiments in a remote Git repository: + +```dvc +$ dvc exp list --all origin +11-bigrams-experiment: + exp-e6c97 + exp-86dd6 +``` + +We can see that in our Git remote named `origin`, that two experiments are +available in the remote. `dvc exp pull` can be used to pull these experiments +from the Git remote, and `dvc exp push` can be used to push our local +experiments to the Git remote. From 9c55c8933068e2ccdadae4dc5e49c6ce049b0aaa Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:45:47 +0900 Subject: [PATCH 21/28] document `exp push` --- content/docs/command-reference/exp/push.md | 105 +++++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 content/docs/command-reference/exp/push.md diff --git a/content/docs/command-reference/exp/push.md b/content/docs/command-reference/exp/push.md new file mode 100644 index 0000000000..894aa17cd7 --- /dev/null +++ b/content/docs/command-reference/exp/push.md @@ -0,0 +1,105 @@ +# exp push + +Push a local experiment to a Git remote. + +## Synopsis + +```usage +usage: dvc experiments push [-h] [-q | -v] + [-f] [--no-cache] [-r ] + [-j ] [--run-cache] + + +positional arguments: + Git remote name or Git URL. + Experiment to push. + +optional arguments: + -f, --force Replace experiment in the Git remote if it already exists. + --no-cache Do not push cached outputs for this experiment to DVC remote storage. + -r , --remote + Name of the DVC remote to use when pushing cached outputs. + -j , --jobs + Number of jobs to run simultaneously when pushing to DVC remote storage. + --run-cache Push run history for all stages. +``` + +## Description + +This command can be used to push a local experiment to a remote Git repository. +`dvc exp push` is analogous to `git push` for experiments. + +By default, any DVC cache files associated with the experiment will also be +pushed to a DVC remote. `--no-cache` can be used to disable this behavior. The +`-r`, `-j`, and `--run-cache` options can be used to control this behavior, just +like in `dvc push`. + +## Options + +- `` - remote Git repository to push to. `git_remote` can either be + a Git remote name (i.e. `origin`) or a full Git repository URL (i.e. + `https://github.com/iterative/example-get-started.git`). + +- `` - name of the experiment to push. + +- `-f`, `--force` - replace experiment in the Git remote if it already exists. + This option is equivalent to `git push --force`. By default, if the experiment + already exists in the Git remote, the push operation will only complete if the + remote experiment can be fast-forwarded. + +- `--no-cache` - do not push DVC cache files associated with this experiment to + a DVC remote. + +- `-r `, `--remote ` - name of the + [remote storage](/doc/command-reference/remote) to push DVC cache files to + (see `dvc remote list`). If this option is not provided, the default DVC + remote will be used. + +- `--run-cache` - uploads all available history of stage runs to the remote + repository. + +- `-j `, `--jobs ` - parallelism level for DVC to upload data to + remote storage. The default value is `4 * cpu_count()`. For SSH remotes, the + default is `4`. Note that the default value can be set using the `jobs` config + option with `dvc remote modify`. Using more jobs may improve the overall + transfer speed. + +- `-h`, `--help` - shows the help message and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information from executing the + `dvc pull` command. + +## Example: Pushing an experiment + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have run 3 experiments in our project workspace: + +```dvc +$ dvc exp list --all +11-bigrams-experiment: + exp-e6c97 + exp-1dad0 + exp-1df77 +``` + +We would now like to push one of these experiments into a remote Git repository: + +```dvc +$ dvc exp push origin exp-e6c97 +Pushed experiment 'exp-e6c97' to Git remote 'origin'. +``` + +Finally, we can now see that the pushed experiment exists in the remote Git +repository: + +```dvc +$ dvc exp list --all origin +master: + exp-e6c97 +``` From 5244a6295f09f6cc6e28305629c40ba14e315d55 Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:48:30 +0900 Subject: [PATCH 22/28] document `exp push` --- content/docs/command-reference/exp/push.md | 12 ++++++++---- content/docs/sidebar.json | 2 +- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/content/docs/command-reference/exp/push.md b/content/docs/command-reference/exp/push.md index 894aa17cd7..36732ae97a 100644 --- a/content/docs/command-reference/exp/push.md +++ b/content/docs/command-reference/exp/push.md @@ -15,12 +15,16 @@ positional arguments: Experiment to push. optional arguments: - -f, --force Replace experiment in the Git remote if it already exists. - --no-cache Do not push cached outputs for this experiment to DVC remote storage. + -f, --force Replace experiment in the Git remote if it already + exists. + --no-cache Do not push cached outputs for this experiment to DVC + remote storage. -r , --remote - Name of the DVC remote to use when pushing cached outputs. + Name of the DVC remote to use when pushing cached + outputs. -j , --jobs - Number of jobs to run simultaneously when pushing to DVC remote storage. + Number of jobs to run simultaneously when pushing to + DVC remote storage. --run-cache Push run history for all stages. ``` diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 578b0c95b1..d7dfbce279 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -227,7 +227,7 @@ }, { "label": "exp push", - "slug": "pull" + "slug": "push" }, { "label": "exp resume", From ddf70033c0665b78762671770db601262c90a46f Mon Sep 17 00:00:00 2001 From: Peter Rowlands Date: Tue, 15 Dec 2020 21:55:09 +0900 Subject: [PATCH 23/28] document `dvc exp pull` --- content/docs/command-reference/exp/pull.md | 108 +++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 content/docs/command-reference/exp/pull.md diff --git a/content/docs/command-reference/exp/pull.md b/content/docs/command-reference/exp/pull.md new file mode 100644 index 0000000000..6b832ac5fe --- /dev/null +++ b/content/docs/command-reference/exp/pull.md @@ -0,0 +1,108 @@ +# exp pull + +Pull an experiment from a Git remote into the local repository. + +## Synopsis + +```usage +usage: dvc experiments pull [-h] [-q | -v] + [-f] [--no-cache] [-r ] + [-j ] [--run-cache] + + +positional arguments: + Git remote name or Git URL. + Experiment to pull. + +optional arguments: + -f, --force Replace local experiment if it already exists. + --no-cache Do not pull cached outputs for this experiment to DVC + remote storage. + -r , --remote + Name of the DVC remote to use when pulling cached + outputs. + -j , --jobs + Number of jobs to run simultaneously when pulling from + DVC remote storage. + --run-cache Pull run history for all stages. +``` + +## Description + +This command can be used to pull an experiment from a remote Git repository into +the local repository. `dvc exp pull` is analogous to `git fetch` or `git pull` +for experiments. + +By default, any DVC cache files associated with the experiment will also be +pulled from DVC remote storage. `--no-cache` can be used to disable this +behavior. The `-r`, `-j`, and `--run-cache` options can be used to control this +behavior, just like in `dvc pull`. + +## Options + +- `` - remote Git repository to pull from. `git_remote` can either + be a Git remote name (i.e. `origin`) or a full Git repository URL (i.e. + `https://github.com/iterative/example-get-started.git`). + +- `` - name of the experiment to pull. + +- `-f`, `--force` - replace local experiment if it already exists. This option + is equivalent to `git fetch --force`. By default, if the experiment already + exists in the local repository, the pull operation will only complete if the + local experiment can be fast-forwarded. + +- `--no-cache` - do not pull DVC cache files associated with this experiment + from DVC remote storage. + +- `-r `, `--remote ` - name of the + [remote storage](/doc/command-reference/remote) to pull DVC cache files from + (see `dvc remote list`). If this option is not provided, the default DVC + remote will be used. + +- `--run-cache` - downloads all available history of stage runs from the remote + repository (to the cache only, like `dvc fetch --run-cache`). + +- `-j `, `--jobs ` - parallelism level for DVC to upload data to + remote storage. The default value is `4 * cpu_count()`. For SSH remotes, the + default is `4`. Note that the default value can be set using the `jobs` config + option with `dvc remote modify`. Using more jobs may improve the overall + transfer speed. + +- `-h`, `--help` - shows the help message and exit. + +- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no + problems arise, otherwise 1. + +- `-v`, `--verbose` - displays detailed tracing information from executing the + `dvc pull` command. + +## Example: Pulling an experiment + +> This example is based on our +> [Get Started](/doc/tutorials/get-started/experiments), where you can find the +> actual source code. + +Let's say we have cloned a remote Git repository, and would now like to fetch an +experiment that someone else has already pushed to that remote Git repository. + +```dvc +$ dvc exp list --all origin +master: + exp-e6c97 +``` + +To pull the specified experiment: + +```dvc +$ dvc exp pull origin exp-e6c97 +Pulled experiment 'exp-e6c97' from Git remote 'origin'. +``` + +Finally, we can now see that the pulled experiment exists in the local +repository: + +```dvc +$ dvc exp list --all +master: + exp-e6c97 +``` From e260ceed6935820cb47927740074645c85a4c715 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 16 Dec 2020 13:40:49 -0600 Subject: [PATCH 24/28] Update content/docs/command-reference/exp/index.md --- content/docs/command-reference/exp/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 3e1e89a1df..5325829f99 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -1,6 +1,6 @@ # experiments -A set of commands to generate and compare local experiments: +A set of commands to generate and manage experiments: [apply](/doc/command-reference/exp/apply), [branch](/doc/command-reference/exp/branch), [diff](/doc/command-reference/exp/diff), [gc](/doc/command-reference/exp/gc), From 73a4d51839bc25d5cb4a2a5116e930c6de453731 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 16 Dec 2020 14:48:28 -0600 Subject: [PATCH 25/28] Update content/docs/command-reference/exp/index.md --- content/docs/command-reference/exp/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/command-reference/exp/index.md b/content/docs/command-reference/exp/index.md index 5325829f99..f4a2f3c3b4 100644 --- a/content/docs/command-reference/exp/index.md +++ b/content/docs/command-reference/exp/index.md @@ -11,7 +11,8 @@ A set of commands to generate and manage experiments: [run](/doc/command-reference/exp/run), and [show](/doc/command-reference/exp/show). -Note that `experiments` is also aliased to `exp`. +Note that `dvc experiments` is also aliased to `dvc exp` including subcommands +e.g. `dvc exp list`. ## Synopsis From 5c3de95a600d9203bee8b2425280c1d35a026664 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 17 Dec 2020 21:25:09 -0600 Subject: [PATCH 26/28] Update content/docs/command-reference/gc.md --- content/docs/command-reference/gc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/gc.md b/content/docs/command-reference/gc.md index 13a5f40d24..e017833965 100644 --- a/content/docs/command-reference/gc.md +++ b/content/docs/command-reference/gc.md @@ -76,7 +76,7 @@ The default remote is cleaned (see `dvc config core.remote`) unless the that is never referenced from the workspace or from any Git commit can still be stored in the project's cache). -- `--all-experiments` same as `a`, `T`, or `--all-commits`, but applies to +- `--all-experiments` same as `a`, `T`, or `--all-commits`, but applies to all [experiments](/doc/command-reference/exp). This keeps all the data used in all local experiments (including intermediate checkpoints) in the project. From c2f110b4bd3e11b285acba0d1646d186745b032a Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Thu, 17 Dec 2020 21:32:16 -0600 Subject: [PATCH 27/28] Update content/docs/command-reference/gc.md --- content/docs/command-reference/gc.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/docs/command-reference/gc.md b/content/docs/command-reference/gc.md index e017833965..3e79aa0570 100644 --- a/content/docs/command-reference/gc.md +++ b/content/docs/command-reference/gc.md @@ -77,8 +77,8 @@ The default remote is cleaned (see `dvc config core.remote`) unless the be stored in the project's cache). - `--all-experiments` same as `a`, `T`, or `--all-commits`, but applies to all - [experiments](/doc/command-reference/exp). This keeps all the data used in all - local experiments (including intermediate checkpoints) in the project. + [experiments](/doc/command-reference/exp). This preserves the cache for all + data used in every local experiment (including intermediate checkpoints). - `-p `, `--projects ` - if a single remote or a single cache is shared among different projects (e.g. a configuration like the one described From 3b24bab6d0ec8f142b87f77eab8c5ef8f92dc016 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Mon, 25 Jan 2021 23:32:43 -0600 Subject: [PATCH 28/28] cmd: add --all-exps to gc -w per https://github.com/iterative/dvc.org/pull/1926#pullrequestreview-555149734 --- content/docs/command-reference/gc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/gc.md b/content/docs/command-reference/gc.md index a3a7886a2a..44d9d625d2 100644 --- a/content/docs/command-reference/gc.md +++ b/content/docs/command-reference/gc.md @@ -54,7 +54,7 @@ The default remote is cleaned (see `dvc config core.remote`) unless the - `-w`, `--workspace` - keep _only_ files and directories referenced in the workspace. Note that this behavior is implied in `--all-tags`, - `--all-branches`, and `--all-commits`. + `--all-branches`, `--all-commits`, and `--all-commits`. - `-a`, `--all-branches` - keep cached objects referenced in all Git branches, and in the workspace (implying `-w`). Useful if branches are used to track