[WIP] cmd: experiment

content/docs/command-reference/exp/apply.md

@@ -0,0 +1,138 @@
+# exp apply
+
+Apply the changes from a previously run experiment into the workspace.
+
+## Synopsis
+
+```usage
+usage: dvc exp apply [-h] [-q | -v] experiment
+
+positional arguments:
+  experiment     Experiment to be applied.
+```
+
+## Description
+
+This command will apply the changes from an experiment to the workspace. It is
+usually run when you are ready to Git commit the results of an experiment into
+the project.
+
+This command will fail if the specified experiment was not derived from the Git
+commit which is currently (Git) checkout-ed in the workspace.
+
+## Options
+
+- `experiment` - name or revision of the experiment to be checked out.
+
+- `-h`, `--help` - shows the help message and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
+  problems arise, otherwise 1.
+
+- `-v`, `--verbose` - displays detailed tracing information from executing the
+  `dvc pull` command.
+
+## Example: Apply and promote an experiment
+
+> This example is based on our
+> [Get Started](/doc/tutorials/get-started/experiments), where you can find the
+> actual source code.
+
+Let's say we have run 3 experiments in our project workspace:
+
+```dvc
+$ dvc exp show --include-params=featurize
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ Experiment            ┃ Created      ┃     auc ┃ featurize.max_features ┃ featurize.ngrams ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ workspace             │ -            │ 0.61314 │ 1500                   │ 2                │
+│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-e6c97         │ Oct 21, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-1dad0         │ Oct 09, 2020 │ 0.57756 │ 2000                   │ 2                │
+│ └── exp-1df77         │ Oct 09, 2020 │ 0.51676 │ 500                    │ 2                │
+└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘
+```
+
+We now wish to commit the results of experiment `1dad0d2` into our project.
+
+```dvc
+$ dvc exp apply exp-1dad0
+Changes for experiment 'exp-1dad0' have been applied to your current workspace.
+```
+
+We can inspect these changes with Git:
+
+```dvc
+$ git status
+On branch master
+Your branch is up to date with 'origin/master'.
+
+Changes not staged for commit:
+        modified:   dvc.lock
+        modified:   params.yaml
+        modified:   prc.json
+        modified:   scores.json
+
+$ git diff params.yaml scores.json
+diff --git a/params.yaml b/params.yaml
+index 4c4d898..faf781a 100644
+--- a/params.yaml
++++ b/params.yaml
+@@ -3,7 +3,7 @@ prepare:
+   seed: 20170428
+
+ featurize:
+-  max_features: 1500
++  max_features: 2000
+   ngrams: 2
+
+ train:
+diff --git a/scores.json b/scores.json
+index c995f24..c640c4e 100644
+--- a/scores.json
++++ b/scores.json
+@@ -1 +1 @@
+-{"auc": 0.6131382960762474}
+\ No newline at end of file
++{"auc": 0.5775633054725381}
+\ No newline at end of file
+```
+
+and with DVC:
+
+```
+$ dvc status
+Data and pipelines are up to date.
+
+$ dvc diff
+Modified:
+    data/features/
+    data/features/test.pkl
+    data/features/train.pkl
+    model.pkl
+    prc.json
+    scores.json
+
+files summary: 0 added, 0 deleted, 5 modified, 0 not in cache
+```
+
+To promote this experiment we simply `git add` and `git commit` the changes:
+
+```dvc
+$ git add .
+$ git commit -m "promote experiment exp-1dad0"
+[master 0412386] promote experiment exp-1dad0
+```
+
+Finally, we can now see that the promoted experiment is the new tip of our
+master branch:
+
+```dvc
+$ dvc exp show --include-params=featurize
+┏━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ Experiment ┃ Created  ┃     auc ┃ featurize.max_features ┃ featurize.ngrams ┃
+┡━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ workspace  │ -        │ 0.57756 │ 2000                   │ 2                │
+│ master     │ 04:31 PM │ 0.57756 │ 2000                   │ 2                │
+└────────────┴──────────┴─────────┴────────────────────────┴──────────────────┘
+```

content/docs/command-reference/exp/branch.md

@@ -0,0 +1,74 @@
+# exp branch
+
+Apply the changes from a previously run experiment into the workspace.
+
+## Synopsis
+
+```usage
+usage: dvc exp branch [-h] [-q | -v] experiment branch
+
+positional arguments:
+  experiment     Experiment to be promoted.
+  branch         Git branch name to use.
+```
+
+## Description
+
+This command will directly promote an experiment to a Git branch.
+
+## Options
+
+- `experiment` - name of the experiment to be promoted.
+
+- `branch` - Git branch name to use for the promoted experiment.
+
+- `-h`, `--help` - shows the help message and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
+  problems arise, otherwise 1.
+
+- `-v`, `--verbose` - displays detailed tracing information from executing the
+  `dvc pull` command.
+
+## Example: Promote an experiment
+
+> This example is based on our
+> [Get Started](/doc/tutorials/get-started/experiments), where you can find the
+> actual source code.
+
+Let's say we have run 3 experiments in our project workspace:
+
+```dvc
+$ dvc exp show --include-params=featurize
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ Experiment            ┃ Created      ┃     auc ┃ featurize.max_features ┃ featurize.ngrams ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ workspace             │ -            │ 0.61314 │ 1500                   │ 2                │
+│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-e6c97         │ Oct 21, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-1dad0         │ Oct 09, 2020 │ 0.57756 │ 2000                   │ 2                │
+│ └── exp-1df77         │ Oct 09, 2020 │ 0.51676 │ 500                    │ 2                │
+└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘
+```
+
+We now wish to promote the results of experiment `1dad0d2` to a new Git branch
+`my-branch`.
+
+```dvc
+$ dvc exp branch exp-1dad0 my-branch                                                                                                             ⏎
+Git branch 'my-branch' has been created from experiment 'exp-70841'.
+To switch to the new branch run:
+
+        git checkout my-branch
+```
+
+We can inspect the result with Git:
+
+```dvc
+$ git --no-pager branch -l
+* master
+  my-branch
+```
+
+The resulting branch can now be merged, rebased, pushed, etc. like any other Git
+branch.

content/docs/command-reference/exp/diff.md

@@ -0,0 +1,105 @@
+# exp diff
+
+Show changes in [metrics](/doc/command-reference/metrics) and
+[parameters](/doc/command-reference/params) between experiments.
+
+## Synopsis
+
+```usage
+usage: dvc exp diff [-h] [-q | -v] [--all] [--show-json] [--show-md]
+                    [--old] [--no-path] [--precision <n>] [a_rev] [b_rev]
+
+positional arguments:
+  a_rev            Old experiment to compare (defaults to HEAD)
+  b_rev            New experiment to compare (defaults to the current workspace)
+```
+
+## Description
+
+This command simply combines the output of both `dvc metrics diff` and
+`dvc params diff` for quickly comparing two experiments.
+
+`a_rev` and `b_rev` can be experiment hashes, Git commit hashes, tag, or branch
+names. If only `a_rev` is provided and `a_rev` is an experiment hash, it will be
+compared to the baseline commit from which the experiment was derived.
+
+## Options
+
+- `--all` - prints all parameters including not changed.
+
+- `--show-json` - prints the command's output in easily parsable JSON format,
+  instead of a human-readable table.
+
+- `--show-md` - prints the command's output in the Markdown table format.
+
+- `--old` - show old metric value in addition to the new value.
+
+- `--no-path` - don't show metric or param path in the result table. This option
+  is useful when only one metrics or params file is in use or there is no
+  intersection between the metric or param names.
+
+- `--precision <n>` -
+  [round](https://docs.python.org/3/library/functions.html#round) metrics to `n`
+  digits precision after the decimal point. Rounds to 5 digits by default.
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
+  problems arise, otherwise 1.
+
+- `-v`, `--verbose` - displays detailed tracing information.
+
+## Example
+
+> This example is based on our
+> [Get Started](/doc/tutorials/get-started/experiments), where you can find the
+> actual source code.
+
+Let's say we have run 3 experiments in our project workspace:
+
+```dvc
+$ dvc exp show --include-params=featurize
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ Experiment            ┃ Created      ┃     auc ┃ featurize.max_features ┃ featurize.ngrams ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ workspace             │ -            │ 0.61314 │ 1500                   │ 2                │
+│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-e6c97         │ Oct 21, 2020 │ 0.61314 │ 1500                   │ 2                │
+│ ├── exp-1dad0         │ Oct 09, 2020 │ 0.57756 │ 2000                   │ 2                │
+│ └── exp-1df77         │ Oct 09, 2020 │ 0.51676 │ 500                    │ 2                │
+└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┘
+```
+
+To compare a experiment `exp-1dad0` to its baseline (`11-bigrams-experiment`):
+
+```dvc
+$ dvc exp diff exp-1dad0
+Path         Metric    Value    Change
+scores.json  auc       0.61314  0.035575
+
+Path         Param                   Value    Change
+params.yaml  featurize.max_features  1500     -500
+```
+
+To compare two experiments:
+
+```dvc
+$ dvc exp diff exp-1dad0 exp-1df77
+Path         Metric    Value    Change
+scores.json  auc       0.51676  -0.060799
+
+Path         Param                   Value    Change
+params.yaml  featurize.max_features  500      -1500
+```
+
+To compare an experiment to a Git tag:
+
+```dvc
+$ dvc exp diff exp-1dad0 8-ml-pipeline
+Path         Metric    Value    Change
+scores.json  auc       None     diff not supported
+
+Path         Param                   Value    Change
+params.yaml  featurize.max_features  500      -1500
+params.yaml  featurize.ngrams        1        -1
+```