guide: DVC Experiments Overview

content/docs/command-reference/exp/apply.md

@@ -20,7 +20,7 @@ can be referenced by name or hash (see `dvc exp run` for details).
 
 This is typically used after choosing a target `experiment` with `dvc exp show`
 or `dvc exp diff`, and before committing it to Git (making it
-[persistent](/doc/user-guide/experiment-management#persistent-experiments)).
+[persistent](/doc/user-guide/experiment-management/dvc-experiments#persistent-experiments)).
 
 `dvc exp apply` changes any files (code, data, <abbr>parameters</abbr>,
 <abbr>metrics</abbr>, etc.) needed to reflect the experiment conditions and

content/docs/command-reference/exp/branch.md

@@ -18,9 +18,9 @@ positional arguments:
 Makes a named Git
 [`branch`](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging)
 containing the target `experiment` (making it
-[persistent](/doc/user-guide/experiment-management#persistent-experiments)). For
-[checkpoint experiments](/doc/command-reference/exp/run#checkpoints), the new
-branch will contain multiple commits (the checkpoints).
+[persistent](/doc/user-guide/experiment-management/dvc-experiments#persistent-experiments)).
+For [checkpoint experiments](/doc/command-reference/exp/run#checkpoints), the
+new branch will contain multiple commits (the checkpoints).
 
 The new `branch` will be based on the experiment's parent commit (`HEAD` at the
 time that the experiment was run). Note that DVC **does not** switch into the

content/docs/command-reference/exp/run.md

@@ -22,10 +22,10 @@ Provides a way to execute and track <abbr>experiments</abbr> in your
 <abbr>project</abbr> without polluting it with unnecessary commits, branches,
 directories, etc.
 
-> `dvc exp run` is equivalent to `dvc repro` for experiments. It has the same
-> behavior when it comes to `targets` and stage execution (restores the
-> dependency graph, etc.). See the command [options](#options) for more on the
-> differences.
+> `dvc exp run` is equivalent to `dvc repro` for <abbr>experiments</abbr>. It
+> has the same behavior when it comes to `targets` and stage execution (restores
+> the dependency graph, etc.). See the command [options](#options) for more on
+> the differences.
 
 Before running an experiment, you'll probably want to make modifications such as
 data and code updates, or <abbr>hyperparameter</abbr> tuning. For the latter,
@@ -44,7 +44,7 @@ option.
 Experiments are custom
 [Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References)
 (found in `.git/refs/exps`) with a single commit based on `HEAD` (not checked
-out by DVC). Note that these commits are not pushed to the Git remote by default
+out by DVC). Note that these commits are not pushed to Git remotes by default
 (see `dvc exp push`).
 
 </details>
@@ -55,8 +55,8 @@ and compare multiple experiments, use `dvc exp show` or `dvc exp diff`
 to restore the results of any other experiment instead.
 
 Successful experiments can be made
-[persistent](/doc/user-guide/experiment-management#persistent-experiments) by
-committing them to the Git repo. Unnecessary ones can be removed with
+[persistent](/doc/user-guide/experiment-management/dvc-experiments#persistent-experiments)
+by committing them to the Git repo. Unnecessary ones can be removed with
 `dvc exp remove`or `dvc exp gc` (or abandoned).
 
 > Note that experiment data will remain in the <abbr>cache</abbr> until you use

content/docs/sidebar.json

@@ -147,6 +147,7 @@
         "slug": "experiment-management",
         "source": "experiment-management/index.md",
         "children": [
+          "dvc-experiments",
           "running-experiments",
           "sharing-experiments",
           "cleaning-experiments",

content/docs/user-guide/experiment-management/cleaning-experiments.md

@@ -2,7 +2,10 @@
 
 Although DVC uses minimal resources to keep track of the experiments, they may
 clutter tables and the workspace. DVC allows to remove specific experiments from
-the workspace or delete all not-yet-persisted experiments at once.
+the workspace or delete all not-yet-[persisted] experiments at once.
+
+[persisted]:
+  /doc/user-guide/experiment-management/dvc-experiments#persistent-experiments
 
 ## Removing specific experiments
 

content/docs/user-guide/experiment-management/dvc-experiments.md

@@ -0,0 +1,36 @@
+## DVC Experiments
+
+_New in DVC 2.0_
+
+`dvc exp` commands let you automatically track a variation to an established
+[data pipeline](/doc/command-reference/dag) baseline. You can create multiple
+isolated experiments this way, as well as review, compare, and restore them
+later, or roll back to the baseline. The basic workflow goes like this:
+
+- Modify stage <abbr>parameters</abbr> or other dependencies (e.g. input data,
+  source code) of committed stages.
+- [Run experiments] with `dvc exp run` (instead of `repro`) to execute the
+  pipeline. The results are reflected in your <abbr>workspace</abbr>, and
+  tracked automatically.
+- Use `dvc metrics` to identify the best experiment(s).
+- Visualize, compare experiments with `dvc exp show` or `dvc exp diff`. Repeat
+  🔄
+- Use `dvc exp apply` to roll back to the best one.
+- Make the selected experiment persistent by committing its results to Git. This
+  cleans the slate so you can repeat the process.
+
+[run experiments]: /doc/user-guide/experiment-management/running-experiments
+
+## Persistent Experiments
+
+When your experiments are good enough to save or share, you may want to store
+them persistently as Git commits in your <abbr>repository</abbr>.
+
+Whether the results were produced with `dvc repro` directly, or after a
+`dvc exp` workflow, `dvc.yaml` and `dvc.lock` will define the experiment as a
+new project version. The right <abbr>outputs</abbr> (including
+[metrics](/doc/command-reference/metrics)) should also be present, or available
+via `dvc checkout`.
+
+Use `dvc exp apply` and `dvc exp branch` to persist experiments in your Git
+history.

content/docs/user-guide/experiment-management/index.md

@@ -1,89 +1,72 @@
 # Experiment Management
 
-_New in DVC 2.0_
-
 Data science and ML are iterative processes that require a large number of
 attempts to reach a certain level of a metric. Experimentation is part of the
 development of data features, hyperspace exploration, deep learning
-optimization, etc. DVC helps you codify and manage all of your
-<abbr>experiments</abbr>, supporting these main approaches:
-
-1. Create [experiments](#experiments) that derive from your latest project
-   version without having to track them manually. DVC does that automatically,
-   letting you list and compare them. The best ones can be made persistent, and
-   the rest archived.
-2. Place in-code [checkpoints](#checkpoints-in-source-code) that mark a series
-   of variations, forming a deep experiment. DVC helps you capture them at
-   runtime, and manage them in batches.
-3. Make experiments or checkpoints [persistent](#persistent-experiments) by
-   committing them to your <abbr>repository</abbr>. Or create these versions
-   from scratch like typical project changes.
-
-   At this point you may also want to consider the different
-   [ways to organize](#organization-patterns) experiments in your project (as
-   Git branches, as folders, etc.).
-
-DVC also provides specialized features to codify and analyze experiments.
+optimization, etc.
+
+Some of DVC's base features already help you codify and analyze experiments.
 [Parameters](/doc/command-reference/params) are simple values you can tweak in a
-human-readable text file, which cause different behaviors in your code and
-models. On the other end, [metrics](/doc/command-reference/metrics) (and
+formatted text file; They cause different behaviors in your code and models. On
+the other end, [metrics](/doc/command-reference/metrics) (and
 [plots](/doc/command-reference/plots)) let you define, visualize, and compare
-meaningful measures for the experimental results.
-
-> 👨‍💻 See [Get Started: Experiments](/doc/start/experiments) for a hands-on
-> introduction to DVC experiments.
+quantitative measures of your results.
 
-## Experiments
+<details>
 
-`dvc exp` commands let you automatically track a variation to an established
-[data pipeline](/doc/command-reference/dag). You can create multiple isolated
-experiments this way, as well as review, compare, and restore them later, or
-roll back to the baseline. The basic workflow goes like this:
+## 💡 Run Cache: Automatic Log of Stage Runs
 
-- Modify stage <abbr>parameters</abbr> or other dependencies (e.g. input data,
-  source code) of committed stages.
-- Use `dvc exp run` (instead of `repro`) to execute the pipeline. The results
-  are reflected in your <abbr>workspace</abbr>, and tracked automatically.
-- Use [metrics](/doc/command-reference/metrics) to identify the best
-  experiment(s).
-- Visualize, compare experiments with `dvc exp show` or `dvc exp diff`. Repeat
-  🔄
-- Use `dvc exp apply` to roll back to the best one.
-- Make the selected experiment persistent by committing its results to Git. This
-  cleans the slate so you can repeat the process.
-
-## Checkpoints in source code
+Every time you [reproduce](/doc/command-reference/repro) a pipeline with DVC, it
+logs the unique signature of each stage run (in `.dvc/cache/runs` by default).
+If it never happened before, the stage command(s) are executed normally. Every
+subsequent time a [stage](/doc/command-reference/run) runs under the same
+conditions, the previous results can be restored instantly, without wasting time
+or computing resources.
 
-To track successive steps in a longer experiment, you can register checkpoints
-from your code at runtime. This allows you, for example, to track the progress
-in deep learning techniques such as evolving neural networks.
+✅ This built-in feature is called <abbr>run-cache</abbr> and it can
+dramatically improve performance. It's enabled out-of-the-box (can be disabled),
+which means DVC is already saving all of your tests and experiments behind the
+scene. But there's no easy way to explore it.
 
-This kind of experiments track a series of variations (the checkpoints) and its
-execution can be stopped and resumed as needed. You interact with them using
-`dvc exp run` and its `--rev`, `--reset` options (see also the `checkpoint`
-field in `dvc.yaml` `outs`).
+</details>
 
-> 📖 To learn more, see the dedicated
-> [Checkpoints](/doc/user-guide/experiment-management/checkpoints) guide.
+## DVC Experiments
 
-## Persistent experiments
+_New in DVC 2.0_
 
-When your experiments are good enough to save or share, you may want to store
-them persistently as Git commits in your <abbr>repository</abbr>.
+DVC experiment management features are designed to support these main
+approaches:
+
+1. [Run] and capture [experiments] that derive from your latest project version
+   without polluting your Git history. DVC tracks them for you, letting you list
+   and compare them. The best ones can be made persistent, and the rest left as
+   history or cleared.
+1. [Queue] and process series of experiments based on a parameter search or
+   other modifications to your baseline.
+1. Generate [checkpoints] during your code execution to analyze the internal
+   progress of deep experiments. DVC captures them at runtime, and can manage
+   them in batches.
+1. Make experiments [persistent] by committing them to your
+   <abbr>repository</abbr> history.
+
+[run]: /doc/user-guide/experiment-management/running-experiments
+[experiments]: /doc/user-guide/experiment-management/dvc-experiments
+[queue]:
+  /doc/user-guide/experiment-management/running-experiments#the-experiments-queue
+[checkpoints]: /doc/user-guide/experiment-management/checkpoints
+[persistent]:
+  /doc/user-guide/experiment-management/dvc-experiments#persistent-experiments
+
+📖 More information in the
+[full guide](/doc/user-guide/experiment-management/dvc-experiments).
 
-Whether the results were produced with `dvc repro` directly, or after a
-`dvc exp` workflow (refer to previous sections), the `dvc.yaml` and `dvc.lock`
-pair in the <abbr>workspace</abbr> will codify the experiment as a new project
-version. The right <abbr>outputs</abbr> (including
-[metrics](/doc/command-reference/metrics)) should also be present, or available
-via `dvc checkout`.
+> 👨‍💻 See [Get Started: Experiments](/doc/start/experiments) for a hands-on
+> introduction to DVC experiments.
 
-### Organization patterns
+### Organization Patterns
 
-DVC takes care of arranging `dvc exp` experiments and the data
-<abbr>cache</abbr> under the hood. But when it comes to full-blown persistent
-experiments, it's up to you to decide how to organize them in your project.
-These are the main alternatives:
+It's up to you to decide how to organize completed experiments. These are the
+main alternatives:
 
 - **Git tags and branches** - use the repo's "time dimension" to distribute your
   experiments. This makes the most sense for experiments that build on each
@@ -99,15 +82,6 @@ These are the main alternatives:
   Completely independent experiments live in separate directories, while their
   progress can be found in different branches.
 
-## Automatic log of stage runs (run-cache)
-
-Every time you `dvc repro` pipelines or `dvc exp run` experiments, DVC logs the
-unique signature of each stage run (to `.dvc/cache/runs` by default). If it
-never happened before, the stage command(s) are executed normally. Every
-subsequent time a [stage](/doc/command-reference/run) runs under the same
-conditions, the previous results can be restored instantly, without wasting time
-or computing resources.
-
-✅ This built-in feature is called <abbr>run-cache</abbr> and it can
-dramatically improve performance. It's enabled out-of-the-box (but can be
-disabled with the `--no-run-cache` command option).
+DVC takes care of arranging `dvc exp` experiments and the data
+<abbr>cache</abbr> under the hood so there's no need to decide on the above
+until your experiments are made [persistent].

content/docs/user-guide/experiment-management/running-experiments.md

@@ -226,7 +226,8 @@ Note that Git-ignored files/dirs are explicitly excluded from queued/temp runs
 to avoid committing unwanted files into Git (e.g. once successful experiments
 are [persisted]).
 
-[persisted]: /doc/user-guide/experiment-management#persistent-experiments
+[persisted]:
+  /doc/user-guide/experiment-management/dvc-experiments#persistent-experiments
 
 > 💡 To include untracked files, stage them with `git add` first (before
 > `dvc exp run`) and `git reset` them afterwards.