guide: Managing Experiments

content/docs/sidebar.json

@@ -142,7 +142,11 @@
         "label": "Experiment Management",
         "slug": "experiment-management",
         "source": "experiment-management/index.md",
-        "children": ["sharing-experiments", "checkpoints"]
+        "children": [
+          "running-experiments",
+          "sharing-experiments",
+          "checkpoints"
+        ]
       },
       "setup-google-drive-remote",
       "large-dataset-optimization",

content/docs/user-guide/experiment-management/checkpoints.md

@@ -1,5 +1,140 @@
 # Checkpoints
 
+<!--
+## Checkpoints
+
+To track successive steps in a longer or deeper <abbr>experiment</abbr>, you can
+register checkpoints from your code. Each `dvc exp run` will resume from the
+last checkpoint.
+
+Checkpoints provide a way to train models iteratively, keeping the metrics,
+models and other artifacts associated with each epoch.
+
+### Adding checkpoints to the pipeline
+
+There are various ways to add checkpoints to a project. In common, these all
+involve marking a stage <abbr>output</abbr> with `checkpoint: true` in
+`dvc.yaml`. This is needed so that the experiment can resume later, based on the
+<abbr>cached</abbr> output(s).
+
+If you are adding a new stage with `dvc stage add`, you can mark its output(s)
+with `--checkpoints` (`-c`) option. DVC will add a `checkpoint: true` to the
+stage output in `dvc.yaml`.
+
+Otherwise, if you are adding a checkpoint to an already existing project, you
+can edit `dvc.yaml` and add a `checkpoint: true` to the stage output as shown
+below:
+
+```yaml
+stages:
+  ...
+  train:
+    ...
+    outs:
+      - model.pt:
+          checkpoint: true
+  ...
+```
+
+### Adding checkpoints to Python code
+
+DVC is agnostic when it comes to the language you use in your project.
+Checkpoints are basically a mechanism to associate outputs of a pipeline with
+its metrics. Reading the model from previous iteration and writing a new model
+as a file are not handled by DVC. DVC captures the signal produced by the
+machine learning experimentation code and stores each successive checkpoint.
+
+> 💡 DVC provides several automated ways to capture checkpoints for popular ML
+> libraries in [DVClive](https://dvc.org/doc/dvclive). It may be more productive
+> to use checkpoints via DVClive. Here we discuss adding checkpoints to a
+> project manually.
+
+If you are writing the project in Python, the easiest way to signal DVC to
+capture the checkpoint is to use `dvc.api.make_checkpoint()` function. It
+creates a checkpoint and records all artifacts changed after the previous
+checkpoint as another experiment.
+
+The following snippet shows an example that uses a Keras custom callback class.
+The callback signals DVC to create a checkpoint at the end of each checkpoint.
+
+```python
+class DVCCheckpointsCallback(tf.keras.callbacks.Callback):
+    def on_epoch_end(self, epoch, logs=None):
+            dvc.api.make_checkpoint()
+...
+
+history = model.fit(
+          ...
+        callbacks=[DVCCheckpointsCallback(), ...]
+        )
+```
+
+A similar approach can be taken in PyTorch when using a loop to train a model:
+
+```python
+   for epoch in range(1, EPOCHS+1):
+      ...
+      for x_batch, y_batch in train_loader:
+          train(model, x_batch, y_batch)
+      torch.save(model.state_dict(), "model.pt")
+      # Evaluate and checkpoint.
+      evaluate(model, x_test, y_test)
+      dvc.api.make_checkpoint()
+    ...
+```
+
+Even if you're not using these libraries, you can use checkpoints in your
+project at each epoch/step by first recording all intermediate artifacts and
+metrics, then calling `dvc.api.make_checkpoint()`.
+
+### Adding checkpoints to non-Python code
+
+If you use another language in your project, you can mimic the behavior of
+`make_checkpoint`. In essence `make_checkpoint` creates a special file named
+`DVC_CHECKPOINT` inside `.dvc/tmp/` to signal DVC, and waits the file to be
+removed.
+
+```r
+
+dvcroot <- Sys.getenv("DVC_ROOT")
+
+if (dvcroot != "") {
+    signalfilepath = file.path(dvcroot, ".dvc", "tmp", "DVC_CHECKPOINT")
+    file.create(signalfilepath)
+    while (file.exists(signalfilepath)) {
+      Sys.sleep(0.01)
+    }
+
+}
+
+```
+
+The following Julia snippet creates a signal file to create a checkpoint.
+
+```julia
+
+dvc_root =  get(ENV, "DVC_ROOT", "")
+
+if dvc_root != ""
+   signal_file_path = joinpath(dvc_root, ".dvc", "tmp", "DVC_CHECKPOINT")
+   open(signal_file_path, "w") do io
+           write(io, "")
+   end;
+   while isfile(signal_file_path)
+        sleep()
+   end;
+```
+
+<details>
+
+### How are checkpoints captured?
+
+Instead of a single commit, checkpoint experiments have multiple commits under
+the custom Git reference (in `.git/refs/exps`), similar to a branch.
+
+</details>
+-->
+
 ML checkpoints are an important part of deep learning because ML engineers like
 to save the model files at certain points during a training process.
 

content/docs/user-guide/experiment-management/index.md

@@ -1,5 +1,18 @@
 # Experiment Management
 
+<!--
+Machine Learning and Data Science projects usually involve experimentation.
+These experiments' goals can range from finding good hyperparameters to testing
+for data and concept drift. DVC 2.0 introduced a new set of commands to manage
+experiments with minimum boilerplate. It allows to run experiments defined by
+pipelines, track their associated data and model files, set parameters for each,
+push experiment parameters and code to Git remotes without committing them,
+create branches and persist them in Git.
+
+Each experiment represents a project variation based on the changes in your
+current <abbr>workspace</abbr>.
+-->
+
 _New in DVC 2.0_
 
 Data science and ML are iterative processes that require a large number of

content/docs/user-guide/experiment-management/managing-experiments.md

@@ -0,0 +1,68 @@
+# Managing Experiments
+
+Once you have defined and/or [run experiments] in your project, you can use
+several features of DVC to see, compare, reproduce, share, or remove them.
+
+[run experiments]: /doc/user-guide/experiment-management/running-experiments
+
+## Experiment names
+
+Experiments created with `dvc exp run` will have an auto-generated name like
+`exp-bfe64` by default. It can be customized using the `--name` (`-n`) option:
+
+```dvc
+$ dvc exp run --name cnn-512 --set-param model.conv_units=512
+```
+
+When you create an experiment, DVC generates a Git-like SHA-1 hash from its
+contents. This is shown when you [queue experiments] with `--queue`:
+
+[queue experiments]:
+  /doc/user-guide/experiment-management/running-experiments#the-experiments-queue
+
+```dvc
+$ dvc exp run --queue -S model.conv_units=32
+Queued experiment '6518f17' for future execution.
+```
+
+After running queued experiment, DVC uses the regular name mentioned earlier.
+
+> Note that you can set a queued experiment's name in advance:
+>
+> ```dvc
+> $ dvc exp run --queue --name cnn-512 -S model.conv_units=512
+> Queued experiment '86bd8f9' for future execution.
+> ```
+
+You can refer to experiments in `dvc exp apply` or `dvc exp branch` either with
+regular experiment names or by their SHA hashes.
+
+## Listing experiments
+
+Use `dvc exp show` to see both run and queued experiments:
+
+```dvc
+$ dvc exp show --no-pager --no-timestamp \
+      --include-metrics loss --include-params model.conv_units
+```
+
+```dvctable
+┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ neutral:**Experiment**              ┃ metric:**loss**    ┃ param:**model.conv_units** ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ workspace               │ 0.23534 │ 64               │
+│ 3973b6b                 │ -       │ 16               │
+│ ├── aeaabb0 [exp-cb13f] │ 0.23534 │ 64               │
+│ ├── d0ee7ce [exp-5dccf] │ 0.23818 │ 32               │
+│ ├── 1533e4d [exp-88874] │ 0.24039 │ 128              │
+│ ├── b1f41d3 [cnn-256]   │ 0.23296 │ 256              │
+│ ├── 07e927f [exp-6c06d] │ 0.23279 │ 24               │
+│ ├── b2b8586 [exp-2a1d5] │ 0.25036 │ 16               │
+│ └── *86bd8f9            │ -       │ 512              │
+└─────────────────────────┴─────────┴──────────────────┘
+```
+
+When an experiment is not run yet, only the former hash is shown (marked with
+`*`).
+
+<!-- WIP -->