diff --git a/content/docs/command-reference/exp/run.md b/content/docs/command-reference/exp/run.md index 4dd24b6896..c2b5506b44 100644 --- a/content/docs/command-reference/exp/run.md +++ b/content/docs/command-reference/exp/run.md @@ -1,6 +1,7 @@ # exp run -Run or resume an [experiment](/doc/command-reference/exp). +Run or resume a +[DVC Experiment](/doc/user-guide/experiment-management/experiments-overview). ## Synopsis @@ -22,136 +23,46 @@ Provides a way to execute and track experiments in your project without polluting it with unnecessary commits, branches, directories, etc. -> `dvc exp run` is equivalent to `dvc repro` for experiments. It has the same -> behavior when it comes to `targets` and stage execution (restores the -> dependency graph, etc.). See the command [options](#options) for more on the -> differences. +> `dvc exp run` has the same behavior as `dvc repro` when it comes to `targets` +> and stage execution (restores the dependency graph, etc.). See the command +> [options](#options) for more on the differences. -Before running an experiment, you'll probably want to make modifications such as -data and code updates, or hyperparameter tuning. For the latter, -you can use the `--set-param` (`-S`) option of this command to change -`dvc param` values on-the fly. +Use the `--set-param` (`-S`) option as a shortcut to change +parameter values [on-the-fly] before running the experiment. -Each experiment creates and tracks a project variation based on your -workspace changes. Experiments will have a unique, auto-generated -name like `exp-bfe64` by default, which can be customized using the `--name` -(`-n`) option. +It's possible to [queue experiments] for later execution with the `--queue` +flag. To actually run them, use `dvc exp run --run-all`. Queued experiments are +run sequentially by default, but can be run in parallel using the `--jobs` +option. -
- -### ⚙️ How does DVC track experiments? - -Experiments are custom -[Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References) -(found in `.git/refs/exps`) with a single commit based on `HEAD` (not checked -out by DVC). Note that these commits are not pushed to Git remotes by default -(see `dvc exp push`). - -
- -The results of the last `dvc exp run` can be seen in the workspace. To display -and compare multiple experiments, use `dvc exp show` or `dvc exp diff` -(`plots diff` also accepts experiment names as `revisions`). Use `dvc exp apply` -to restore the results of any other experiment instead. - -Successful experiments can be made -[persistent](/doc/user-guide/experiment-management#persistent-experiments) by -committing them to the Git repo. Unnecessary ones can be removed with -`dvc exp remove`or `dvc exp gc` (or abandoned). - -> Note that experiment data will remain in the cache until you use -> regular `dvc gc` to clean it up. - -## Checkpoints - -To track successive steps in a longer or deeper experiment, you can -register checkpoints from your code. Each `dvc exp run` will resume from the -last checkpoint. - -First, mark at least stage output with `checkpoint: true` in -`dvc.yaml`. This is needed so that the experiment can resume later, based on the -cached output(s) (circular dependency). - -⚠️ Note that using `checkpoint` in `dvc.yaml` makes it incompatible with -`dvc repro`. - -Then, use the `dvc.api.make_checkpoint()` function (Python code), or write a -signal file (any programming language) following the same steps as that -function. - -You can now use `dvc exp run` to begin the experiment. All checkpoints -registered at runtime will be preserved, even if the process gets interrupted -(e.g. with `[Ctrl] C`, or by an error). Without interruption, a "wrap-up" -checkpoint will be added (if needed), so that changes to pipeline outputs don't -remain in the workspace. - -Subsequent uses of `dvc exp run` will continue from the latest checkpoint (using -the latest cached versions of all outputs). - -
- -### ⚙️ How are checkpoints captured? - -Instead of a single commit, checkpoint experiments have multiple commits under -the custom Git reference (in `.git/refs/exps`), similar to a branch. +> ⚠️ Parallel runs are experimental and may be unstable. Make sure you're using +> a number of jobs that your environment can handle (no more than the CPU +> cores). -
- -List previous checkpoints with `dvc exp show`. To resume from a previous -checkpoint, you must first `dvc exp apply` it before using `dvc exp run`. For -`--queue` or `--temp` runs (see next section), use `--rev` instead to specify -the checkpoint to continue from. - -Alternatively, use `--reset` to start over (discards previous checkpoints and -their outputs). This is useful for re-training ML models, for example. - -## Queueing and parallel execution - -The `--queue` option lets you create an experiment as usual, except that nothing -is actually run. Instead, the experiment is put in a wait-list for later -execution. `dvc exp show` will mark queued experiments with an asterisk `*`. +It's also possible to run special [checkpoint experiments] that log the +execution progress (useful for deep learning ML). The `--rev` and `--reset` +options have special uses for these. -> Note that queuing an experiment that uses checkpoints implies `--reset`, -> unless a `--rev` is provided (refer to the previous section). - -Use `dvc exp run --run-all` to process the queue. This is done outside your -workspace (in temporary dirs in `.dvc/tmp/exps`) to preserve any -changes between/after queueing runs. - -💡 You can also run a single experiment outside the workspace with -`dvc exp run --temp`, for example to continue working on the project meanwhile -(e.g. on another terminal). - -> ⚠️ Note that only tracked files and directories will be included in -> `--queue/temp` experiments. To include untracked files, stage them with -> `git add` first (before `dvc exp run`). Feel free to `git reset` them -> afterwards. Git-ignored files/dirs are explicitly excluded from runs outside -> the workspace to avoid committing unwanted files into experiments. - -
+> 📖 See the [Running Experiments] guide for more details on all these features. -### ⚙️ How are experiments queued? +[Review] run experiments with `dvc exp show`. Successful ones can be [made +persistent] by restoring them via `dvc exp branch` or `dvc exp apply` and +committing them to the Git repo. Unnecessary ones can be [cleared] with +`dvc exp gc`. -A custom [Git stash](https://www.git-scm.com/docs/git-stash) is used to queue -pre-experiment commits. - -
- -Adding `-j` (`--jobs`), experiment queues can be run in parallel for better -performance (creates a tmp dir for each job). - -⚠️ Parallel runs are experimental and may be unstable at this time. ⚠️ Make sure -you're using a number of jobs that your environment can handle (no more than the -CPU cores). - -> Note that each job runs the entire pipeline (or `targets`) serially. DVC makes -> no attempt to distribute stage commands among jobs. The order in which they -> were queued is also not preserved when running them. +[on-the-fly]: #example-modify-parameters-on-the-fly +[queue experiments]: + /doc/user-guide/experiment-management/running-experiments#the-experiments-queue +[checkpoint experiments]: /doc/user-guide/experiment-management/checkpoints +[running experiments]: /doc/user-guide/experiment-management/running-experiments +[review]: /doc/user-guide/experiment-management/comparing-experiments +[made persistent]: /doc/user-guide/experiment-management/persisting-experiments +[cleared]: /doc/user-guide/experiment-management/cleaning-experiments ## Options > In addition to the following, `dvc exp run` accepts all the options in -> `dvc repro`, with the exception that `--no-commit` has no effect here. +> `dvc repro`, with the exception that `--no-commit` has no effect. - `-S [:]=`, `--set-param [:]=` - set the value of @@ -169,8 +80,10 @@ CPU cores). - `--queue` - place this experiment at the end of a line for future execution, but don't actually run it yet. Use `dvc exp run --run-all` to process the - queue. For checkpoint experiments, this implies `--reset` unless a `--rev` is - provided. + queue. + + > For checkpoint experiments, this implies `--reset` unless a `--rev` is + > provided. - `--run-all` - run all queued experiments (see `--queue`) and outside your workspace (in `.dvc/tmp/exps`). Use `-j` to execute them @@ -180,10 +93,14 @@ CPU cores). parallel. Only has an effect along with `--run-all`. Defaults to 1 (the queue is processed serially). + > Note that since queued experiments are run isolated from each other, common + > stages may sometimes be executed several times depending on the state of the + > [run-cache] at that time. + - `-r `, `--rev ` - continue an experiment from a specific checkpoint name or hash (`commit`) in `--queue` or `--temp` runs. -- `--reset` - deletes `checkpoint` outputs before running this experiment +- `--reset` - deletes `checkpoint: true` outputs before running this experiment (regardless of `dvc.lock`). Useful for ML model re-training. - `-f`, `--force` - reproduce pipelines even if no changes were found (same as @@ -198,10 +115,12 @@ CPU cores). - `-v`, `--verbose` - displays detailed tracing information. +[run-cache]: /doc/user-guide/project-structure/internal-files#run-cache + ## Examples -> These examples are based on our [Get Started](/doc/start/experiments), where -> you can find the actual source code. +> This is based on our [Get Started](/doc/start/experiments), where you can find +> the actual source code.
@@ -256,19 +175,16 @@ experiment we just ran (`exp-44136`). ## Example: Modify parameters on-the-fly -You could modify a params file just like any other dependency and -run an experiment on that basis. Since this is a common need, `dvc exp run` -comes with the `--set-param` (`-S`) option built-in to update existing -parameters. This saves you the need to manually edit the params file. +`dvc exp run--set-param` (`-S`) saves you the need to manually edit the params +file before running an experiment. ```dvc $ dvc exp run -S prepare.split=0.25 -S featurize.max_features=2000 ... Reproduced experiment(s): exp-18bf6 -Experiment results have been applied to your workspace. ``` -To see the results, we can use `dvc exp diff` which compares both params and +To see the results, you can use `dvc exp diff`. It compares both params and metrics to the previous project version: ```dvc diff --git a/content/docs/user-guide/experiment-management/cleaning-experiments.md b/content/docs/user-guide/experiment-management/cleaning-experiments.md index 8a54ee91ec..5064ec7bee 100644 --- a/content/docs/user-guide/experiment-management/cleaning-experiments.md +++ b/content/docs/user-guide/experiment-management/cleaning-experiments.md @@ -2,9 +2,9 @@ Although DVC uses minimal resources to keep track of the experiments, they may clutter tables and the workspace. DVC allows to remove specific experiments from -the workspace or delete all not-yet-[persisted] experiments at once. +the workspace or delete the ones that are not [final] yet. -[persisted]: /doc/user-guide/experiment-management/persisting-experiments +[final]: /doc/user-guide/experiment-management/persisting-experiments ## Removing specific experiments @@ -30,10 +30,13 @@ these to keep rather than which of these to remove. You can use `dvc exp gc` to select a set of experiments to keep and the rest of them are _garbage collected._ -This command takes a _scope_ argument. The scope can be `workspace`, -`all-branches`, `all-tags`, `all-commits`. In garbage collection, the scope -determines the experiments to _keep_, i.e., experiments out of the scope of the -given flag are removed. +This command takes a `scope` argument. It accepts "workspace", "all-branches", +"all-tags", or "all-commits". This determines the experiments to _keep_, i.e. +experiments not in scope are removed. + +> ⚠️ Note that experiment remains in the cache until you use +> regular `dvc gc` separately to clean it up (if it's not needed by committed +> versions). ### Keeping experiments in the workspace diff --git a/content/docs/user-guide/experiment-management/comparing-experiments.md b/content/docs/user-guide/experiment-management/comparing-experiments.md index 6a274687ca..33ddd29002 100644 --- a/content/docs/user-guide/experiment-management/comparing-experiments.md +++ b/content/docs/user-guide/experiment-management/comparing-experiments.md @@ -109,6 +109,11 @@ $ dvc exp show `dvc exp show` only tabulates experiments in the workspace and in `HEAD`. You can use `--all` flag to show all the experiments in the project instead. +Note that [queued experiments] will be marked with an asterisk `*`. + +[queued experiments]: + /doc/user-guide/experiment-management/running-experiments#the-experiments-queue + ## Customize the table of experiments The table output may become cluttered if you have a large number of parameters diff --git a/content/docs/user-guide/experiment-management/experiments-overview.md b/content/docs/user-guide/experiment-management/experiments-overview.md index 82251c59e0..0eff5fbe53 100644 --- a/content/docs/user-guide/experiment-management/experiments-overview.md +++ b/content/docs/user-guide/experiment-management/experiments-overview.md @@ -61,7 +61,7 @@ experiments. This includes the locations for expected dependencies metrics, etc.). These assume [sane defaults] but can be customized with the options of `dvc exp init`. -💡 We recommend adding the `-i` flag to use its `--interactive` mode. This will +💡 We recommend adding the `-i` flag to use its [interactive mode]. This will ask you how to run the experiments, and guide you through customizing the aforementioned locations (optional). @@ -70,3 +70,4 @@ begin using DVC Experiments. Now you can move on to [running experiments][run] (next). [sane defaults]: /doc/command-reference/exp/init#description +[interactive mode]: /doc/command-reference/exp/init#example-interactive-mode diff --git a/content/docs/user-guide/experiment-management/running-experiments.md b/content/docs/user-guide/experiment-management/running-experiments.md index 0edca46ab0..276a0f97f1 100644 --- a/content/docs/user-guide/experiment-management/running-experiments.md +++ b/content/docs/user-guide/experiment-management/running-experiments.md @@ -8,176 +8,107 @@ details. > experimentation, you may want to check the basics in > [Get Started: Experiments](/doc/start/experiments/) first. -## The pipeline +## Pipelines files -DVC relies on pipelines that codify experiment workflows (code, -stages, parameters, outputs, etc.) in a -`dvc.yaml` file. These contain the commands to run the experiments. +DVC relies on `dvc.yaml` files that contain the commands to run the +experiment(s). These files codify _pipelines_ that specify the +stages of experiment workflows (code, dependencies, +outputs, etc.). > 📖 See [Get Started: Data Pipelines](/doc/start/data-pipelines) for an intro -> to this topic. -> Here we assume that there's already a working `dvc.yaml` file in the -> project. +> to this topic. -[ug-pipeline-files]: /doc/user-guide/project-structure/pipelines-files +### Running the pipeline(s) -### Running the pipeline - -You can run the pipeline using default settings with `dvc exp run`: +You can run the pipeline using `dvc exp run`. It uses `./dvc.yaml` (in the +current directory) by default: ```dvc $ dvc exp run +... +Reproduced experiment(s): exp-44136 ``` -DVC keeps track of the dependency graph and runs only the stages with changed -dependencies or missing outputs. - -> Example: for a pipeline composed of `prepare`, `train`, and `evaluate` stages, -> if a dependency of `prepare` stage has changed, the downstream stages -> (`train`, `evaluate`) are also run. - -### Running specific stages - -By default DVC uses `./dvc.yaml` (in the current directory). You can specify -`dvc.yaml` files in other directories, or even specific stages to run. These are -given as the last argument to the `dvc exp run`. Examples: - -```dvc -$ dvc exp run my-project/dvc.yaml # a specific dvc.yaml file - -$ dvc exp run extract # a specific stage (from `./dvc.yaml`) - -$ dvc exp run my-project/dvc.yaml:extract - # ^ a stage from a specific dvc.yaml file -``` - -> 📖 See [reproduction `targets`](/doc/command-reference/repro#options) for all -> the details. - -### Running stages independently - -In some cases you may need to run a stage without invoking its dependents. The -`--single-item` (`-s`) flag allows to run the command of a single stage. - -> Example: for a pipeline composed of `prepare`, `train`, and `evaluate` stages -> and you only want to run the `train` stage to check its outputs, you can do so -> by: -> -> ```dvc -> $ dvc exp run --single-stage train -> ``` - -### Running all pipelines - -DVC projects support more than a single pipeline in one or more -`dvc.yaml` files. In this case, you can run all pipelines with a single command: - -```dvc -$ dvc exp run --all-pipelines -``` - -> Note that the order in which pipelines are executed is not guaranteed; Only -> the internal order of stage execution is. - -> (ℹ️) When your `dvc.yaml` files are organized inside recursive subfolders, you -> can selectively run the pipeline(s) using `--recursive` (takes a parent -> directory as argument). +DVC keeps track of the [dependency graph] among stages. It only runs the ones +with changed dependencies or outputs missing from the cache. You +can limit this to certain [reproduction targets] or even single stages +(`--single-item` flag). -### Running stages interactively +DVC projects actually supports more than one pipeline, in one or +more `dvc.yaml` files. The `--all-pipelines` option lets you run them all at +once. -When you want to have more granular control over which stages are run, you can -use the `--interactive` option. This flag allows you to confirm each stage -before running. - -```dvc -$ dvc exp run --interactive -Going to reproduce stage: 'train'... continue? [y/n] -``` +> 📖 `dvc exp run` is an experiment-specific alternative to `dvc repro` where +> you can learn more about these and other pipeline-related options. -> Note that `dvc exp run` is an experimentation-specific alternative to -> `dvc repro`. +[reproduction targets]: /doc/command-reference/repro#options +[dependency graph]: /doc/command-reference/dag#directed-acyclic-graph -## (Hyper)parameters +## Tuning (hyper)parameters -Parameters are the values that modify the underlying code's -behavior, producing different experiment results. Machine learning -experimentation, for example, involves searching hyperparameters that improve -the resulting model metrics. +Parameters are the values that modify the behavior of coded processes -- in this +case producing different experiment results. Machine learning experimentation +often involves defining and searching hyperparameter spaces to improve the +resulting model metrics. -In DVC projects, parameters should be read by the code from _parameter files_ -(`params.yaml` by default). DVC parses these files to track individual param -values. When a tracked param is changed, `dvc exp run` invalidates any stages -that depend on it, and reruns the experiment. +In DVC project source code, parameters should be read from _params +files_ (`params.yaml` by default) and defined in `dvc.yaml`. When a tracked +param value has changed, `dvc exp run` invalidates any stages that depend on it, +and reproduces them. -> Parameters can be defined in `dvc.yaml` directly or through `dvc stage add`. > 📖 See `dvc params` for more details. -For a params file named `params.yaml` with the contents - -```yaml -model: - learning_rate: 0.0001 -``` - -You can specify the parameter dependency as - -```dvc -$ dvc stage add -n train \ - --parameter model.learning_rate \ - --outs ... -``` - -> ⚠️ DVC does not check whether the parameters are actually used in your code. - -
- -#### Non-default parameter files - -DVC allows param files in YAML 1.2, JSON, TOML, and Python formats. When your -parameters file is named something other than `params.yaml`, you need to specify -it in both stage description and `dvc exp run`. For example using -`myparams.toml`: - -```dvc -$ dvc stage add -n train \ - -p myparams.toml:learning_rate \ - ... - -$ dvc exp run -S myparams.toml:learning_rate = 0.0001 -``` - -
- -### Updating experiment parameters on-the-fly - -DVC allows to update parameters from command line when running experiments. The -`--set-param` (`-S`) option takes an existing parameter name and its value, and -updates the params file before the run. +You could manually edit a params file and run an experiment on that basis. Since +this is a common sequence, the built-in option `dvc exp run --set-param` (`-S`) +is provided as a shortcut. It takes an existing param name and value, and +updates the file on-the-fly before execution. ```dvc $ cat params.yaml model: learning_rate: 0.001 + units=64 $ dvc exp run --set-param model.learning_rate=0.0002 ... + +$ dvc exp run -S learning_rate=0.001 -S units=128 # set multiple params +... ``` -> Note that parameters are attached to experiments so you can view them together -> with `dvc exp show` and `dvc exp diff`. +## Experiment results -To set more than one param for the same experiment, use the `-S` option multiple -times: +The results of the last `dvc exp run` can be seen in the workspace. +They are stored and tracked internally by DVC. -```dvc -$ dvc exp run -S learning_rate=0.001 -S units=128 -``` +To display and compare multiple experiments along with their +parameters and metrics, use `dvc exp show` or +`dvc exp diff`. `plots diff` also accepts experiments as `revisions`. See +[Reviewing and Comparing Experiments][reviewing] for more details. + +Use `dvc exp apply` to restore the results of any other experiment instead. See +[Bring experiment results to your workspace][apply] for more info. + +[reviewing]: /doc/user-guide/experiment-management/comparing-experiments +[apply]: + /doc/user-guide/experiment-management/persisting-experiments#bring-experiment-results-to-your-workspace ## The experiments queue The `--queue` option of `dvc exp run` tells DVC to append an experiment for later execution. Nothing is actually run yet. +```dvc +$ dvc exp run --queue -S units=10 +Queued experiment '1cac8ca' for future execution. +$ dvc exp run --queue -S units=64 +Queued experiment '23660bb' for future execution. +$ dvc exp run --queue -S units=128 +Queued experiment '3591a5c' for future execution. +$ dvc exp run --queue -S units=256 +Queued experiment '4109ead' for future execution. +``` +
### How are experiments queued? @@ -191,32 +122,29 @@ is found in `.git/refs/exps`, and earlier ones are in its [reflog].
+Run them all with the `--run-all` flag: + ```dvc -$ dvc exp run --queue -S units=10 -Queued experiment '1cac8ca' for future execution. -$ dvc exp run --queue -S units=64 -Queued experiment '23660bb' for future execution. -$ dvc exp run --queue -S units=128 -Queued experiment '3591a5c' for future execution. -$ dvc exp run --queue -S units=256 -Queued experiment '4109ead' for future execution. +$ dvc exp run --run-all +... ``` -Each experiment is derived from the workspace at the time it's queued. If you -make changes in the workspace afterwards, they won't be reflected in queued -experiments (once run). +> Note that the order of execution is independent of their creation order. + +Their execution happens outside your workspace in temporary +directories for isolation, so each experiment is derived from the workspace at +the time it was queued.
-### How are queued experiments isolated? (Temporary directories) +### How are experiments isolated? -To guarantee that queued experiments derive from their original workspace, DVC -creates a copy of it in `.dvc/tmp/exps/`, where the experiment will run. All -these workspaces share the main project cache. +DVC creates a copy of the experiment's original workspace in `.dvc/tmp/exps/` +and runs it there. All workspaces share the single project cache, +however. -If you want to isolate an experiments this way without queuing it, you can use -the `--temp` option. This allows you to continue working while a long experiment -runs. +💡 To isolate any experiment (without queuing it), you can use the `--temp` +flag. This allows you to continue working while a long experiment runs, e.g.: ```dvc $ nohup dvc exp run --temp & @@ -224,55 +152,32 @@ $ nohup dvc exp run --temp & nohup: ignoring input and appending output to 'nohup.out' ``` -> The above example creates a `nohup.log` file in the original workspace with -> the output of the DVC process. - -Note that Git-ignored files/dirs are explicitly excluded from queued/temp runs -to avoid committing unwanted files into Git (e.g. once successful experiments -are [persisted]). +Note that Git-ignored files/dirs are excluded from queued/temp runs to avoid +committing unwanted files into Git (e.g. once successful experiments are +[persisted]). To include untracked files, stage them with `git add` first +(before `dvc exp run`) and `git reset` them afterwards. [persisted]: /doc/user-guide/experiment-management/persisting-experiments -> 💡 To include untracked files, stage them with `git add` first (before -> `dvc exp run`) and `git reset` them afterwards. -
-Run them all one-by-one with the `--run-all` flag. The order of execution is -independent of their creation order. - -```dvc -$ dvc exp run --run-all -``` - -To remove all experiments from the queue and start over, you can use -`dvc exp remove --queue`. +💡 To clear the experiments queue and start over, use `dvc exp remove --queue`. -### Running experiments in parallel - -DVC allows to run queued experiments in parallel by specifying a number of -execution processes (`--jobs`): - -```dvc -$ dvc exp run --run-all --jobs 4 -``` - -> Note that since each experiment runs in an independent temporary directory, -> common stages may sometimes be executed several times depending -> on the state of the [run-cache] at that time. - -[run-cache]: /doc/user-guide/project-structure/internal-files#run-cache - -⚠️ Parallel runs are experimental and may be unstable at this time. ⚠️ Make sure -you're using a number of jobs that your environment can handle (no more than the -CPU cores). +> 📖 See the `dvc exp run` reference for more options related to experiments +> queue, such as running them in parallel with `--jobs`. ## Checkpoint experiments To track successive steps in a longer or deeper experiment, you can -register [checkpoints](/doc/user-guide/experiment-management/checkpoints) from -your code. Running checkpoint experiments is no different than with regular -ones, e.g.: +register "checkpoints" from your code. These combine DVC Experiments with code +logging. The latter can be achieved either with [DVCLive](/doc/dvclive), by +using `dvc.api.make_checkpoint()` (Python code), or writing signal files (any +programming language) following the same steps as `make_checkpoint()`. + +> 📖 See [Checkpoints](/doc/user-guide/experiment-management/checkpoints) to +> learn more about this feature. + +Running checkpoint experiments is no different than with regular ones, e.g.: ```dvc $ dvc exp run -S param=value @@ -294,6 +199,3 @@ their outputs). This is useful for re-training ML models, for example. > Note that queuing an experiment that uses checkpoints implies `--reset`, > unless a `--rev` is provided (refer to the previous section). - -> 📖 See [Checkpoints](/doc/user-guide/experiment-management/checkpoints) to -> learn more about this feature.