diff --git a/content/docs/command-reference/params/index.md b/content/docs/command-reference/params/index.md index 285685c1d5..6309309a12 100644 --- a/content/docs/command-reference/params/index.md +++ b/content/docs/command-reference/params/index.md @@ -30,10 +30,11 @@ stages: learn: cmd: ./deep.py params: - - epochs + - epochs # track specific parameter (from params.yaml) - tuning.learning-rate - - myparams.toml: + - myparams.toml: # track specific params from custom file - batch_size + - config.json: # track all parameters in this file ``` In contrast to a regular dependency, a parameter dependency is not @@ -47,9 +48,9 @@ dependency (e.g. a config file), and any change in it invalidates all of them The default **parameters file** name is `params.yaml`, but any other YAML 1.2, JSON, TOML, or [Python](#examples-python-parameters-file) files can be used -additionally (listed under `params:` with a sub-list of param values, as shown -in the sample above) . These files are typically written manually (or they can -be generated) and they can be versioned directly with Git. +additionally (listed under `params:` as shown in the sample above). These files +are typically written manually (or they can be generated) and they can be +versioned directly with Git. **Parameter values** should be organized in tree-like hierarchies (dictionaries) inside params files (see [Examples](#examples)). DVC will interpret param names diff --git a/content/docs/command-reference/run.md b/content/docs/command-reference/run.md index 5f534eaee5..6a29bd51b3 100644 --- a/content/docs/command-reference/run.md +++ b/content/docs/command-reference/run.md @@ -197,16 +197,16 @@ $ dvc run -n second_stage './another_script.sh $MYENVVAR' outputs are not tracked by DVC (same as with `-O` above). - `-c `, `--checkpoints ` - the same as `-o` but also marks the - output as a [checkpoint](/doc/command-reference/exp/run#checkpoints). Implies - `--no-exec`. This makes the stage incompatible with `dvc repro`. - -- `-p [:]`, `--params [:]` - specify a set - of [parameter dependencies](/doc/command-reference/params) the stage depends - on, from a parameters file. This is done by sending a comma separated list as - argument, e.g. `-p learning_rate,epochs`. The default parameters file name is - `params.yaml`, but this can be redefined with a prefix in the argument sent to - this option, e.g. `-p parse_params.yaml:threshold`. See `dvc params` to learn - more about parameters. + output as a [checkpoint](/doc/command-reference/exp/run#checkpoints). This + makes the stage incompatible with `dvc repro`. Implies `--no-exec`. + +- `-p [:]`, `--params [:]` - specify one + or more [parameter dependencies](/doc/command-reference/params) from a + parameters file `path` (`./params.yaml` by default). This is done by sending a + comma separated list (`params_list`) as argument, e.g. + `-p learning_rate,epochs`. A custom params file can be defined with a prefix, + e.g. `-p params.json:threshold`. Or use the prefix alone with `:` to use all + the parameters found in that file, e.g. `-p myparams.toml:`. - `-m `, `--metrics ` - specify a metrics file produced by this stage. This option behaves like `-o` but registers the file in a `metrics` diff --git a/content/docs/command-reference/stage/add.md b/content/docs/command-reference/stage/add.md index 218ead53d2..e58e637f40 100644 --- a/content/docs/command-reference/stage/add.md +++ b/content/docs/command-reference/stage/add.md @@ -195,13 +195,13 @@ data science experiments. output as a [checkpoint](/doc/command-reference/exp/run#checkpoints). This makes the stage incompatible with `dvc repro`. -- `-p [:]`, `--params [:]` - specify a set - of [parameter dependencies](/doc/command-reference/params) the stage depends - on, from a parameters file. This is done by sending a comma separated list as - argument, e.g. `-p learning_rate,epochs`. The default parameters file name is - `params.yaml`, but this can be redefined with a prefix in the argument sent to - this option, e.g. `-p parse_params.yaml:threshold`. See `dvc params` to learn - more about parameters. +- `-p [:]`, `--params [:]` - specify one + or more [parameter dependencies](/doc/command-reference/params) from a + parameters file `path` (`./params.yaml` by default). This is done by sending a + comma separated list (`params_list`) as argument, e.g. + `-p learning_rate,epochs`. A custom params file can be defined with a prefix, + e.g. `-p params.json:threshold`. Or use the prefix alone with `:` to use all + the parameters found in that file, e.g. `-p myparams.toml:`. - `-m `, `--metrics ` - specify a metrics file produced by this stage. This option behaves like `-o` but registers the file in a `metrics` diff --git a/content/docs/user-guide/project-structure/pipelines-files.md b/content/docs/user-guide/project-structure/pipelines-files.md index 674b6c906f..c064dc5524 100644 --- a/content/docs/user-guide/project-structure/pipelines-files.md +++ b/content/docs/user-guide/project-structure/pipelines-files.md @@ -51,8 +51,21 @@ If it writes files or dirs, they can be defined as outputs ### Parameter dependencies [Parameters](/doc/command-reference/params) are a special type of stage -dependency. They consist of a name/value pair to find in a YAML, JSON, TOML, or -Python parameters file (`params.yaml` by default). Example: +dependency. They consist of a list of params to track in one of these formats: + +1. A param key/value pair that can be found in `params.yaml` (default params + file); +2. A dictionary named by the file path to a custom params file, and with a list + of param key/value pairs to find in it; +3. An empty set (give no value or use `null`) named by the file path to a params + file: to track all the params in it dynamically. + + + +Note that file paths used must be to valid YAML, JSON, TOML, or Python +parameters file. + + ```yaml stages: @@ -61,8 +74,11 @@ stages: deps: - raw.txt params: - - threshold + - threshold # track specific param (from params.yaml) - passes + - myparams.yaml: # track specific params from custom file + - epochs + - config.json: # track all parameters in this file outs: - clean.txt ```