Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

params: support for tracking all params in a file #3431

Merged
merged 7 commits into from
May 21, 2022
4 changes: 3 additions & 1 deletion content/docs/command-reference/params/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,9 @@ to define [stage](/doc/command-reference/run) dependencies more granularly:
changes to other parts of the params file will not affect the stage. Parameter
dependencies also prevent situations where several stages share a regular
dependency (e.g. a config file), and any change in it invalidates all of them
(see `dvc status`), causing unnecessary re-executions upon `dvc repro`.
(see `dvc status`), causing unnecessary re-executions upon `dvc repro`. However,
you can track all the parameters in a file by setting _parameter value_ to be
`null`.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

The default **parameters file** name is `params.yaml`, but any other YAML 1.2,
JSON, TOML, or [Python](#examples-python-parameters-file) files can be used
Expand Down
16 changes: 14 additions & 2 deletions content/docs/user-guide/project-structure/pipelines-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,17 @@ If it writes files or dirs, they can be defined as <abbr>outputs</abbr>
### Parameter dependencies

[Parameters](/doc/command-reference/params) are a special type of stage
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
dependency. They consist of a name/value pair to find in a YAML, JSON, TOML, or
Python parameters file (`params.yaml` by default). Example:
dependency. They consist of a list of params to track in one of these formats:

1. A param key/value pair that can be found in `params.yaml` (default params
file);
2. A dictionary named by the file path to a custom params file, and with a list
of param key/value pairs to find in it;
3. An empty set (give no value or use `null`) named by the file path to a params
file: to track all the params in it dynamically.

> Note that file paths used must be to valid YAML, JSON, TOML, or Python
> parameters file.

```yaml
stages:
Expand All @@ -63,6 +72,9 @@ stages:
params:
- threshold
- passes
- custom_params.yaml: # track only specified parameters
- epochs
- config.json: # track all parameters
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
outs:
- clean.txt
```
Expand Down