Misc. updates (2.0ish)

content/blog/2020-04-06-april-20-dvc-heartbeat.md

@@ -14,10 +14,10 @@ descriptionLong: |
   projects by our users and big ideas about best practices in ML and data
   science.
 picture: 2020-04-06/april_header.png
-pictureComment:
-  A view from [Barrancas del
-  Cobre](https://en.wikipedia.org/wiki/Copper_Canyon), shot by Jorge Orpinel
-  Pérez. Jorge has mastered the art of working on DVC remotely.
+pictureComment: |
+  A view from
+  [Barrancas del Cobre](https://en.wikipedia.org/wiki/Copper_Canyon), shot by
+  Jorge Orpinel Pérez. Jorge has mastered the art of working on DVC remotely.
 author: elle_obrien
 commentsUrl: https://discuss.dvc.org/t/april-20-heartbeat/347
 tags:

content/docs/command-reference/diff.md

@@ -44,18 +44,13 @@ for example when `dvc init` was used with the `--no-scm` option.
 
 ## Options
 
-- `--targets <paths>` - limit command scope to these paths. When specifying
-  arguments for `--targets` before `a_rev`/`b_rev`, you should use `--` after
-  this option's arguments, e.g.:
+- `--targets <paths>` - limit command scope to these paths.
 
-  ```dvc
-  $ dvc diff --targets t1.json t2.yaml -- HEAD v1
-  ```
-
-  Alternatively, you can also run the above statement as:
+  When specifying arguments for `--targets` before `a_rev`/`b_rev`, you should
+  use `--` after this option's arguments (POSIX terminals), e.g.:
 
   ```dvc
-  $ dvc diff HEAD v1 --targets t1.json t2.json
+  $ dvc diff --targets t1.json t2.yaml -- HEAD v1
   ```
 
 - `--show-json` - prints the command's output in easily parsable JSON format,

content/docs/command-reference/metrics/diff.md

@@ -36,19 +36,15 @@ lists all the current metrics, without comparisons.
 
 ## Options
 
-- `--targets <paths>` - limit command scope to these metrics files. Using `-R`,
-  directories to search metrics files in can also be given. When specifying
-  arguments for `--targets` before `revisions`, you should use `--` after this
-  option's arguments, e.g.:
+- `--targets <paths>` - specify the command's scope to these metrics files. It
+  accepts `paths` to any valid metrics file, regardless of whether it's used by
+  DVC. Using `-R`, directories to search metrics files in can also be given.
 
-  ```dvc
-  $ dvc metrics diff --targets t1.json t2.yaml -- HEAD v1
-  ```
-
-  Alternatively, you can also run the above statement as:
+  When specifying arguments for `--targets` before `revisions`, you should use
+  `--` after this option's arguments (POSIX terminals), e.g.:
 
   ```dvc
-  $ dvc metrics diff HEAD v1 --targets t1.json t2.json
+  $ dvc metrics diff --targets t1.json t2.yaml -- HEAD v1
   ```
 
 - `-R`, `--recursive` - determines the metrics files to use by searching each

content/docs/command-reference/params/diff.md

@@ -1,7 +1,7 @@
 # params diff
 
-Show changes in [parameter dependencies](/doc/command-reference/params) between
-commits in the <abbr>DVC repository</abbr>, or between a commit and the
+Show changes in [parameters](/doc/command-reference/params) between commits in
+the <abbr>DVC repository</abbr>, or between a commit and the
 <abbr>workspace</abbr>.
 
 ## Synopsis
@@ -19,35 +19,30 @@ positional arguments:
 
 ## Description
 
-This command provides a quick way to compare parameter values among experiments
-in the repository history. Requires that Git is being used to version the
-project params.
+Provides a quick way to compare parameter values among experiments in the
+repository history. Requires that Git is being used to version the project
+params.
 
-> Parameter dependencies are defined with the `-p` option in `dvc run`. See also
-> `dvc params`.
+> Parameter dependencies are defined in the `params` field of `dvc.yaml` (e.g.
+> with the the `-p` (`--params`) option of `dvc run`).
 
 Without arguments, this command compares parameters currently present in the
 <abbr>workspace</abbr> (uncommitted changes) with the latest committed version.
-
-Supported parameter _value_ types are: string, integer, float, and arrays. DVC
-itself does not ascribe any specific meaning for these values.
-
-❗ By default it only shows parameters that were changed.
+This includes everything in the default parameters file (`params.yaml`) as well
+as all `params` found in [DVC files](/doc/user-guide/dvc-files). Only params
+that have changes are listed.
 
 ## Options
 
-- `--targets <paths>` - limit command scope to these params files. When
-  specifying arguments for `--targets` before `revisions`, you should use `--`
-  after this option's arguments, e.g.:
+- `--targets <paths>` - specify the command's scope to these params files. It
+  accepts `paths` to any valid parameters file, regardless of whether it's used
+  by DVC.
 
-  ```dvc
-  $ dvc params diff --targets m1.json m2.yaml -- HEAD v1
-  ```
-
-  Alternatively, you can also run the above statement as:
+  When specifying arguments for `--targets` before `a_rev`/`b_rev`, you should
+  use `--` after this option's arguments (POSIX terminals), e.g.:
 
   ```dvc
-  $ dvc params diff HEAD v1 --targets m1.json m2.json
+  $ dvc params diff --targets m1.json m2.yaml -- HEAD v1
   ```
 
 - `--all` - prints all parameters including not changed.

content/docs/command-reference/params/index.md

@@ -18,44 +18,64 @@ positional arguments:
 
 In order to track parameters and hyperparameters associated to machine learning
 experiments in <abbr>DVC projects</abbr>, DVC provides a different type of
-dependencies: _parameters_. Parameters are defined using the the `-p`
-(`--params`) option of `dvc run`, using simple names like `epochs`,
+dependencies: _parameters_. They usually have simple names like `epochs`,
 `learning-rate`, `batch_size`, etc.
 
-In contrast to a regular <abbr>dependency</abbr>, a parameter is not a file (or
-directory). Instead, it consists of a _parameter name_ (or key) to find inside a
-YAML 1.2, JSON, TOML, or [Python](#examples-python-parameters-file) _parameters
-file_. Multiple parameter dependencies can be specified from one or more
-parameters files.
+To start tracking parameters, list them under the `params` field of `dvc.yaml`
+stages (manually or with the the `-p`/`--params` option of `dvc run`). For
+example:
 
-The default parameters file name is `params.yaml`. Parameters should be
-organized as a tree hierarchy inside, as DVC will locate param names by their
-tree path. Parameters files have to be manually written, or generated, and these
-can be versioned directly with Git.
+```yaml
+stages:
+  mystage:
+    cmd: ./myscript.sh
+    params:
+      - foo
+      - bar.baz
+      - myparams.toml:
+          - qux
+```
 
-Supported parameter _value_ types are: string, integer, float, and arrays. DVC
-itself does not ascribe any specific meaning for these values. They are
-user-defined, and serve as a way to generalize and parametrize an machine
-learning algorithms or data processing code.
+> By default, parameters are read from `params.yaml`. Other params files can be
+> listed too, with sub-lists of the params found in them (as shown above).
 
-DVC saves the param names and their latest values in the `dvc.yaml` file. These
-values will be compared to the ones in the params files to determine if the
-stage is invalidated upon pipeline [reproduction](/doc/command-reference/repro).
+In contrast to a regular <abbr>dependency</abbr>, a parameter is not a file or
+directory. Instead, it consists of a _parameter name_ (or key) in a
+[parameters file](#parameters-file), where the _parameter value_ should be
+found. This allows you to define [stage](/doc/command-reference/run)
+dependencies more granularly: changes to other parts of the params file will not
+affect the stage. Parameter dependencies also prevent situations where several
+stages share a regular dependency (e.g. a config file), and any change in it
+invalidates all these stages, causing unnecessary re-executions upon
+`dvc repro`.
 
-> Note that DVC does not pass the parameter values to stage commands. The
-> associated command executed by `dvc run` or `dvc repro` will have to open and
-> parse the parameters file by itself, and use the params specified with `-p`.
+The `dvc params diff` command is available to show parameter changes, displaying
+their current and previous [values](#parameter-values).
+
+### Parameters files
 
-The parameters concept helps to define [stage](/doc/command-reference/run)
-dependencies more granularly. A particular parameter or set of parameters will
-be required for the stage invalidation (see `dvc status` and `dvc repro`).
-Changes to other parts of the dependency file will not affect the stage. This
-prevents situations where several stages share a (configuration) file as a
-common dependency, and any change in this dependency invalidates all these
-stages and causes their reproduction unnecessarily.
+The default params file name is `params.yaml`, but any other YAML 1.2, JSON,
+TOML, or [Python](#examples-python-parameters-file) files can be used
+additionally. These files are typically written manually (or they can be
+generated) and they can be versioned directly with Git.
 
-`dvc params diff` is available to show changes in parameters, displaying the
-param names as well as their current and previous values.
+### Parameter values
+
+Param values should be organized in tree-like hierarchies (dictionaries) inside
+param files (see [Examples](#examples)). DVC will interpret param names as the
+tree path to find those values.
+
+Supported types are: string, integer, float, and arrays. Note that DVC does not
+ascribe any specific meaning to these values.
+
+DVC saves parameter names and values in the project's
+[DVC files](/doc/user-guide/dvc-files) in order to track them over time. They
+will be compared to the latest params files to determine if the stage is
+outdated upon `dvc repro` (or `dvc status`).
+
+> Note that DVC does not pass the parameter values to stage commands. The
+> commands executed by DVC will have to load and parse the parameters file by
+> itself.
 
 ## Options
 
@@ -95,8 +115,8 @@ $ dvc run -n train -d users.csv -o model.pkl \
 > Note that we could use the same parameter addressing with JSON, TOML, or
 > Python parameters files.
 
-The `train.py` script will have some code to parse the needed parameters. For
-example:
+The `train.py` script will have some code to parse and load the needed
+parameters. For example:
 
 ```py
 import yaml
@@ -109,11 +129,12 @@ epochs = params['train']['epochs']
 layers = params['train']['layers']
 ```
 
-You can find that each parameter and it's value were saved to `dvc.yaml`. These
-values will be compared to the ones in the parameters files whenever `dvc repro`
-is used, to determine if dependency to the params file is invalidated:
+You can find that each parameter and their values were saved to `dvc.yaml` and
+`dvc.lock`. These are compared to the values in the params files when
+`dvc repro` is used to determine if the parameter dependency has changed.
 
 ```yaml
+# dvc.yaml
 stages:
   train:
     cmd: python train.py
@@ -127,7 +148,7 @@ stages:
 ```
 
 Alternatively, the entire group of parameters `train` can be referenced, instead
-of specifying each of the group parameters separately:
+of specifying each of the params separately:
 
 ```dvc
 $ dvc run -n train -d users.csv -o model.pkl \

content/docs/command-reference/plots/index.md

@@ -125,8 +125,8 @@ All metrics files given to `dvc plots show` and `dvc plots diff` as input are
 combined together into a single data array for injection into a template file.
 There are two important fields that DVC adds to the plot data:
 
-- `index` - self-incrementing, zero-based counter for the data rows/values. In
-  many cases it corresponds to a machine learning training epoch or step number.
+- `index` - zero-based counter for the data rows/values. In many cases it
+  corresponds to a machine learning training epoch or step number.
 
 - `rev` - Git commit hash, tag, or branch of the metrics file. This helps
   distinguish between different versions when using the `dvc plots diff`

content/docs/user-guide/contributing/docs.md

@@ -202,5 +202,4 @@ We also use "emoji" symbols sparingly for visibility on certain notes. Mainly:
   and "Note that..." notes)
 - 💡 Useful notes and tips, often related to external tools and integrations
 
-> Some other emojis currently in use here and there: ⚡✅🙏🐛⭐❗ (among
-> others).
+> Some other emojis currently in use here and there: ⚡✅🙏🐛⭐ (among others).

content/docs/user-guide/dvc-files.md

@@ -143,9 +143,12 @@ stages:
     metrics:
       - performance.json
   training:
-    desc: Training stage description
-    cmd: python train.py
+    desc: Train model with Python
+    cmd:
+      - pip install -r requirements.txt
+      - python train.py --out ${model_file}
     deps:
+      - requirements.txt
       - train.py
       - features
     outs:
@@ -163,15 +166,19 @@ stages:
 by the user with the `--name` (`-n`) option of `dvc run`. Each stage can contain
 the following fields:
 
-- `cmd` (always present): Executable command defined in this stage
+- `cmd` (always present): One or more commands executed by the stage (may
+  contain either a single value, or a list). Commands are executed sequentially
+  until all are finished or until one of them fails (see `dvc repro` for
+  details).
 - `wdir`: Working directory for the stage command to run in (relative to the
   file's location). If this field is not present explicitly, it defaults to `.`
   (the file's location).
 - `deps`: List of <abbr>dependency</abbr> file or directory paths of this stage
   (relative to `wdir` which defaults to the file's location). See
   [Dependency entries](#dependency-entries) above for more details.
-- `params`: List of <abbr>parameter</abbr> dependency keys (field names) that
-  are read from a YAML, JSON, TOML, or Python file (`params.yaml` by default)
+- `params`: List of <abbr>parameter</abbr> dependency keys (field names) to
+  track in `params.yaml`. The list may also contain other YAML, JSON, TOML, or
+  Python file names, with a sub-list of the param names to track in them.
 - `outs`: List of <abbr>output</abbr> file or directory paths of this stage
   (relative to `wdir` which defaults to the file's location). See
   [Output entries](#output-entries) above for more details.
@@ -207,8 +214,8 @@ For every `dvc.yaml` file, a matching `dvc.lock` (YAML) file usually exists.
 It's created or updated by DVC commands such as `dvc run` and `dvc repro`.
 `dvc.lock` describes the latest pipeline state. It has several purposes:
 
-- Tracking of intermediate and final results of a pipeline — similar to
-  [`.dvc` files](#dvc-files).
+- Tracking of intermediate and final <abbr>outputs</abbr> of a pipeline —
+  similar to [`.dvc` files](#dvc-files).
 - Allow DVC to detect when stage definitions, or their dependencies have
   changed. Such conditions invalidate stages, requiring their reproduction (see
   `dvc status`, `dvc repro`).