metrics: 1.0 update

config/prismjs/dvc-commands.js

@@ -29,10 +29,7 @@ module.exports = [
   'pipeline',
   'move',
   'metrics show',
-  'metrics remove',
-  'metrics modify',
   'metrics diff',
-  'metrics add',
   'metrics',
   'params diff',
   'params',

content/docs/command-reference/metrics/diff.md

@@ -7,10 +7,10 @@ commits in the <abbr>DVC repository</abbr>, or between a commit and the
 ## Synopsis
 
 ```usage
-usage: dvc metrics diff [-h] [-q | -v]
-                        [--targets [<path> [<path> ...]]]
-                        [-t <type>] [-x <path>] [-R] [--all]
-                        [--show-json] [--show-md] [a_ref] [b_ref]
+usage: dvc metrics diff [-h] [-q | -v] [--targets [<paths> [<paths> ...]]]
+                        [-R] [--all] [--show-json] [--show-md] [--no-path]
+                        [--old]
+                        [a_rev] [b_rev]
 
 positional arguments:
   a_rev                 Old Git commit to compare (defaults to HEAD)
@@ -45,28 +45,19 @@ They're calculated between two commits (hash, branch, tag, or any
   target directory and its subdirectories for DVC-files to inspect. If there are
   no directories among the `targets`, this option is ignored.
 
-- `-t <type>`, `--type <type>` - specify a type of the metric file. Accepted
-  values are: `json`. It will be saved into the corresponding DVC-file, and used
-  to determine how to handle displaying metrics. See `dvc metrics show` for more
-  details.
-
-  This option will override any `type` and `xpath` values defined in the
-  corresponding DVC-file. If no `type` is provided or found in the DVC-file, DVC
-  will try to detect it based on file extension.
-
-- `-x <path>`, `--xpath <path>` - specify a path within a metric file to show
-  changes for a specific metric value only. Should be used if the metric file
-  contains multiple numbers and you want to use only one of them. Only a single
-  path is allowed. It will override `xpath` defined in the corresponding
-  DVC-file. See `dvc metrics show` for more details.
-
 - `--all` - list all metrics, even those without changes.
 
 - `--show-json` - prints the command's output in easily parsable JSON format,
   instead of a human-readable table.
 
 - `--show-md` - prints the command's output in Markdown table format.
 
+- `--old` - Show old metric value in addition to the new value.
+
+- `--no-path` - Don't show metric path in the result table. This option is
+  useful when only one metrics file is in use or there is no intersection
+  between the metrics names.
+
 - `-h`, `--help` - prints the usage/help message, and exit.
 
 - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no

content/docs/command-reference/metrics/index.md

@@ -1,24 +1,18 @@
 # metrics
 
-A set of commands to add, manage, collect, and display project metrics:
-[add](/doc/command-reference/metrics/add),
-[show](/doc/command-reference/metrics/show),
-[diff](/doc/command-reference/metrics/diff),
-[modify](/doc/command-reference/metrics/modify), and
-[remove](/doc/command-reference/metrics/remove).
+A set of commands to manage, collect, and display project metrics:
+[show](/doc/command-reference/metrics/show), and
+[diff](/doc/command-reference/metrics/diff).
 
 ## Synopsis
 
 ```usage
 usage: dvc metrics [-h] [-q | -v]
-                   {show,add,modify,remove,diff} ...
+                   {show,diff} ...
 
 positional arguments:
   COMMAND
     show                Print metrics, with optional formatting.
-    add                 Mark a DVC-tracked file as a metric.
-    modify              Modify metric default formatting.
-    remove              Remove metric mark on a DVC-tracked file.
     diff                Show changes in metrics between commits
 ```
 
@@ -27,21 +21,76 @@ positional arguments:
 In order to track metrics associated to machine learning experiments, DVC has
 the ability to mark a certain stage <abbr>outputs</abbr> as files containing
 metrics to track (see the `--metrics` option of `dvc run`). Metrics are
-project-specific floating-point values e.g. `AUC`, `ROC`, etc.
-
-Supported file formats: JSON. Metrics can be organized in a tree hierarchy in a
-JSON file. DVC addresses the metrics by the tree path.
+project-specific floating-point or integer values e.g. `AUC`, `ROC`,
+`False-Positives` etc.
+
+[Show](/doc/command-reference/metrics/show), and
+[diff](/doc/command-reference/metrics/diff), commands are available to manage
+<abbr>DVC project</abbr> metrics.
+
+## Formats and metrics types
+
+Supported file formats: JSON and YAML. Metrics can be organized in a tree
+hierarchy in a JSON file or a YAML file. DVC addresses the metrics by the tree
+path. In the example below five metrics are presented: `train.accuracy`,
+`train.loss`, `train.TN`, `train.FP` and `time_real`.
+
+```json
+{
+  "train": {
+    "accuracy": 0.9886999726295471,
+    "loss": 0.041855331510305405,
+    "TN": 473,
+    "FP": 845
+  },
+  "time_real": 344.61309599876404
+}
+```
 
 DVC itself does not ascribe any specific meaning for these numbers. Usually
 these numbers are produced by the model training or model evaluation code and
 serve as a way to compare and pick the best performing experiment.
 
-[Add](/doc/command-reference/metrics/add),
-[show](/doc/command-reference/metrics/show),
-[diff](/doc/command-reference/metrics/diff),
-[modify](/doc/command-reference/metrics/modify), and
-[remove](/doc/command-reference/metrics/remove) commands are available to set up
-and manage <abbr>DVC project</abbr> metrics.
+## Default metrics files
+
+`dvc metrics show` and `dvc metrics diff` commands by default use all metrics
+files that are specified in `dvc.yaml`. So, there no need to specify metrics
+file name to see the deault metrics. The metrics files can be added to
+`dvc.yaml` by options `--metrics` (`-m`) or `--metrics-no-cache` (`-M`) of
+`dvc run` command or manualy to `metrics` section of a stage:
+
+```yaml
+stages:
+  train:
+    cmd: python train.py
+    deps:
+      - users.csv
+    params:
+      - epochs
+      - dropout
+      - lr
+    outs:
+      - model.pkl
+    metrics:
+      - summary.json:
+          cache: false
+```
+
+`False` value of the additional `cache` option specifies that this is not data
+file, it should not be moved to DVC cache and it should be commited to Git
+history instead.
+
+### Difference between continuous and scalar metrics
+
+DVC has two concepts for metrics for representing result of machine learning
+training or data processing:
+
+1. `dvc metrics` to represent scalar numbers such as AUC, true positive rate and
+   others.
+2. `dvc plots` to visualize continuous metrics such as AUC curves, loss
+   functions, confusion matrices, and others.
+
+In this section, only scalar metrics are covered.
 
 ## Options
 
@@ -54,7 +103,7 @@ and manage <abbr>DVC project</abbr> metrics.
 
 ## Examples
 
-First, let's create a simple DVC-file:
+First, let's create a simple DVC pipeline stage:
 
 ```dvc
 $ dvc run -d code/evaluate.py -M data/eval.json \
@@ -69,9 +118,7 @@ Now let's print metric values that we are tracking in this <abbr>project</abbr>,
 using `dvc metrics show`:
 
 ```dvc
-$ dvc metrics show -a
-
-  master:
+$ dvc metrics show
       data/eval.json:
 		{
 		    "AUC": 0.65115,
@@ -80,30 +127,24 @@ $ dvc metrics show -a
 		}
 ```
 
-We can also give DVC an `xpath` for the metric file, so that it outputs only the
-AUC value. For JSON metrics, we use
-[JSONPath](https://goessner.net/articles/JsonPath/index.html) expressions to
-filter data out of metric files:
-
-```dvc
-$ dvc metrics show --xpath AUC data/eval.json
-      data/eval.json: {'AUC': 0.65115}
-```
-
-The `xpath` filter can be saved as the default way to display a metrics file:
+When metrics file changes in the user workspace witout commiting it to the Git,
+the `dvc metrics diff` command show the difference between metrics values:
 
 ```dvc
-$ dvc metrics modify data/eval.json --xpath AUC
-$ dvc metrics show
-      data/eval.json: {'AUC': 0.65115}
+$ dvc metrics diff
+Path       Metric    Value    Change
+eval.json  ACU       0.66729  0.01614
+eval.json  error     0.16982  0.00322
+eval.json  TP        516      -12
 ```
 
-And finally let's remove `data/eval.json` from the project metrics:
+Metrics commited to the Git history can be shown by referencing the commit name
+or any aliase:
 
 ```dvc
-$ dvc metrics remove data/eval.json
-$ dvc metrics show
-
-Failed to show metrics: No metric files in this repository.
-Use 'dvc metrics add' to add a metric file to track.
+$ dvc metrics diff HEAD c7bef55
+Path       Metric    Value    Change
+eval.json  ACU       0.66729  0.01614
+eval.json  error     0.16982  0.00322
+eval.json  TP        516      -12
 ```