From 7347cefaaaf3375a0534c8a68369e230552317f9 Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Sat, 28 Mar 2020 06:25:19 -0700 Subject: [PATCH 1/8] Get rid of CSV/TSV metrics --- content/docs/command-reference/metrics/add.md | 65 +++++++++---------- .../docs/command-reference/metrics/diff.md | 54 ++++++--------- .../docs/command-reference/metrics/index.md | 37 +++++++---- .../docs/command-reference/metrics/modify.md | 58 ++++++++--------- .../docs/command-reference/metrics/remove.md | 50 ++++++-------- .../docs/command-reference/metrics/show.md | 10 +-- 6 files changed, 125 insertions(+), 149 deletions(-) diff --git a/content/docs/command-reference/metrics/add.md b/content/docs/command-reference/metrics/add.md index cd2a3d790c..f93c4e5c2d 100644 --- a/content/docs/command-reference/metrics/add.md +++ b/content/docs/command-reference/metrics/add.md @@ -18,12 +18,12 @@ defines the given `path` as an output, marking `path` as a metric file to track. Note that outputs can also be marked as metrics via the `-m` or `-M` options of -`dvc run`. +`dvc run`. We recommend using `-M` option to keep metrics in Git history. -While any text file can be tracked as a metric file, we recommend using TSV, -CSV, or JSON formats. DVC provides a way to parse those formats to get to a -specific value, if the file contains multiple metrics. See the -[options](#options) below and `dvc metrics show` for more info. +While any text file can be tracked as a metric file, we recommend using JSON +formats. DVC provides a way to parse this formats to get to a specific value, if +the file contains multiple metrics. See the [options](#options) below and +`dvc metrics diff` for more info. > Note that [external output](/doc/user-guide/managing-external-data) cannot be > marked as project metrics. @@ -31,29 +31,23 @@ specific value, if the file contains multiple metrics. See the ## Options - `-t `, `--type ` - specify a type for the metric file. Accepted - values are: `raw` (default), `json`, `tsv`, `htsv`, `csv`, `hcsv`. It will be - saved into the corresponding DVC-file, and used by `dvc metrics show` to - determine how to handle displaying metrics. + values are: `raw` (default), `json`. It will be saved into the corresponding + DVC-file, and used by `dvc metrics show` to determine how to handle displaying + metrics. `raw` means that no additional parsing is applied, and `--xpath` is ignored. - `htsv`/`hcsv` are the same as `tsv`/`csv`, but the values in the first row of - the file will be used as the field names and should be used to address columns - in the `--xpath` option. - `-x `, `--xpath ` - specify a path within a metric file to get a specific metric value. Should be used if the metric file contains multiple numbers and you want to use only one of them. Only a single path is allowed. It will be saved into the corresponding DVC-file, and used by - `dvc metrics show` to determine how to display metrics. The accepted value - depends on the metric file type (`--type` option): + `dvc metrics show` and `dvc metrics diff` to determine how to display metrics. + The accepted value depends on the metric file type (`--type` option): - For `json` - see [JSONPath](https://goessner.net/articles/JsonPath/) or [jsonpath-ng](https://github.com/h2non/jsonpath-ng) to know the syntax. For example, `"AUC"` extracts the value from the following JSON-formatted metric file: `{"AUC": "0.624652"}`. - - For `tsv`/`csv` - `row,column` e.g. `1,2`. Indices are 0-based. - - For `htsv`/`hcsv` - `row,column name` e.g. `0,Name`. Row index is 0-based. - First row is used to specify column names and is not included into index. - `-h`, `--help` - prints the usage/help message, and exit. @@ -68,22 +62,24 @@ Let's first create a regular output with the `-o` option of `dvc run`: ```dvc -$ dvc run -o metrics.txt "echo 0.9643 > metrics.txt" +$ dvc run -o metrics.json \ + 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' ``` -Even when we named this output file `metrics.txt`, DVC won't know that it's a -metric if we don't specify so. The content of stage file `metrics.txt.dvc` (a +Even when we named this output file `metrics.json`, DVC won't know that it's a +metric if we don't specify so. The content of stage file `metrics.json.dvc` (a [DVC-file](/doc/user-guide/dvc-file-format)) should look like this: (Notice the `metric: false` field.) ```yaml -cmd: echo 0.9643 > metrics.txt -md5: f75f291b02ab38530ba659c1e10e577f +md5: 906ea9489e432c85d085b248c712567b +cmd: echo {\"AUC\":0.9643, \"TP\":527} > metrics.json outs: - - cache: true - md5: 235d585fcea283135682457b15c76101 + - md5: 0f0e67dc927aa69cd3fc37435ee1304f + path: metrics.json + cache: true metric: false - path: metrics.txt + persist: false ``` If you run `dvc metrics show` now, you should get an error message: @@ -97,27 +93,26 @@ ERROR: failed to show metrics - no metric files in Now, let's mark the output as a metric: ```dvc -$ dvc metrics add metrics.txt - -Saving information to 'metrics.txt.dvc'. +$ dvc metrics add metrics.json ``` -This command updates `metrics.txt.dvc` to specify that `metrics.txt` is actually -a metric file: +This command updates `metrics.json.dvc` to specify that `metrics.json` is +actually a metric file: ```yaml -cmd: echo 0.9643 > metrics.txt -md5: f75f291b02ab38530ba659c1e10e577f +md5: 906ea9489e432c85d085b248c712567b +cmd: echo {\"AUC\":0.9643, \"TP\":527} > metrics.json outs: - - cache: true - md5: 235d585fcea283135682457b15c76101 + - md5: 0f0e67dc927aa69cd3fc37435ee1304f + path: metrics.json + cache: true metric: type: raw - path: metrics.txt + persist: false ``` And if you run `dvc metrics show` you should now see a report like this: ```dvc -metrics.txt: 0.9643 + metrics.json: {"AUC":0.9643, "TP":527} ``` diff --git a/content/docs/command-reference/metrics/diff.md b/content/docs/command-reference/metrics/diff.md index 8cc97f0a0f..48324f992d 100644 --- a/content/docs/command-reference/metrics/diff.md +++ b/content/docs/command-reference/metrics/diff.md @@ -42,9 +42,8 @@ They're calculated between two commits (hash, branch, tag, or any no directories among the `targets`, this option is ignored. - `-t `, `--type ` - specify a type of the metric file. Accepted - values are: `raw` (default), `json`, `tsv`, `htsv`, `csv`, `hcsv`. It will be - used to determine how to parse and format metics for display. See - `dvc metrics show` for more details. + values are: `raw` (default), `json`. It will be used to determine how to parse + and format metics for display. See `dvc metrics show` for more details. This option will override `type` and `xpath` defined in the corresponding DVC-file. If no `type` is provided or found in the DVC-file, DVC will try to @@ -68,35 +67,31 @@ They're calculated between two commits (hash, branch, tag, or any ## Examples -Let's employ a simple workspace with some data, code, ML models, -pipeline stages, such as the DVC project created in our -[Get Started](/doc/tutorials/get-started) section. Then we can see what happens -with `dvc install` in different situations. - -
- -### Click and expand to setup the project - -Start by cloning our example repo if you don't already have it: +Start by creating a simple metrics file and commit it: ```dvc -$ git clone https://github.com/iterative/example-get-started -$ cd example-get-started +$ dvc run -M metrics.json \ + 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' +$ git add metrics.json metrics.json.dvc +$ git commit -m "Add metrics file" ``` -
- -Notice that we have an `auc.metric` metric file: - ``` -$ cat auc.metric -0.602818 +$ cat metrics.json +{"AUC":0.9643, "TP":527} ``` Now let's mock a change in our AUC metric: ``` -$ echo '0.5' > auc.metric +$ echo {\"AUC\":0.9671, \"TP\":531} > metrics.json + +$ git diff +--- a/metrics.json ++++ b/metrics.json +@@ -1 +1 @@ +-{"AUC":0.9643, "TP":527} ++{"AUC":0.9671, "TP":531} ``` To see the change, let's run `dvc metrics diff`. This compares our current @@ -104,17 +99,8 @@ To see the change, let's run `dvc metrics diff`. This compares our current had in the previous commit: ``` -$ git diff ---- a/auc.metric -+++ b/auc.metric -@@ -1 +1 @@ --0.602818 -+0.5 - $ dvc metrics diff - Path Metric Value Change -auc.metric 0.500 -0.103 + Path Metric Value Change +metrics.json TP 531 4 +metrics.json AUC 0.967 0.003 ``` - -> Note that metric files are typically versioned with Git, so we can use both -> `git diff` and `dvc metrics diff` to understand their changes, as seen above. diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index c53418a653..0ef825889a 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -22,12 +22,17 @@ positional arguments: ## Description -DVC has the ability to mark a certain stage outputs as files -containing metrics to track. (See the `--metrics` option of `dvc run`.) Metrics -are project-specific numeric values e.g. `AUC`, `ROC`, etc. DVC itself does not -ascribe any specific meaning for these numbers. Usually these numbers are -produced by the model evaluation script and serve as a way to compare and pick -the best performing experiment. +In order to track metrics associated to machie learning experiments DVC has the +ability to mark a certain stage outputs as files containing metrics +to track. (See the `--metrics` option of `dvc run`.) Metrics are +project-specific floating-point values e.g. `AUC`, `ROC`, etc. + +Supported file formats: JSON. Metrics can be organized in a tree hierarchy in a +JSON file. DVC addresses the metrics by the tree path. + +DVC itself does not ascribe any specific meaning for these numbers. Usually +these numbers are produced by the model training or model evaluation code and +serve as a way to compare and pick the best performing experiment. [Add](/doc/command-reference/metrics/add), [show](/doc/command-reference/metrics/show), @@ -65,7 +70,12 @@ Now let's print metric values that we are tracking in this project: $ dvc metrics show -a master: - data/eval.json: {"AUC": "0.624652"} + data/eval.json: + { + "AUC": 0.65115, + "error": 0.17304, + "TP": 528 + } ``` We can also tell DVC an `xpath` for the metric file, so that it can output only @@ -74,11 +84,16 @@ the value of AUC. In the case of JSON, use selectively extract data out of metric files: ```dvc -$ dvc metrics modify data/eval.json --type json --xpath AUC -$ dvc metrics show +$ dvc metrics show --xpath AUC data/eval.json + data/eval.json: {'AUC': 0.65115} +``` - master: - data/eval.json: 0.624652 +The xpath filter can be saved for a metrics file: + +```dvc +$ dvc metrics modify data/eval.json --xpath AUC +$ dvc metrics show + data/eval.json: {'AUC': 0.65115} ``` And finally let's remove `data/eval.json` from the project metrics: diff --git a/content/docs/command-reference/metrics/modify.md b/content/docs/command-reference/metrics/modify.md index 257afcaacd..432c0a13cf 100644 --- a/content/docs/command-reference/metrics/modify.md +++ b/content/docs/command-reference/metrics/modify.md @@ -33,14 +33,11 @@ ERROR: failed to modify metric file settings - ## Options - `-t `, `--type ` - specify a type for the metric file. Accepted - values are: `raw` (default), `json`, `tsv`, `htsv`, `csv`, `hcsv`. It will be - saved into the corresponding DVC-file, and used by `dvc metrics show` to - determine how to handle displaying metrics. + values are: `raw` (default), `json`. It will be saved into the corresponding + DVC-file, and used by `dvc metrics show` to determine how to handle displaying + metrics. `raw` means that no additional parsing is applied, and `--xpath` is ignored. - `htsv`/`hcsv` are the same as `tsv`/`csv`, but the values in the first row of - the file will be used as the field names and should be used to address columns - in the `--xpath` option. - `-x `, `--xpath ` - specify a path within a metric file to get a specific metric value. Should be used if the metric file contains multiple @@ -53,9 +50,6 @@ ERROR: failed to modify metric file settings - [jsonpath-ng](https://github.com/h2non/jsonpath-ng) to know the syntax. For example, `"AUC"` extracts the value from the following JSON-formatted metric file: `{"AUC": "0.624652"}`. - - For `tsv`/`csv` - `row,column` e.g. `1,2`. Indices are 0-based. - - For `htsv`/`hcsv` - `row,column name` e.g. `0,Name`. Row index is 0-based. - First row is used to specify column names and is not included into index. - `-h`, `--help` - prints the usage/help message, and exit. @@ -70,52 +64,52 @@ Let's first imagine we have a [stage](/doc/command-reference/run) with a generic metric file initially. The dummy command below simulates this imaginary setup: ```dvc -$ dvc run -M metrics.csv "echo auc, 0.9567 > metrics.csv" +$ dvc run -o metrics.json \ + 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' ``` -The resulting stage file `metrics.csv.dvc` should look like this: +The resulting stage file `metrics.json.dvc` should look like this: ```yaml -md5: 6ed9b798bf460e1aa80b27388425a07d -cmd: echo auc, 0.9567 > metrics.csv -wdir: . +md5: 906ea9489e432c85d085b248c712567b +cmd: echo {\"AUC\":0.9643, \"TP\":527} > metrics.json outs: - - md5: 13ee80c6b3e238c5097427c2114ae6e4 - path: metrics.csv - cache: false - metric: true + - md5: 0f0e67dc927aa69cd3fc37435ee1304f + path: metrics.json + cache: true + metric: false persist: false ``` -And if we run `dvc metrics show metrics.csv`, we will get the complete contents +And if we run `dvc metrics show metrics.json`, we will get the complete contents of the file: ```dvc -$ dvc metrics show metrics.csv - metrics.csv: auc 0.9567 +$ dvc metrics show metrics.json + metrics.json: {"AUC":0.9643, "TP":527} ``` -Okay. Let's now imagine we are interested only in the numeric value, the second -column of the CSV file. We can specify the `CSV` type (`-t`) and an `xpath` -(`-x`) to extract the second column: +Okay. Let's now imagine we are interested only in a single value of true +posivives - TP. We can specify the `JSON` type (`-t`) and an `xpath` (`-x`) to +extract the TP value: ```dvc -$ dvc metrics modify -t csv -x '0,1' metrics.csv +$ dvc metrics modify -t json -x TP metrics.json ``` -After this change `dvc metrics show` should always select only the value itself, -and exclude names: +After this change `dvc metrics show` should always select only the specified +value: ```dvc -$ dvc metrics show metrics.csv - metrics.csv: [' 0.9567'] +$ dvc metrics show metrics.json + metrics.json: {'TP': 527} ``` -Notice that the `metric` field in the `metrics.csv.dvc` stage file changed to +Notice that the `metric` field in the `metrics.json.dvc` stage file changed to include this information: ```yaml metric: - type: csv - xpath: 0,1 + type: json + xpath: TP ``` diff --git a/content/docs/command-reference/metrics/remove.md b/content/docs/command-reference/metrics/remove.md index c40a87182f..1e22fcb9cb 100644 --- a/content/docs/command-reference/metrics/remove.md +++ b/content/docs/command-reference/metrics/remove.md @@ -46,43 +46,39 @@ Let's first create a dummy non-cached metric output with the `-M` option of `dvc run`: ```dvc -$ dvc run -M metrics.tsv \ - "echo -e 'time\tauc\n2019-02-13\t0.9643' > metrics.tsv" -... -Saving information to 'metrics.tsv.dvc'. -$ cat metrics.tsv -time auc -2019-02-13 0.9643 +$ dvc run -M metrics.json \ + 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' +$ cat metrics.json +{"AUC":0.9671, "TP":531} ``` -To extract the AUC value out of `metrics.tsv`, we can use the `-t` and `-x` +To extract the AUC value out of `metrics.json`, we can use the `-t` and `-x` options that `dvc metrics show` command supports: ```dvc -$ dvc metrics show -t htsv -x 0,auc metrics.tsv - metrics.tsv: ['0.9643'] +$ dvc metrics show -t json -x AUC metrics.json + metrics.json: {'AUC': 0.9671} ``` -If you check the `metrics.tsv.dvc` file, you should see that `metric: true` is +If you check the `metrics.json.dvc` file, you should see that `metric: true` is set: ```yaml -md5: 6f910c9000bb03492d1e66035ba8faf6 -cmd: echo -e 'time\tauc\n2019-02-13\t0.9643' > metrics.tsv -wdir: . +ndex 7ad71f8..e7f94e9 100644 +md5: 1443725f6d0bd5b77aa8d5fc36e886ef +cmd: echo {\"AUC\":0.9643, \"TP\":527} > metrics.json outs: - - md5: 7ce0bc12da7f88c1493763cdd4c3f684 - path: metrics.tsv - cache: false - metric: true - persist: false +- md5: 0f0e67dc927aa69cd3fc37435ee1304f + path: metrics.json + cache: false + metric: true + persist: false ``` Now, let's reset the `metric` field with the `dvc metrics remove` command: ```dvc -$ dvc metrics remove metrics.tsv -Saving information to 'metrics.tsv.dvc'. +$ dvc metrics remove metrics.json ``` Let's check the outputs section (`outs`) of same @@ -90,8 +86,8 @@ Let's check the outputs section (`outs`) of same ```yaml outs: - - md5: 7ce0bc12da7f88c1493763cdd4c3f684 - path: metrics.tsv + - md5: 0f0e67dc927aa69cd3fc37435ee1304f + path: metrics.json cache: false metric: persist: false @@ -101,10 +97,6 @@ As you can see, nothing has changed, except the `metric` field. And both files are still here: ```dvc -$ tree -. -├── metrics.tsv -└── metrics.tsv.dvc - -0 directories, 2 files +$ ls metrics.json metrics.json.dvc +metrics.json metrics.json.dvc ``` diff --git a/content/docs/command-reference/metrics/show.md b/content/docs/command-reference/metrics/show.md index 60d0499301..26f6b6b030 100644 --- a/content/docs/command-reference/metrics/show.md +++ b/content/docs/command-reference/metrics/show.md @@ -43,13 +43,10 @@ compares them with a previous version. ## Options - `-t `, `--type ` - specify a type for the metric file. Accepted - values are: `raw` (default), `json`, `tsv`, `htsv`, `csv`, `hcsv`. It will be - used to determine how to parse and format metics for display. + values are: `raw` (default), `json`. It will be used to determine how to parse + and format metics for display. `raw` means that no additional parsing is applied, and `--xpath` is ignored. - `htsv`/`hcsv` are the same as `tsv`/`csv`, but the values in the first row of - the file will be used as the field names and should be used to address columns - in the `--xpath` option. This option will override `type` and `xpath` defined in the corresponding DVC-file. If no `type` is provided or found in the DVC-file, DVC will try to @@ -69,9 +66,6 @@ compares them with a previous version. only the values for model versions if they meet the given conditions from the metric file: `{"metrics": [{"dataset": "train", "deviation_mse": 0.173461, "value_mse": 0.421601}]}` - - For `tsv`/`csv` - `row,column` e.g. `1,2`. Indices are 0-based. - - For `htsv`/`hcsv` - `row,column name` e.g. `0,Name`. Row index is 0-based. - First row is used to specify column names and is not included into index. If multiple metric files exist in the project, the same parser and path will be applied to all of them. If `xpath` for a particular metric From c1d9bf74f76f31c86fe7fc6353970f49ef2829e9 Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Sat, 28 Mar 2020 06:42:30 -0700 Subject: [PATCH 2/8] Update metrics synopsis --- .../docs/command-reference/metrics/index.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index 0ef825889a..3e29f2dd3d 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -10,14 +10,22 @@ A set of commands to add, manage, collect, and display project metrics: ## Synopsis ```usage -usage: dvc metrics [-h] [-q | -v] {show,add,modify,remove} ... +usage: dvc metrics [-h] [-q | -v] {show,add,modify,remove,diff} ... positional arguments: - COMMAND - show Output metric values. - add Tag file as a metric file. - modify Modify metric file values. - remove Remove files's metric tag. + {show,add,modify,remove,diff} + Use `dvc metrics CMD --help` to display command- + specific help. + show Print metrics, with optional formatting. + add Mark a DVC-tracked file as a metric. + modify Modify metric default formatting. + remove Remove metric mark on a DVC-tracked file. + diff Show changes in metrics between commits + +optional arguments: + -h, --help show this help message and exit + -q, --quiet Be quiet. + -v, --verbose Be verbose ``` ## Description From 00d458affd651f13e71f1cdcdf6c274680ce231c Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 1 Apr 2020 01:53:42 -0600 Subject: [PATCH 3/8] Update content/docs/command-reference/metrics/index.md --- content/docs/command-reference/metrics/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index 3e29f2dd3d..900a6f92a9 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -30,7 +30,7 @@ optional arguments: ## Description -In order to track metrics associated to machie learning experiments DVC has the +In order to track metrics associated to machie learning experiments, DVC has the ability to mark a certain stage outputs as files containing metrics to track. (See the `--metrics` option of `dvc run`.) Metrics are project-specific floating-point values e.g. `AUC`, `ROC`, etc. From ab2ab3044cb4c7731c35388b0a581bf839e0e339 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 1 Apr 2020 02:07:58 -0600 Subject: [PATCH 4/8] Update content/docs/command-reference/metrics/index.md --- content/docs/command-reference/metrics/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index 900a6f92a9..a66a27f1f8 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -96,7 +96,7 @@ $ dvc metrics show --xpath AUC data/eval.json data/eval.json: {'AUC': 0.65115} ``` -The xpath filter can be saved for a metrics file: +The `xpath` filter can be saved for a metrics file: ```dvc $ dvc metrics modify data/eval.json --xpath AUC From 547cd211c13692acbd34c1c56834ee46e7173b01 Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Wed, 1 Apr 2020 02:41:29 -0600 Subject: [PATCH 5/8] Update content/docs/command-reference/metrics/modify.md --- content/docs/command-reference/metrics/modify.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/metrics/modify.md b/content/docs/command-reference/metrics/modify.md index 432c0a13cf..c129c91f39 100644 --- a/content/docs/command-reference/metrics/modify.md +++ b/content/docs/command-reference/metrics/modify.md @@ -90,7 +90,7 @@ $ dvc metrics show metrics.json ``` Okay. Let's now imagine we are interested only in a single value of true -posivives - TP. We can specify the `JSON` type (`-t`) and an `xpath` (`-x`) to +posivives (TP). We can specify the `JSON` type (`-t`) and an `xpath` (`-x`) to extract the TP value: ```dvc From ed251fbb9f4c9254a97223665dfc5b178ec76049 Mon Sep 17 00:00:00 2001 From: Dmitry Petrov Date: Fri, 3 Apr 2020 17:03:02 -0700 Subject: [PATCH 6/8] After-review changes --- content/docs/command-reference/metrics/add.md | 9 ++++----- content/docs/command-reference/metrics/diff.md | 5 +++-- content/docs/command-reference/metrics/modify.md | 16 ++++++++-------- content/docs/command-reference/metrics/show.md | 4 ++-- 4 files changed, 17 insertions(+), 17 deletions(-) diff --git a/content/docs/command-reference/metrics/add.md b/content/docs/command-reference/metrics/add.md index f93c4e5c2d..dc20400374 100644 --- a/content/docs/command-reference/metrics/add.md +++ b/content/docs/command-reference/metrics/add.md @@ -33,9 +33,8 @@ the file contains multiple metrics. See the [options](#options) below and - `-t `, `--type ` - specify a type for the metric file. Accepted values are: `raw` (default), `json`. It will be saved into the corresponding DVC-file, and used by `dvc metrics show` to determine how to handle displaying - metrics. - - `raw` means that no additional parsing is applied, and `--xpath` is ignored. + metrics. `raw` means that no additional parsing is applied, and `--xpath` is + ignored. - `-x `, `--xpath ` - specify a path within a metric file to get a specific metric value. Should be used if the metric file contains multiple @@ -58,11 +57,11 @@ the file contains multiple metrics. See the [options](#options) below and ## Examples -Let's first create a regular output with the `-o` option of +Let's first create a regular output with the `-O` option of `dvc run`: ```dvc -$ dvc run -o metrics.json \ +$ dvc run -O metrics.json \ 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' ``` diff --git a/content/docs/command-reference/metrics/diff.md b/content/docs/command-reference/metrics/diff.md index 48324f992d..ac77cd1bb6 100644 --- a/content/docs/command-reference/metrics/diff.md +++ b/content/docs/command-reference/metrics/diff.md @@ -42,8 +42,9 @@ They're calculated between two commits (hash, branch, tag, or any no directories among the `targets`, this option is ignored. - `-t `, `--type ` - specify a type of the metric file. Accepted - values are: `raw` (default), `json`. It will be used to determine how to parse - and format metics for display. See `dvc metrics show` for more details. + values are: `raw` (default), `json`. It will be saved into the corresponding + DVC-file, and used to determine how to handle displaying metrics. See + `dvc metrics show` for more details. This option will override `type` and `xpath` defined in the corresponding DVC-file. If no `type` is provided or found in the DVC-file, DVC will try to diff --git a/content/docs/command-reference/metrics/modify.md b/content/docs/command-reference/metrics/modify.md index c129c91f39..938ec21ba8 100644 --- a/content/docs/command-reference/metrics/modify.md +++ b/content/docs/command-reference/metrics/modify.md @@ -34,8 +34,8 @@ ERROR: failed to modify metric file settings - - `-t `, `--type ` - specify a type for the metric file. Accepted values are: `raw` (default), `json`. It will be saved into the corresponding - DVC-file, and used by `dvc metrics show` to determine how to handle displaying - metrics. + DVC-file, and used by `dvc metrics show` and `dvc metrics diff` to determine + how to handle displaying metrics. `raw` means that no additional parsing is applied, and `--xpath` is ignored. @@ -64,20 +64,20 @@ Let's first imagine we have a [stage](/doc/command-reference/run) with a generic metric file initially. The dummy command below simulates this imaginary setup: ```dvc -$ dvc run -o metrics.json \ +$ dvc run -M metrics.json \ 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' ``` The resulting stage file `metrics.json.dvc` should look like this: ```yaml -md5: 906ea9489e432c85d085b248c712567b -cmd: echo {\"AUC\":0.9643, \"TP\":527} > metrics.json +md5: c607baf8e350957c2a6db134cfe1c2e2 +cmd: 'echo {\"AUC\": 0.9643, \"TP\": 527} > metrics.json' outs: - - md5: 0f0e67dc927aa69cd3fc37435ee1304f + - md5: 2d975ad7af38fe0511163e60b80cb1b4 path: metrics.json - cache: true - metric: false + cache: false + metric: true persist: false ``` diff --git a/content/docs/command-reference/metrics/show.md b/content/docs/command-reference/metrics/show.md index 26f6b6b030..1c45f15a26 100644 --- a/content/docs/command-reference/metrics/show.md +++ b/content/docs/command-reference/metrics/show.md @@ -43,8 +43,8 @@ compares them with a previous version. ## Options - `-t `, `--type ` - specify a type for the metric file. Accepted - values are: `raw` (default), `json`. It will be used to determine how to parse - and format metics for display. + values are: `raw` (default), `json`. It will be saved into the corresponding + DVC-file, and used to determine how to handle displaying metrics. `raw` means that no additional parsing is applied, and `--xpath` is ignored. From f81735aa2c13e0b27695eb440f0ef2a61bed84ef Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 7 Apr 2020 03:19:32 -0500 Subject: [PATCH 7/8] Update content/docs/command-reference/metrics/index.md --- content/docs/command-reference/metrics/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index a66a27f1f8..423b71da3e 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -13,7 +13,7 @@ A set of commands to add, manage, collect, and display project metrics: usage: dvc metrics [-h] [-q | -v] {show,add,modify,remove,diff} ... positional arguments: - {show,add,modify,remove,diff} +COMMAND Use `dvc metrics CMD --help` to display command- specific help. show Print metrics, with optional formatting. From 4584c7ce4f643c9f4ea4f788977c1a930f01541b Mon Sep 17 00:00:00 2001 From: Jorge Orpinel Date: Tue, 7 Apr 2020 03:42:34 -0500 Subject: [PATCH 8/8] Update content/docs/command-reference/metrics/index.md --- content/docs/command-reference/metrics/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/docs/command-reference/metrics/index.md b/content/docs/command-reference/metrics/index.md index 423b71da3e..8b32b17d70 100644 --- a/content/docs/command-reference/metrics/index.md +++ b/content/docs/command-reference/metrics/index.md @@ -96,7 +96,7 @@ $ dvc metrics show --xpath AUC data/eval.json data/eval.json: {'AUC': 0.65115} ``` -The `xpath` filter can be saved for a metrics file: +The `xpath` filter can be saved as the default way to display a metrics file: ```dvc $ dvc metrics modify data/eval.json --xpath AUC