Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated exp diff tables #2988

Merged
merged 3 commits into from
Nov 8, 2021
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 21 additions & 21 deletions content/docs/command-reference/exp/diff.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ difference ("Change") for numeric values. Example:

```dvc
$ dvc exp diff
Path Metric Value Change
scores.json roc_auc 0.93345 0.018087
scores.json grade B-
Path Metric HEAD workspace Change
scores.json roc_auc 0.915363 0.93345 0.018087
scores.json grade B B-

Path Param Value Change
params.yaml max_features 3000 1500
Path Param HEAD workspace Change
params.yaml max_features 1500 3000 1500
```

> This is similar to combining the reports from `dvc params diff` and
Expand Down Expand Up @@ -60,9 +60,6 @@ all the current experiments (without comparisons).
- `--md` - prints the command's output in the Markdown table format
([GFM](https://github.github.com/gfm/#tables-extension-)).

- `--old` - include the "Old" value column in addition to the new "Value" (and
"Change"). Useful when the values are not numeric, for example

- `--no-path` - hide the "Path" column that lists the param/metrics file
location. Useful when only one metrics or params file exists, for example

Expand Down Expand Up @@ -101,21 +98,23 @@ to its baseline (`10-bigrams-experiment`, current `HEAD`) like this:

```dvc
$ dvc exp diff exp-1dad0
Path Metric Value Change
scores.json auc 0.61314 0.035575
Path Param Value Change
params.yaml featurize.max_features 1500 -500
Path Metric HEAD exp-1dad0 Change
scores.json auc 0.577565 0.61314 0.035575

Path Param HEAD exp-1dad0 Change
params.yaml featurize.max_features 2000 1500 -500
```

To compare two specific experiments (values are shown for the second one by
default):

```dvc
$ dvc exp diff exp-1dad0 exp-1df77
Path Metric Value Change
scores.json auc 0.51676 -0.060799
Path Param Value Change
params.yaml featurize.max_features 500 -1500
Path Metric exp-1dad0 exp-1df77 Change
scores.json auc 0.577559 0.51676 -0.060799

Path Param exp-1dad0 exp-1df77 Change
params.yaml featurize.max_features 2000 500 -1500
```

To compare an experiment to the
Expand All @@ -124,9 +123,10 @@ tag (or any other [revision](https://git-scm.com/docs/revisions)):

```dvc
$ dvc exp diff exp-1dad0 7-ml-pipeline
Path Metric Value Change
scores.json auc None diff not supported
Path Param Value Change
params.yaml featurize.max_features 500 -1500
params.yaml featurize.ngrams 1 -1
Path Metric exp-1dad0 7-ml-pipeline Change
scores.json auc 0.577559 None diff not supported

Path Param exp-1dad0 7-ml-pipeline Change
params.yaml featurize.max_features 2000 500 -1500
params.yaml featurize.ngrams 2 1 -1
```
Original file line number Diff line number Diff line change
Expand Up @@ -362,29 +362,29 @@ their metrics and params.

```dvc
$ dvc exp diff
Path Metric Value Change
metrics.json acc 0.9151 0.0024
metrics.json loss 0.23867 0.0020977
Path Metric HEAD workspace Change
metrics.json acc 0.9127 0.9151 0.0024
metrics.json loss 0.23657 0.23867 0.0020977

Path Param Value Change
params.yaml model.conv_units 256 240
Path Param HEAD workspace Change
params.yaml model.conv_units 16 256 240
```

Without experiment names, `dvc exp diff` shows the change between the last
experiment and the last commit. So if the command doesn't print an output, there
might be no experiment since the previous commit or it didn't produce changes in
results. If you want to see all the parameters and metrics regardless of whether
they have changed, you can use `--all` flag.
experiment (`workspace`) and the last commit (`HEAD`). So if the command doesn't
print an output, there might be no experiment since the previous commit or it
didn't produce changes in results. If you want to see all the parameters and
metrics regardless of whether they have changed, you can use `--all` flag.

```dvc
$ dvc exp diff --all
Path Metric Value Change
metrics.json acc 0.9151 0.0024
metrics.json loss 0.23867 0.0020977
Path Metric HEAD workspace Change
metrics.json acc 0.9127 0.9151 0.0024
metrics.json loss 0.23657 0.23867 0.0020977

Path Param Value Change
params.yaml model.conv_units 256 240
params.yaml train.epochs 10 0
Path Param HEAD workspace Change
params.yaml model.conv_units 16 256 240
params.yaml train.epochs 10 10 0
```

## Compare an experiment with the workspace
Expand All @@ -395,12 +395,12 @@ names to `dvc exp diff`.

```
$ dvc exp diff cnn-128 cnn-64
Path Metric Value Change
metrics.json acc 0.9153 0.00020003
metrics.json loss 0.23385 -0.0048174
Path Metric cnn-128 cnn-64 Change
metrics.json acc 0.9151 0.9153 0.00020003
metrics.json loss 0.23867 0.23385 -0.0048174

Path Param Value Change
params.yaml model.conv_units 64 -192
Path Param cnn-128 cnn-64 Change
params.yaml model.conv_units 128 64 -64
```

## Customize the comparison output
Expand All @@ -412,12 +412,12 @@ the `--precision` option (5 by default).

```dvc
$ dvc exp diff exp-25a26 cnn-64 --no-path --precision 2
Metric Value Change
acc 0.92 0.0002
loss 0.23 -0.0048
Metric exp-25a26 cnn-64 Change
acc 0.92 0.92 0.0002
loss 0.23 0.23 -0.0048

Param Value Change
model.conv_units 64 -192
Param exp-25a26 Value Change
model.conv_units 256 64 -192
```

### Get the comparison in JSON
Expand Down Expand Up @@ -482,15 +482,15 @@ table to embed in the reports directly.

```dvc
$ dvc exp diff exp-25a26 cnn-64 --md
| Path | Metric | Value | Change |
| ------------ | ------ | ------- | ---------- |
| metrics.json | acc | 0.9153 | 0.00020003 |
| metrics.json | loss | 0.23385 | -0.0048174 |
| Path | Metric | exp-25a26 | cnn-64 | Change |
| ------------ | ------ | --------- | ------- | ---------- |
| metrics.json | acc | 0.9151 | 0.9153 | 0.00020003 |
| metrics.json | loss | 0.23867 | 0.23385 | -0.0048174 |


| Path | Param | Value | Change |
| ----------- | ---------------- | ----- | ------ |
| params.yaml | model.conv_units | 64 | -192 |
| Path | Param | exp-25a26 | cnn-64 | Change |
| ----------- | ---------------- | --------- | ------ | ------ |
| params.yaml | model.conv_units | 256 | 64 | -192 |
```

You can use this output to automatically update the documents with a command
Expand Down