Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GS:Visualization and Plots #3050

Merged
merged 69 commits into from
Apr 27, 2022
Merged
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
2b04b63
Added initial paragraphs
iesahin Nov 30, 2021
e865161
added three tentative tasks
iesahin Nov 30, 2021
f6bc44e
added visualization to the sidebar
iesahin Dec 14, 2021
b9013dc
moved visualization below the experiments in the sidebar
iesahin Dec 14, 2021
07086b9
added first two ways to visualize
iesahin Dec 14, 2021
7175d17
a paragraph for dvclive & keras
iesahin Dec 15, 2021
ff00094
changes required in the code for dvclive
iesahin Dec 15, 2021
6771178
simplified the callback
iesahin Dec 15, 2021
77dbdf5
added dvclive info and reformatted
iesahin Dec 15, 2021
42bf550
added install reference to README
iesahin Dec 17, 2021
10e0344
added dvclive image
iesahin Dec 28, 2021
b6c8b3e
linked dvclive image
iesahin Dec 28, 2021
1952a9b
fixed dvc plots command
iesahin Dec 28, 2021
653ffaa
Added confusion image link
iesahin Dec 28, 2021
219a59c
fixed dvc plots command
iesahin Dec 28, 2021
f4df213
added confusion image
iesahin Dec 28, 2021
93fdb93
adding image for confusion matrxi
iesahin Dec 28, 2021
925f472
add confusion image
iesahin Dec 28, 2021
7053759
updated misclassification image
iesahin Jan 5, 2022
b343d34
revised
iesahin Feb 9, 2022
dc6a2ed
added links
iesahin Feb 9, 2022
51b9820
updated the initial sentences
iesahin Feb 14, 2022
7dd9675
fixed vega-lite link
iesahin Feb 14, 2022
07dcd51
fixed dvclive link
iesahin Feb 14, 2022
4f3b0f1
fixed readme link
iesahin Feb 14, 2022
c90918d
comma
iesahin Feb 14, 2022
9a7ab68
removed "now"
iesahin Feb 14, 2022
5ff8661
removed older metrics, plots, vis from the sidebar
iesahin Feb 14, 2022
6b3f660
rephrase
iesahin Feb 14, 2022
8c9f957
Moved the initial text to the end as summary
iesahin Feb 15, 2022
29ceb79
added space for highlighters
iesahin Feb 15, 2022
99b4487
enlarged the confusion image
iesahin Feb 17, 2022
63af2d9
revised after review
iesahin Mar 14, 2022
ca6ae91
revision after review
iesahin Mar 14, 2022
fc49d48
revise after review
iesahin Mar 14, 2022
2eb2e6a
moved the description of image
iesahin Mar 21, 2022
4e6cf30
updated misclassification code link and added description
iesahin Mar 21, 2022
a11b4c5
added section titles
iesahin Mar 21, 2022
3af2c42
Update content/docs/start/visualization.md
jorgeorpinel Apr 11, 2022
b4ee07c
Update content/docs/start/visualization.md
jorgeorpinel Apr 11, 2022
ad9b474
Update content/docs/start/visualization.md
jorgeorpinel Apr 11, 2022
e894d0b
Update content/docs/start/visualization.md
jorgeorpinel Apr 12, 2022
18c45f0
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
4ca9196
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
987105f
added confusion matrix link
iesahin Apr 13, 2022
314791d
plots file
iesahin Apr 13, 2022
f34c59f
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
3b28572
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
a153b70
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
603674a
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
63f9dc0
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
4427ff3
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
1f70192
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
eb1853a
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
79f2bf3
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
4cacc6b
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
80af071
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
debec87
Update content/docs/start/visualization.md
iesahin Apr 13, 2022
fcd0cd5
restyle and minor fix
iesahin Apr 13, 2022
18274b3
fit ... to ... -> use .. in ..
iesahin Apr 13, 2022
2ae5379
deleted the older file
iesahin Apr 13, 2022
b4681fe
Apply suggestions from code review
jorgeorpinel Apr 14, 2022
6f5264d
renamed confusion.png to misclassified.png in the example
iesahin Apr 20, 2022
f26f366
rename one more
iesahin Apr 20, 2022
c1f4961
Update content/docs/start/visualization.md
iesahin Apr 20, 2022
b0ed669
Update content/docs/start/visualization.md
iesahin Apr 27, 2022
7e411e2
Update content/docs/start/visualization.md
iesahin Apr 27, 2022
cf02e81
Revert "removed older metrics, plots, vis from the sidebar"
iesahin Apr 27, 2022
b2a832e
readd metric-params-plots
iesahin Apr 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added first two ways to visualize
iesahin committed Apr 13, 2022

Verified

This commit was signed with the committer’s verified signature.
pawanjay176 Pawan Dhananjay
commit 07086b9ed2925d7d018bc9f349ab8448bd82da53
64 changes: 60 additions & 4 deletions content/docs/start/visualization.md
Original file line number Diff line number Diff line change
@@ -4,17 +4,73 @@ In parallel with versioning parameters and metrics by associating them with
experiments, DVC allows to generate plots from multivariate outputs. There are
several options to generate and keep track of the plots:

iesahin marked this conversation as resolved.
Show resolved Hide resolved
- DVC, with its DVCLive integration can produce plots automatically during
training.

- DVC can generate HTML files that includes interactive Vega-Lite plots, from
the data series in JSON, YAML, CSV, or TSV.

- DVC can keep track of image files produced as plot outputs from the
training/evaluation scripts.

- DVC, with its DVCLive integration can produce plots automatically during
training.
We'll add visualization to [`example-dvc-experiments`][ede] project. You may
refer to the [earlier section] to install it.

One of the important plots that shows the classification performance is the
iesahin marked this conversation as resolved.
Show resolved Hide resolved
confusion matrix. DVC uses Vega-Lite to plot the text files. In order to produce
a confusion matrix, Vega expects a CSV file in the form:
iesahin marked this conversation as resolved.
Show resolved Hide resolved

```csv
actual,predicted
0,0
0,2
...
```

We add the following loop at the end of the training function:

```python
y_prob = m.predict(x_valid)
y_pred = y_prob.argmax(axis=-1)
os.makedirs("plots")
with open("plots/confusion.csv", "w") as f:
f.write("actual,predicted\n")
sx = y_valid.shape[0]
for i in range(sx):
f.write(f"{y_valid.argmax()},{y_pred[i]}\n")
```

After running the experiment, with `dvc exp run`, DVC will produce
`plots/confusion.csv` file that can be shown in the web browser:

```dvc
$ dvc plots show plots/confusion.csv --template confusion
file:///Users/.../example-dvc-experiments/plots/confusion.json.html
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
```

The file contains the following plot that shows the performance of the CNN
classifier:

iesahin marked this conversation as resolved.
Show resolved Hide resolved
<<IMAGE_GOES_HERE>>

Now, let's produce another plot to see misclassified examples from each class.
This procedure is a bit more complex than creating a CSV file, and skipped here,
but you can see the code in [Github][misclassified-example-code]

We can see the misclassification examples in the browser:

```dvc
$ dvc plots show plots/misclassified.png --open
```

DVC automatically shows the image in an HTML page.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

<<IMAGE_GOES_HERE>>

A major requirement for deep learning projects is to see in which epoch training loss and validation loss differs. DVC helps in that regard with its integrations to major deep learning libraries via DVCLive.

We use the following code snippet to see the training loss

In this document, we'll focus on the first option and briefly describe the other
two. Please see `dvc plots` and the DVCLive documentation for details.

- [ ] Configure the plots to use dvclive output
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
- [ ] Generate a plot file and open it in the browser