Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvclive 1.0 #4085

Merged
merged 71 commits into from
Nov 5, 2022
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
86e5e93
initial refactor
daavoo Aug 24, 2022
dcf97b2
Mention API Reference
daavoo Sep 5, 2022
a84ff5b
Fix link
daavoo Sep 5, 2022
11de5a8
Add redirect
daavoo Oct 5, 2022
e765b93
Update content/docs/dvclive/how-to/checkpoints.md
daavoo Oct 5, 2022
2082fed
Remove how-to folder
daavoo Oct 6, 2022
ed31bb3
Add outputs content
daavoo Oct 7, 2022
8e2b6a9
Reorder sidebar
daavoo Oct 7, 2022
8e3882b
Add tabs with popular frameworks
daavoo Oct 7, 2022
2b61de3
Remove outdated `dvclive with DVC`
daavoo Oct 7, 2022
5d217da
clean
daavoo Oct 7, 2022
c06f3f1
Add params
daavoo Oct 7, 2022
790111c
Clarify value without DVC
daavoo Oct 7, 2022
d8f4817
Update outputs to current status
daavoo Oct 7, 2022
542457b
Update content/docs/dvclive/get-started.md
daavoo Oct 11, 2022
a885104
Reduce wording
daavoo Oct 11, 2022
71f0eaf
Add Python API tab
daavoo Oct 11, 2022
dc3328a
More details
daavoo Oct 11, 2022
9ddd4cb
Updates from review
daavoo Oct 11, 2022
d8a73e7
Capitalize pages
daavoo Oct 11, 2022
878ecc6
Mention studio for sharing experiments
daavoo Oct 11, 2022
d34b0a7
Merge branch 'main' into dvclive-refactor
daavoo Oct 11, 2022
f4d21e0
New section for Studio
daavoo Oct 11, 2022
8b382ef
Add sample output
daavoo Oct 11, 2022
54cc0fe
Move ml-frameworks to api-refeference
daavoo Oct 17, 2022
88bfcb6
Drop resume page
daavoo Oct 17, 2022
035747e
Apply suggestions from code review
daavoo Oct 18, 2022
9ca4088
Restyled by prettier (#4050)
restyled-io[bot] Oct 18, 2022
942349a
Suggestions from code review
daavoo Oct 18, 2022
3be1c39
caps
daavoo Oct 18, 2022
6c3e0c7
Add sections
daavoo Oct 18, 2022
df82e7a
Add example output
daavoo Oct 18, 2022
f0bcfe0
Add admon
daavoo Oct 18, 2022
98416c6
Share experiment -> Share Results
daavoo Oct 18, 2022
5cc4b57
Update output structure.
daavoo Oct 26, 2022
ae8d127
Revisit output names
daavoo Oct 27, 2022
fa1ca04
path -> dir
daavoo Oct 27, 2022
8c73db4
Add make_report call
daavoo Oct 27, 2022
db49ca6
Rename log -> log_metric
daavoo Oct 27, 2022
f1d9c10
Update log_sklearn_plot. (#4074)
daavoo Oct 27, 2022
9a08d4f
Merge branch 'main' into dvclive-1.0
daavoo Oct 28, 2022
1c51b05
Apply suggestions from code review
daavoo Oct 28, 2022
8a33ffd
Replace dvc with cli
daavoo Oct 28, 2022
1e6642c
Remove tree
daavoo Oct 28, 2022
5cd7701
Update output dir in frameworks
daavoo Oct 28, 2022
e2c1e83
grammarly
daavoo Oct 28, 2022
8a79e7b
Revisit frameworks naming.
daavoo Oct 31, 2022
dd05deb
Remplaze `path` with `dir`
daavoo Oct 31, 2022
200b744
Rename `outputs` to `how-it-works`
daavoo Oct 31, 2022
98406ff
Rename `Output Folder Structure` to `How it Works`
daavoo Oct 31, 2022
3e0e9d7
Rename Scalars to Metrics
daavoo Oct 31, 2022
c9f3693
Simplify `Live` Attributes description
daavoo Oct 31, 2022
eee1a66
Renamed folder to directory
daavoo Oct 31, 2022
925e89f
Revisit `step` definition
daavoo Oct 31, 2022
9092878
Fix tree
daavoo Oct 31, 2022
8186c07
Remove cat
daavoo Oct 31, 2022
5b50a9e
Replace vscode plots with data series
daavoo Oct 31, 2022
fa57f41
Updates with new summary behavior and step property
daavoo Nov 4, 2022
07b1f5b
Fix links
daavoo Nov 4, 2022
84e6df6
Consistent framework tabs
daavoo Nov 4, 2022
1680073
Update content/docs/dvclive/api-reference/live/log_metric.md
Nov 4, 2022
1e2fd17
start: dvclive 1.0 updates to exp viz page
Nov 4, 2022
f4d7812
Update content/docs/start/experiment-management/visualization.md
Nov 4, 2022
a3e5c81
Revisit dvclive usage outside doc/dvclive
daavoo Nov 4, 2022
c77bfa4
Focus on context manager usage.
daavoo Nov 4, 2022
91202c0
hint about next_step
daavoo Nov 4, 2022
df71440
Merge branch 'main' of github.com:iterative/dvc.org into dvclive-1.0
rogermparent Nov 4, 2022
be394a2
Update content/docs/dvclive/api-reference/live/next_step.md
daavoo Nov 4, 2022
6b9d2c9
Link for `Live.metrics_file`
daavoo Nov 4, 2022
3fd57c5
Restyled by prettier (#4097)
restyled-io[bot] Nov 4, 2022
11d6166
Updates per https://github.com/iterative/example-repos-dev/pull/143
daavoo Nov 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/docs/command-reference/exp/init.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,8 @@ See the [Pipelines guide] for more on that topic.
default value (`plots/`).

- `--live` - set the path to the directory where the metrics and plots
[produced by DVCLive](https://dvc.org/doc/dvclive/dvclive-with-dvc#outputs)
will be found. Overrides the default values for `--metrics` and `--plots`.
[produced by DVCLive](https://dvc.org/doc/dvclive/how-it-works) will be found.
Overrides the default values for `--metrics` and `--plots`.

- `--explicit` - do not assume default locations of project dependencies and
outputs. You'll have to provide specific locations via other options or
Expand Down
2 changes: 1 addition & 1 deletion content/docs/command-reference/plots/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Plots have to be defined either at the stage level, or at the pipeline level in
`dvc.yaml`.

[ml pipeline]: /doc/start/data-management/data-pipelines
[dvclive]: /doc/dvclive/dvclive-with-dvc
[dvclive]: /doc/dvclive

<admon icon="book">

Expand Down
31 changes: 18 additions & 13 deletions content/docs/dvclive/api-reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ to its corresponding page.
### Initialize DVCLive

```python
from dvclive import Live

live = Live()
with Live() as live:
```

See [`Live()`](/doc/dvclive/api-reference/live) for details.
Expand All @@ -48,13 +46,13 @@ live.log_params(params)
See `Live.log_param()` / `Live.log_params()`.

</tab>
<tab title="Scalars">
<tab title="Metrics">

```python
live.log("acc", 0.9)
live.log_metric("acc", 0.9)
```

See `Live.log()`.
See `Live.log_metric()`.

</tab>
<tab title="Images">
Expand All @@ -67,15 +65,15 @@ live.log_image("image.png", img)
See `Live.log_image()`.

</tab>
<tab title="Plots">
<tab title="SKLearn Plots">

```python
y_true = [0, 0, 1, 1]
y_pred = [0.2, 0.5, 0.3, 0.8]
live.log_plot("roc", y_true, y_score)
live.log_sklearn_plot("roc", y_true, y_score)
```

See `Live.log_plot()`.
See `Live.log_sklearn_plot()`.

</tab>
</toggle>
Expand All @@ -86,7 +84,14 @@ See `Live.log_plot()`.
live.next_step()
```

See `Live.next_step()` and `Live.set_step()` for details.
See `Live.next_step()`.

Under the hood, `Live.next_step()` calls `Live.make_summary()` and
`Live.make_report()`.

If you want to decouple the `step` update from the rest of the calls, you can
manually modify the `Live.step` property and call `Live.make_summary()` /
`Live.make_report()`.
Comment on lines +92 to +94
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking suggested rewording:

Suggested change
If you want to decouple the `step` update from the rest of the calls, you can
manually modify the `Live.step` property and call `Live.make_summary()` /
`Live.make_report()`.
If you want to update the `step` to a different number or don't want to call
`Live.make_summary()` / `Live.make_report()`, you can manually modify the
`Live.step` property (for example, `live.step = 1`).

Also, why no mention of make_checkpoint()?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why no mention of make_checkpoint()?

I'm also curious as to why this is not considered part of the https://dvc.org/doc/dvclive/api-reference#basic-workflow .


## Putting it all together

Expand All @@ -106,13 +111,13 @@ for epoch in range(NUM_EPOCHS):
metrics = evaluate_model(...)

for metric_name, value in metrics.items():
live.log(metric_name, value)
live.log_metric(metric_name, value)

live.next_step()
```

## Outputs

After you run your training code, all the logged data will be stored in the
`dvclive` folder. Check the [DVCLive outputs](/doc/dvclive/outputs) page for
more details.
`dvclive` directory. Check the [DVCLive outputs](/doc/dvclive/how-it-works) page
for more details.
29 changes: 0 additions & 29 deletions content/docs/dvclive/api-reference/live/get_step.md

This file was deleted.

58 changes: 31 additions & 27 deletions content/docs/dvclive/api-reference/live/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ class Live:

def __init__(
self,
path: Optional[str] = None,
dir: str = "dvclive",
resume: bool = False,
report: Optional[str] = "auto",
):
Expand All @@ -18,7 +18,8 @@ class Live:
```py
from dvclive import Live

live = Live()
with Live() as live:
...
```

## Description
Expand All @@ -28,52 +29,55 @@ metrics and other metadata.

<admon type="warn">

`Live()` will remove all existing DVCLive related files under `path` unless
`Live()` will remove all existing DVCLive related files under `dir` unless
`resume=True`.

</admon>

## Attributes
You can use `Live()` as a context manager. When exiting the context manager,
`Live.make_summary()` and `Live.make_report()` will be called.
daavoo marked this conversation as resolved.
Show resolved Hide resolved

## Properties

- `step` - See `Live.next_step()`.

- `summary` - See `Live.make_summary()`.

- `dir` - Location of the directory to store
[outputs](/doc/dvclive/get-started#outputs).
[outputs](/doc/dvclive/how-it-works).

- `summary_path` - `{Live.dir}.json`. Location of the
[summary](/doc/dvclive/api-reference/live/log#description).
- `metrics_file` - `{Live.dir}/metrics.json`.

- `report_path` - `{Live.dir}/report.{format}`. Location of the
[metrics report](/doc/dvclive/api-reference/live/make_report). The `format`
can be HTML or Markdown depending on the value of the `report` parameter.
- `params_file` - `{Live.dir}/params.yaml`.

## Parameters
- `plots_dir` - `{Live.dir}/plots`.

- `path` - Where to save DVCLive's outputs. _Default_: `None`. If `None`,
`"dvclive"` will be used.
- `report_file` - `{Live.dir}/report.{format}`. The `format` can be HTML
(`.html`) or Markdown (`.md`) depending on the value of the `report`
parameter.

- `resume` - If `True`, DVCLive will try to read the previous `step` from the
`path` directory and start from that point. _Default_: `False`.
## Parameters

<admon type="info">
- `dir` - Where to save DVCLive's outputs. _Default_: `dvclive`.

If you are not using steps, don't set `resume=True` since DVCLive will
preserve previous run's files and assume that `step` has been enabled.
- `resume` - If `True`, DVCLive will try to read the previous `step` from the
`metrics_file` and start from that point. _Default_: `False`.

</admon>
- `report` - Any of `auto`, `html`, `md` or `None`. See `Live.make_report()`.

- `report` - If `auto`, `html`, or `md`, DVCLive will call `Live.make_report()`
on each step update. The `auto` mode (default) will use `md` format if a `CI`
env var is present, otherwise it will use `html`.
The `auto` mode (default) will use `md` format if the `CI` env var is present
and [matplotlib](https://matplotlib.org/) is installed, otherwise it will use
`html`.

If `report` is `None`, `Live.make_report()` won't be called automatically.
If `report` is `None`, `Live.make_report()` won't generate anything.

## Methods

- `Live.get_step()`
- `Live.log()`
- `Live.log_image()`
- `Live.log_metric()`
- `Live.log_param()`
- `Live.log_params()`
- `Live.log_plot()`
- `Live.log_sklearn_plot()`
- `Live.make_report()`
- `Live.make_summary()`
- `Live.next_step()`
- `Live.set_step()`
84 changes: 0 additions & 84 deletions content/docs/dvclive/api-reference/live/log.md

This file was deleted.

40 changes: 11 additions & 29 deletions content/docs/dvclive/api-reference/live/log_image.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,48 +26,30 @@ live.log_image("pil.png", img_pil)

Supported values for `val` are:

- A valid numpy array (convertible to image via
- A valid NumPy array (convertible to image via
[PIL.Image.fromarray](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.fromarray))
- A `PIL.Image` instance.

The images will be saved in `{Live.dir}/images/{name}`:
The images will be saved in `{Live.plots_dir}/images/{name}`:

```dvc
$ tree
└── dvclive
```
dvclive
└── plots
└── images
├── numpy.png
└── pil.png
```

<admon type="tip">

The logged images can be visualized with `dvc plots`.

</admon>

### Step updates

The first `step` update (with `Live.next_step()` or `Live.set_step()`) will move
the saved file from `{Live.dir}/images/{name}` to
`{Live.dir}/images/{step}/{name}`.
The logged images can be visualized with `dvc plots`:

Each subsequent call to `live.log_image(name, val)` will save the image under
the folder `{Live.dir}/images/{step}/{name}` corresponding to the current
`step`.

```dvc
$ tree
└── dvclive
└── images
├── 0
│ ├── numpy.png
│ └── pil.png
└── 1
├── numpy.png
└── pil.png
```cli
$ dvc plots diff dvclive/plots
```

</admon>

## Parameters

- `name` - name of the image file that this command will output
Expand All @@ -80,4 +62,4 @@ $ tree
have a supported type.

- `dvclive.error.DataAlreadyLoggedError` - thrown if the provided `name` has
already been logged whithin the same `step`.
already been logged within the same `step`.
Loading