Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dvclive: Updates for implicit no step #3147

Merged
merged 11 commits into from
Jan 27, 2022
5 changes: 4 additions & 1 deletion content/docs/dvclive/api-reference/live/get_step.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,7 @@ while live.get_step() < 3:
DVCLive uses `step` to track the progress of each metric logged with
`Live.log()`.

The `step` value can be updated with `Live.next_step()` or `Live.set_step()`.
The current `step` value can be retrieved with `Live.get_step()`.

In order to update the `step` value, you can use `Live.next_step()` or
`Live.set_step()`.
23 changes: 14 additions & 9 deletions content/docs/dvclive/api-reference/live/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,25 +23,24 @@ live = Live()

## Description

Its parameters are focused on configuring the behavior of subsequent calls to
[`Live()` methods](#methods).
A `Live()` instance is required in order to log machine learning metrics and
other metadata.

⚠️ `Live()` will remove all existing DVCLive related files under `path` unless
`resume=True`.

## Attributes

- `dir` - Location of the
[_metrics logs_](/doc/dvclive/get-started#metrics-logs) directory.
- `dir` - Location of the [metrics history](/doc/dvclive/get-started#history)
directory.
- `summary_path` - Location of the
[_metrics summary_](/doc/dvclive/get-started#metrics-summary).
[summary](/doc/dvclive/get-started#metrics-summary).
- `html_path` - Location of the
[_html report_](/doc/dvclive/dvclive-with-dvc#html-report).
[html report](/doc/dvclive/dvclive-with-dvc#html-report).

## Parameters

- `path` - Configure where to save _metrics logs_ and _metrics summary_
(generated by `Live.log()`). _Default_: `None`.
- `path` - Where to save DVCLive's outputs. _Default_: `None`.

If `None` and DVC is enabled (see
[DVCLive with DVC](/docs/dvclive/dvclive-with-dvc)), the `path` set by DVC
Expand All @@ -50,10 +49,16 @@ Its parameters are focused on configuring the behavior of subsequent calls to
- `resume` - If `True`, DVCLive will try to read the previous `step` from the
`path` directory and start from that point. _Default_: `False`.

⚠️ If you are not using steps, don't set to `True` since DVCLive will preserve
previous run's files and assume that `step` has been enabled.

- `summary` - If `True`, upon each `Live.log()` call, DVCLive will generate a
_metrics summary_ (usable by `dvc metrics`). The _summary_ will be located at
summary (usable by `dvc metrics`). The summary will be located at
`{path}.json`. _Default_: `True`.
daavoo marked this conversation as resolved.
Show resolved Hide resolved

⚠️ If you are not using steps, don't set to `False` since `Live.log()` won't
be generating any output.

## Exceptions

- `dvclive.error.ConfigMismatchError` - thrown if the provided `path` does not
Expand Down
33 changes: 16 additions & 17 deletions content/docs/dvclive/api-reference/live/log.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
# Live.log()

Generates [_metrics logs_](/doc/dvclive/get-started#metrics-logs) (usable by
`dvc plots`) by saving the given `name`: `val` pair to a `.tsv` file.
Logs the given scalar `val` associating it with the given `name`.

```py
def log(name: str, val: float, step: int = None):
def log(name: str, val: float):
```

#### Usage:
Expand All @@ -19,10 +18,16 @@ live.log("loss", 0.9)

## Description

The first call to `live.log(name, val)` will create a new file in
`{path}/{name}.tsv` including the header and first row.
If `summary` is True, `live.log(name, val)` will update the
[summary](/doc/dvclive/get-started#summary) with the latest value logged.

For example `live.log("loss", 0.9)` will create `{path}/loss.tsv`:
💡 The summary `{path}.json` is usable by `dvc metrics`.

### Step updates
dberenbaum marked this conversation as resolved.
Show resolved Hide resolved

The first `step` update (with `Live.next_step()` or `Live.set_step()`) will
create a new [metric history](/doc/dvclive/get-started#history) in
`{path}/{name}.tsv`:

```
timestamp step loss
Expand All @@ -32,23 +37,17 @@ timestamp step loss
Each subsequent call to `live.log(name, val)` will add a new row to
`{path}/{name}.tsv`.

The created file `{path}/{name}.tsv` is usable by `dvc plots`.
💡 The metric history `{path}/{name}.tsv` is usable by `dvc plots`.

💡 If `name` contains slashes (e.g. `train/loss`), the required subdirectories
will be created and the file will be saved inside the last one (e.g.
If `name` contains slashes (e.g. `train/loss`), the required subdirectories will
be created and the file will be saved inside the last one (e.g.
`{path}/train/loss.tsv`).

If `summary` is True, `Live.log()` DVCLive will update the
[_metrics summary_](/doc/dvclive/get-started#metrics-summary) with the latest
value logged.

The updated summary `{path}.json` is usable by `dvc metrics`.

## Parameters

- `name` - The _metrics logs_ will be saved in `{path}/{name}.tsv`.
- `name` - Name of the metric being logged.

- `val` - The value to be added in the `name` column of a new row.
- `val` - The value to be logged.

## Exceptions

Expand Down
8 changes: 5 additions & 3 deletions content/docs/dvclive/api-reference/live/next_step.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ for step in range(3):
## Description

DVCLive uses the `step` value to track the progress of each metric logged with
`Live.log()`. You can use `Live.next_step()` to increase the `step` by 1 (one).
`Live.log()`.

You can use `Live.next_step()` to increase the `step` by 1 (one).

Each metric logged in between `Live.next_step()` (or `Live.set_step()`) calls
will be associated to the updated `step` value.
Expand All @@ -32,8 +34,8 @@ When `dvclive` is used alongside `DVC`, each `Live.next_step()` call will have
additional effects.

By default, on each `Live.next_step()` call, `DVC` will prepare an
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with all the _metrics
logs_ logged in `path`.
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the [metrics
history(/doc/dvclive/get-started#history).

In addition, when
[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in
Expand Down
10 changes: 6 additions & 4 deletions content/docs/dvclive/api-reference/live/set_step.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ for step in [0, 10, 20]:
## Description

DVCLive uses the `step` value to track the progress of each metric logged with
`Live.log()`. You can use `Live.set_step()` to set `step` to any value.
`Live.log()`.

You can use `Live.set_step()` to set `step` to any value.

Each metric logged in between `Live.set_step()` (or `Live.next_step()`) calls
will be associated to the provided `step` value.
Expand All @@ -33,8 +35,8 @@ When `dvclive` is used alongside `DVC`, each `Live.set_step()` call will have
additional effects.

By default, on each `Live.set_step()` call, `DVC` will prepare an
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with all the _metrics
logs_ logged in `path`.
[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the
[metrics history](/doc/dvclive/get-started#lhistory).

In addition, when
[checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in
Expand All @@ -45,7 +47,7 @@ the <abbr>pipeline</abbr>, `DVC` will
## Example

Given the [Usage](#usage) code snippet above, the
[metrics logs](/doc/dvclive/get-started#metrics-logs) generated for `metric_1`
[metric history](/doc/dvclive/get-started#history) generated for `metric_1`
would be:

```dvc
Expand Down
15 changes: 10 additions & 5 deletions content/docs/dvclive/dvclive-with-dvc.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Even though DVCLive does not require DVC, they can integrate in several useful
ways:

- The [_outputs_](#outputs) DVCLive produces can be fed as
`dvc plots`/`dvc metrics`, making it easier to add metrics logging to DVC
`dvc metrics`/`dvc plots`, making it easier to add metrics logging to DVC
<abbr>stages</abbr>. Those same outputs can be visualized in
[_DVC Studio_](#dvc-studio)
- You can monitor model performance in realtime with the
Expand Down Expand Up @@ -85,11 +85,12 @@ dvc.lock training_metrics train.py
dvc.yaml training_metrics.json
```

The `.tsv` files generated under `training_metrics` can be visualized with
`dvc plots`.
The [metrics history](/doc/dvclive/get-started#history) generated under
`training_metrics` can be visualized with `dvc plots`.

In addition, `training_metrics.json` can be used by `dvc metrics` and visualized
with `dvc exp show`/`dvc exp diff`.
In addition, the [metrics summary](/doc/dvclive/get-started#summary)
`training_metrics.json` can be used by `dvc metrics` and visualized with
`dvc exp show`/`dvc exp diff`.

### DVC Studio

Expand All @@ -111,6 +112,8 @@ plot for metrics automatically updated during the model training!

![](/img/dvclive-html.gif)

> If you don't update the step number, the HTML report won't be generated.

### Checkpoints

When used alongside DVC, DVCLive can create _checkpoint_ signal files used by
Expand All @@ -121,3 +124,5 @@ This will save the metrics, plots, models, etc. associated to each

You can learn more about how to use them in the
[Checkpoints User Guide](/docs/user-guide/experiment-management/checkpoints).

> If you don't update the step number, checkpoints won't be created.
54 changes: 30 additions & 24 deletions content/docs/dvclive/get-started.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
# Get Started

DVCLive is a simple Python library whose interface consists of three main steps.
DVCLive is a Python library for logging machine learning metrics and other
metadata in simple file formats, which is fully compatible with DVC.

## Steps
> If you use one of the supported [ML Frameworks](/doc/dvclive/ml-frameworks),
> you can jump directly to it's corresponding page.

To get it up and running you just need to follow these steps:
## Workflow

### 1. Initialize DVCLive
### Initialize DVCLive

```python
from dvclive import Live
Expand All @@ -16,25 +18,25 @@ live = Live()

See [`Live()`](/doc/dvclive/api-reference/live) for details.

### 2. Log metrics
### Log data

```python
live.log(metric_name, value)
live.log("acc", 0.9)
```

See `Live.log()` for details.

### 3. Increase the step number
### (Optionally) Update the step number

```python
live.next_step()
```

See `Live.next_step()` for details.
See `Live.next_step()` and `Live.set_step()` for details.

## Putting all together

Using the above steps, you can easily include DVCLive in your training code:
Joining the above snippets, you can include DVCLive in your training code:

```python
# train.py
Expand Down Expand Up @@ -64,9 +66,25 @@ dvclive train.py
dvclive.json
```

### Metrics Logs
### Summary

For each `{metric_name}`, DVCLive produces metrics _logs_ under
When [`summary`](/doc/dvclive/api-reference/live/#parameters) is enabled (True
by default), DVCLive generates a summary with the latest metrics:

```dvc
$ cat dvclive.json
{
"step": 2,
"{metric_name}": 0.8907166719436646,
}
```

> If you don't update the step number, the `step` entry won't be present in the
> summary.

### History

In addition, for each `{metric_name}`, DVCLive stores the metric's history under
`dvclive/{metric_name}.tsv`:

```dvc
Expand All @@ -77,19 +95,7 @@ timestamp step {metric_name}
1614129198848 2 0.8907166719436646
```

### Metrics Summary

In addition, when [`summary`](/doc/dvclive/api-reference/live/#parameters) is
enabled (True by default), DVCLive generates a metrics _summary_ with the latest
metrics:

```dvc
$ cat dvclive.json
{
"step": 2,
"{metric_name}": 0.8907166719436646
}
```
> If you don't update the step number, the metrics history won't be generated.

## What next?

Expand Down
6 changes: 3 additions & 3 deletions content/docs/dvclive/index.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# DVCLive Documentation

[DVCLive](https://github.com/iterative/dvclive) is an open-source Python library
for monitoring the progress of metrics during training of machine learning
models. You can simply
[DVCLive](https://github.com/iterative/dvclive) is a Python library for logging
machine learning metrics and other metadata in simple file formats, which is
fully compatible with DVC. You can
[install it](https://github.com/iterative/dvclive#installation) with `pip`.

<cards>
Expand Down