diff --git a/content/docs/dvclive/api-reference/live/get_step.md b/content/docs/dvclive/api-reference/live/get_step.md index ab3c5b2f62..86c3124b5b 100644 --- a/content/docs/dvclive/api-reference/live/get_step.md +++ b/content/docs/dvclive/api-reference/live/get_step.md @@ -23,4 +23,7 @@ while live.get_step() < 3: DVCLive uses `step` to track the progress of each metric logged with `Live.log()`. -The `step` value can be updated with `Live.next_step()` or `Live.set_step()`. +The current `step` value can be retrieved with `Live.get_step()`. + +In order to update the `step` value, you can use `Live.next_step()` or +`Live.set_step()`. diff --git a/content/docs/dvclive/api-reference/live/index.md b/content/docs/dvclive/api-reference/live/index.md index d7af4d8c08..bb3885533b 100644 --- a/content/docs/dvclive/api-reference/live/index.md +++ b/content/docs/dvclive/api-reference/live/index.md @@ -23,25 +23,24 @@ live = Live() ## Description -Its parameters are focused on configuring the behavior of subsequent calls to -[`Live()` methods](#methods). +A `Live()` instance is required in order to log machine learning metrics and +other metadata. ⚠️ `Live()` will remove all existing DVCLive related files under `path` unless `resume=True`. ## Attributes -- `dir` - Location of the - [_metrics logs_](/doc/dvclive/get-started#metrics-logs) directory. +- `dir` - Location of the [metrics history](/doc/dvclive/get-started#history) + directory. - `summary_path` - Location of the - [_metrics summary_](/doc/dvclive/get-started#metrics-summary). + [summary](/doc/dvclive/get-started#metrics-summary). - `html_path` - Location of the - [_html report_](/doc/dvclive/dvclive-with-dvc#html-report). + [html report](/doc/dvclive/dvclive-with-dvc#html-report). ## Parameters -- `path` - Configure where to save _metrics logs_ and _metrics summary_ - (generated by `Live.log()`). _Default_: `None`. +- `path` - Where to save DVCLive's outputs. _Default_: `None`. If `None` and DVC is enabled (see [DVCLive with DVC](/docs/dvclive/dvclive-with-dvc)), the `path` set by DVC @@ -50,10 +49,16 @@ Its parameters are focused on configuring the behavior of subsequent calls to - `resume` - If `True`, DVCLive will try to read the previous `step` from the `path` directory and start from that point. _Default_: `False`. + ⚠️ If you are not using steps, don't set to `True` since DVCLive will preserve + previous run's files and assume that `step` has been enabled. + - `summary` - If `True`, upon each `Live.log()` call, DVCLive will generate a - _metrics summary_ (usable by `dvc metrics`). The _summary_ will be located at + summary (usable by `dvc metrics`). The summary will be located at `{path}.json`. _Default_: `True`. + ⚠️ If you are not using steps, don't set to `False` since `Live.log()` won't + be generating any output. + ## Exceptions - `dvclive.error.ConfigMismatchError` - thrown if the provided `path` does not diff --git a/content/docs/dvclive/api-reference/live/log.md b/content/docs/dvclive/api-reference/live/log.md index f1b0dd53ac..91c9d1e0d7 100644 --- a/content/docs/dvclive/api-reference/live/log.md +++ b/content/docs/dvclive/api-reference/live/log.md @@ -1,10 +1,9 @@ # Live.log() -Generates [_metrics logs_](/doc/dvclive/get-started#metrics-logs) (usable by -`dvc plots`) by saving the given `name`: `val` pair to a `.tsv` file. +Logs the given scalar `val` associating it with the given `name`. ```py - def log(name: str, val: float, step: int = None): + def log(name: str, val: float): ``` #### Usage: @@ -19,10 +18,16 @@ live.log("loss", 0.9) ## Description -The first call to `live.log(name, val)` will create a new file in -`{path}/{name}.tsv` including the header and first row. +If `summary` is True, `live.log(name, val)` will update the +[summary](/doc/dvclive/get-started#summary) with the latest value logged. -For example `live.log("loss", 0.9)` will create `{path}/loss.tsv`: +💡 The summary `{path}.json` is usable by `dvc metrics`. + +### Step updates + +The first `step` update (with `Live.next_step()` or `Live.set_step()`) will +create a new [metric history](/doc/dvclive/get-started#history) in +`{path}/{name}.tsv`: ``` timestamp step loss @@ -32,23 +37,17 @@ timestamp step loss Each subsequent call to `live.log(name, val)` will add a new row to `{path}/{name}.tsv`. -The created file `{path}/{name}.tsv` is usable by `dvc plots`. +💡 The metric history `{path}/{name}.tsv` is usable by `dvc plots`. -💡 If `name` contains slashes (e.g. `train/loss`), the required subdirectories -will be created and the file will be saved inside the last one (e.g. +If `name` contains slashes (e.g. `train/loss`), the required subdirectories will +be created and the file will be saved inside the last one (e.g. `{path}/train/loss.tsv`). -If `summary` is True, `Live.log()` DVCLive will update the -[_metrics summary_](/doc/dvclive/get-started#metrics-summary) with the latest -value logged. - -The updated summary `{path}.json` is usable by `dvc metrics`. - ## Parameters -- `name` - The _metrics logs_ will be saved in `{path}/{name}.tsv`. +- `name` - Name of the metric being logged. -- `val` - The value to be added in the `name` column of a new row. +- `val` - The value to be logged. ## Exceptions diff --git a/content/docs/dvclive/api-reference/live/next_step.md b/content/docs/dvclive/api-reference/live/next_step.md index 35f4ff9337..402e2e9191 100644 --- a/content/docs/dvclive/api-reference/live/next_step.md +++ b/content/docs/dvclive/api-reference/live/next_step.md @@ -21,7 +21,9 @@ for step in range(3): ## Description DVCLive uses the `step` value to track the progress of each metric logged with -`Live.log()`. You can use `Live.next_step()` to increase the `step` by 1 (one). +`Live.log()`. + +You can use `Live.next_step()` to increase the `step` by 1 (one). Each metric logged in between `Live.next_step()` (or `Live.set_step()`) calls will be associated to the updated `step` value. @@ -32,8 +34,8 @@ When `dvclive` is used alongside `DVC`, each `Live.next_step()` call will have additional effects. By default, on each `Live.next_step()` call, `DVC` will prepare an -[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with all the _metrics -logs_ logged in `path`. +[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the [metrics +history(/doc/dvclive/get-started#history). In addition, when [checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in diff --git a/content/docs/dvclive/api-reference/live/set_step.md b/content/docs/dvclive/api-reference/live/set_step.md index 5fbf439703..b50003fff4 100644 --- a/content/docs/dvclive/api-reference/live/set_step.md +++ b/content/docs/dvclive/api-reference/live/set_step.md @@ -22,7 +22,9 @@ for step in [0, 10, 20]: ## Description DVCLive uses the `step` value to track the progress of each metric logged with -`Live.log()`. You can use `Live.set_step()` to set `step` to any value. +`Live.log()`. + +You can use `Live.set_step()` to set `step` to any value. Each metric logged in between `Live.set_step()` (or `Live.next_step()`) calls will be associated to the provided `step` value. @@ -33,8 +35,8 @@ When `dvclive` is used alongside `DVC`, each `Live.set_step()` call will have additional effects. By default, on each `Live.set_step()` call, `DVC` will prepare an -[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with all the _metrics -logs_ logged in `path`. +[HTML report](/doc/dvclive/dvclive-with-dvc#html-report) with the +[metrics history](/doc/dvclive/get-started#lhistory). In addition, when [checkpoints](/doc/user-guide/experiment-management/checkpoints) are enabled in @@ -45,7 +47,7 @@ the pipeline, `DVC` will ## Example Given the [Usage](#usage) code snippet above, the -[metrics logs](/doc/dvclive/get-started#metrics-logs) generated for `metric_1` +[metric history](/doc/dvclive/get-started#history) generated for `metric_1` would be: ```dvc diff --git a/content/docs/dvclive/dvclive-with-dvc.md b/content/docs/dvclive/dvclive-with-dvc.md index 0ea99a1cf0..8aaaa38f35 100644 --- a/content/docs/dvclive/dvclive-with-dvc.md +++ b/content/docs/dvclive/dvclive-with-dvc.md @@ -4,7 +4,7 @@ Even though DVCLive does not require DVC, they can integrate in several useful ways: - The [_outputs_](#outputs) DVCLive produces can be fed as - `dvc plots`/`dvc metrics`, making it easier to add metrics logging to DVC + `dvc metrics`/`dvc plots`, making it easier to add metrics logging to DVC stages. Those same outputs can be visualized in [_DVC Studio_](#dvc-studio) - You can monitor model performance in realtime with the @@ -85,11 +85,12 @@ dvc.lock training_metrics train.py dvc.yaml training_metrics.json ``` -The `.tsv` files generated under `training_metrics` can be visualized with -`dvc plots`. +The [metrics history](/doc/dvclive/get-started#history) generated under +`training_metrics` can be visualized with `dvc plots`. -In addition, `training_metrics.json` can be used by `dvc metrics` and visualized -with `dvc exp show`/`dvc exp diff`. +In addition, the [metrics summary](/doc/dvclive/get-started#summary) +`training_metrics.json` can be used by `dvc metrics` and visualized with +`dvc exp show`/`dvc exp diff`. ### DVC Studio @@ -111,6 +112,8 @@ plot for metrics automatically updated during the model training! ![](/img/dvclive-html.gif) +> If you don't update the step number, the HTML report won't be generated. + ### Checkpoints When used alongside DVC, DVCLive can create _checkpoint_ signal files used by @@ -121,3 +124,5 @@ This will save the metrics, plots, models, etc. associated to each You can learn more about how to use them in the [Checkpoints User Guide](/docs/user-guide/experiment-management/checkpoints). + +> If you don't update the step number, checkpoints won't be created. diff --git a/content/docs/dvclive/get-started.md b/content/docs/dvclive/get-started.md index 7452286dcb..ff27ddc195 100644 --- a/content/docs/dvclive/get-started.md +++ b/content/docs/dvclive/get-started.md @@ -1,12 +1,14 @@ # Get Started -DVCLive is a simple Python library whose interface consists of three main steps. +DVCLive is a Python library for logging machine learning metrics and other +metadata in simple file formats, which is fully compatible with DVC. -## Steps +> If you use one of the supported [ML Frameworks](/doc/dvclive/ml-frameworks), +> you can jump directly to it's corresponding page. -To get it up and running you just need to follow these steps: +## Workflow -### 1. Initialize DVCLive +### Initialize DVCLive ```python from dvclive import Live @@ -16,25 +18,25 @@ live = Live() See [`Live()`](/doc/dvclive/api-reference/live) for details. -### 2. Log metrics +### Log data ```python -live.log(metric_name, value) +live.log("acc", 0.9) ``` See `Live.log()` for details. -### 3. Increase the step number +### (Optionally) Update the step number ```python live.next_step() ``` -See `Live.next_step()` for details. +See `Live.next_step()` and `Live.set_step()` for details. ## Putting all together -Using the above steps, you can easily include DVCLive in your training code: +Joining the above snippets, you can include DVCLive in your training code: ```python # train.py @@ -64,9 +66,25 @@ dvclive train.py dvclive.json ``` -### Metrics Logs +### Summary -For each `{metric_name}`, DVCLive produces metrics _logs_ under +When [`summary`](/doc/dvclive/api-reference/live/#parameters) is enabled (True +by default), DVCLive generates a summary with the latest metrics: + +```dvc +$ cat dvclive.json +{ + "step": 2, + "{metric_name}": 0.8907166719436646, +} +``` + +> If you don't update the step number, the `step` entry won't be present in the +> summary. + +### History + +In addition, for each `{metric_name}`, DVCLive stores the metric's history under `dvclive/{metric_name}.tsv`: ```dvc @@ -77,19 +95,7 @@ timestamp step {metric_name} 1614129198848 2 0.8907166719436646 ``` -### Metrics Summary - -In addition, when [`summary`](/doc/dvclive/api-reference/live/#parameters) is -enabled (True by default), DVCLive generates a metrics _summary_ with the latest -metrics: - -```dvc -$ cat dvclive.json -{ - "step": 2, - "{metric_name}": 0.8907166719436646 -} -``` +> If you don't update the step number, the metrics history won't be generated. ## What next? diff --git a/content/docs/dvclive/index.md b/content/docs/dvclive/index.md index d37e429cc0..1881adc228 100644 --- a/content/docs/dvclive/index.md +++ b/content/docs/dvclive/index.md @@ -1,8 +1,8 @@ # DVCLive Documentation -[DVCLive](https://github.com/iterative/dvclive) is an open-source Python library -for monitoring the progress of metrics during training of machine learning -models. You can simply +[DVCLive](https://github.com/iterative/dvclive) is a Python library for logging +machine learning metrics and other metadata in simple file formats, which is +fully compatible with DVC. You can [install it](https://github.com/iterative/dvclive#installation) with `pip`.