diff --git a/content/docs/command-reference/exp/init.md b/content/docs/command-reference/exp/init.md index c7d70c1af4..62ded78211 100644 --- a/content/docs/command-reference/exp/init.md +++ b/content/docs/command-reference/exp/init.md @@ -148,8 +148,8 @@ See the [Pipelines guide] for more on that topic. default value (`plots/`). - `--live` - set the path to the directory where the metrics and plots - [produced by DVCLive](https://dvc.org/doc/dvclive/dvclive-with-dvc#outputs) - will be found. Overrides the default values for `--metrics` and `--plots`. + [produced by DVCLive](https://dvc.org/doc/dvclive/how-it-works) will be found. + Overrides the default values for `--metrics` and `--plots`. - `--explicit` - do not assume default locations of project dependencies and outputs. You'll have to provide specific locations via other options or diff --git a/content/docs/command-reference/plots/index.md b/content/docs/command-reference/plots/index.md index da9348b971..de62e5dcc0 100644 --- a/content/docs/command-reference/plots/index.md +++ b/content/docs/command-reference/plots/index.md @@ -33,7 +33,7 @@ Plots have to be defined either at the stage level, or at the pipeline level in `dvc.yaml`. [ml pipeline]: /doc/start/data-management/data-pipelines -[dvclive]: /doc/dvclive/dvclive-with-dvc +[dvclive]: /doc/dvclive diff --git a/content/docs/dvclive/api-reference/index.md b/content/docs/dvclive/api-reference/index.md index f3a48f6438..0751c4b9ae 100644 --- a/content/docs/dvclive/api-reference/index.md +++ b/content/docs/dvclive/api-reference/index.md @@ -20,9 +20,7 @@ to its corresponding page. ### Initialize DVCLive ```python -from dvclive import Live - -live = Live() +with Live() as live: ``` See [`Live()`](/doc/dvclive/api-reference/live) for details. @@ -48,13 +46,13 @@ live.log_params(params) See `Live.log_param()` / `Live.log_params()`. - + ```python -live.log("acc", 0.9) +live.log_metric("acc", 0.9) ``` -See `Live.log()`. +See `Live.log_metric()`. @@ -67,15 +65,15 @@ live.log_image("image.png", img) See `Live.log_image()`. - + ```python y_true = [0, 0, 1, 1] y_pred = [0.2, 0.5, 0.3, 0.8] -live.log_plot("roc", y_true, y_score) +live.log_sklearn_plot("roc", y_true, y_score) ``` -See `Live.log_plot()`. +See `Live.log_sklearn_plot()`. @@ -86,7 +84,14 @@ See `Live.log_plot()`. live.next_step() ``` -See `Live.next_step()` and `Live.set_step()` for details. +See `Live.next_step()`. + +Under the hood, `Live.next_step()` calls `Live.make_summary()` and +`Live.make_report()`. + +If you want to decouple the `step` update from the rest of the calls, you can +manually modify the `Live.step` property and call `Live.make_summary()` / +`Live.make_report()`. ## Putting it all together @@ -106,7 +111,7 @@ for epoch in range(NUM_EPOCHS): metrics = evaluate_model(...) for metric_name, value in metrics.items(): - live.log(metric_name, value) + live.log_metric(metric_name, value) live.next_step() ``` @@ -114,5 +119,5 @@ for epoch in range(NUM_EPOCHS): ## Outputs After you run your training code, all the logged data will be stored in the -`dvclive` folder. Check the [DVCLive outputs](/doc/dvclive/outputs) page for -more details. +`dvclive` directory. Check the [DVCLive outputs](/doc/dvclive/how-it-works) page +for more details. diff --git a/content/docs/dvclive/api-reference/live/get_step.md b/content/docs/dvclive/api-reference/live/get_step.md deleted file mode 100644 index d5af8291a7..0000000000 --- a/content/docs/dvclive/api-reference/live/get_step.md +++ /dev/null @@ -1,29 +0,0 @@ -# Live.get_step() - -Returns the current `step` value. - -```py -def get_step() -> int: -``` - -## Usage - -```py -from dvclive import Live - -live = Live() - -while live.get_step() < 3: - live.log("metric", 0.9) - live.next_step() -``` - -## Description - -DVCLive uses `step` to track the progress of the data logged with `Live.log()` -and/or `Live.log_image()`. - -The current `step` value can be retrieved with `Live.get_step()`. - -In order to update the `step` value, you can use `Live.next_step()` or -`Live.set_step()`. diff --git a/content/docs/dvclive/api-reference/live/index.md b/content/docs/dvclive/api-reference/live/index.md index ea268dadaf..2bfc7e86b1 100644 --- a/content/docs/dvclive/api-reference/live/index.md +++ b/content/docs/dvclive/api-reference/live/index.md @@ -7,7 +7,7 @@ class Live: def __init__( self, - path: Optional[str] = None, + dir: str = "dvclive", resume: bool = False, report: Optional[str] = "auto", ): @@ -18,7 +18,8 @@ class Live: ```py from dvclive import Live -live = Live() +with Live() as live: + ... ``` ## Description @@ -28,52 +29,55 @@ metrics and other metadata. -`Live()` will remove all existing DVCLive related files under `path` unless +`Live()` will remove all existing DVCLive related files under `dir` unless `resume=True`. -## Attributes +You can use `Live()` as a context manager. When exiting the context manager, +`Live.make_summary()` and `Live.make_report()` will be called. + +## Properties + +- `step` - See `Live.next_step()`. + +- `summary` - See `Live.make_summary()`. - `dir` - Location of the directory to store - [outputs](/doc/dvclive/get-started#outputs). + [outputs](/doc/dvclive/how-it-works). -- `summary_path` - `{Live.dir}.json`. Location of the - [summary](/doc/dvclive/api-reference/live/log#description). +- `metrics_file` - `{Live.dir}/metrics.json`. -- `report_path` - `{Live.dir}/report.{format}`. Location of the - [metrics report](/doc/dvclive/api-reference/live/make_report). The `format` - can be HTML or Markdown depending on the value of the `report` parameter. +- `params_file` - `{Live.dir}/params.yaml`. -## Parameters +- `plots_dir` - `{Live.dir}/plots`. -- `path` - Where to save DVCLive's outputs. _Default_: `None`. If `None`, - `"dvclive"` will be used. +- `report_file` - `{Live.dir}/report.{format}`. The `format` can be HTML + (`.html`) or Markdown (`.md`) depending on the value of the `report` + parameter. -- `resume` - If `True`, DVCLive will try to read the previous `step` from the - `path` directory and start from that point. _Default_: `False`. +## Parameters - +- `dir` - Where to save DVCLive's outputs. _Default_: `dvclive`. - If you are not using steps, don't set `resume=True` since DVCLive will - preserve previous run's files and assume that `step` has been enabled. +- `resume` - If `True`, DVCLive will try to read the previous `step` from the + `metrics_file` and start from that point. _Default_: `False`. - +- `report` - Any of `auto`, `html`, `md` or `None`. See `Live.make_report()`. -- `report` - If `auto`, `html`, or `md`, DVCLive will call `Live.make_report()` - on each step update. The `auto` mode (default) will use `md` format if a `CI` - env var is present, otherwise it will use `html`. + The `auto` mode (default) will use `md` format if the `CI` env var is present + and [matplotlib](https://matplotlib.org/) is installed, otherwise it will use + `html`. - If `report` is `None`, `Live.make_report()` won't be called automatically. + If `report` is `None`, `Live.make_report()` won't generate anything. ## Methods -- `Live.get_step()` -- `Live.log()` - `Live.log_image()` +- `Live.log_metric()` - `Live.log_param()` - `Live.log_params()` -- `Live.log_plot()` +- `Live.log_sklearn_plot()` - `Live.make_report()` +- `Live.make_summary()` - `Live.next_step()` -- `Live.set_step()` diff --git a/content/docs/dvclive/api-reference/live/log.md b/content/docs/dvclive/api-reference/live/log.md deleted file mode 100644 index e5865c143c..0000000000 --- a/content/docs/dvclive/api-reference/live/log.md +++ /dev/null @@ -1,84 +0,0 @@ -# Live.log() - -Logs the given scalar `val` associating it with the given `name`. - -```py - def log(name: str, val: float): -``` - -## Usage - -```py -from dvclive import Live - -live = Live() - -live.log("loss", 0.1) -live.log("acc", 0.9) -``` - -## Description - -On each `live.log(name, val)` call DVCLive will create or update the `name` -entry in `{Live.dir}.json` with the corresponding `val`: - -```dvc -$ cat dvclive.json -{ - "loss": 0.1, - "acc": 0.9 -} -``` - - - -The summary `{Live.dir}.json` is usable by `dvc metrics` and -`dvc exp show`/`dvc exp diff`. - - - -### Step updates - -The first `step` update (with `Live.next_step()` or `Live.set_step()`) will -create a _metrics history_ file in `{Live.dir}/scalars/{name}.tsv`: - -``` -timestamp step loss -1623671484747 0 0.9 -``` - -Each subsequent call to `live.log(name, val)` will add a new row to -`{Live.dir}/scalars/{name}.tsv`. - -```dvc -$ tree -├── dvclive -│ └── scalars -│ ├── acc.tsv -│ └── loss.tsv -└── dvclive.json -``` - - - -The metrics history (`{Live.dir}/scalars/{name}.tsv`) is usable by `dvc plots`. - - - -If `name` contains slashes (e.g. `train/loss`), the required subdirectories will -be created and the file will be saved inside the last one (e.g. -`{Live.dir}/scalars/train/loss.tsv`). - -## Parameters - -- `name` - Name of the scalar being logged. - -- `val` - The value to be logged. - -## Exceptions - -- `dvclive.error.InvalidDataTypeError` - thrown if the provided `val` does not - have a supported type. - -- `dvclive.error.DataAlreadyLoggedError` - thrown if the provided `name` has - already been logged within the same `step`. diff --git a/content/docs/dvclive/api-reference/live/log_image.md b/content/docs/dvclive/api-reference/live/log_image.md index 89467467fd..ccca01eaa8 100644 --- a/content/docs/dvclive/api-reference/live/log_image.md +++ b/content/docs/dvclive/api-reference/live/log_image.md @@ -26,15 +26,15 @@ live.log_image("pil.png", img_pil) Supported values for `val` are: -- A valid numpy array (convertible to image via +- A valid NumPy array (convertible to image via [PIL.Image.fromarray](https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.fromarray)) - A `PIL.Image` instance. -The images will be saved in `{Live.dir}/images/{name}`: +The images will be saved in `{Live.plots_dir}/images/{name}`: -```dvc -$ tree -└── dvclive +``` +dvclive +└── plots └── images ├── numpy.png └── pil.png @@ -42,32 +42,14 @@ $ tree -The logged images can be visualized with `dvc plots`. - - - -### Step updates - -The first `step` update (with `Live.next_step()` or `Live.set_step()`) will move -the saved file from `{Live.dir}/images/{name}` to -`{Live.dir}/images/{step}/{name}`. +The logged images can be visualized with `dvc plots`: -Each subsequent call to `live.log_image(name, val)` will save the image under -the folder `{Live.dir}/images/{step}/{name}` corresponding to the current -`step`. - -```dvc -$ tree -└── dvclive - └── images - ├── 0 - │ ├── numpy.png - │ └── pil.png - └── 1 - ├── numpy.png - └── pil.png +```cli +$ dvc plots diff dvclive/plots ``` + + ## Parameters - `name` - name of the image file that this command will output @@ -80,4 +62,4 @@ $ tree have a supported type. - `dvclive.error.DataAlreadyLoggedError` - thrown if the provided `name` has - already been logged whithin the same `step`. + already been logged within the same `step`. diff --git a/content/docs/dvclive/api-reference/live/log_metric.md b/content/docs/dvclive/api-reference/live/log_metric.md new file mode 100644 index 0000000000..debfefe3ab --- /dev/null +++ b/content/docs/dvclive/api-reference/live/log_metric.md @@ -0,0 +1,102 @@ +# Live.log_metric() + +```py + def log_metric(name: str, val: float): +``` + +## Usage + +```py +from dvclive import Live + +with Live() as live: + live.log_metric("train/loss", 0.4) + live.log_metric("val/loss", 0.9) +``` + +## Description + +On each `live.log_metric(name, val)` call DVCLive will create a _metrics +history_ file in `{Live.plots_dir}/metrics/{name}.tsv`: + +``` +dvclive +├── metrics.json +└── plots + └── metrics + ├── train + │ └── loss.tsv + └── val + └── loss.tsv +``` + +```cli +$ cat dvclive/plots/metrics/train/loss.tsv +timestamp step loss +1623671484747 0 0.4 +``` + + + +The metrics history can be visualized with `dvc plots`: + +``` +dvc plots diff dvclive/plots +``` + + + +Each subsequent call to `live.log_metric(name, val)` will add a new row to +`{Live.plots_dir}/metrics/{name}.tsv`: + +```python +live.next_step() +live.log_metric("train/loss", 0.2) +live.log_metric("val/loss", 0.4) +``` + +```ts +timestamp step loss +1623671484747 0 0.4 +1623671484892 1 0.2 +``` + +In addition, DVCLive will store the latest value logged in `Live.summary`, so it +can be serialized with calls to `live.make_summary()`, `live.next_step()` or +when exiting the `with` block: + +```json +{ + "step": 1, + "train": { + "loss": 0.2 + }, + "val": { + "loss": 0.4 + } +} +``` + + + +The metrics summary can be visualized with `dvc metrics`: + +``` +dvc metrics diff dvclive/metrics.json +``` + + + +## Parameters + +- `name` - Name of the metric being logged. + +- `val` - The value to be logged. + +## Exceptions + +- `dvclive.error.InvalidDataTypeError` - thrown if the provided `val` does not + have a supported type. + +- `dvclive.error.DataAlreadyLoggedError` - thrown if the provided `name` has + already been logged within the same `step`. diff --git a/content/docs/dvclive/api-reference/live/log_param.md b/content/docs/dvclive/api-reference/live/log_param.md index a2360c0fcd..d7cf7df5d6 100644 --- a/content/docs/dvclive/api-reference/live/log_param.md +++ b/content/docs/dvclive/api-reference/live/log_param.md @@ -1,7 +1,5 @@ # Live.log_param() -Logs the given parameter `val` associating it with the given `name`. - ```py def log_param(name: str, val: ParamLike): ``` @@ -24,12 +22,21 @@ live.log_param("n_iter", 50) On each `live.log_param(name, val)` call, DVCLive will write the `name` parameter to `{Live.dir}/params.yaml` with the corresponding `val`: -```dvc -$ cat dvclive/params.yaml +```yaml lr: 1e-3 n_iter: 50 ``` + + +The logged params can be visualized with `dvc params`: + +```cli +$ dvc params diff dvclive/params.yaml +``` + + + ## Parameters - `name` - Name of the parameter being logged. diff --git a/content/docs/dvclive/api-reference/live/log_params.md b/content/docs/dvclive/api-reference/live/log_params.md index 6f6814b73b..f395d039db 100644 --- a/content/docs/dvclive/api-reference/live/log_params.md +++ b/content/docs/dvclive/api-reference/live/log_params.md @@ -1,7 +1,5 @@ # Live.log_params() -Logs the given group of parameters `params`. - ```py def log_params(params: Dict[ParamLike]): ``` @@ -36,12 +34,21 @@ live.log_params(params) On each `live.log_params(params)` call, DVCLive will write keys/values pairs in the `params` dict to `{Live.dir}/params.yaml`: -```dvc -$ cat dvclive/params.yaml +```yaml lr: 1e-3 n_iter: 50 ``` + + +The logged params can be visualized with `dvc params`: + +```cli +dvc params diff dvclive/params.yaml +``` + + + ## Parameters - `params` - Dictionary with name/value pairs of parameters to be logged. diff --git a/content/docs/dvclive/api-reference/live/log_plot.md b/content/docs/dvclive/api-reference/live/log_sklearn_plot.md similarity index 60% rename from content/docs/dvclive/api-reference/live/log_plot.md rename to content/docs/dvclive/api-reference/live/log_sklearn_plot.md index 2e2fa75616..af148d9c29 100644 --- a/content/docs/dvclive/api-reference/live/log_plot.md +++ b/content/docs/dvclive/api-reference/live/log_sklearn_plot.md @@ -1,11 +1,17 @@ -# Live.log_plot() +# Live.log_sklearn_plot() Generates a [scikit learn plot](https://scikit-learn.org/stable/visualizations.html) and -saves the data in `{Live.dir}/plots/{name}.json`. +saves the data in `{Live.dir}/plots/sklearn/{name}.json`. ```py -def log_plot(self, name: str, labels, predictions, **kwargs): +def log_sklearn_plot( + self, + kind: Literal['calibration', 'confusion_matrix', 'precision_recall', 'roc'], + labels, + predictions, + name: Optional[str] = None, + **kwargs): ``` ## Usage @@ -18,40 +24,20 @@ live = Live() y_true = [0, 0, 1, 1] y_pred = [1, 0, 1, 0] y_score = [0.1, 0.4, 0.35, 0.8]] -live.log_plot("roc", y_true, y_score) -live.log_plot("confusion_matrix", y_true, y_pred) +live.log_sklearn_plot("roc", y_true, y_score) +live.log_sklearn_plot("confusion_matrix", y_true, y_pred, name="cm.json") ``` ## Description -Uses `name` to determine which plot should be generated. See +Uses `kind` to determine the type of plot to be generated. See [supported plots](#supported-plots). - - -The generated `{Live.dir}/plots/{name}.json` can be visualized with `dvc plots`. - - - -### Step updates - -`Live.log_plot()` can be currently only used when `step` is `None`. - -If you perform `step` updates in your code, you can later use -`Live.set_step(None)` in order to be able to use `Live.log_plot()`. - -```python -for epoch in range(NUM_EPOCHS): - live.log(metric_name, value) - live.next_step() - -live.set_step(None) -live.log_plot("roc", y_true, y_score) -``` +If `name` is not provided, `kind` will be used as the default name. ## Supported plots -`name` must be one of the supported plots: +`kind` must be one of the supported plots: @@ -63,19 +49,19 @@ plot. Calls [sklearn.calibration.calibration_curve](https://scikit-learn.org/stable/modules/generated/sklearn.calibration.calibration_curve.html) -and stores the data at `{Live.dir}/plots/calibratrion.json` in a format +and stores the data at `{Live.dir}/plots/sklearn/calibratrion.json` in a format compatible with `dvc plots`. ```py y_true = [0, 0, 1, 1] y_score = [0.1, 0.4, 0.35, 0.8] -live.log_plot("calibration", y_true, y_score) +live.log_sklearn_plot("calibration", y_true, y_score) ``` Example usage with `dvc plots`: -```dvc -$ dvc plots show 'dvclive/plots/calibration.json' \ +```cli +$ dvc plots show 'dvclive/plots/sklearn/calibration.json' \ -x prob_pred -y prob_true \ --x-label 'Mean Predicted Probability' \ --y-label 'Fraction of Positives' \ @@ -91,21 +77,22 @@ $ dvc plots show 'dvclive/plots/calibration.json' \ Generates a [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) plot. -Stores the labels and predictions in `{Live.dir}/plots/confusion_matrix.json`, -with the format expected by the confusion matrix -[template](/doc/user-guide/experiment-management/visualizing-plots#plot-templates-data-series-only) -of `dvc plots`. +Stores the labels and predictions in +`{Live.dir}/plots/sklearn/confusion_matrix.json`, with the format expected by +the confusion matrix +[template](/doc/user-guide/visualizing-plots#plot-templates-data-series-only) of +`dvc plots`. ```py y_true = [1, 1, 2, 2] y_pred = [2, 1, 1, 2] -live.log_plot("confusion_matrix", y_true, y_pred) +live.log_sklearn_plot("confusion_matrix", y_true, y_pred) ``` Example usage with `dvc plots`: -```dvc -$ dvc plots show 'dvclive/plots/confusion_matrix.json' \ +```cli +$ dvc plots show 'dvclive/plots/sklearn/confusion_matrix.json' \ -x actual -y predicted \ --template confusion ``` @@ -122,19 +109,19 @@ plot. Calls [sklearn.metrics.det_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.det_curve.html) -and stores the data at `{Live.dir}/plots/det.json` in a format compatible with -`dvc plots`. +and stores the data at `{Live.dir}/plots/sklearn/det.json` in a format +compatible with `dvc plots`. ```py y_true = [1, 1, 2, 2] y_score = [0.1, 0.4, 0.35, 0.8] -live.log_plot("det", y_true, y_score) +live.log_sklearn_plot("det", y_true, y_score) ``` Example usage with `dvc plots`: -```dvc -$ dvc plots show 'dvclive/plots/det.json' \ +```cli +$ dvc plots show 'dvclive/plots/sklearn/det.json' \ -x fpr -y fnr \ --title 'DET Curve' ``` @@ -151,19 +138,19 @@ plot. Calls [sklearn.metrics.precision_recall_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html) -and stores the data at `{Live.dir}/plots/precision_recall.json` in a format -compatible with `dvc plots`. +and stores the data at `{Live.dir}/plots/sklearn/precision_recall.json` in a +format compatible with `dvc plots`. ```py y_true = [1, 1, 2, 2] y_score = [0.1, 0.4, 0.35, 0.8] -live.log_plot("precision_recall", y_true, y_score) +live.log_sklearn_plot("precision_recall", y_true, y_score) ``` Example usage with `dvc plots`: -```dvc -$ dvc plots show 'dvclive/plots/precision_recall.json' \ +```cli +$ dvc plots show 'dvclive/plots/sklearn/precision_recall.json' \ -x recall -y precision \ --title 'Precision Recall Curve' ``` @@ -180,19 +167,19 @@ plot. Calls [sklearn.metrics.roc_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html#sklearn.metrics.roc_curve) -and stores the data at `{Live.dir}/plots/roc.json` in a format compatible with -`dvc plots`. +and stores the data at `{Live.dir}/plots/sklearn/roc.json` in a format +compatible with `dvc plots`. ```py y_true = [1, 1, 2, 2] y_score = [0.1, 0.4, 0.35, 0.8] -live.log_plot("roc", y_true, y_score) +live.log_sklearn_plot("roc", y_true, y_score) ``` Example usage with `dvc plots`: -```dvc -$ dvc plots show 'dvclive/plots/roc.json' \ +```cli +$ dvc plots show 'dvclive/plots/sklearn/roc.json' \ -x fpr -y tpr \ --title 'ROC Curve' ``` @@ -205,19 +192,20 @@ $ dvc plots show 'dvclive/plots/roc.json' \ ## Parameters -- `name` - a [supported plot type](#supported-plots) +- `kind` - a [supported plot type](#supported-plots) - `labels` - array of ground truth labels - `predictions` - array of predicted labels (for `confusion_matrix`) or predicted probabilities (for other plots) +- `name` - Optional name of the output file. If not provided, `kind` will be + used as name. + - `**kwargs` - additional arguments to be passed to the internal scikit-learn function being called ## Exceptions -- `dvclive.error.InvalidPlotTypeError` - thrown if the provided `name` does not +- `dvclive.error.InvalidPlotTypeError` - thrown if the provided `kind` does not correspond to any of the supported plots. - -- `RuntimeError` - thrown if `Live.log_plot()` is used and `step` is not `None`. diff --git a/content/docs/dvclive/api-reference/live/make_report.md b/content/docs/dvclive/api-reference/live/make_report.md index c376f1ed0b..18dd3c7e26 100644 --- a/content/docs/dvclive/api-reference/live/make_report.md +++ b/content/docs/dvclive/api-reference/live/make_report.md @@ -1,6 +1,6 @@ # Live.make_report() -Generates a metrics report from the logged data. +Generates a report from the logged data. ```py def make_report() @@ -12,7 +12,7 @@ def make_report() from dvclive import Live live = Live() -live.log_plot("confusion_matrix", [0, 0, 1, 1], [1, 0, 0, 1]) +live.log_sklearn_plot("confusion_matrix", [0, 0, 1, 1], [1, 0, 0, 1]) live.make_report() ``` @@ -25,10 +25,3 @@ The `format` can be HTML or Markdown depending on the value of the `report` argument passed to [`Live()`](/doc/dvclive/api-reference/live#parameters). ![](/img/dvclive-html.gif) - - - -This function gets called internally on each `step` update by default (unless -`report=None` is passed to `Live()`). - - diff --git a/content/docs/dvclive/api-reference/live/make_summary.md b/content/docs/dvclive/api-reference/live/make_summary.md new file mode 100644 index 0000000000..db12e9d05b --- /dev/null +++ b/content/docs/dvclive/api-reference/live/make_summary.md @@ -0,0 +1,37 @@ +# Live.make_summary() + +Serializes a summary of the logged metrics (`Live.summary`) to +[`Live.metrics_file`](/doc/dvclive/api-reference/live#properties). + +```py +def make_summary() +``` + +## Usage + +```py +from dvclive import Live + +live = Live() +live.summary["foo"] = 1.0 +live.make_summary() +``` + +## Description + +The `Live.summary` object will contain the latest value of each metric logged +with `Live.log_metric()`. It can be also modified manually, as in the snippet +above. + + + +`Live.next_step()` will call `Live.make_summary()` internally, so you don't need +to call both. + + + +The summary is usable by `dvc metrics`: + +``` +dvc metrics diff dvclive/metrics.json +``` diff --git a/content/docs/dvclive/api-reference/live/next_step.md b/content/docs/dvclive/api-reference/live/next_step.md index dbb146bb52..ad10098f14 100644 --- a/content/docs/dvclive/api-reference/live/next_step.md +++ b/content/docs/dvclive/api-reference/live/next_step.md @@ -1,6 +1,7 @@ # Live.next_step() -Signals that the current step has ended and increases step value by 1 (one). +Signals that the current iteration has ended and increases `step` value by 1 +(one). ```py def next_step() @@ -14,33 +15,35 @@ from dvclive import Live live = Live() for step in range(3): - live.log("metric", 0.9) + live.log_metric("metric", 0.9) live.next_step() ``` ## Description -DVCLive uses `step` to track the progress of the data logged with `Live.log()` -and/or `Live.log_image()`. +DVCLive uses `step` to track the history of the metrics logged with +`Live.log_metric()`. You can use `Live.next_step()` to increase the `step` by 1 (one). -Each metric logged in between `Live.next_step()` (or `Live.set_step()`) calls -will be associated to the updated `step` value. +In addition to increasing the `step` number, it will call `Live.make_report()` +and `Live.make_summary()`. - +### Manual step updates -Each `Live.next_step()` will call `Live.make_report()` internally by default -(unless `report=None` is passed to `Live()`). +If you want custom `step` intervals or don't want to call `Live.make_summary()` +/ `Live.make_report()`, you can manually modify the `Live.step` property: - - -### DVC integration +```py +from dvclive import Live -When `dvclive` is used alongside `DVC`, each `Live.next_step()` call will have -additional effects. +live = Live() -When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are -enabled in the pipeline, DVC will -[create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each -`Live.next_step()` call. +for custom_step in [0, 15, 20]: + live.step = step + live.log_metric("metric_1", 0.9) + live.log_metric("metric_2", 0.7) + live.make_summary() +# Create report only at the end instead of at each iteration +live.make_report() +``` diff --git a/content/docs/dvclive/api-reference/live/set_step.md b/content/docs/dvclive/api-reference/live/set_step.md deleted file mode 100644 index 80770f2c54..0000000000 --- a/content/docs/dvclive/api-reference/live/set_step.md +++ /dev/null @@ -1,61 +0,0 @@ -# Live.set_step() - -Signals that the current step has ended and sets step to the given value. - -```py -def set_step(step: int): -``` - -## Usage - -```py -from dvclive import Live - -live = Live() - -for step in [0, 10, 20]: - live.set_step(step) - live.log("metric_1", 0.9) - live.log("metric_2", 0.7) -``` - -## Description - -DVCLive uses `step` to track the progress of the data logged with `Live.log()` -and/or `Live.log_image()`. - -You can use `Live.set_step()` to set `step` to any value. - -Each metric logged in between `Live.set_step()` (or `Live.next_step()`) calls -will be associated to the provided `step` value. - - - -Each `Live.set_step()` will call `Live.make_report()` internally by default -(unless `report=None` is passed to `Live()`). - - - -### DVC integration - -When `dvclive` is used alongside `DVC`, each `Live.set_step()` call will have -additional effects. - -When [checkpoints](/doc/user-guide/experiment-management/checkpoints) are -enabled in the pipeline, DVC will -[create a new checkpoint](/doc/dvclive/dvclive-with-dvc#checkpoints) on each -`Live.set_step()` call. - -## Example - -Given the [Usage](#usage) code snippet above, the -[metrics history](/doc/dvclive/api-reference/live/log#step-updates) generated -for `metric_1` would be: - -```dvc -$ cat dvclive/metric_1.tsv -timestamp step metric_1 -1614129197192 0 0.9 -1614129198031 10 0.9 -1614129198848 20 0.9 -``` diff --git a/content/docs/dvclive/api-reference/ml-frameworks/catalyst.md b/content/docs/dvclive/api-reference/ml-frameworks/catalyst.md index 2c865e4134..6d128497b1 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/catalyst.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/catalyst.md @@ -6,31 +6,29 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/catalyst.py) -int the callbacks list passed to your +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/catalyst.py) +in the callbacks list passed to your [`Runner`](https://catalyst-team.github.io/catalyst/core/runner.html): ```python -from dvclive.catalyst import DvcLiveCallback +from dvclive.catalyst import DVCLiveCallback ... runner.train( model=model, criterion=criterion, optimizer=optimizer, loaders=loaders, - callbacks=[DvcLiveCallback()]) + callbacks=[DVCLiveCallback()]) ``` -The [history](/doc/dvclive/api-reference/live/log#step-updates) of each -`{metric}` will be stored in: +Each metric will be logged to: ```py -{Live.dir}/scalars/{split}/{metric}.tsv +{Live.plots_dir}/metrics/{split}/{metric}.tsv ``` Where: -- `{Live.dir}` is the - [`dir` attribute of `Live`](/doc/dvclive/api-reference/live#attributes). +- `{Live.plots_dir}` is defined in [`Live`]. - `{split}` can be either `train` or `valid`. - `{metric}` is the name provided by the framework. @@ -39,15 +37,21 @@ Where: - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. + +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples -- Using `model_file`. +- Using `live` to pass an existing [`Live`] instance. ```python -from dvclive.catalyst import DvcLiveCallback +from dvclive import Live +from dvclive.catalyst import DVCLiveCallback + +live = Live("custom_dir") runner.train( model=model, @@ -55,14 +59,31 @@ runner.train( optimizer=optimizer, loaders=loaders, num_epochs=2, - callbacks=[DvcLiveCallback(model_file="model.pth")]) + callbacks=[ + DVCLiveCallback(live=live)]) + +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `model_file`. ```python -from dvclive.catalyst import DvcLiveCallback +runner.train( + model=model, + criterion=criterion, + optimizer=optimizer, + loaders=loaders, + num_epochs=2, + callbacks=[DVCLiveCallback(model_file="model.pth")]) +``` +[`live`]: /docs/dvclive/api-reference/live + +- Using `**kwargs` to customize the new [`Live`] instance. + +```python runner.train( model=model, criterion=criterion, @@ -70,5 +91,7 @@ runner.train( loaders=loaders, num_epochs=2, callbacks=[ - DvcLiveCallback(model_file="model.pth", path="custom_path")]) + DVCLiveCallback(model_file="model.pth", dir="custom_dir")]) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/fastai.md b/content/docs/dvclive/api-reference/ml-frameworks/fastai.md index d6e055824e..ba5f194a45 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/fastai.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/fastai.md @@ -6,32 +6,30 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/fastai.py) -int the callbacks list passed to your +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/fastai.py) +in the callbacks list passed to your [`Learner`](https://docs.fast.ai/learner.html#Learner): ```python -from dvclive.fastai import DvcLiveCallback +from dvclive.fastai import DVCLiveCallback ... learn = tabular_learner(data_loader, metrics=accuracy) learn.fit_one_cycle( n_epoch=2, - cbs=[DvcLiveCallback()]) + cbs=[DVCLiveCallback()]) ``` -The [history](/doc/dvclive/api-reference/live/log#step-updates) of each -`{metric}` will be stored in: +Each metric will be logged to: ```py -{Live.dir}/scalars/{split}/{metric}.tsv +{Live.plots_dir}/metrics/{split}/{metric}.tsv ``` Where: -- `{Live.dir}` is the - [`dir` attribute of `Live`](/doc/dvclive/api-reference/live#attributes). +- `{Live.plots_dir}` is defined in [`Live`]. - `{split}` can be either `train` or `eval`. - `{metric}` is the name provided by the framework. @@ -40,29 +38,46 @@ Where: - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. + +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples -- Using `model_file`. +- Using `live` to pass an existing [`Live`] instance. ```python -from dvclive.fastai import DvcLiveCallback +from dvclive import Live +from dvclive.fastai import DVCLiveCallback + +live = Live("custom_dir") learn = tabular_learner(data_loader, metrics=accuracy) learn.fit_one_cycle( n_epoch=2, - cbs=[DvcLiveCallback(model_file="model.pth")]) + cbs=[DVCLiveCallback(live=live)]) + +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `model_file`. ```python -from dvclive.fastai import DvcLiveCallback +learn.fit_one_cycle( + n_epoch=2, + cbs=[DVCLiveCallback(model_file="model.pth")]) +``` -learn = tabular_learner(data_loader, metrics=accuracy) +- Using `**kwargs` to customize the new [`Live`] instance. + +```python learn.fit_one_cycle( n_epoch=2, - cbs=[DvcLiveCallback(model_file="model.pth", path="custom_path")]) + cbs=[DVCLiveCallback(model_file="model.pth", dir="custom_dir")]) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/huggingface.md b/content/docs/dvclive/api-reference/ml-frameworks/huggingface.md index cac5269353..d7f0d38b60 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/huggingface.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/huggingface.md @@ -6,12 +6,12 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/huggingface.py) -int the callbacks list passed to your +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/huggingface.py) +in the callbacks list passed to your [`Trainer`](https://huggingface.co/transformers/main_classes/trainer.html): ```python -from dvclive.huggingface import DvcLiveCallback +from dvclive.huggingface import DVCLiveCallback ... @@ -22,21 +22,19 @@ from dvclive.huggingface import DvcLiveCallback tokenizer=tokenizer, compute_metrics=compute_metrics, ) -trainer.add_callback(DvcLiveCallback()) +trainer.add_callback(DVCLiveCallback()) trainer.train() ``` -The [history](/doc/dvclive/api-reference/live/log#step-updates) of each -`{metric}` will be stored in: +Each metric will be logged to: ```py -{Live.dir}/scalars/{split}/{metric}.tsv +{Live.plots_dir}/metrics/{split}/{metric}.tsv ``` Where: -- `{Live.dir}` is the - [`dir` attribute of `Live`](/doc/dvclive/api-reference/live#attributes). +- `{Live.plots_dir}` is defined in [`Live`]. - `{split}` can be either `train` or `eval`. - `{metric}` is the name provided by the framework. @@ -45,43 +43,46 @@ Where: - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. + +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples -- Using `model_file`. +- Using `live` to pass an existing [`Live`] instance. ```python -from dvclive.huggingface import DvcLiveCallback +from dvclive import Live +from dvclive.huggingface import DVCLiveCallback + +live = Live("custom_dir") trainer = Trainer( - model, - args, - train_dataset=train_data, - eval_dataset=eval_data, - tokenizer=tokenizer, - compute_metrics=compute_metrics, -) + model, args, + train_dataset=train_data, eval_dataset=eval_data, tokenizer=tokenizer) trainer.add_callback( - DvcLiveCallback(model_file="my_model_path")) -trainer.train() + DVCLiveCallback(live=live)) + +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `model_file`. ```python -from dvclive.huggingface import DvcLiveCallback - -trainer = Trainer( - model, - args, - train_dataset=train_data, - eval_dataset=eval_data, - tokenizer=tokenizer, - compute_metrics=compute_metrics, -) trainer.add_callback( - DvcLiveCallback(model_file="my_model_path", path="custom_path")) + DVCLiveCallback(model_file="my_model_file")) trainer.train() ``` + +- Using `**kwargs` to customize the new [`Live`] instance. + +```python +trainer.add_callback( + DVCLiveCallback(model_file="my_model_file", dir="custom_dir")) +``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/keras.md b/content/docs/dvclive/api-reference/ml-frameworks/keras.md index 0f37886586..c2f0c1f735 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/keras.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/keras.md @@ -6,31 +6,29 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/keras.py) -int the callbacks list passed to your +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/keras.py) +in the callbacks list passed to your [`Model`](https://keras.io/api/models/model/): ```python -from dvclive.keras import DvcLiveCallback +from dvclive.keras import DVCLiveCallback ... model.fit( train_dataset, epochs=num_epochs, validation_data=validation_dataset, - callbacks=[DvcLiveCallback()]) + callbacks=[DVCLiveCallback()]) ``` -The [history](/doc/dvclive/api-reference/live/log#step-updates) of each -`{metric}` will be stored in: +Each metric will be logged to: ```py -{Live.dir}/scalars/{split}/{metric}.tsv +{Live.plots_dir}/metrics/{split}/{metric}.tsv ``` Where: -- `{Live.dir}` is the - [`dir` attribute of `Live`](/doc/dvclive/api-reference/live#attributes). +- `{Live.plots_dir}` is defined in [`Live`]. - `{split}` can be either `train` or `eval`. - `{metric}` is the name provided by the framework. @@ -39,39 +37,55 @@ Where: - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `save_weights_only` (`False` by default) - if True, then only the model's - weights will be saved (`model.save_weights(model_file)`), else the full model - is saved (`model.save(model_file)`) +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples -- Using `model_file` and `save_weights_only`. +- Using `live` to pass an existing [`Live`] instance. ```python -from dvclive.keras import DvcLiveCallback +from dvclive import Live +from dvclive.keras import DVCLiveCallback + +live = Live("custom_dir") + +model.fit( + train_dataset, + epochs=num_epochs, + validation_data=validation_dataset, + callbacks=[DVCLiveCallback(live=live)]) +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() +``` + +- Using `model_file` and `save_weights_only`. + +```python model.fit( train_dataset, epochs=num_epochs, validation_data=validation_dataset, - callbacks=[DvcLiveCallback( + callbacks=[DVCLiveCallback( model_file="my_model_weights.h5", save_weights_only=True)]) ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `**kwargs` to customize the new [`Live`] instance. ```python -from dvclive.keras import DvcLiveCallback - model.fit( train_dataset, epochs=num_epochs, validation_data=validation_dataset, - callbacks=[DvcLiveCallback( + callbacks=[DVCLiveCallback( model_file="my_model_weights.h5", - path="custom_path")]) + dir="custom_dir")]) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/lightgbm.md b/content/docs/dvclive/api-reference/ml-frameworks/lightgbm.md index 12dd728df2..f4f700d18c 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/lightgbm.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/lightgbm.md @@ -6,17 +6,17 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/lgbm.py) +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/lgbm.py) in the callbacks list passed to the `lightgbm.train` call: ```python -from dvclive.lgbm import DvcLiveCallback +from dvclive.lgbm import DVCLiveCallback ... lightgbm.train( param, train_data, valid_sets=[validation_data], num_round=5, - callbacks=[DvcLiveCallback()]) + callbacks=[DVCLiveCallback()]) ``` ## Parameters @@ -24,11 +24,34 @@ lightgbm.train( - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. + +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples +- Using `live` to pass an existing [`Live`] instance. + +```python +from dvclive import Live +from dvclive.lgbm import DVCLiveCallback + +live = Live("custom_dir") + +lightgbm.train( + param, + train_data, + valid_sets=[validation_data], + num_round=5, + callbacks=[DVCLiveCallback(live=live)]) + +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() +``` + - Using `model_file`. ```python @@ -37,10 +60,10 @@ lightgbm.train( train_data, valid_sets=[validation_data], num_round=5, - callbacks=[DvcLiveCallback(model_file="lgbm_model.txt")]) + callbacks=[DVCLiveCallback(model_file="lgbm_model.txt")]) ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `**kwargs` to customize the new [`Live`] instance. ```python lightgbm.train( @@ -48,7 +71,9 @@ lightgbm.train( train_data, valid_sets=[validation_data], num_round=5, - callbacks=[DvcLiveCallback( + callbacks=[DVCLiveCallback( model_file="lgbm_model.txt", - path="custom_path")]) + dir="custom_dir")]) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/mmcv.md b/content/docs/dvclive/api-reference/ml-frameworks/mmcv.md index a12c8c5d81..05963d82c1 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/mmcv.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/mmcv.md @@ -14,7 +14,6 @@ the following in the config file of your log_config = dict( interval=100, hooks=[ - dict(type='TextLoggerHook') dict(type='TextLoggerHook'), dict(type='DvcliveLoggerHook') ] @@ -26,8 +25,8 @@ log_config = dict( - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. ## Examples @@ -43,7 +42,7 @@ log_config = dict( ) ``` -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `**kwargs` to customize [`Live`]. ```python log_config = dict( @@ -53,8 +52,10 @@ log_config = dict( dict( type='DvcliveLoggerHook', model_file="my_model.pth", - path="custom_path" + dir="custom_dir" ) ] ) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/pytorch-lightning.md b/content/docs/dvclive/api-reference/ml-frameworks/pytorch-lightning.md index 8e8379f199..0afef9e876 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/pytorch-lightning.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/pytorch-lightning.md @@ -6,31 +6,29 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Pass the -[`DvcLiveLogger`](https://github.com/iterative/dvclive/blob/main/src/dvclive/lightning.py) +[`DVCLiveLogger`](https://github.com/iterative/dvclive/blob/main/src/dvclive/lightning.py) to your [`Trainer`](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html): ```python -from dvclive.lightning import DvcLiveLogger +from dvclive.lightning import DVCLiveLogger ... - dvclive_logger = DvcLiveLogger() +dvclive_logger = DVCLiveLogger() trainer = Trainer(logger=dvclive_logger) trainer.fit(model) ``` -The [history](/doc/dvclive/api-reference/live/log#step-updates) of each -`{metric}` will be stored in: +Each metric will be logged to: ```py -{Live_dir}/scalars/{split}/{iter_type}/{metric}.tsv +{Live.plots_dir}/metrics/{split}/{iter_type}/{metric}.tsv ``` Where: -- `{Live.dir}` is the - [`dir` attribute of `Live`](/doc/dvclive/api-reference/live#attributes). +- `{Live.plots_dir}` is defined in [`Live`]. - `{split}` can be either `train` or `eval`. - `{iter_type}` can be either `epoch` or `step`. - `{metric}` is the name provided by the framework. @@ -46,30 +44,28 @@ Where: [`Live`](/docs/dvclive/api-reference/live) object to be used instead of initializing a new one. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `experiment` is used, the arguments are ignored. ## Examples -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `**kwargs` to customize [`Live`]. ```python -from dvclive.lightning import DvcLiveLogger +from dvclive.lightning import DVCLiveLogger -dvclive_logger = DvcLiveLogger( - path='my_logs_path' -) trainer = Trainer( - logger=dvclive_logger, -) + logger=DVCLiveLogger(dir='my_logs_dir')) trainer.fit(model) ``` By default, PyTorch Lightning creates a directory to store checkpoints using the -logger's name (`DvcLiveLogger`). You can change the checkpoint path or disable +logger's name (`DVCLiveLogger`). You can change the checkpoint path or disable checkpointing at all as described in the [PyTorch Lightning documentation](https://pytorch-lightning.readthedocs.io/en/latest/common/checkpointing.html) + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/api-reference/ml-frameworks/pytorch.md b/content/docs/dvclive/api-reference/ml-frameworks/pytorch.md index c4fed51f53..75ca9b13a5 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/pytorch.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/pytorch.md @@ -5,9 +5,9 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage -You need to add `Live.log()` calls to each place where you would like to log -metrics and one single `Live.next_step()` call to indicate that the epoch has -ended. +You need to add `Live.log_metric()` calls to each place where you would like to +log metrics and one single `Live.next_step()` call to indicate that the epoch +has ended. let's consider the following example, extracted from the [official PyTorch ImageNet example](https://github.com/pytorch/examples/blob/main/imagenet/main.py): @@ -19,14 +19,14 @@ live = Live() for epoch in range(args.start_epoch, args.epochs): lr = adjust_learning_rate(optimizer, epoch, args) - live.log("learning_rate", lr) + live.log_metric("learning_rate", lr) train_acc1 = train( train_loader, model, criterion, optimizer, epoch, args) - live.log("train/accuracy", train_acc1) + live.log_metric("train/accuracy", train_acc1) val_acc1 = validate(val_loader, model, criterion, args) - live.log("validation/accuracy", val_acc1) + live.log_metric("validation/accuracy", val_acc1) is_best = val_acc1 > best_acc1 best_acc1 = max(val_acc1, best_acc1) diff --git a/content/docs/dvclive/api-reference/ml-frameworks/tensorflow.md b/content/docs/dvclive/api-reference/ml-frameworks/tensorflow.md index 5ba3ccf115..3b89c41672 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/tensorflow.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/tensorflow.md @@ -12,9 +12,9 @@ If you prefer the Keras API, check the -You need to add `Live.log()` calls to each place where you would like to log -metrics and one single `Live.next_step()` call to indicate that the epoch has -ended. +You need to add `Live.log_metric()` calls to each place where you would like to +log metrics and one single `Live.next_step()` call to indicate that the epoch +has ended. let's consider the following example, extracted from the [official TensorFlow guide](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch): @@ -34,13 +34,13 @@ for epoch in range(epochs): optimizer.apply_gradients(zip(grads, model.trainable_weights)) train_acc_metric.update_state(y_batch_train, logits) - live.log("train/accuracy", float(train_acc_metric.result()) + live.log_metric("train/accuracy", float(train_acc_metric.result()) train_acc_metric.reset_states() for x_batch_val, y_batch_val in val_dataset: val_logits = model(x_batch_val, training=False) val_acc_metric.update_state(y_batch_val, val_logits) - live.log("val/accuracy", float(val_acc_metric.result()) + live.log_metric("val/accuracy", float(val_acc_metric.result()) val_acc_metric.reset_states() live.next_step() diff --git a/content/docs/dvclive/api-reference/ml-frameworks/xgboost.md b/content/docs/dvclive/api-reference/ml-frameworks/xgboost.md index 2b7cea2ff2..004b94a69e 100644 --- a/content/docs/dvclive/api-reference/ml-frameworks/xgboost.md +++ b/content/docs/dvclive/api-reference/ml-frameworks/xgboost.md @@ -6,17 +6,17 @@ DVCLive allows you to add experiment tracking capabilities to your ## Usage Include the -[`DvcLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/xgb.py) +[`DVCLiveCallback`](https://github.com/iterative/dvclive/blob/main/src/dvclive/xgb.py) in the callbacks list passed to the `xgboost.train` call: ```python -from dvclive.xgb import DvcLiveCallback +from dvclive.xgb import DVCLiveCallback ... xgboost.train( param, dtrain, num_round=5, evals=[(dval, "eval_data")] - callbacks=[DvcLiveCallback("eval_data")], + callbacks=[DVCLiveCallback("eval_data")], ) ``` @@ -25,12 +25,35 @@ xgboost.train( - `model_file` - (`None` by default) - The name of the file where the model will be saved at the end of each `step`. -- `**kwargs` - Any additional arguments will be passed to - [`Live`](/docs/dvclive/api-reference/live). +- `live` - (`None` by default) - Optional [`Live`] instance. If `None`, a new + instance will be created using `**kwargs`. + +- `**kwargs` - Any additional arguments will be used to instantiate a new + [`Live`] instance. If `live` is used, the arguments are ignored. ## Examples -- Using `**kwargs` to customize [`Live`](/docs/dvclive/api-reference/live). +- Using `live` to pass an existing [`Live`] instance. + +```python +from dvclive import Live +from dvclive.xgb import DVCLiveCallback + +live = Live("custom_dir") + +xgboost.train( + param, + dtrain, + num_round=5, + callbacks=[DVCLiveCallback("eval_data", live=live)], + evals=[(dval, "eval_data")]) + +# Log additional metrics after training +live.summary["additional_metric"] = 1.0 +live.make_summary() +``` + +- Using `**kwargs` to customize [`Live`]. ```python xgboost.train( @@ -38,8 +61,10 @@ xgboost.train( dtrain, num_round=5, callbacks=[ - DvcLiveCallback( + DVCLiveCallback( "eval_data", - path="custom_path")], + dir="custom_dir")], evals=[(dval, "eval_data")]) ``` + +[`live`]: /docs/dvclive/api-reference/live diff --git a/content/docs/dvclive/get-started.md b/content/docs/dvclive/get-started.md index 4b12912559..72f1dbc5b4 100644 --- a/content/docs/dvclive/get-started.md +++ b/content/docs/dvclive/get-started.md @@ -14,11 +14,13 @@ First of all, you need to add DVCLive to your Python script: ```python # train.py -from dvclive.keras import DvcLiveCallback +from dvclive.keras import DVCLiveCallback + ... + model.fit( train_dataset, validation_data=validation_dataset, - callbacks=[DvcLiveCallback()]) + callbacks=[DVCLiveCallback()]) ``` @@ -27,18 +29,11 @@ model.fit( ```python # train.py -from dvclive.huggingface import DvcLiveCallback - -. . . - - trainer = Trainer( - model, args, - train_dataset=train_data, - eval_dataset=eval_data, - tokenizer=tokenizer, - compute_metrics=compute_metrics, -) -trainer.add_callback(DvcLiveCallback()) +from dvclive.huggingface import DVCLiveCallback + +... + +trainer.add_callback(DVCLiveCallback()) trainer.train() ``` @@ -47,12 +42,11 @@ trainer.train() ```python # train.py -from dvclive.lightning import DvcLiveLogger +from dvclive.lightning import DVCLiveLogger -. . . -dvclive_logger = DvcLiveLogger() +... -trainer = Trainer(logger=dvclive_logger) +trainer = Trainer(logger=DVCLiveLogger()) trainer.fit(model) ``` @@ -64,18 +58,15 @@ trainer.fit(model) # train.py from dvclive import Live -live = Live() - -live.log_param("epochs", NUM_EPOCHS) - -for epoch in range(NUM_EPOCHS): - train_model(...) - metrics = evaluate_model(...) - - for metric_name, value in metrics.items(): - live.log(metric_name, value) +with Live() as live: + live.log_param("epochs", NUM_EPOCHS) - live.next_step() + for epoch in range(NUM_EPOCHS): + train_model(...) + metrics = evaluate_model(...) + for metric_name, value in metrics.items(): + live.log_metric(metric_name, value) + live.next_step() ``` @@ -89,13 +80,13 @@ more details and other supported frameworks. Once you have added DVCLive to our python code, you can run the script as you would usually do: -```dvc +```cli $ python train.py ``` -DVCLive will generate a [report](/doc/dvclive/outputs#report) in +DVCLive will generate a [report](/doc/dvclive/api-reference/live/make_report) in `dvclive/report.html` containing all the logged data. It will be automatically -updated during training on each `step` update: +updated during training: ![HTML report](/img/dvclive-html.gif). @@ -103,17 +94,19 @@ In addition, the logged data will be stored in plain text files. For this simple example, it would look as follows: ``` -dvclive.json dvclive -├── scalars -│ └── metric.tsv +├── metrics.json +├── params.yaml +├── plots +│ └── metrics +│ └── accuracy.tsv └── report.html ``` You can find more details about the structure of files generated by DVCLive in -the [Output Folder Structure](/doc/dvclive/outputs) page. +the [How it Works](/doc/dvclive/how-it-works) page. @@ -125,7 +118,7 @@ ways to compare and visualize our experiments. You can find below an example `dvc.yaml` containing a single stage `train`, that runs the script from previous steps and tracks the -[DVCLive outputs](/doc/dvclive/outputs) as `dvc metrics` and `dvc plots`: +[DVCLive outputs](/doc/dvclive/how-it-works) as `dvc metrics` and `dvc plots`: ```yaml stages: @@ -134,10 +127,10 @@ stages: deps: - train.py metrics: - - dvclive.json: + - dvclive/metrics.json: cache: false plots: - - dvclive/scalars: + - dvclive/plots: cache: false ``` @@ -147,7 +140,7 @@ You can now [run experiments](/doc/user-guide/experiment-management/running-experiments) with `dvc exp run`: -```dvc +```cli $ dvc exp run Running stage 'train': > python train.py @@ -157,7 +150,7 @@ Running stage 'train': ### Compare and Visualize DVC Experiments After following the above steps, you have enabled different ways of monitoring -the training, comparing, and visualizing the experiments: +the training, comparing, and visualizing the experiments. #### DVC CLI @@ -184,13 +177,13 @@ and views. ![Experiments view](https://github.com/iterative/vscode-dvc/raw/main/extension/resources/walkthrough/images/experiments-table.png) -![Plots view](https://github.com/iterative/vscode-dvc/raw/main/extension/resources/walkthrough/images/plots-trends.png) +![Plots view](https://github.com/iterative/vscode-dvc/raw/main/extension/resources/walkthrough/images/plots-data-series.png) ### Share Results After the experiment has finished and you have committed and pushed the results, [Iterative Studio](/doc/studio) will automatically parse the outputs generated by DVCLive, allowing you to -[share your experiments online](https://dvc.org/doc/studio/get-started): +[share your experiments online](/doc/studio/get-started): ![Studio view](/img/dvclive-studio-plots.png) diff --git a/content/docs/dvclive/how-it-works.md b/content/docs/dvclive/how-it-works.md new file mode 100644 index 0000000000..5b5091096d --- /dev/null +++ b/content/docs/dvclive/how-it-works.md @@ -0,0 +1,66 @@ +# How it Works + +DVCLive will store the logged data under the directory (`dir`) passed to +[`Live()`](/doc/dvclive/api-reference/live). If not provided, `dvclive` will be +used by default. + +The contents of the directory will depend on the methods used: + +| Method | Writes to | +| ------------------------------------------------------------------------- | -------------------------- | +| [live.log_metric](/doc/dvclive/api-reference/live/log_metric) | `dvclive/plots/metrics` | +| [live.log_image](/doc/dvclive/api-reference/live/log_image) | `dvclive/plots/images` | +| [live.log_param](/doc/dvclive/api-reference/live/log_param) | `dvclive/params.yaml` | +| [live.log_sklearn_plot](/doc/dvclive/api-reference/live/log_sklearn_plot) | `dvclive/plots/sklearn` | +| [live.make_report](/doc/dvclive/api-reference/live/make_report) | `dvclive/report.{md/html}` | +| [live.make_summary](/doc/dvclive/api-reference/live/make_summary) | `dvclive/metrics.json` | + + + +`live.next_step()` takes care of calling `live.make_report()` and +`live.make_summary`, in addition to increasing the `step` number. + + + +## Example + +To illustrate with an example, given the following script: + +```python +import random + +from dvclive import Live +from PIL import Image + +EPOCHS = 2 + +with Live() as live: + live.log_param("epochs", EPOCHS) + + for i in range(EPOCHS): + live.log_metric("metric", i + random.random()) + live.log_metric("nested/metric", i + random.random()) + live.log_image("img.png", Image.new("RGB", (50, 50), (i, i, i))) + live.next_step() + + live.log_sklearn_plot("confusion_matrix", [0, 0, 1, 1], [0, 1, 0, 1]) + live.summary["additional_metric"] = 1.0 +``` + +The resulting structure will be: + +``` +dvclive +├── metrics.json +├── params.yaml +├── plots +│ ├── images +│ │ └── img.png +│ ├── metrics +│ │ ├── metric.tsv +│ │ └── nested +│ │ └── metric.tsv +│ └── sklearn +│ └── confusion_matrix.json +└── report.html +``` diff --git a/content/docs/dvclive/index.md b/content/docs/dvclive/index.md index 629e612118..0163509c6b 100644 --- a/content/docs/dvclive/index.md +++ b/content/docs/dvclive/index.md @@ -2,7 +2,9 @@ [DVCLive](https://github.com/iterative/dvclive) is a Python library for logging machine learning metrics and other metadata in simple file formats, which is -fully compatible with DVC. You can +fully compatible with DVC, the +[VS Code Extension](https://marketplace.visualstudio.com/items?itemName=Iterative.dvc), +and [Iterative Studio](https://studio.iterative.ai/). You can [install it](https://github.com/iterative/dvclive#installation) with `pip`. diff --git a/content/docs/dvclive/outputs.md b/content/docs/dvclive/outputs.md deleted file mode 100644 index fb661495e8..0000000000 --- a/content/docs/dvclive/outputs.md +++ /dev/null @@ -1,83 +0,0 @@ -# Output Folder Structure - -DVCLive will store the logged data under the folder (`path`) passed to -[`Live()`](/doc/dvclive/api-reference/live). If not provided, `dvclive` will be -used by default. - -The resulting structure of the folder depends on the methods used for logging -data: - -| Method | Writes to | -| --------------------------------------------------------------- | ---------------------------------- | -| [live.log](/doc/dvclive/api-reference/live/log) | `dvclive/scalars` & `dvclive.json` | -| [live.log_image](/doc/dvclive/api-reference/live/log_image) | `dvclive/images` | -| [live.log_param](/doc/dvclive/api-reference/live/log_param) | `dvclive/params.yaml` | -| [live.log_plot](/doc/dvclive/api-reference/live/log_plot) | `dvclive/plots` | -| [live.make_report](/doc/dvclive/api-reference/live/make_report) | `dvclive/report.{md/html}` | - -## Example - -To illustrate with an example, given the following script: - -```python -import random - -from dvclive import Live -from PIL import Image -from PIL import ImageDraw - -live = Live() - -live.log_param("param", random.random()) - -for i in range(2): - live.log("metric", i + random.random()) - live.log("nested/metric", i + random.random()) - - img = Image.new("RGB", (50, 50), (255, 255, 255)) - draw = ImageDraw.Draw(img) - draw.text((20, 20), f"{i + random.random():.1f}", (0,0,0)) - live.log_image("img.png", img) - - live.next_step() # calls live.make_report() - -live.set_step(None) -live.log_plot("confusion_matrix", [0, 0, 1, 1], [0, 1, 0, 1]) -``` - -The resulting structure will be: - -``` -dvclive.json -dvclive -├── params.yaml -├── plots -│ └── confusion_matrix.json -├── images -│ ├── 0 -│ │ └── img.png -│ └── 1 -│ └── img.png -├── scalars -│ ├── nested -│ │ └── metric.tsv -│ └── metric.tsv -└── report.html -``` - -## Report - -If and when `step` is updated, DVCLive generates or updates a report in -`dvclive/report.{format}` which will contain the logged data. - -![](/img/dvclive-html.gif) - -The `format` can be HTML or Markdown depending on the value of the `report` -argument passed to [`Live()`](/doc/dvclive/api-reference/live#parameters). - - - -Reports are generated if you update the step number or call `Live.make_report()` -directly. - - diff --git a/content/docs/sidebar.json b/content/docs/sidebar.json index 297aee5ee0..7b1c3e24e6 100644 --- a/content/docs/sidebar.json +++ b/content/docs/sidebar.json @@ -628,8 +628,8 @@ "slug": "get-started" }, { - "slug": "outputs", - "label": "Output Folder Structure" + "slug": "how-it-works", + "label": "How it Works" }, { "slug": "api-reference", @@ -640,18 +640,14 @@ "label": "Live()", "source": "api-reference/live/index.md", "children": [ - { - "slug": "get_step", - "label": "get_step()" - }, - { - "slug": "log", - "label": "log()" - }, { "slug": "log_image", "label": "log_image()" }, + { + "slug": "log_metric", + "label": "log_metric()" + }, { "slug": "log_param", "label": "log_param()" @@ -661,20 +657,20 @@ "label": "log_params()" }, { - "slug": "log_plot", - "label": "log_plot()" + "slug": "log_sklearn_plot", + "label": "log_sklearn_plot()" }, { "slug": "make_report", "label": "make_report()" }, { - "slug": "next_step", - "label": "next_step()" + "slug": "make_summary", + "label": "make_summary()" }, { - "slug": "set_step", - "label": "set_step()" + "slug": "next_step", + "label": "next_step()" } ] }, diff --git a/content/docs/start/experiment-management/experiments.md b/content/docs/start/experiment-management/experiments.md index ba2d51ebc0..c8d11eb820 100644 --- a/content/docs/start/experiment-management/experiments.md +++ b/content/docs/start/experiment-management/experiments.md @@ -58,7 +58,7 @@ metrics and plots to evaluate the models. `dvc exp init` has sane defaults about the names of these elements to initialize a project: ```dvc -$ dvc exp init python src/train.py +$ dvc exp init --live dvclive --plots plots python src/train.py ``` Here, `python src/train.py` specifies how you run experiments. It could be any @@ -242,13 +242,13 @@ generate these files. Please refer to the documentation for details. `dvc exp show` and `dvc metrics` are used to tabulate the experiments and Git commits with their associated metrics. In the above tables, `loss` and `acc` -values are metrics found in [`metrics.json`] file. +values are metrics found in [`dvclive/metrics.json`] file. Metrics files are interpreted specially also in [Iterative Studio](https://studio.iterative.ai). [`metrics.json`]: - https://github.com/iterative/example-dvc-experiments/blob/main/metrics.json + https://github.com/iterative/example-dvc-experiments/blob/main/dvclive/metrics.json diff --git a/content/docs/start/experiment-management/visualization.md b/content/docs/start/experiment-management/visualization.md index 53f8059d24..7101b82989 100644 --- a/content/docs/start/experiment-management/visualization.md +++ b/content/docs/start/experiment-management/visualization.md @@ -72,7 +72,7 @@ $ dvc plots show --open plots/misclassified.png ## Autogenerating plots from deep learning code -An important issue for deep learning projects is to observe in which epoch do +An important issue for deep learning projects is to observe in which epoch training and validation loss differ. DVC helps in that regard with its Python integrations to deep learning libraries via [DVCLive]. @@ -81,7 +81,7 @@ callback that visualizes the training and validation loss for each epoch. We first import the callback from DVCLive. ```python -from dvclive.keras import DvcLiveCallback +from dvclive.keras import DVCLiveCallback ``` Then we add this callback to the @@ -91,12 +91,12 @@ call. ```python model.fit( ... - callbacks=[DvcLiveCallback()], + callbacks=[DVCLiveCallback()], ...) ``` With these two changes, the model metrics are automatically logged to -`dvclive.json` and plotted in `training_metrics/index.html`: +`dvclive/metrics.json` and plotted in `dvclive/report.html`: ![dvclive](/img/start_visualization_dvclive.png) @@ -106,8 +106,8 @@ these default values. In summary, DVC provides more than one option to use visualization in your workflow: -- DVC can generate HTML files that includes interactive plots from data series - in JSON, YAML, CSV, or TSV format. +- DVC can generate HTML files that include interactive plots from data series in + JSON, YAML, CSV, or TSV format. - DVC can keep track of image files produced as [plot outputs] from the training/evaluation scripts. @@ -115,4 +115,4 @@ workflow: - [DVCLive] integrations can produce plots automatically during training. [plot outputs]: /doc/user-guide/experiment-management/visualizing-plots -[dvclive]: /doc/dvclive/dvclive-with-dvc +[dvclive]: /doc/dvclive/ diff --git a/content/docs/user-guide/experiment-management/checkpoints.md b/content/docs/user-guide/experiment-management/checkpoints.md index 2457e1e15f..ad9d4cb0af 100644 --- a/content/docs/user-guide/experiment-management/checkpoints.md +++ b/content/docs/user-guide/experiment-management/checkpoints.md @@ -122,11 +122,11 @@ stages: - model.pt: checkpoint: true metrics: - - dvclive.json: + - dvclive/metrics.json: cache: false persist: true plots: - - dvclive/scalars: + - dvclive/plots: cache: false persist: true ``` @@ -181,7 +181,7 @@ Then update the following lines of code in the `main` method inside of the training epoch loop. ```git -+ dvclive = Live() ++ live = Live() # Iterate over training epochs. for i in range(1, EPOCHS+1): @@ -197,15 +197,15 @@ for i in range(1, EPOCHS+1): metrics = evaluate(model, x_test, y_test) for k, v in metrics.items(): print('Epoch %s: %s=%s'%(i, k, v)) -+ dvclive.log(k, v) -+ dvclive.next_step() ++ live.log_metric(k, v) ++ live.next_step() ``` The line `torch.save(model.state_dict(), "model.pt")` updates the checkpoint file. -You can read about what the line `dvclive.log(k, v)` does in the `Live.log()` -reference. +You can read about what the line `live.log_metric(k, v)` does in the +`Live.log_metric()` reference. The `Live.next_step()` line tells DVC that it can take a snapshot of the entire workspace and version it with Git. It's important that with this approach only @@ -231,19 +231,16 @@ Generating lock file 'dvc.lock' Updating lock file 'dvc.lock' Checkpoint experiment iteration 'd99d81c'. -file:///Users/milecia/Repos/checkpoints-tutorial/dvclive_dvc_plots/index.html Epoch 2: loss=1.25374174118042 Epoch 2: acc=0.7738 Updating lock file 'dvc.lock' Checkpoint experiment iteration '963b396'. -file:///Users/milecia/Repos/checkpoints-tutorial/dvclive_dvc_plots/index.html Epoch 3: loss=0.7242147922515869 Epoch 3: acc=0.8284 Updating lock file 'dvc.lock' Checkpoint experiment iteration 'd630b92'. -file:///Users/milecia/Repos/checkpoints-tutorial/dvclive_dvc_plots/index.html Epoch 4: loss=0.5083536505699158 Epoch 4: acc=0.8538 Updating lock file 'dvc.lock' @@ -375,10 +372,10 @@ table with the checkpoints you want to compare. You'll see something similar to this in your terminal. ``` -Path Metric d90179a 726d32f Change -dvclive.json acc 0.9044 0.8185 -0.0859 -dvclive.json loss 0.33246 0.83515 0.50269 -dvclive.json step 6 8 2 +Path Metric d90179a 726d32f Change +dvclive/metrics.json acc 0.9044 0.8185 -0.0859 +dvclive/metrics.json loss 0.33246 0.83515 0.50269 +dvclive/metrics.json step 6 8 2 ``` _These are the same numbers you see in the metrics table, just in a different @@ -453,7 +450,7 @@ allow you to add these changes to Git, making them [persistent]: ```dvc To track the changes with git, run: - git add dvclive.json dvc.yaml .gitignore train.py dvc.lock + git add dvclive/metrics.json dvc.yaml .gitignore train.py dvc.lock ... ``` @@ -468,13 +465,13 @@ Changes to be committed: (use "git restore --staged ..." to unstage) new file: .gitignore new file: dvc.lock - new file: dvclive.json + new file: dvclive/metrics.json Untracked files: (use "git add ..." to include in what will be committed) data/ - dvclive.html - dvclive/ + dvclive/report.html + dvclive/plots model.pt plots.html predictions.json diff --git a/content/docs/user-guide/experiment-management/visualizing-plots.md b/content/docs/user-guide/experiment-management/visualizing-plots.md index dd24bbe5fa..5482231696 100644 --- a/content/docs/user-guide/experiment-management/visualizing-plots.md +++ b/content/docs/user-guide/experiment-management/visualizing-plots.md @@ -53,8 +53,8 @@ workflow is: To create valid plots files, you can: -- Use [DVCLive](/doc/dvclive/dvclive-with-dvc) in your Python code to log the - data automatically in a DVC-compatible format. +- Use [DVCLive](/doc/dvclive/) in your Python code to log the data automatically + in a DVC-compatible format. - Generate a JSON, YAML 1.2, CSV, or TSV data series file yourself. - Save an JPEG, GIF, or PNG image file to render directly in your reports (helpful for custom visualizations that would be hard to configure in DVC).