live: initial docs draft #2227

pared · 2021-02-22T16:37:10Z

You may disregard these recommendations if you used the Edit on GitHub button from dvc.org to improve a doc in place.

❗ Please read the guidelines in the Contributing to the Documentation list if you make any substantial changes to the documentation or JS engine.

🐛 Please make sure to mention Fix #issue (if applicable) in the description of the PR. This causes GitHub to close it automatically when the PR is merged.

Please choose to allow us to edit your branch when creating the PR.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Matches iterative/dvc/pull/4902

content/docs/dvclive/index.md

jorgeorpinel · 2021-03-01T18:43:17Z

should also come up with a good name - "ML Logger - Dvclive"
if the pitch is a lightweight standalone logger with no dependencies, including DVC in the name seems confusing.

Like @dberenbaum mentioned, we did discuss this in some Slack thread. My take was that "LogML" or "TraceML" were the best options.

dberenbaum · 2021-03-02T21:42:08Z

@pared Love that you added the new section for dvc usage!

I have an example repo in https://github.com/iterative/dvc-checkpoints-mnist that shows how to integrate dvclive with a checkpoints experiment. Maybe it's worth linking that here as an example of how it can be used in place of make_checkpoint with some additional benefits? Also, maybe we should include something similar in the checkpoints docs recommending people use dvclive to capture checkpoints?

Thoughts @pared @pmrowla @jorgeorpinel?

dberenbaum

I have a few comments, but this is a great start! I'll reiterate that it's better to have some documentation than none, so don't take any of my comments as blockers.

dberenbaum · 2021-03-03T00:29:11Z

content/docs/dvclive/index.md

+  training summary during training. When used in DVC project user does not have
+  to call `dvclive.init` method.


When used in DVC project user does not have
to call dvclive.init method.

I'd probably leave this out since it's not that much of an incentive. I would either just leave the first sentence or add in some other benefit, like "When used in a DVC project, it can log metrics and create checkpoints during training that are tracked by DVC."

Removing, I guess the proper place for that is usage-with-dvc section

dberenbaum · 2021-03-03T00:30:22Z

content/docs/dvclive/usage.md

+
+The training code (`train.py` file):
+
+```python


I should update the example repo I have with this example. Keras really makes for a clean, minimal example.

Yeah, but on the other hand we need to prepare Callback, while in torch everything nicely goes into the training loop

dberenbaum · 2021-03-03T00:34:50Z

content/docs/dvclive/usage.md

+In order to do that, we will need to provide proper
+[`Callback`](https://keras.io/api/callbacks/) for `fit` method:


This is nice, but would it be easier for people getting started to see a manual training loop first rather than a callback? This doesn't need to be a blocker, but maybe something to consider as an enhancement.

I think that if we link your torch example, it will be fine.
As i read manual loop tutorial for keras, it seems to me that we will occlude the main point of the tutorial with step by step execution (pushing data through model, calling optimizer, updating gradients). While in this case we have everything hidden under fit call. Callbacks are not that obvious concept, though I think its easier to understand it than reading through whole training loop.

Still, that makes me think that maybe we need not one example, but more? MNIST classifier targeted for DL frameworks users, and something different for, for example, classic ML practitioners? Maybe something with Iris dataset?

Yeah, we will need multiple examples. I'm still not sure myself how dvclive would work for "classic" ML or whether that's a useful scenario to consider. For now, let's keep what you have, and if you think there's a natural way to link to the dvc-checkpoints-mnist example, we could have that as an additional use case. Otherwise, we can find a way to add that or a similar example in the future.

dberenbaum · 2021-03-03T00:38:49Z

content/docs/dvclive/usage-with-dvc.md

+Let's use the code prepared in previous example and try to make it work with
+dvc. Training file `train.py` content:
+
+```python


I mentioned that we might want to link to https://github.com/iterative/dvc-checkpoints-mnist, but alternatively you could incorporate checkpoints in this example. That might be too much to dive right into, but I think most use cases for dvclive with dvc will include checkpoints.

Then maybe we need one more section? We integrated the project with DVC and dvclive, lets make some experiments?

I think the changes to introduce checkpoints would be to write the model out in the callback, and to add the model output to dvc with stage add --checkpoint model .... I don't see checkpoints as necessarily being tied to experiments. It adds versioning of the model output that aligns with the metrics dvclive is tracking.

However, as you mentioned below, let's think about how we can introduce multiple examples to cover all of this. I think the example you have is a great starting point.

dberenbaum · 2021-03-03T00:39:54Z

content/docs/dvclive/usage-with-dvc.md

+Now, lets use dvc to run the project:
+
+```dvc
+$ dvc run -n train --live training_metrics -d train.py python train.py


Would it be better to use stage add && repro/exp run here? @jorgeorpinel?

Good point, I will introduce that.

dberenbaum · 2021-03-03T00:42:31Z

content/docs/dvclive/usage-with-dvc.md

+In `dvc.yaml` there is new stage defined, containing information about the
+`dvclive` outputs:
+
+```bash
+$ cat dvc.yaml
+
+stages:
+  train:
+    cmd: python train.py
+    deps:
+    - train.py
+    live:
+      training_metrics:
+        summary: true
+        html: true
+```


Using stage add above also gives you a reason to move this section up and separate it from the run/repro part and explanation of the dvclive outputs.

content/docs/dvclive/index.md

Co-authored-by: Dave Berenbaum <[email protected]>

dberenbaum

Once the changes around stage add are implemented, I think this should be ready to merge.

dberenbaum · 2021-03-03T16:16:37Z

content/docs/dvclive/usage-with-dvc.md

+Let's use the code prepared in previous example and try to make it work with
+dvc. Training file `train.py` content:
+
+```python


I think the changes to introduce checkpoints would be to write the model out in the callback, and to add the model output to dvc with stage add --checkpoint model .... I don't see checkpoints as necessarily being tied to experiments. It adds versioning of the model output that aligns with the metrics dvclive is tracking.

However, as you mentioned below, let's think about how we can introduce multiple examples to cover all of this. I think the example you have is a great starting point.

dberenbaum · 2021-03-03T16:19:14Z

content/docs/dvclive/usage.md

+In order to do that, we will need to provide proper
+[`Callback`](https://keras.io/api/callbacks/) for `fit` method:


Yeah, we will need multiple examples. I'm still not sure myself how dvclive would work for "classic" ML or whether that's a useful scenario to consider. For now, let's keep what you have, and if you think there's a natural way to link to the dvc-checkpoints-mnist example, we could have that as an additional use case. Otherwise, we can find a way to add that or a similar example in the future.

jorgeorpinel · 2021-03-03T19:47:26Z

content/docs/dvclive/index.md

+[dvclive](https://cml.dev) is an open-source python library for monitoring
+[dvclive](https://cml.dev) is an open-source python library for monitoring the
+progress of metrics during training of machine learning projects.


during training of machine learning models

Is this too specific? What's the broader use case?

I don't think so? Are you asking whether there are non-ML use cases (not likely), or is there something else you think might be too specific?

Are you asking whether there are non-ML use cases (not likely)

Non model-training use cases. E.g. feature extraction or any other stage

Good question. I think it's safe to say that dvclive is a narrower product built specifically for model training.

content/docs/dvclive/index.md

jorgeorpinel

Reviewed the index @pared ☝️. That plus changing sidebar.json + Dave's review should be good enough as a first iteration to merge I think.

Co-authored-by: Jorge Orpinel <[email protected]>

content/docs/dvclive/usage.md

jorgeorpinel

Dvclive + DVC copy edits (will commit):

content/docs/dvclive/dvclive-with-dvc.md

Restyle live: initial docs draft

content/docs/dvclive/dvclive-with-dvc.md

jorgeorpinel · 2021-03-09T09:16:09Z

I have a few minor questions/notes starting at #2227 (review) (cc @pared please scroll through all the resolved comments) but otherwise LGTM — feel free to merge.

jorgeorpinel · 2021-03-09T09:25:01Z

content/docs/dvclive/dvclive-with-dvc.md

+The value passed to `--live` (`training_metrics`) became the directory `path`
+for Dvclive to write logs in. Other supported command options for DVC
+integration:
+
+- `--live-no-summary` - passes `summary=False` to Dvclive.
+- `--live-no-html` - passes `html=False` to Dvclive.


Oh yeah also we need to add these to https://dvc.org/doc/command-reference/stage/add#options ...

Done in 4042082.

Co-authored-by: Jorge Orpinel <[email protected]>

pared · 2021-03-09T10:32:33Z

@jorgeorpinel
I made note out of Further integrations as per your comment

Also I did a issue to fix up missing entries for stages and stage related commands

I am not authorized to merge

per #2227 (review)

rel #2227 (review)

per #2227 (review)

shcheklein had a problem deploying to dvc-org-live-docs-l8ir7gutynb3 February 22, 2021 16:37 Failure

pared force-pushed the live_docs branch from b548e9b to 04014f9 Compare February 22, 2021 16:41

shcheklein temporarily deployed to dvc-org-live-docs-l8ir7gutynb3 February 22, 2021 16:42 Inactive

pared mentioned this pull request Feb 22, 2021

[WIP] logs: initial #1966

Closed

shcheklein temporarily deployed to dvc-org-live-docs-ab0swizph76a February 24, 2021 01:19 Inactive

pared force-pushed the live_docs branch from 35f0107 to 39b625f Compare February 24, 2021 01:19

shcheklein temporarily deployed to dvc-org-live-docs-rtsrgkphj5qf February 24, 2021 01:19 Inactive

jorgeorpinel reviewed Feb 24, 2021

View reviewed changes

content/docs/dvclive/index.md Outdated Show resolved Hide resolved

pared mentioned this pull request Mar 2, 2021

docs: fix README and create documentation entry for dvclive iterative/dvclive#45

Closed

jorgeorpinel mentioned this pull request Mar 2, 2021

2.0 docs #2026

Closed

12 tasks

pared force-pushed the live_docs branch from 39b625f to 05c75e0 Compare March 2, 2021 18:22

dberenbaum approved these changes Mar 3, 2021

View reviewed changes

restyled-io bot mentioned this pull request Mar 3, 2021

Restyle [WIP] live: initial docs draft #2256

Closed

pared and others added 5 commits March 3, 2021 11:30

live: initial docs draft

36d7286

live: initial usage section

cbe791d

dvclive docs: add usage with DVC section

d13ddf0

Update content/docs/dvclive/index.md

edd4138

Co-authored-by: Dave Berenbaum <[email protected]>

fixup

dde7364

pared force-pushed the live_docs branch from 90409a3 to dde7364 Compare March 3, 2021 10:30

dberenbaum reviewed Mar 3, 2021

View reviewed changes

dberenbaum requested a review from jorgeorpinel March 3, 2021 16:20

jorgeorpinel reviewed Mar 3, 2021

View reviewed changes

content/docs/dvclive/index.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Mar 3, 2021

View reviewed changes

content/docs/dvclive/index.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Mar 3, 2021

View reviewed changes

Update content/docs/dvclive/index.md

2285569

Co-authored-by: Jorge Orpinel <[email protected]>

restyled-io bot mentioned this pull request Mar 3, 2021

Restyle [WIP] live: initial docs draft #2260

Closed

Update content/docs/dvclive/index.md

4e9148c

Co-authored-by: Jorge Orpinel <[email protected]>

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

content/docs/dvclive/usage.md Outdated Show resolved Hide resolved

Update content/docs/dvclive/usage.md

7c06934

jorgeorpinel temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 08:39 Inactive

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

content/docs/dvclive/usage.md Show resolved Hide resolved

Update content/docs/dvclive/usage.md

05ae874

jorgeorpinel temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 08:42 Inactive

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

content/docs/dvclive/usage.md Show resolved Hide resolved

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

dvclive: DVC copy edits

402ef5b

jorgeorpinel temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 09:13 Inactive

Restyled by prettier

b0f5481

restyled-io bot mentioned this pull request Mar 9, 2021

Restyle live: initial docs draft #2280

Merged

Merge pull request #2280 from iterative/restyled/live_docs

1bc36e8

Restyle live: initial docs draft

jorgeorpinel temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 09:14 Inactive

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

content/docs/dvclive/dvclive-with-dvc.md Outdated Show resolved Hide resolved

jorgeorpinel reviewed Mar 9, 2021

View reviewed changes

Update content/docs/dvclive/usage.md

837ba4a

Co-authored-by: Jorge Orpinel <[email protected]>

shcheklein temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 10:09 Inactive

fixup

1c60f27

pared mentioned this pull request Mar 9, 2021

dvclive: doc improvements #2282

Closed

2 tasks

shcheklein temporarily deployed to dvc-org-live-docs-b9xyay1xiojb March 9, 2021 10:30 Inactive

jorgeorpinel merged commit 5e456a2 into master Mar 9, 2021

jorgeorpinel deleted the live_docs branch March 10, 2021 19:15

jorgeorpinel added a commit that referenced this pull request Mar 28, 2021

guide: add live field to dvc.yaml guide

5814b32

per #2227 (review)

jorgeorpinel added a commit that referenced this pull request Mar 28, 2021

dvclive: reformat Python code

3617e6f

per #2227 (review)

jorgeorpinel added a commit that referenced this pull request Mar 28, 2021

dvclive: emphasize note about exp signal files

eda7a86

rel #2227 (review)

jorgeorpinel added a commit that referenced this pull request Mar 28, 2021

ref: add Dvclive options to stage add

4042082

per #2227 (review)

jorgeorpinel mentioned this pull request Mar 28, 2021

dvclive: docs followup #2333

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

live: initial docs draft #2227

live: initial docs draft #2227

pared commented Feb 22, 2021

jorgeorpinel commented Mar 1, 2021 •

edited

Loading

dberenbaum commented Mar 2, 2021

dberenbaum left a comment

dberenbaum Mar 3, 2021

pared Mar 3, 2021

dberenbaum Mar 3, 2021

pared Mar 3, 2021

dberenbaum Mar 3, 2021

pared Mar 3, 2021

dberenbaum Mar 3, 2021

dberenbaum Mar 3, 2021

pared Mar 3, 2021

dberenbaum Mar 3, 2021

dberenbaum Mar 3, 2021

pared Mar 3, 2021

dberenbaum Mar 3, 2021

dberenbaum left a comment

dberenbaum Mar 3, 2021

dberenbaum Mar 3, 2021

This comment was marked as resolved.

jorgeorpinel Mar 3, 2021

dberenbaum Mar 3, 2021

jorgeorpinel Mar 3, 2021 •

edited

Loading

dberenbaum Mar 3, 2021

jorgeorpinel left a comment •

edited

Loading

jorgeorpinel left a comment

jorgeorpinel commented Mar 9, 2021

jorgeorpinel Mar 9, 2021 •

edited

Loading

jorgeorpinel Mar 28, 2021

pared commented Mar 9, 2021

		training summary during training. When used in DVC project user does not have
		to call `dvclive.init` method.

		In order to do that, we will need to provide proper
		[`Callback`](https://keras.io/api/callbacks/) for `fit` method:

live: initial docs draft #2227

live: initial docs draft #2227

Conversation

pared commented Feb 22, 2021

jorgeorpinel commented Mar 1, 2021 • edited Loading

dberenbaum commented Mar 2, 2021

dberenbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dberenbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorgeorpinel left a comment • edited Loading

Choose a reason for hiding this comment

jorgeorpinel left a comment

Choose a reason for hiding this comment

jorgeorpinel commented Mar 9, 2021

jorgeorpinel Mar 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pared commented Mar 9, 2021

jorgeorpinel commented Mar 1, 2021 •

edited

Loading

jorgeorpinel Mar 3, 2021 •

edited

Loading

jorgeorpinel left a comment •

edited

Loading

jorgeorpinel Mar 9, 2021 •

edited

Loading