dvclive: Updates for implicit no step

[daavoo]

P.R. for iterative/dvclive#203

[dberenbaum]

Looks good! Can you also update https://dvc.org/doc/dvclive/api-reference/live/log to mention the metrics summary and/or no_step?

[jorgeorpinel]

A bit confused by the concept of DVCLive without steps (or why that addresses iterative/dvclive#182). Plus I see there's a discussion on the semantics in iterative/dvclive#203 (comment) so I'll wait until when/if you request my review to take a closer look. Thanks

[daavoo]

A bit confused by the concept of DVCLive without steps (or why that addresses iterative/dvclive#182).

One of the outputs of DVCLive are linear plots, containing metric values associated to each iteration (step) of the training process.

When exploring DVCLive for ML scenarios where there are no iterations we realize that these outputs don't have any use so in iterative/dvclive#203 we introduce an implicit no step mode.

This makes DVCLive to don't generate linear plots unless next_step or set_step is called in the code, addressing iterative/dvclive#182.

[dberenbaum]

Lots of comments since this is a big change.

This also probably requires an update to the dvclive readme. We probably need some process to keep that updated.

[dberenbaum]

It wouldn't be obvious to me what linear plots means here (or throughout the rest of the docs), and it will be really confusing with iterative/dvclive#189. What was wrong with metrics logs?

Some other ideas:

• metrics trends
• metrics steps
• trend logs
• step logs

[daavoo]

• What was wrong with metrics logs?

I just find metrics logs or the other alternatives a little abstract. I think that "data series", "scalar plots", or line/linear plots better indicate what it's being saved.

[dberenbaum]

Without dvc plots, they are .tsv files, and that's what's actually being saved. IMO plots is a bit unnatural and DVC-specific. It also seems to me like the key feature of these are that they capture the metrics over time at each step, showing training progression. What would describe that? What about something like "step .tsv files"? I don't really like it but not sure what's better.

Do you feel the same about "metrics summary"? That seems similarly abstract to me.

[daavoo]

It also seems to me like the key feature of these are that they capture the metrics over time at each step, showing training progression. What would describe that?

Metrics history?

Do you feel the same about "metrics summary"? That seems similarly abstract to me.

I don't know why but I don't feel the same for summary. Maybe because I can't think of alternatives 😅

[dberenbaum]

I think "metrics history" is pretty good. What do you think?

[daavoo]

Updated with metrics history. Overall I like how it reads

[jorgeorpinel]

I don't see how "metrics logs" is abstract. It's pretty concrete and specific to what those TSV files are. It's just a bit confusing in the current Get Started since there's a "Log Metrics" section and later a "Metrics Logs" subsection. Probably there's no need for those subsections (under Outputs): just use paragraphs, then the terms don't become titles.

"Metrics logs" does also read funny because of the final s in both words though so maybe be more descriptive like TSV log files of your metrics or something like that (and metrics summary file (JSON format)).

"Metrics history" to me sounds like it could relate to the Git history more? Not sure

[daavoo]

For me, history makes more sense than logs when combined with the other output, named summary. It feels like summary and history are "connected".

history is also a term used in Keras with similar meaning as here, so data scientist might be already familiar with it.

Probably there's no need for those subsections (under Outputs): just use paragraphs, then the terms don't become titles.

In subsequent P.R. I will remove the outputs subsections https://dvc-org-dvclive-data-hpgunbngf.herokuapp.com/doc/dvclive/get-started#outputs

[dberenbaum]

Looks good! Just checking on previous comment from @jorgeorpinel about keeping the summaries at the top of the api methods.