Demo updates #1618

daavoo · 2022-04-26T21:13:55Z

Pending on DVC and DVCLive releases

shcheklein · 2022-04-27T05:43:16Z

demo/params.yaml

@@ -1,3 +1,3 @@
-seed: 473987


Q: why removing this? (to make it cleaner?)

(to make it cleaner?)

Yep. It's just additional column in table not really a parameter to configure

demo/.gitignore

demo/train.py

demo/dvc.yaml

shcheklein

Thanks and thanks, @daavoo , you saved me a lot of time! :)

I've put some comments, it works great for me locally. I've put some comments. Default (?) naming is confusing. And we need to get some images - I can even work on this, but I would appreciate some ideas.

demo/requirements.txt

codeclimate · 2022-04-28T23:52:23Z

demo/train.py

-    return metrics
+    return metrics, predictions
+
+def get_confusion_image(predictions, dataset):


Function get_confusion_image has a Cognitive Complexity of 18 (exceeds 5 allowed). Consider refactoring.

codeclimate · 2022-04-28T23:55:58Z

Code Climate has analyzed commit d91ec4d and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (85% is the threshold).

This pull request will bring the total coverage in the repository to 96.7% (0.0% change).

View more on Code Climate.

mattseddon · 2022-04-28T23:58:44Z

🙏🏻 @daavoo

mattseddon · 2022-04-29T01:11:37Z

demo/dvc.lock

-      md5: e04749646f33be203f210c5f1ea63a2a.dir
-      size: 10783
-      nfiles: 1
+      md5: c7a5760efd52d3759d8e546ab867f4a6


@daavoo can you push this to the remote if you still have it. Please and thank you.

mattseddon · 2022-04-29T01:41:47Z

demo/train.py

        for k, v in metrics.items():
            live.log(k, v)
        live.next_step()

+    live.set_step(None)
+    missclassified = get_confusion_image(predictions, mnist_test)
+    live.log_image("missclassified.jpg", missclassified)


@daavoo there is a change in behaviour here for us. We used to get a single image for every checkpoint. It was saved when the checkpoint was written and it meant that we could have "live updates" in the comparison table. Due to the way that log_image works if we log an image for each checkpoint we now see the following:

Is the old behaviour of being able to see only a single plot for each checkpoint desirable? Would a DS use that functionality? My feeling is that it would be more useful than having all previous images shown and/or only having a single image provided at the end of the epochs.

WDYT?

Is the old behaviour of being able to see only a single plot for each checkpoint desirable? Would a DS use that functionality?

It's dependent on the use case. But I can say that it is more desirable to be able to see all the images from previous checkpoints in the UI, not only the latest.

We introduced the behavior in DVCLive to cover this, but still lack a UI component that makes sense for visualizing it.

Tensorboard implements this functionality with a slider, and it's a very popular feature:

For now, I will remove the log_image usage from DVCLive in favor of plain image saving.

I would say we can create a feature request out of this (we were discussing before, what is the best way to present images from the checkpoint)

mattseddon · 2022-04-29T01:49:24Z

demo/train.py

        for k, v in metrics.items():
            live.log(k, v)
        live.next_step()

+    live.set_step(None)


I think having this here also leads to the old "experiment is logged with the last epoch number" error

Taking a look. I wasn't sure about logging the image with dvclive. We could remove It and just save the image directly

demo/dvc.yaml

mattseddon · 2022-04-29T02:47:53Z

demo/dvclive/scalars/acc.tsv

@@ -0,0 +1,10 @@
+timestamp	step	acc


[Q] After running an experiment for diff we now have:

/demo add-exp-data !7 ❯ dvc diff Added: training_metrics/images/missclassified.jpg training_metrics/report.html training_metrics/scalars/acc.tsv training_metrics/scalars/loss.tsv Modified: model.pt predictions.json training_metrics.json training_metrics/ files summary: 4 added, 3 modified

but in dvc list . -R --dvc-only we only get:

❯ dvc list . --dvc-only -R data/MNIST/raw/t10k-images-idx3-ubyte data/MNIST/raw/t10k-images-idx3-ubyte.gz data/MNIST/raw/t10k-labels-idx1-ubyte data/MNIST/raw/t10k-labels-idx1-ubyte.gz data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-images-idx3-ubyte.gz data/MNIST/raw/train-labels-idx1-ubyte data/MNIST/raw/train-labels-idx1-ubyte.gz model.pt

Should these files be tracked by DVC so that we can showcase the SCM view/decorations accordingly or is that bad practice?

full output from list:

❯ dvc list . -R 0.87841s  .env  3.0.0 12:43:12 .DS_Store .dvcignore .gitignore .vscode/extensions.json .vscode/settings.json data/MNIST/.gitignore data/MNIST/raw.dvc data/MNIST/raw/t10k-images-idx3-ubyte data/MNIST/raw/t10k-images-idx3-ubyte.gz data/MNIST/raw/t10k-labels-idx1-ubyte data/MNIST/raw/t10k-labels-idx1-ubyte.gz data/MNIST/raw/train-images-idx3-ubyte data/MNIST/raw/train-images-idx3-ubyte.gz data/MNIST/raw/train-labels-idx1-ubyte data/MNIST/raw/train-labels-idx1-ubyte.gz dvc.lock dvc.yaml dvclive.json dvclive/scalars/acc.tsv dvclive/scalars/loss.tsv model.pt params.yaml predictions.json requirements.txt train.py training_metrics.json training_metrics/images/missclassified.jpg training_metrics/report.html training_metrics/scalars/acc.tsv training_metrics/scalars/loss.tsv

Its completely use case dependent. we usually set cache: false for files that are small enough to be tracked by git.

If It makes more sense for the VSCode demo, there is nothing wrong with removing the cache: false lines and track everything with DVC

mattseddon added 🏠 housekeeping A: integration Area: DVC integration layer labels Apr 26, 2022

shcheklein reviewed Apr 27, 2022

View reviewed changes

demo/.gitignore Show resolved Hide resolved

shcheklein reviewed Apr 27, 2022

View reviewed changes

demo/train.py Show resolved Hide resolved

shcheklein reviewed Apr 27, 2022

View reviewed changes

demo/dvc.yaml Outdated Show resolved Hide resolved

shcheklein requested changes Apr 27, 2022

View reviewed changes

daavoo force-pushed the demo-updates branch from 281e8ef to 07b36aa Compare April 27, 2022 12:18

Demo updates

f90889b

daavoo force-pushed the demo-updates branch from 07b36aa to f90889b Compare April 27, 2022 12:21

daavoo requested a review from shcheklein April 27, 2022 12:49

daavoo commented Apr 27, 2022

View reviewed changes

demo/requirements.txt Show resolved Hide resolved

mattseddon reviewed Apr 28, 2022

View reviewed changes

demo/requirements.txt Show resolved Hide resolved

daavoo mentioned this pull request Apr 28, 2022

live: Revisit output names iterative/dvclive#246

Closed

Merge branch 'main' into demo-updates

ac0d266

daavoo marked this pull request as ready for review April 28, 2022 18:29

dvc.yaml minor: add newline

d0be7a5

shcheklein approved these changes Apr 28, 2022

View reviewed changes

fix integration tests

63e1948

codeclimate bot reviewed Apr 28, 2022

View reviewed changes

exclude demo from code climate

d91ec4d

mattseddon assigned daavoo Apr 28, 2022

mattseddon merged commit efa7dfd into main Apr 28, 2022

mattseddon deleted the demo-updates branch April 28, 2022 23:59

mattseddon reviewed Apr 29, 2022

View reviewed changes

demo/dvc.yaml Show resolved Hide resolved

mattseddon reviewed Apr 29, 2022

View reviewed changes

This was referenced Apr 29, 2022

set_step(None) generates additional checkpoint iterative/dvclive#248

Closed

More demo updates #1633

Merged

mattseddon mentioned this pull request May 2, 2022

Keep workspace revision in sync with checkpoints when experiment is running in the workspace #1639

Merged

daavoo mentioned this pull request May 2, 2022

plots: How to display images and plots per step #1640

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Demo updates #1618

Demo updates #1618

daavoo commented Apr 26, 2022 •

edited

Loading

shcheklein Apr 27, 2022

daavoo Apr 27, 2022

shcheklein left a comment •

edited

Loading

codeclimate bot Apr 28, 2022

codeclimate bot commented Apr 28, 2022

mattseddon commented Apr 28, 2022

mattseddon Apr 29, 2022

mattseddon Apr 29, 2022 •

edited

Loading

daavoo Apr 29, 2022

shcheklein Apr 29, 2022

mattseddon Apr 29, 2022

daavoo Apr 29, 2022

mattseddon Apr 29, 2022 •

edited

Loading

daavoo Apr 29, 2022

Demo updates #1618

Demo updates #1618

Conversation

daavoo commented Apr 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shcheklein left a comment • edited Loading

Choose a reason for hiding this comment

codeclimate bot Apr 28, 2022

Choose a reason for hiding this comment

codeclimate bot commented Apr 28, 2022

mattseddon commented Apr 28, 2022

Choose a reason for hiding this comment

mattseddon Apr 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattseddon Apr 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daavoo commented Apr 26, 2022 •

edited

Loading

shcheklein left a comment •

edited

Loading

mattseddon Apr 29, 2022 •

edited

Loading

mattseddon Apr 29, 2022 •

edited

Loading