Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demo updates #1618

Merged
merged 5 commits into from
Apr 28, 2022
Merged

Demo updates #1618

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f90889b
Demo updates
daavoo Apr 26, 2022
ac0d266
Merge branch 'main' into demo-updates
mattseddon Apr 28, 2022
d0be7a5
dvc.yaml minor: add newline
shcheklein Apr 28, 2022
63e1948
fix integration tests
mattseddon Apr 28, 2022
d91ec4d
exclude demo from code climate
mattseddon Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions demo/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
/model.pt
/data/MNIST/processed
/.vscode/settings.json
/.env
daavoo marked this conversation as resolved.
Show resolved Hide resolved
/logs
/logs_dvc_plots
/plots
.env
/training_metrics/report.html
/model.pt
30 changes: 15 additions & 15 deletions demo/dvc.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,25 @@ stages:
size: 66544866
nfiles: 10
- path: train.py
md5: b6291190036da6a4c61d408826d1c7d5
size: 3895
md5: 3e52f9876bad40d131d536c9ef9f09b8
size: 4842
params:
params.yaml:
lr: 0.001
seed: 473987
epochs: 15
lr: 0.003
weight_decay: 0
outs:
- path: logs
md5: e97c9b6938efba82e955301e1bbfa4c6.dir
size: 572
nfiles: 2
- path: missclassified.jpg
- path: model.pt
md5: d06ada7ceb6a17c0e9d88c178de52a33
size: 410191
- path: plots
md5: e04749646f33be203f210c5f1ea63a2a.dir
size: 10783
nfiles: 1
md5: c7a5760efd52d3759d8e546ab867f4a6
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daavoo can you push this to the remote if you still have it. Please and thank you.

size: 439383
- path: predictions.json
md5: e6b7b45e9dcf958ff5ddaf8fd920919f
md5: 77955a85635a5273fd80ea4cfeca5822
size: 310000
- path: training_metrics
md5: 9ad74f1e8e0760fc5108bd684b2cd281.dir
size: 45614
nfiles: 4
- path: training_metrics.json
md5: 1d702331f65c5bc23c7552306d2d3861
size: 69
19 changes: 10 additions & 9 deletions demo/dvc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,21 @@ stages:
- data/MNIST
- train.py
params:
- seed
- lr
- weight_decay
- params.yaml:
outs:
- model.pt:
checkpoint: true
metrics:
- training_metrics.json:
cache: false
persist: true
plots:
- plots
- training_metrics:
cache: false
- missclassified.jpg:
daavoo marked this conversation as resolved.
Show resolved Hide resolved
cache: false
- predictions.json:
cache: false
template: confusion
x: actual
y: predicted
live:
logs:
summary: true
html: true
y: predicted
5 changes: 5 additions & 0 deletions demo/dvclive.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"step": 14,
"loss": 1.2490187883377075,
"acc": 0.6258
}
10 changes: 10 additions & 0 deletions demo/dvclive/scalars/acc.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
timestamp step acc
Copy link
Member

@mattseddon mattseddon Apr 29, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] After running an experiment for diff we now have:

/demo add-exp-data !7 ❯ dvc diff
Added:                                                                                                                                               
    training_metrics/images/missclassified.jpg
    training_metrics/report.html
    training_metrics/scalars/acc.tsv
    training_metrics/scalars/loss.tsv

Modified:
    model.pt
    predictions.json
    training_metrics.json
    training_metrics/

files summary: 4 added, 3 modified

but in dvc list . -R --dvc-only we only get:

❯ dvc list . --dvc-only -R
data/MNIST/raw/t10k-images-idx3-ubyte
data/MNIST/raw/t10k-images-idx3-ubyte.gz
data/MNIST/raw/t10k-labels-idx1-ubyte
data/MNIST/raw/t10k-labels-idx1-ubyte.gz
data/MNIST/raw/train-images-idx3-ubyte
data/MNIST/raw/train-images-idx3-ubyte.gz
data/MNIST/raw/train-labels-idx1-ubyte
data/MNIST/raw/train-labels-idx1-ubyte.gz
model.pt

Should these files be tracked by DVC so that we can showcase the SCM view/decorations accordingly or is that bad practice?

full output from list:

❯ dvc list . -R                                                                0.87841s  .env  3.0.0 12:43:12
.DS_Store
.dvcignore
.gitignore
.vscode/extensions.json
.vscode/settings.json
data/MNIST/.gitignore
data/MNIST/raw.dvc
data/MNIST/raw/t10k-images-idx3-ubyte
data/MNIST/raw/t10k-images-idx3-ubyte.gz
data/MNIST/raw/t10k-labels-idx1-ubyte
data/MNIST/raw/t10k-labels-idx1-ubyte.gz
data/MNIST/raw/train-images-idx3-ubyte
data/MNIST/raw/train-images-idx3-ubyte.gz
data/MNIST/raw/train-labels-idx1-ubyte
data/MNIST/raw/train-labels-idx1-ubyte.gz
dvc.lock
dvc.yaml
dvclive.json
dvclive/scalars/acc.tsv
dvclive/scalars/loss.tsv
model.pt
params.yaml
predictions.json
requirements.txt
train.py
training_metrics.json
training_metrics/images/missclassified.jpg
training_metrics/report.html
training_metrics/scalars/acc.tsv
training_metrics/scalars/loss.tsv

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its completely use case dependent. we usually set cache: false for files that are small enough to be tracked by git.

If It makes more sense for the VSCode demo, there is nothing wrong with removing the cache: false lines and track everything with DVC

1651005963170 6 0.176
1651005974309 7 0.2092
1651005985776 8 0.2653
1651005996579 9 0.3531
1651006007761 10 0.2096
1651006018496 11 0.3589
1651006029593 12 0.5413
1651006040567 13 0.4922
1651006051500 14 0.6258
10 changes: 10 additions & 0 deletions demo/dvclive/scalars/loss.tsv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
timestamp step loss
1651005963169 6 2.3462729454040527
1651005974308 7 2.169138193130493
1651005985775 8 2.08247447013855
1651005996578 9 1.884981393814087
1651006007760 10 2.1960153579711914
1651006018495 11 1.8895779848098755
1651006029593 12 1.3949280977249146
1651006040566 13 1.6361356973648071
1651006051499 14 1.2490187883377075
214 changes: 0 additions & 214 deletions demo/logs.html
View file
Open in desktop

This file was deleted.

5 changes: 0 additions & 5 deletions demo/logs.json
View file
Open in desktop

This file was deleted.

4 changes: 2 additions & 2 deletions demo/params.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
seed: 473987
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: why removing this? (to make it cleaner?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(to make it cleaner?)

Yep. It's just additional column in table not really a parameter to configure

lr: 0.001
lr: 0.003
weight_decay: 0
epochs: 15
2 changes: 1 addition & 1 deletion demo/predictions.json
View file
Open in desktop

Large diffs are not rendered by default.

11 changes: 4 additions & 7 deletions demo/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
dvc[s3]==2.10.1
dvclive==0.6.0
torch
torchvision
ruamel.yaml
matplotlib
numpy
dvc==2.10.2
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
dvclive[image]==0.7.3
shcheklein marked this conversation as resolved.
Show resolved Hide resolved
torch==1.11.0
torchvision==0.12.0
Loading