Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plots: make output structure consistent between plot types #326

Closed
Tracked by #223
dberenbaum opened this issue Oct 14, 2022 · 2 comments
Closed
Tracked by #223

plots: make output structure consistent between plot types #326

dberenbaum opened this issue Oct 14, 2022 · 2 comments
Assignees
Labels
A: log_image Area: `live.log_image`

Comments

@dberenbaum
Copy link
Collaborator

Follow up to #322.

The difference in behavior between log_image and log_sklearn_plot bothers me:

├── plots
│   ├── images
│   │   ├── 0
│   │   │   └── img.png
│   │   └── 1
│   │       └── img.png
│   └── sklearn
│       └── confusion_matrix.json

Now that they both get saved under plots, it seems more odd that they have such different behavior and output structure. images create a new folder for each step, while sklearn throws an exception when used with multiple steps. #271 will only make this more complicated.

For 1.0, dvclive should be more consistently opinionated. Since the focus has shifted a bit from serverless live tracking to more of a convenience for dvc logging, I would vote to overwrite plots at each step since this is how tools like dvc plots diff, vs code, and studio expect them (this was my mistake pushing for the nested structure originally). That would mean the output would look like:

├── plots
│   ├── images
│   │   └── img.png # flatten and overwrite at each step.
│   └── sklearn
│       └── confusion_matrix.json # enable overwriting at each step.
@daavoo
Copy link
Contributor

daavoo commented Oct 17, 2022

I think it makes sense as default behavior, but doesn't really fix the use case of wanting to visualize image updates across epochs (iterative/vscode-dvc#1640).

A nested structure makes sense for that use case and anyhow if users want that structure they can manually generate it (and we can document it) as follows:

live.log_image(f"{live.get_step()}/img.png", img)

@daavoo daavoo added the A: log_image Area: `live.log_image` label Oct 17, 2022
@daavoo daavoo mentioned this issue Oct 17, 2022
13 tasks
@daavoo daavoo self-assigned this Oct 17, 2022
@daavoo daavoo added this to DVC Oct 18, 2022
@daavoo daavoo moved this to Backlog in DVC Oct 18, 2022
@daavoo daavoo moved this from Backlog to In Progress in DVC Oct 18, 2022
@dberenbaum
Copy link
Collaborator Author

live.log_image(f"{live.get_step()}/img.png", img)

What about live.log_image(f"img/{live.get_step()}.png", img)? That gives me all steps in a single dir, which is easier for me to browse and easier for us to handle in VS Code, Studio, etc.

daavoo added a commit that referenced this issue Oct 27, 2022
It was initially introduced for supporting different logging format between step and not step updates.

For `live.log_image`, "step" mode now overwrites the path instead of creating subfolder by step.

For `live.log`, the "no step" was meant to not generate the `.tsv` file but only the `.json`.
Added a public property `summary` so "no step" scenarios can work as follows:

```
live = Live()

live.summary["foo"] = 1
live.make_summary()
```

Closes #326

Apply suggestions from code review

Co-authored-by: Paweł Redzyński <[email protected]>
daavoo added a commit that referenced this issue Oct 27, 2022
…s. (#331)

It was initially introduced for supporting different logging format between step and not step updates.

For `live.log_image`, "step" mode now overwrites the path instead of creating subfolder by step.

For `live.log`, the "no step" was meant to not generate the `.tsv` file but only the `.json`.
Added a public property `summary` so "no step" scenarios can work as follows:

```
live = Live()

live.summary["foo"] = 1
live.make_summary()
```

Closes #326

Apply suggestions from code review

Co-authored-by: Paweł Redzyński <[email protected]>

Co-authored-by: Paweł Redzyński <[email protected]>
@daavoo daavoo closed this as completed Oct 28, 2022
Repository owner moved this from In Progress to Done in DVC Oct 28, 2022
daavoo added a commit that referenced this issue Oct 31, 2022
…s. (#331)

It was initially introduced for supporting different logging format between step and not step updates.

For `live.log_image`, "step" mode now overwrites the path instead of creating subfolder by step.

For `live.log`, the "no step" was meant to not generate the `.tsv` file but only the `.json`.
Added a public property `summary` so "no step" scenarios can work as follows:

```
live = Live()

live.summary["foo"] = 1
live.make_summary()
```

Closes #326

Apply suggestions from code review

Co-authored-by: Paweł Redzyński <[email protected]>

Co-authored-by: Paweł Redzyński <[email protected]>
daavoo added a commit that referenced this issue Nov 4, 2022
…s. (#331)

It was initially introduced for supporting different logging format between step and not step updates.

For `live.log_image`, "step" mode now overwrites the path instead of creating subfolder by step.

For `live.log`, the "no step" was meant to not generate the `.tsv` file but only the `.json`.
Added a public property `summary` so "no step" scenarios can work as follows:

```
live = Live()

live.summary["foo"] = 1
live.make_summary()
```

Closes #326

Apply suggestions from code review

Co-authored-by: Paweł Redzyński <[email protected]>

Co-authored-by: Paweł Redzyński <[email protected]>
daavoo added a commit that referenced this issue Nov 4, 2022
…s. (#331)

It was initially introduced for supporting different logging format between step and not step updates.

For `live.log_image`, "step" mode now overwrites the path instead of creating subfolder by step.

For `live.log`, the "no step" was meant to not generate the `.tsv` file but only the `.json`.
Added a public property `summary` so "no step" scenarios can work as follows:

```
live = Live()

live.summary["foo"] = 1
live.make_summary()
```

Closes #326

Apply suggestions from code review

Co-authored-by: Paweł Redzyński <[email protected]>

Co-authored-by: Paweł Redzyński <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: log_image Area: `live.log_image`
Projects
No open projects
Archived in project
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants