Improve Custom Plots

[julieg18]

Tasks are sorted by importance.

Tasks

Preview Give feedback

1. Remove Trends section, instead having trends plots be plots you can create in "Custom". This will simplify the code logic and duplication.
2. Update README/Walkthrough
3. Remove the possibility of a user selecting options that create a plot that is already made. Currently an error popup will appear if a user does this. (See #3342 discussion)
4. Add commit data into custom plots
5. Update get started screen (see Fix plots "get started" walkthrough #3382 (comment))
6. Add command pallete commands
7. Have metric vs param plots use scatter plots instead of linear plots
8. Make sure that custom spec does not generate errors in webview/src/plots/components/App.test.tsx (Move custom plot spec creation to backend #3575 (comment))
9. Delete old Persistance metrics state (Cleanup plots workspace state #3518)
10. Adjust custom plots order state (Cleanup plots workspace state #3518)
11. Investigate loading section test (Move trends plots inside of "Custom" section #3404 (comment))
12. Update walkthrough images after other PRs are merged (https://github.com/iterative/vscode-dvc/pull/3404/files#r1133343393)
13. Check if plot spec creation can be moved to extension (Move trends plots inside of "Custom" section #3404 (comment))
14. Consolidate type checking to the smallest space/amount of functions (Move trends plots inside of "Custom" section #3404 (comment)) (Most of this has been done already but more to do)
15. Add code to add 1 trend and 1 metric vs param custom plot to e2e test (https://github.com/iterative/vscode-dvc/pull/3404/files#r1133364032)
16. Improve Jest testing on plots section (see #3342 comment)

Followup to #3342

[mattseddon]

Addition of a custom plot with no experiments present breaks the empty state screen:

Screen.Recording.2023-03-02.at.12.26.27.pm.mov

Edit: doesn't seem like the empty screen is accessible at all now.

[mattseddon]

Lower priority but we should change "iteration" to step and make it 0 based (same as DVCLive) for the checkpoint trends plots:

[julieg18]

Add commit data into custom plots

@mattseddon, what do you mean by commit data? I know we have rows that we refer to as commits but custom plots rely on experiments 🤔

[mattseddon]

Add commit data into custom plots

@mattseddon, what do you mean by commit data? I know we have rows that we refer to as commits but custom plots rely on experiments 🤔

Regarding custom plots (not trends) commits and experiments contain the same data. Is there a reason to exclude the baselines (commits) from custom plots?

[dberenbaum]

I mentioned in yesterday's discussion that I don't think we should be using linear plots. There can be multiple points per parameter with different metric values, which makes the linear plot look odd. For example, using this table of experiments:

We get custom plots that look like this:

I think we should use something like the smooth template we have for other plots, where we show the points but generate a smoothed line:

[shcheklein]

Sounds reasonable to me @dberenbaum , thanks for the feedback!

[daavoo]

I think we should use something like the smooth template we have for other plots, where we show the points but generate a smoothed line:

I think we should not show a line at all, and use pure scatter plot template, like:

https://vega.github.io/vega-lite/examples/point_color_with_shape.html

[dberenbaum]

@daavoo I thought it might be helpful to have a trend line since it could be noisy with many experments, and a typical use case is to try to see a rough pattern of how the metric varies with changes to the hyperparameter. Any reason you prefer not to show a line at all?

To clarify, unlike the screenshot I included above, I would only expect a single line on these plots.

[daavoo]

I thought it might be helpful to have a trend line since it could be noisy with many experments, and a typical use case is to try to see a rough pattern of how the metric varies with changes to the hyperparameter.

That sounds like parallel coordinates 😅

Any reason you prefer not to show a line at all?

No strong reason. Just don't try to do too much in one type of plot.

It just feels cleaner / aligned with existing tools, at least as default (wandb does allow to optionally insert line into scatter plot, but it also provides different options on how to add the line).

[dberenbaum]

Okay, I don't see it as a blocker, so I'm fine to go with a scatter plot for now. We could always add either a loess or linear regression line later and have an option to show/hide it.

[julieg18]

Tried three different plots:

1. A smooth plot based off our dvc templates
2. scatter plot
3. scatter plot with a regression line

The smooth plot seemed kind of buggy though I'm not sure if it's due to the experiment values or me configuring the plot incorrectly. Scatter plots seem simple enough to create though:

Screen.Recording.2023-04-03.at.6.45.14.PM.mov

Screen.Recording.2023-04-03.at.6.46.35.PM.mov

What do we think will work best? cc @iterative/vs-code, @shcheklein