Persisting experiments from table: share as a branch and commit and share

[SoyGema]

Context

Running experiments from workshop. Find a parameter that impacts model performance. Testing persistance UI table features. In table , access to context menu, and then click in share as a branch

Share as a branch

Pop up menu appears and git checkout is done

ISSUE : Inconsistency of branches from CLI and VSCode. CLI tells me that Im in that branch, but I can´t find it . Unclear if I have to publish first or if the extension does it for me . I´m probably missing something . Can you help ?

[SoyGema]

Commit and share (1)

From the UI under the 7f33f67 experiment, access to context menu and click Commit and share. VSCode shows a message . It pops p an error . Apparently is expecting the experiment above he one I want to commit . 479e5d3

Don´t know if this helps , but I checked that I commited all things before this
Unclear if this is related, but under left menu I selected the experiment and clicked apply to Workspace. The following error was displayed.

Commit and share(2)

From the UI under the 479e5d3 experiment, access to context menu and click Commit and share. VSCode shows a message . It pops p an error . Apparently is expecting the experiment to derive from '67a731e' , one that I removed previously from the table .

[SoyGema]

Thanks for having a look at this!
Please don´t hesitate to point me out in the right direction if I did something wrong .
And thanks for bringing persistance to machine learning experiments !

[mattseddon]

Share as a branch

What the command does (happy path) sequentially:

1. Creates a branch from the selected experiment.
2. Applies experiment to workspace.
3. Pushes DVC data to DVC remote.
4. Pushes the Git branch to the Git remote

Looking at the code I just realised that these steps are not outlined to the user. Only the CLI output is shown. Would some more messaging help? I.e "Creating branch..."

ISSUE : Inconsistency of branches from CLI and VSCode. CLI tells me that Im in that branch, but I can´t find it . Unclear if I have to publish first or if the extension does it for me . I´m probably missing something . Can you help ?

Looking closer at this image:

The information you are displaying does not come from this extension. It comes from VS Code's built-in Git extension. Under these circumstances I would first try this button:

[mattseddon]

Commit and share

From your screenshots it looks like you are trying to commit and share an experiment from the previous commit. It seems that it is not possible to apply an experiment from a previous commit onto the workspace using DVC. I'll do some more investigation before raising a bug there 👍🏻.

Edit: iterative/dvc#8481 is already open.

[dberenbaum]

Commit and share

What are the steps for this action? I'm surprised that it runs dvc exp apply. It's not clear to me that this action will try to overwrite my workspace with the results of that experiment, especially since it's sandwiched between branching options.

[mattseddon]

@dberenbaum

What commit and share does (happy path) sequentially:

Applies experiment to workspace.
Pushes DVC data to DVC remote.
Git stage entire workspace.
Git commit.

Order of operation for share as branch: #2855 (comment)

[mattseddon]

[Q] @dberenbaum do you think we should limit these actions to the current commit or keep them available for any of the experiments displayed?

[dberenbaum]

[Q] @dberenbaum do you think we should limit these actions to the current commit or keep them available for any of the experiments displayed?

How much work does that take?

@pmrowla @karajan1001 How much work would it take to address iterative/dvc#8481?

I would rather do the latter, but if it's much easier to disable in VS Code in the short term, then it makes sense.

[shcheklein]

@dberenbaum do we know why we have this limitation? (what is the core reason). I'm not sure I understand what is the difference between commits tbh.

[dberenbaum]

Since the exp ref stashes changes from some baseline commit, it's safe to apply that stash on top of the commit from which it originated. If it is applied on top of some other commit, it could create conflicts or maybe even just unexpected results.

When experiments was first implemented, it was sufficient to apply on top of the baseline commit, and applying on top of other commits might have been considered too dangerous. Now that we see it used more, I think the pros outweigh the cons, especially since we should be able to rollback in case of conflicts.

That does complicate things for VS Code, though, since there's no guarantee the apply will work (the same could be true for a dirty workspace even on top of the baseline commit).

[shcheklein]

It's not a stash though, right? It's more or less a proper hidden commit?

In this sense, there should be not difference to which ref to hard reset to to my mind. It's not like applying on top, it should be more like:

git stash

git revert <ref> # not sure this is the right way to do this, but I think you get the idea. We need to keep the history, but apply an experiment exactly, means change files back if needed

wdyt?

[dberenbaum]

Hey @shcheklein, I moved this discussion back into iterative/dvc#8481.

In the short term, should we limit these actions to the current commit since it seems like there's still product questions on how dvc exp apply should work for other commits?

Edit: sounds like this is not an easy fix on the vs code side, so limiting to the current commit is not necessarily the best approach.

[karajan1001]

[Q] @dberenbaum do you think we should limit these actions to the current commit or keep them available for any of the experiments displayed?

How much work does that take?

@pmrowla @karajan1001 How much work would it take to address iterative/dvc#8481?

I would rather do the latter, but if it's much easier to disable in VS Code in the short term, then it makes sense.

about 1-2 days?

[shcheklein]

@karajan1001 @dberenbaum any thought / further actions on this?

[dberenbaum]

Nothing planned yet since @karajan1001 still has a few queue bugs he's working on AFAIK. I think there's still some product questions we can address in the meantime in iterative/dvc#8481.

[mattseddon]

Should be fixed by iterative/dvc#8481. More than likely that this functionality will either:

1. Get pushed back into DVC
2. Get dropped completely.

Closing for now.