-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
exp show: checkpoint experiment summary #6194
Comments
Related (#6174 (comment)):
To me it's like if I queue an experiment based on |
This is entirely dependent on what the user's stage code is doing. We just capture the state of the user's workspace, in theory the user can write to any file they want to before we capture that state, including params files. And as we've discussed before, with the way the exp feature actually works, in theory users could leverage DVC to generate commits within an experiment ref that are not actually checkpoint experiments at all (i.e. just use them as a DVC-managed git branches). We would still want |
Won't any git-tracked or dvc-tracked changes generate a new experiment ref?
Good point, but is there any mechanism other than checkpoints where users could generate multiple commits to the same experiment? Obviously, it's possible users could manually mess with the experiments, but this seems like it would be an unexpected use. |
Not currently, but there have been discussions about potentially allowing users to use the make_checkpoint signaling without |
For what purpose? It seems like holding dependencies fixed is the defining feature of an experiment. If parameters or dependencies change within an experiment, what is the meaning of an experiment? Would this ever be desirable? |
Being able to track any kind of intermediate state during a stage or pipeline seems like it would be useful, it doesn't have to be restricted to intermediate checkpoints or circular deps. |
Even in checkpoints that are breaking down a multi-part stage without any kind of circular dependency, why would parameters change? Also, this scenario probably breaks all kinds of things in A hyperparameter search script is one scenario where parameters might get changed within a stage, but my personal preference would be for each result to be tracked as a separate experiment, although I'm not sure how that would work. Otherwise, each checkpoint would include the history of all previous checkpoints even though they would be independent events in that scenario. Also, making them modified/independent experiments seems more consistent with current behavior where any parameter modifications generate a modified experiment. |
cc @daavoo |
In
dvc exp show
, the last checkpoint for an experiment also contains the experiment name:It would be useful to have the experiment on a separate row to keep the experiment and individual checkpoints distinct.
The table combines info that's really about the experiment with info that's about the individual checkpoint. For example, parameters should be static for all checkpoints in an experiment, right?
Also, it opens up options to show things like summary metrics for an experiment, or to collapse the checkpoints if the user just wants a summary of each experiment.
The text was updated successfully, but these errors were encountered: