Experiments proof of concept #4199
Conversation
|
Not really ready yet, but some basic things work: |
* only includes metrics + params (no code/data information)
* just shows combined output from `metrics diff` and `plots diff` for now
* experiment related commands will be accessible but hidden from the default command help messages
* experiments only enabled in test environment (DVC_TEST) or when core.experiments config option is true
| experiments_parser = subparsers.add_parser( | ||
| "experiments", | ||
| parents=[parent_parser], | ||
| description=append_doc_link(EXPERIMENTS_HELP, "experiments"), | ||
| formatter_class=argparse.RawDescriptionHelpFormatter, | ||
| ) |
There was a problem hiding this comment.
help param intentionally omitted here for now so that the command is hidden from the default dvc/dvc --help usage output
There was a problem hiding this comment.
In the long term, we should think of some way for this. Even though we had it hidden, users were still trying out new 1.0 docs in 0.94 dvc. Something like [experimental] flags come to mind but would be difficult to read it at this higher-level (add_parser), or could introduce DVC_EXPERIMENTAL_XXX_ENABLE envs or maybe, even simpler name mangling till it's ready.
Not a big issue right now, just sharing the issue that I had. 🙂. Definitely something we should consider, as we mostly work on HEAD.
pmrowla
changed the title
|
First post has been updated with details. All behavior changes are disabled unless |
|
This feature piqued my interest. Is there a writeup somewhere of what an "experiment" is, and how it relates to a pipeline and its results? I didn't see any docs/explanation in the PR itself. |
Related to #2799.
❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.
❌ I will check DeepSource, CodeClimate, and other sanity checks below. (We consider them recommendatory and don't expect everything to be addressed. Please fix things that actually improve code or fix bugs.)
Thank you for the contribution - we'll try to review it as soon as possible. 🙏
Implements initial POC support using a single git clone. UI is rough expected to change a lot.
dvc repro -e/--experiment- creates a new experiment. Experiment is derived from the current HEAD commit in the user's repo and contains any changes in the user's current workspace.dvc experiments show- show experiment table. By default table includes the user's workspace, the current HEAD commit and any experiments derived from HEAD.--all-commits/--all-tags/--all-branchescan be used to display more experiments.dvc experiments diff- diff 2 experiments. Currently just concats the output fromdvc metrics diffanddvc plots diff.dvc experiments checkout <experiment_rev>- applies the changes from the specified experiment to the user's current workspace. If the changes would generate a git merge conflict, this command does nothing.--forcecan be used to replace the user's current workspace with the experiment changes. This has only been tested in cases where user checks out an experiment into a workspace with the original commit the experiment was derived from.Note: using
--forcecurrently does agit reset --hardin your workspace, the behavior you most likely want is togit stashyour workspace changes before usingdvc experiments checkoutdvc --helpoutput, but runningdvc experiments --helpcan be used to show the usage for the experiments commands.core.experimentsconfig option is set to true (or if DVC is run inside a testing/CI environment).General thoughts/questions for next iteration:
experiments diff- mimicmetrics/params diffformats or use theshowtable with added deltas?experiments checkoutfeels a bit awkward to use, maybe it should be arepro --experiment <rev>command to reproduce a specific experiment instead?