Dropping experiments

[dberenbaum]

Users who are running experiments may find the number of experiments and checkpoints creates clutter, or that they have made a mistake or otherwise don't want to keep or share certain experiments. A user may want to drop experiments. For example:

• User tries some new parameter or continues training checkpoints to find that performance only gets worse and wants to drop one or more of the most recent experiments or checkpoints.
• User makes a typo and isn't interested in keeping the latest experiment run.
• User no longer needs older experiments that explored a parameter space that is no longer under consideration.
• User queues experiments and decides not to run one or more of them.

dvc exp gc may be useful for dropping older experiments but has some drawbacks:

• Can't drop newer experiments
• Can't specify what experiments to keep
• Can't specify particular experiments or revisions to drop

A proposed command to enable users to delete experiments could look like:
dvc exp remove experiment/rev [experiment/rev ...]

Questions:

• What if someone wants to remove experiments in bulk, especially in cases like clearing the queue or dropping all experiments since the last commit? Could this be handled with dvc exp reset or dvc exp run --reset? Would a git-like reset to drop everything after a specified experiment/revision make sense?
• What about removing pushed experiments from a remote? I think this is less essential, but could be supported at some point with a --remote option.

Thoughts @pmrowla @dmpetrov ?

[dberenbaum]

See also #5413

[pmrowla]

A proposed command to enable users to delete experiments could look like:
dvc exp remove experiment/rev [experiment/rev ...]

This is fine and makes sense to have.

• What if someone wants to remove experiments in bulk, especially in cases like clearing the queue or dropping all experiments since the last commit?

It sounds like what we want here is just something like

dvc exp gc --rev HEAD

which would remove everything except for experiments related to the current commit (or any other specified revision)

for clearing the queue

dvc exp gc --all-commits -w

will clear the queue and keep everything else, but it probably makes sense to have a cleaner shortcut like

dvc exp rm --queue

• What about removing pushed experiments from a remote? I think this is less essential, but could be supported at some point with a --remote option.

Not sure if removing pushed experiments from a remote should be tied to exp rm or if it should just be a push flag the same way that you delete remote git branches using git push -d:

dvc exp push -d/--delete [email protected]/... <exp_name> [<exp_name>...]

[jorgeorpinel]

While I'm not opposed to possibly useful utilities like this, as I understand it ephemeral experiments are meant to be messy. You can just ignore all the clutter and apply or branch the good ones — maybe focus on being able to apply several exps to a separate "clean" branch instead? Up to you

Would it remove the cached outputs and/or run-cache entry too though? I remember we also wanted a new command (or redesign of dvc gc) to allow removing specific data from cache/remote and this could have some overlap with that.

[pmrowla]

Would it remove the cached outputs and/or run-cache entry too though? I remember we also wanted a new command (or redesign of dvc gc) to allow removing specific data from cache/remote and this could have some overlap with that.

Removing cached outputs still requires dvc gc, there's a dvc gc flag for for preserving cached outs that are used by experiments. Basically removing cached outputs when removing an experiment would still require doing the gc logic to check if that cache object is referred to by anything else in DVC (and is slow), so it's better for us to just require the explicit dvc gc when the user wants to clean out their cache.

[dberenbaum]

While I'm not oppose to possibly useful utilities like this, as I understand it ephemeral experiments are meant to be messy.

Good point, but I think that's also the reason it's needed. It's realistic to have dozens or even hundreds of experiments and wanting to compare all of them, but if you've made some mistakes along the way and also have a bunch of unneeded experiments, it can get messy to look at the results. I think users will frequently want a clean-looking dvc exp show output to share the results of their experimentation, and being able to clean up the mess first may be important.

[jorgeorpinel]

there's a dvc gc flag for for preserving cached outs that are used by experiments

@pmrowla you mean a new flag in dvc gc? What is that? (Can't find it in https://github.com/iterative/dvc/wiki/Experiments) Thanks

[pmrowla]

@jorgeorpinel there's an --all-experiments flag for preserving experiments outs (works the same as all of the other gc flags)

dvc gc --help
usage: dvc gc [-h] [-q | -v] [-w] [-a] [-T] [--all-commits]
              [--all-experiments] [-c] [-r <name>] [-f] [-j <number>]
              [-p [<paths> [<paths> ...]]]
...
  -w, --workspace       Keep data files used in the current workspace.
  -a, --all-branches    Keep data files for the tips of all Git branches.
  -T, --all-tags        Keep data files for all Git tags.
  --all-commits         Keep data files for all Git commits.
  --all-experiments     Keep data files for all experiments.

The original cmd reference PR included the docs changes for dvc gc: https://github.com/iterative/dvc.org/pull/1926/files#diff-11bc293c440dc80b6e4a37bcac53dd1867ce4a3c86edcfe2262914750ee1c648

[jorgeorpinel]

Ah cool! Will definitely recover that docs change if it's in your original PR (still working on that 😅). Thanks @pmrowla