executor: show experiment results

[dberenbaum]

Meta-issue. Please link related tickets back to this one.

User story

I want to see the results of my executor's experiments on my local machine.

Requirements

• dvc exp show at least has an option to show the results from an experiment completed on an executor without other steps.
• dvc exp show at least has an option to show checkpoints from an experiment running on an executor without other steps.
• dvc executor destroy at least has an option to pull experiment results before destroying the executor instance.

Out of scope

• It's only necessary to show results when requested, not continuously monitor progress.

[dberenbaum]

What's the best way to share the experiment commit and the related dvc-tracked data (see #6265 and #6266)? Should the dvc-tracked data get pulled along with the experiment commit?

[dberenbaum]

May need dvc exp collect or similar to retrieve all the data before doing dvc exp show. To start, we can pull everything (git repo plus dvc-tracked data).

[dberenbaum]

Related request: if experiments have been pushed somewhere, it would be useful to be able to see those results without having to pull all the experiments. See https://discord.com/channels/485586884165107732/485596304961962003/913036645186568192 for an example. Several team members may all be pushing experiments, and we want to compare all the results without having to download everything about the experiments.

[pmrowla]

It won't be possible to show the results without pulling them (in git), but we could automate it (and skip pulling the DVC cache in this case).

This would also require configuring the default git repo to check for pushed experiments. We could consider defaulting to always check origin, but if users are using a pull request workflow, origin is probably a personal fork, where the actual check should be against some different upstream git remote.

[dberenbaum]

It's probably more useful to add a view like this in Studio to see experiments, although I know Studio doesn't currently support dvc exp features. Cc @shcheklein @tapadipti What do you think?

[shcheklein]

It feels that Studio UI is a separate feature to discuss here. It won't replace the need for dvc exp show to be able to show what's happening on remote machines unless I'm missing something.

[dberenbaum]

Needs rescoping