Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiments table: action per item to run experiments #1135

Closed
shcheklein opened this issue Dec 7, 2021 · 1 comment
Closed

Experiments table: action per item to run experiments #1135

shcheklein opened this issue Dec 7, 2021 · 1 comment
Labels
A: experiments Area: experiments table webview and everything related enhancement New feature or request

Comments

@shcheklein
Copy link
Member

shcheklein commented Dec 7, 2021

Originated at #1120 (comment)

Single new experiment

A single experiment to queue, rather than how to submit a whole batch of experiments. For example, there could be a GUI that shows all the parameters, and users can edit their values and click "queue experiment" without needing to edit yaml files or run terminal commands.

We should be able to run experiments based on any of the experiments in the table or workspace.

Batch

Concerns:

As @daavoo alluded, when it comes to larger scale hyperparameter tuning, users probably can't avoid doing this programmatically since there are so many experiments to queue, the methods of defining the hyperparameter space are too varied, and frequently they can't even all be queued up front because later iterations base their values on the results of previous iterations.

I think it makes sense for advanced scenarios and especially when experiments are expensive to run (deep learning), in case of "classical" DS random forest, xgboost it's easier to run a small grid batch .

I would add some basic way to specify a list of values at least and/or simple start-step-end UI.

@shcheklein shcheklein added A: experiments Area: experiments table webview and everything related enhancement New feature or request labels Dec 7, 2021
@shcheklein shcheklein self-assigned this Dec 7, 2021
@shcheklein shcheklein changed the title Experiments table: action per item to batch experiments Experiments table: action per item to run experiments Dec 8, 2021
@shcheklein shcheklein removed the triage label Dec 8, 2021
@shcheklein shcheklein removed their assignment Dec 8, 2021
@rogermparent
Copy link
Contributor

While #1267 is slightly more generic than this Issue, they're so close that it's probably best to merge the two. I've already added the info into the other ticket, so this one can safely be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: experiments Area: experiments table webview and everything related enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants