Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow running the benchmark workflow with different parameter sets #405

Open
rcannood opened this issue Mar 14, 2024 · 0 comments
Open

Allow running the benchmark workflow with different parameter sets #405

rcannood opened this issue Mar 14, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@rcannood
Copy link
Member

The run_benchmark needs to be able to run methods with multiple parameter settings. Perhaps with the addition of something like this?

        - name: "--parameter_sets"
          type: file
          description: |
            A yaml file containing the parameter sets for each method. Example format:

            - method_id: ...
              run_id: ...
              parameters:
                ...
            - method_id: ...
              run_id: ...
              parameters:
                ...

If the run_id is not set, it defaults to the method_id. If no parameters are specified, it defaults to {}.

Alternative formats are possible, such as:

        - name: "--parameter_sets"
          type: file
          description: |
            A yaml file containing the parameter sets for each method. Example format:

           method_1:
             param_set1:
               par_a: ...
               par_b: ...
             param_set2:
               par_a: ...
               par_b: ...
           method_2:
             param_set1:
               par_a: ...
               par_b: ...
             param_set2:
               par_a: ...
               par_b: ...

The paramset_id / run_id would need to be stored in the .uns, and downstream components would need to be adapted accordingly, so let's think about how the api specs would change and which downstream components are affected.

@rcannood rcannood added the enhancement New feature or request label Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants