Making BoTorch's (Discrete) Multifidelity BO Work under Service API

[Abrikosoff]

Hi,

I have another quick question regarding the possibility of do multifidelity BO under the Service API. I now have a use case for BoTorch's multifidelity BO; the tutorials are clear, but as I have already been using Ax's Service API for some time now (and appreciating the ease of use it gives), I was wondering whether the MF workflow can be transplanted to work under the Service API? I'm guessing yes, since on the surface it seems that we only need to pass some acquisition functions (can't call optimize_acqf in Service though, but there are other ways to pass in a qMultiFidelityHypervolumeKnowledgeGradient I guess?)? But it could very well be I'm missing something, so I would really appreciate some expert opinions here!

[Balandat]

Hi @Abrikosoff, yes, I believe this should be possible. I don't have a full tutorial handy, but you should be able to assemble this using a custom GenerationStrategy that uses the modular BoTorch model setup. For this you'll need to:

1. Define "fidelity" parameters as part of the search space when creating the experiment (if you're interested in the discrete case that means setting is_fidelity=True and providing a target_value that is your target fidelity)
2. Construct a custom GenerationStrategy that according to the modular BoTorch model tutorial that uses the Models.BOTORCH_MODULAR model type and specifies "acquisition_class": MultiFidelityAcquisition, "acquisition_options": {"fidelity_weights": {fidelity_param_idx: weight}, "cost_intercept": cost_intercept} and "surrogate": Surrogate(SingleTaskMultiFidelityGP) in the model_kwargs of the GenerationStep. Here fidelity_weights and cost_intercept correspond to the fidelity_weights and fixed_cost args to AffineFidelityCostModel from the botorch multi-fidelity tutorial. This is a bit annoying since fidelity_param_idx is not known at experiment creation time, but this is usually the last of the parameters - we're working on improving the user interface for this.
3. Instantiate the AxClient with the custom GenerationStrategy via the generation_strategy argument.

Hope that helps, let us know how it goes!

[Abrikosoff]

Hi @Abrikosoff, yes, I believe this should be possible. I don't have a full tutorial handy, but you should be able to assemble this using a custom GenerationStrategy that uses the modular BoTorch model setup. For this you'll need to:

1. Define "fidelity" parameters as part of the search space when creating the experiment (if you're interested in the discrete case that means setting [`is_fidelity=True`](https://github.com/facebook/Ax/blob/ba6eb81324f35522054a9bac7b0088b2243f88d9/ax/service/ax_client.py#L281-L282) and providing a `target_value` that is your target fidelity)

2. Construct a custom `GenerationStrategy` that according to the [modular BoTorch model tutorial](https://github.com/facebook/Ax/blob/ba6eb81324f35522054a9bac7b0088b2243f88d9/tutorials/modular_botax.ipynb) that uses the `Models.BOTORCH_MODULAR` model type and specifies `"acquisition_class": MultiFidelityAcquisition`, `"acquisition_options": {"fidelity_weights": {fidelity_param_idx: weight}, "cost_intercept": cost_intercept}` and `"surrogate": Surrogate(SingleTaskMultiFidelityGP)` in the `model_kwargs` of the `GenerationStep`. Here `fidelity_weights` and `cost_intercept` correspond to the `fidelity_weights` and `fixed_cost` args to `AffineFidelityCostModel` from the botorch multi-fidelity tutorial. This is a bit annoying since `fidelity_param_idx` is not known at experiment creation time, but this is usually the last of the parameters - we're working on improving the user interface for this.

3. Instantiate the `AxClient` with the custom `GenerationStrategy` via the [`generation_strategy`](https://github.com/facebook/Ax/blob/ba6eb81324f35522054a9bac7b0088b2243f88d9/ax/service/ax_client.py#L191) argument.

Hope that helps, let us know how it goes!

Hi @Balandat , first of all thanks for your very helpful tips regarding this! I'm now halfway though already; the custom GenerationStrategy is not a problem, but when I try to define a fidelity parameter as part of the search space (in the Service API fashion), as so:

        {
            "name": "fidelity",
            "type": "choice",
            "value": [0.5, 0.75, 1.0],
            "is_fidelity": True,
            "target_value": 1.0,  
        },
        

and run next experiment, I get the error ValueError: Cannot choice-encode fidelity parameter fidelity. However, if I try the original definition without the type parameter, this fails even at the definition step. Intuitively, I thought that this parameter would have been most accurately classed as a Choice parameter, but that does not seem to be the case.

Any idea of what I am doing wrong here? Thanks in advance!

Edit: After performing a more through search of the Issues, I came across this issue: 979. From the fact that the error messages are the same in both cases, would it be correct to assume that support for discrete fidelities are not yet implemented for the Service API?

[saitcakmak]

Closing this since the discussion moved to #2514