using load_state_dict for variational GP in BO loop

[ToennisStef]

Hi
want to set up a BO loop for a optimization Problem with an unknown constraint.
The problem can be understood as optimizing the inputs to a Simulation Problem. At certain input values the Simulation crashes and i don't get any objective value in return.

Im am using a custom variational GP for classifing the Inputs (Simulation Succesful - Simulation Crashed).
I further want to accelerate the training of the model. Therefore i wanted to load the state_dict from the previous run and start the model training from the previously found hyperparameters. But this is not quite straightforward with the variational GP model.
Since I always add an inducing point (training datapoint) for the variational GP after each BO iteration step the loading of the state_dict fails since the dimensionality of the hyperparameters change in each Iteration. Therefore i train the model each iteration "from scratch".

Did somebody already encountered this problem and can help me here?
I am thankfull for any help!
Best regards.

[Balandat]

Hmm why do you add an inducing point for each training data point? What specific model are you using?

Independent of the above, since state dicts are just dictionary of tensor objects, you could write some helper function that pre-processes the previously saved state-dict by adding the additional element to the relevant tensors and then load that into your new model. You could do that manually, or implement it as load_state_dict on your model (a pytorch module), assuming you don't need additional information on top of the old state dict.

[ToennisStef]

Thanks for the response.
This is the model i am using:

class GPClassificationModel(ApproximateGP, GPyTorchModel):
    def __init__(self, train_x, train_y):
        self.train_inputs = (train_x,)
        self.train_targets = train_y

        variational_distribution = CholeskyVariationalDistribution(train_x.size(0))
        variational_strategy = VariationalStrategy(
            self, train_x, variational_distribution
        )
        super(GPClassificationModel, self).__init__(variational_strategy)

        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.MaternKernel(ard_num_dims=train_x.shape[-1]))
        self.likelihood = gpytorch.likelihoods.BernoulliLikelihood()

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

I actually took the model form #640
I tought using all available data as inducing points will increase the accuracy of the model thats why i am using an inducing point for each data point.
Thank you very much! i think i might try to implenent this function :)
Best regards!