CHANGELOG.md

Changelog

The release log for BoTorch.

[0.8.2] - Feb 23, 2023

Compatibility

• Require PyTorch >= 1.12 (#1699).

New Features

• Introduce pathwise sampling API for efficiently sampling functions from (approximate) GP priors and posteriors (#1463).
• Add OneHotToNumeric input transform (#1517).
• Add get_rounding_input_transform utility for constructing rounding input transforms (#1531).
• Introduce EnsemblePosterior (#1636).
• Inducing Point Allocators for Sparse GPs (#1652).
• Pass gen_candidates callable in optimize_acqf (#1655).
• Adding logmeanexp and logdiffexp numerical utilities (#1657).

Other changes

• Warn if inoperable keyword arguments are passed to optimizers (#1421).
• Add BotorchTestCase.assertAllClose (#1618).
• Add sample_shape property to ListSampler (#1624).
• Do not filter out BoTorchWarnings by default (#1630).
• Introduce a DeterministicSampler (#1641).
• Warn when optimizer kwargs are being ignored in BoTorch optim utils _filter_kwargs (#1645).
• Don't use functools.lru_cache on methods (#1650).
• More informative error when someone adds a module without updating the corresponding rst file (#1653).
• Make indices a buffer in AffineInputTransform (#1656).
• Clean up optimize_acqf and _make_linear_constraints (#1660, #1676).
• Support NaN max_reference_point in infer_reference_point (#1671).
• Use _fast_solves in HOGP.posterior (#1682).
• Approximate qPI using MVNXPB (#1684).
• Improve filtering for cache_root in CachedCholeskyMCAcquisitionFunction (#1688).
• Add option to disable retrying on optimization warning (#1696).

Bug fixes

• Fix normalization in Chebyshev scalarization (#1616).
• Fix TransformedPosterior missing batch shape error in _update_base_samples (#1625).
• Detach coefficient and offset in AffineTransform in eval mode (#1642).
• Fix pickle error in TorchPosterior (#1644).
• Fix shape error in optimize_acqf_cyclic (#1648).
• Fixed bug where optimize_acqf didn't work with different batch sizes (#1668).
• Fix EUBO optimization error when two Xs are identical (#1670).
• Bug fix: _filter_kwargs was erroring when provided a function without a __name__ attribute (#1678).

[0.8.1] - Jan 5, 2023

Highlights

• This release includes changes for compatibility with the newest versions of linear_operator and gpytorch.
• Several acquisition functions now have "Log" counterparts, which provide better numerical behavior for improvement-based acquisition functions in areas where the probability of improvement is low. For example, LogExpectedImprovement (#1565) should behave better than ExpectedImprovement. These new acquisition functions are
  • LogExpectedImprovement (#1565).
  • LogNoisyExpectedImprovement (#1577).
  • LogProbabilityOfImprovement (#1594).
  • LogConstrainedExpectedImprovement (#1594).
• Bug fix: Stop ModelListGP.posterior from quietly ignoring Log, Power, and Bilog outcome transforms (#1563).
• Turn off fast_computations setting in linear_operator by default (#1547).

Compatibility

• Require linear_operator == 0.3.0 (#1538).
• Require pyro-ppl >= 1.8.4 (#1606).
• Require gpytorch == 1.9.1 (#1612).

New Features

• Add eta to get_acquisition_function (#1541).
• Support 0d-features in FixedFeatureAcquisitionFunction (#1546).
• Add timeout ability to optimization functions (#1562, #1598).
• Add MultiModelAcquisitionFunction, an abstract base class for acquisition functions that require multiple types of models (#1584).
• Add cache_root option for qNEI in get_acquisition_function (#1608).

Other changes

• Docstring corrections (#1551, #1557, #1573).
• Removal of _fit_multioutput_independent and allclose_mll (#1570).
• Better numerical behavior for fully Bayesian models (#1576).
• More verbose Scipy minimize failure messages (#1579).
• Lower-bound noise inSaasPyroModel to avoid Cholesky errors (#1586).

Bug fixes

• Error rather than failing silently for NaN values in box decomposition (#1554).
• Make get_bounds_as_ndarray device-safe (#1567).

[0.8.0] - Dec 6, 2022

Highlights

This release includes some backwards incompatible changes.

• Refactor Posterior and MCSampler modules to better support non-Gaussian distributions in BoTorch (#1486).
  • Introduced a TorchPosterior object that wraps a PyTorch Distribution object and makes it compatible with the rest of Posterior API.
  • PosteriorList no longer accepts Gaussian base samples. It should be used with a ListSampler that includes the appropriate sampler for each posterior.
  • The MC acquisition functions no longer construct a Sobol sampler by default. Instead, they rely on a get_sampler helper, which dispatches an appropriate sampler based on the posterior provided.
  • The resample and collapse_batch_dims arguments to MCSamplers have been removed. The ForkedRNGSampler and StochasticSampler can be used to get the same functionality.
  • Refer to the PR for additional changes. We will update the website documentation to reflect these changes in a future release.
• #1191 refactors much of botorch.optim to operate based on closures that abstract away how losses (and gradients) are computed. By default, these closures are created using multiply-dispatched factory functions (such as get_loss_closure), which may be customized by registering methods with an associated dispatcher (e.g. GetLossClosure). Future releases will contain tutorials that explore these features in greater detail.

New Features

• Add mixed optimization for list optimization (#1342).
• Add entropy search acquisition functions (#1458).
• Add utilities for straight-through gradient estimators for discretization functions (#1515).
• Add support for categoricals in Round input transform and use STEs (#1516).
• Add closure-based optimizers (#1191).

Other Changes

• Do not count hitting maxiter as optimization failure & update default maxiter (#1478).
• BoxDecomposition cleanup (#1490).
• Deprecate torch.triangular_solve in favor of torch.linalg.solve_triangular (#1494).
• Various docstring improvements (#1496, #1499, #1504).
• Remove __getitem__ method from LinearTruncatedFidelityKernel (#1501).
• Handle Cholesky errors when fitting a fully Bayesian model (#1507).
• Make eta configurable in apply_constraints (#1526).
• Support SAAS ensemble models in RFFs (#1530).
• Deprecate botorch.optim.numpy_converter (#1191).
• Deprecate fit_gpytorch_scipy and fit_gpytorch_torch (#1191).

Bug Fixes

• Enforce use of float64 in NdarrayOptimizationClosure (#1508).
• Replace deprecated np.bool with equivalent bool (#1524).
• Fix RFF bug when using FixedNoiseGP models (#1528).

[0.7.3] - Nov 10, 2022

Highlights

• #1454 fixes a critical bug that affected multi-output BatchedMultiOutputGPyTorchModels that were using a Normalize or InputStandardize input transform and trained using fit_gpytorch_model/mll with sequential=True (which was the default until 0.7.3). The input transform buffers would be reset after model training, leading to the model being trained on normalized input data but evaluated on raw inputs. This bug had been affecting model fits since the 0.6.5 release.
• #1479 changes the inheritance structure of Models in a backwards-incompatible way. If your code relies on isinstance checks with BoTorch Models, especially SingleTaskGP, you should revisit these checks to make sure they still work as expected.

Compatibility

• Require linear_operator == 0.2.0 (#1491).

New Features

• Introduce bvn, MVNXPB, TruncatedMultivariateNormal, and UnifiedSkewNormal classes / methods (#1394, #1408).
• Introduce AffineInputTransform (#1461).
• Introduce a subset_transform decorator to consolidate subsetting of inputs in input transforms (#1468).

Other Changes

• Add a warning when using float dtype (#1193).
• Let Pyre know that AcquisitionFunction.model is a Model (#1216).
• Remove custom BlockDiagLazyTensor logic when using Standardize (#1414).
• Expose _aug_batch_shape in SaasFullyBayesianSingleTaskGP (#1448).
• Adjust PairwiseGP ScaleKernel prior (#1460).
• Pull out fantasize method into a FantasizeMixin class, so it isn't so widely inherited (#1462, #1479).
• Don't use Pyro JIT by default , since it was causing a memory leak (#1474).
• Use get_default_partitioning_alpha for NEHVI input constructor (#1481).

Bug Fixes

• Fix batch_shape property of ModelListGPyTorchModel (#1441).
• Tutorial fixes (#1446, #1475).
• Bug-fix for Proximal acquisition function wrapper for negative base acquisition functions (#1447).
• Handle RuntimeError due to constraint violation while sampling from priors (#1451).
• Fix bug in model list with output indices (#1453).
• Fix input transform bug when sequentially training a BatchedMultiOutputGPyTorchModel (#1454).
• Fix a bug in _fit_multioutput_independent that failed mll comparison (#1455).
• Fix box decomposition behavior with empty or None Y (#1489).

[0.7.2] - Sep 27, 2022

New Features

• A full refactor of model fitting methods (#1134).
  • This introduces a new fit_gpytorch_mll method that multiple-dispatches on the model type. Users may register custom fitting routines for different combinations of MLLs, Likelihoods, and Models.
  • Unlike previous fitting helpers, fit_gpytorch_mll does not pass kwargs to optimizer and instead introduces an optional optimizer_kwargs argument.
  • When a model fitting attempt fails, botorch.fit methods restore modules to their original states.
  • fit_gpytorch_mll throws a ModelFittingError when all model fitting attempts fail.
  • Upon returning from fit_gpytorch_mll, mll.training will be True if fitting failed and False otherwise.
• Allow custom bounds to be passed in to SyntheticTestFunction (#1415).

Deprecations

• Deprecate weights argument of risk measures in favor of a preprocessing_function (#1400),
• Deprecate fit_gyptorch_model; to be superseded by fit_gpytorch_mll.

Other Changes

• Support risk measures in MOO input constructors (#1401).

Bug Fixes

• Fix fully Bayesian state dict loading when there are more than 10 models (#1405).
• Fix batch_shape property of SaasFullyBayesianSingleTaskGP (#1413).
• Fix model_list_to_batched ignoring the covar_module of the input models (#1419).

[0.7.1] - Sep 13, 2022

Compatibility

• Pin GPyTorch >= 1.9.0 (#1397).
• Pin linear_operator == 0.1.1 (#1397).

New Features

• Implement SaasFullyBayesianMultiTaskGP and related utilities (#1181, #1203).

Other Changes

• Support loading a state dict for SaasFullyBayesianSingleTaskGP (#1120).
• Update load_state_dict for ModelList to support fully Bayesian models (#1395).
• Add is_one_to_many attribute to input transforms (#1396).

Bug Fixes

• Fix PairwiseGP on GPU (#1388).

[0.7.0] - Sep 7, 2022

Compatibility

• Require python >= 3.8 (via #1347).
• Support for python 3.10 (via #1379).
• Require PyTorch >= 1.11 (via (#1363).
• Require GPyTorch >= 1.9.0 (#1347).
  • GPyTorch 1.9.0 is a major refactor that factors out the lazy tensor functionality into a new LinearOperator library, which required a number of adjustments to BoTorch (#1363, #1377).
• Require pyro >= 1.8.2 (#1379).

New Features

• Add ability to generate the features appended in the AppendFeatures input transform via a generic callable (#1354).
• Add new synthetic test functions for sensitivity analysis (#1355, #1361).

Other Changes

• Use time.monotonic() instead of time.time() to measure duration (#1353).
• Allow passing Y_samples directly in MARS.set_baseline_Y (#1364).

Bug Fixes

• Patch state_dict loading for PairwiseGP (#1359).
• Fix batch_shape handling in Normalize and InputStandardize transforms (#1360).

[0.6.6] - Aug 12, 2022

Compatibility

• Require GPyTorch >= 1.8.1 (#1347).

New Features

• Support batched models in RandomFourierFeatures (#1336).
• Add a skip_expand option to AppendFeatures (#1344).

Other Changes

• Allow qProbabilityOfImprovement to use batch-shaped best_f (#1324).
• Make optimize_acqf re-attempt failed optimization runs and handle optimization errors in optimize_acqf and gen_candidates_scipy better (#1325).
• Reduce memory overhead in MARS.set_baseline_Y (#1346).

Bug Fixes

• Fix bug where outcome_transform was ignored for ModelListGP.fantasize (#1338).
• Fix bug causing get_polytope_samples to sample incorrectly when variables live in multiple dimensions (#1341).

Documentation

• Add more descriptive docstrings for models (#1327, #1328, #1329, #1330) and for other classes (#1313).
• Expanded on the model documentation at botorch.org/docs/models (#1337).

[0.6.5] - Jul 15, 2022

Compatibility

• Require PyTorch >=1.10 (#1293).
• Require GPyTorch >=1.7 (#1293).

New Features

• Add MOMF (Multi-Objective Multi-Fidelity) acquisition function (#1153).
• Support PairwiseLogitLikelihood and modularize PairwiseGP (#1193).
• Add in transformed weighting flag to Proximal Acquisition function (#1194).
• Add FeasibilityWeightedMCMultiOutputObjective (#1202).
• Add outcome_transform to FixedNoiseMultiTaskGP (#1255).
• Support Scalable Constrained Bayesian Optimization (#1257).
• Support SaasFullyBayesianSingleTaskGP in prune_inferior_points (#1260).
• Implement MARS as a risk measure (#1303).
• Add MARS tutorial (#1305).

Other Changes

• Add Bilog outcome transform (#1189).
• Make get_infeasible_cost return a cost value for each outcome (#1191).
• Modify risk measures to accept List[float] for weights (#1197).
• Support SaasFullyBayesianSingleTaskGP in prune_inferior_points_multi_objective (#1204).
• BotorchContainers and BotorchDatasets: Large refactor of the original TrainingData API to allow for more diverse types of datasets (#1205, #1221).
• Proximal biasing support for multi-output SingleTaskGP models (#1212).
• Improve error handling in optimize_acqf_discrete with a check that choices is non-empty (#1228).
• Handle X_pending properly in FixedFeatureAcquisition (#1233, #1234).
• PE and PLBO support in Ax (#1240, #1241).
• Remove model.train call from get_X_baseline for better caching (#1289).
• Support inf values in bounds argument of optimize_acqf (#1302).

Bug Fixes

• Update get_gp_samples to support input / outcome transforms (#1201).
• Fix cached Cholesky sampling in qNEHVI when using Standardize outcome transform (#1215).
• Make task_feature as required input in MultiTaskGP.construct_inputs (#1246).
• Fix CUDA tests (#1253).
• Fix FixedSingleSampleModel dtype/device conversion (#1254).
• Prevent inappropriate transforms by putting input transforms into train mode before converting models (#1283).
• Fix sample_points_around_best when using 20 dimensional inputs or prob_perturb (#1290).
• Skip bound validation in optimize_acqf if inequality constraints are specified (#1297).
• Properly handle RFFs when used with a ModelList with individual transforms (#1299).
• Update PosteriorList to support deterministic-only models and fix event_shape (#1300).

Documentation

• Add a note about observation noise in the posterior in fit_model_with_torch_optimizer notebook (#1196).
• Fix custom botorch model in Ax tutorial to support new interface (#1213).
• Update MOO docs (#1242).
• Add SMOKE_TEST option to MOMF tutorial (#1243).
• Fix ModelListGP.condition_on_observations/fantasize bug (#1250).
• Replace space with underscore for proper doc generation (#1256).
• Update PBO tutorial to use EUBO (#1262).

[0.6.4] - Apr 21, 2022

New Features

• Implement ExpectationPosteriorTransform (#903).
• Add PairwiseMCPosteriorVariance, a cheap active learning acquisition function (#1125).
• Support computing quantiles in the fully Bayesian posterior, add FullyBayesianPosteriorList (#1161).
• Add expectation risk measures (#1173).
• Implement Multi-Fidelity GIBBON (Lower Bound MES) acquisition function (#1185).

Other Changes

• Add an error message for one shot acquisition functions in optimize_acqf_discrete (#939).
• Validate the shape of the bounds argument in optimize_acqf (#1142).
• Minor tweaks to SAASBO (#1143, #1183).
• Minor updates to tutorials (24f7fda7b40d4aabf502c1a67816ac1951af8c23, #1144, #1148, #1159, #1172, #1180).
• Make it easier to specify a custom PyroModel (#1149).
• Allow passing in a mean_module to SingleTaskGP/FixedNoiseGP (#1160).
• Add a note about acquisitions using gradients to base class (#1168).
• Remove deprecated box_decomposition module (#1175).

Bug Fixes

• Bug-fixes for ProximalAcquisitionFunction (#1122).
• Fix missing warnings on failed optimization in fit_gpytorch_scipy (#1170).
• Ignore data related buffers in PairwiseGP.load_state_dict (#1171).
• Make fit_gpytorch_model properly honor the debug flag (#1178).
• Fix missing posterior_transform in gen_one_shot_kg_initial_conditions (#1187).

[0.6.3] - Mar 28, 2022

New Features

• Implement SAASBO - SaasFullyBayesianSingleTaskGP model for sample-efficient high-dimensional Bayesian optimization (#1123).
• Add SAASBO tutorial (#1127).
• Add LearnedObjective (#1131), AnalyticExpectedUtilityOfBestOption acquisition function (#1135), and a few auxiliary classes to support Bayesian optimization with preference exploration (BOPE).
• Add BOPE tutorial (#1138).

Other Changes

• Use qKG.evaluate in optimize_acqf_mixed (#1133).
• Add construct_inputs to SAASBO (#1136).

Bug Fixes

• Fix "Constraint Active Search" tutorial (#1124).
• Update "Discrete Multi-Fidelity BO" tutorial (#1134).

[0.6.2] - Mar 9, 2022

New Features

• Use BOTORCH_MODULAR in tutorials with Ax (#1105).
• Add optimize_acqf_discrete_local_search for discrete search spaces (#1111).

Bug Fixes

• Fix missing posterior_transform in qNEI and get_acquisition_function (#1113).

[0.6.1] - Feb 28, 2022

New Features

• Add Standardize input transform (#1053).
• Low-rank Cholesky updates for NEI (#1056).
• Add support for non-linear input constraints (#1067).
• New MOO problems: MW7 (#1077), disc brake (#1078), penicillin (#1079), RobustToy (#1082), GMM (#1083).

Other Changes

• Support multi-output models in MES using PosteriorTransform (#904).
• Add Dispatcher (#1009).
• Modify qNEHVI to support deterministic models (#1026).
• Store tensor attributes of input transforms as buffers (#1035).
• Modify NEHVI to support MTGPs (#1037).
• Make Normalize input transform input column-specific (#1047).
• Improve find_interior_point (#1049).
• Remove deprecated botorch.distributions module (#1061).
• Avoid costly application of posterior transform in Kronecker & HOGP models (#1076).
• Support heteroscedastic perturbations in InputPerturbations (#1088).

Performance Improvements

• Make risk measures more memory efficient (#1034).

Bug Fixes

• Properly handle empty fixed_features in optimization (#1029).
• Fix missing weights in VaR risk measure (#1038).
• Fix find_interior_point for negative variables & allow unbounded problems (#1045).
• Filter out indefinite bounds in constraint utilities (#1048).
• Make non-interleaved base samples use intuitive shape (#1057).
• Pad small diagonalization with zeros for KroneckerMultitaskGP (#1071).
• Disable learning of bounds in preprocess_transform (#1089).
• Fix gen_candidates_torch (4079164489613d436d19c7b2df97677d97dfa8dc).
• Catch runtime errors with ill-conditioned covar (#1095).
• Fix compare_mc_analytic_acquisition tutorial (#1099).

[0.6.0] - Dec 8, 2021

Compatibility

• Require PyTorch >=1.9 (#1011).
• Require GPyTorch >=1.6 (#1011).

New Features

• New ApproximateGPyTorchModel wrapper for various (variational) approximate GP models (#1012).
• New SingleTaskVariationalGP stochastic variational Gaussian Process model (#1012).
• Support for Multi-Output Risk Measures (#906, #965).
• Introduce ModelList and PosteriorList (#829).
• New Constraint Active Search tutorial (#1010).
• Add additional multi-objective optimization test problems (#958).

Other Changes

• Add covar_module as an optional input of MultiTaskGP models (#941).
• Add min_range argument to Normalize transform to prevent division by zero (#931).
• Add initialization heuristic for acquisition function optimization that samples around best points (#987).
• Update initialization heuristic to perturb a subset of the dimensions of the best points if the dimension is > 20 (#988).
• Modify apply_constraints utility to work with multi-output objectives (#994).
• Short-cut t_batch_mode_transform decorator on non-tensor inputs (#991).

Performance Improvements

• Use lazy covariance matrix in BatchedMultiOutputGPyTorchModel.posterior (#976).
• Fast low-rank Cholesky updates for qNoisyExpectedHypervolumeImprovement (#747, #995, #996).

Bug Fixes

• Update error handling to new PyTorch linear algebra messages (#940).
• Avoid test failures on Ampere devices (#944).
• Fixes to the Griewank test function (#972).
• Handle empty base_sample_shape in Posterior.rsample (#986).
• Handle NotPSDError and hitting maxiter in fit_gpytorch_model (#1007).
• Use TransformedPosterior for subclasses of GPyTorchPosterior (#983).
• Propagate best_f argument to qProbabilityOfImprovement in input constructors (f5a5f8b6dc20413e67c6234e31783ac340797a8d).

[0.5.1] - Sep 2, 2021

Compatibility

• Require GPyTorch >=1.5.1 (#928).

New Features

• Add HigherOrderGP composite Bayesian Optimization tutorial notebook (#864).
• Add Multi-Task Bayesian Optimziation tutorial (#867).
• New multi-objective test problems from (#876).
• Add PenalizedMCObjective and L1PenaltyObjective (#913).
• Add a ProximalAcquisitionFunction for regularizing new candidates towards previously generated ones (#919, #924).
• Add a Power outcome transform (#925).

Bug Fixes

• Batch mode fix for HigherOrderGP initialization (#856).
• Improve CategoricalKernel precision (#857).
• Fix an issue with qMultiFidelityKnowledgeGradient.evaluate (#858).
• Fix an issue with transforms with HigherOrderGP. (#889)
• Fix initial candidate generation when parameter constraints are on different device (#897).
• Fix bad in-place op in _generate_unfixed_lin_constraints (#901).
• Fix an input transform bug in fantasize call (#902).
• Fix outcome transform bug in batched_to_model_list (#917).

Other Changes

• Make variance optional for TransformedPosterior.mean (#855).
• Support transforms in DeterministicModel (#869).
• Support batch_shape in RandomFourierFeatures (#877).
• Add a maximize flag to PosteriorMean (#881).
• Ignore categorical dimensions when validating training inputs in MixedSingleTaskGP (#882).
• Refactor HigherOrderGPPosterior for memory efficiency (#883).
• Support negative weights for minimization objectives in get_chebyshev_scalarization (#884).
• Move train_inputs transforms to model.train/eval calls (#894).

[0.5.0] - Jun 29, 2021

Compatibility

• Require PyTorch >=1.8.1 (#832).
• Require GPyTorch >=1.5 (#848).
• Changes to how input transforms are applied: transform_inputs is applied in model.forward if the model is in train mode, otherwise it is applied in the posterior call (#819, #835).

New Features

• Improved multi-objective optimization capabilities:
  • qNoisyExpectedHypervolumeImprovement acquisition function that improves on qExpectedHypervolumeImprovement in terms of tolerating observation noise and speeding up computation for large q-batches (#797, #822).
  • qMultiObjectiveMaxValueEntropy acqusition function (913aa0e510dde10568c2b4b911124cdd626f6905, #760).
  • Heuristic for reference point selection (#830).
  • FastNondominatedPartitioning for Hypervolume computations (#699).
  • DominatedPartitioning for partitioning the dominated space (#726).
  • BoxDecompositionList for handling box decompositions of varying sizes (#712).
  • Direct, batched dominated partitioning for the two-outcome case (#739).
  • get_default_partitioning_alpha utility providing heuristic for selecting approximation level for partitioning algorithms (#793).
  • New method for computing Pareto Frontiers with less memory overhead (#842, #846).
• New qLowerBoundMaxValueEntropy acquisition function (a.k.a. GIBBON), a lightweight variant of Multi-fidelity Max-Value Entropy Search using a Determinantal Point Process approximation (#724, #737, #749).
• Support for discrete and mixed input domains:
  • CategoricalKernel for categorical inputs (#771).
  • MixedSingleTaskGP for mixed search spaces (containing both categorical and ordinal parameters) (#772, #847).
  • optimize_acqf_discrete for optimizing acquisition functions over fully discrete domains (#777).
  • Extend optimize_acqf_mixed to allow batch optimization (#804).
• Support for robust / risk-aware optimization:
  • Risk measures for robust / risk-averse optimization (#821).
  • AppendFeatures transform (#820).
  • InputPerturbation input transform for for risk averse BO with implementation errors (#827).
  • Tutorial notebook for Bayesian Optimization of risk measures (#823).
  • Tutorial notebook for risk-averse Bayesian Optimization under input perturbations (#828).
• More scalable multi-task modeling and sampling:
  • KroneckerMultiTaskGP model for efficient multi-task modeling for block-design settings (all tasks observed at all inputs) (#637).
  • Support for transforms in Multi-Task GP models (#681).
  • Posterior sampling based on Matheron's rule for Multi-Task GP models (#841).
• Various changes to simplify and streamline integration with Ax:
  • Handle non-block designs in TrainingData (#794).
  • Acquisition function input constructor registry (#788, #802, #845).
• Random Fourier Feature (RFF) utilties for fast (approximate) GP function sampling (#750).
• DelaunayPolytopeSampler for fast uniform sampling from (simple) polytopes (#741).
• Add evaluate method to ScalarizedObjective (#795).

Bug Fixes

• Handle the case when all features are fixed in optimize_acqf (#770).
• Pass fixed_features to initial candidate generation functions (#806).
• Handle batch empty pareto frontier in FastPartitioning (#740).
• Handle empty pareto set in is_non_dominated (#743).
• Handle edge case of no or a single observation in get_chebyshev_scalarization (#762).
• Fix an issue in gen_candidates_torch that caused problems with acqusition functions using fantasy models (#766).
• Fix HigherOrderGP dtype bug (#728).
• Normalize before clamping in Warp input warping transform (#722).
• Fix bug in GP sampling (#764).

Other Changes

• Modify input transforms to support one-to-many transforms (#819, #835).
• Make initial conditions for acquisition function optimization honor parameter constraints (#752).
• Perform optimization only over unfixed features if fixed_features is passed (#839).
• Refactor Max Value Entropy Search Methods (#734).
• Use Linear Algebra functions from the torch.linalg module (#735).
• Use PyTorch's Kumaraswamy distribution (#746).
• Improved capabilities and some bugfixes for batched models (#723, #767).
• Pass callback argument to scipy.optim.minimize in gen_candidates_scipy (#744).
• Modify behavior of X_pending in in multi-objective acqusiition functions (#747).
• Allow multi-dimensional batch shapes in test functions (#757).
• Utility for converting batched multi-output models into batched single-output models (#759).
• Explicitly raise NotPSDError in _scipy_objective_and_grad (#787).
• Make raw_samples optional if batch_initial_conditions is passed (#801).
• Use powers of 2 in qMC docstrings & examples (#812).

[0.4.0] - Feb 23, 2021

Compatibility

• Require PyTorch >=1.7.1 (#714).
• Require GPyTorch >=1.4 (#714).

New Features

• HigherOrderGP - High-Order Gaussian Process (HOGP) model for high-dimensional output regression (#631, #646, #648, #680).
• qMultiStepLookahead acquisition function for general look-ahead optimization approaches (#611, #659).
• ScalarizedPosteriorMean and project_to_sample_points for more advanced MFKG functionality (#645).
• Large-scale Thompson sampling tutorial (#654, #713).
• Tutorial for optimizing mixed continuous/discrete domains (application to multi-fidelity KG with discrete fidelities) (#716).
• GPDraw utility for sampling from (exact) GP priors (#655).
• Add X as optional arg to call signature of MCAcqusitionObjective (#487).
• OSY synthetic test problem (#679).

Bug Fixes

• Fix matrix multiplication in scalarize_posterior (#638).
• Set X_pending in get_acquisition_function in qEHVI (#662).
• Make contextual kernel device-aware (#666).
• Do not use an MCSampler in MaxPosteriorSampling (#701).
• Add ability to subset outcome transforms (#711).

Performance Improvements

• Batchify box decomposition for 2d case (#642).

Other Changes

• Use scipy distribution in MES quantile bisect (#633).
• Use new closure definition for GPyTorch priors (#634).
• Allow enabling of approximate root decomposition in posterior calls (#652).
• Support for upcoming 21201-dimensional PyTorch SobolEngine (#672, #674).
• Refactored various MOO utilities to allow future additions (#656, #657, #658, #661).
• Support input_transform in PairwiseGP (#632).
• Output shape checks for t_batch_mode_transform (#577).
• Check for NaN in gen_candidates_scipy (#688).
• Introduce base_sample_shape property to Posterior objects (#718).

[0.3.3] - Dec 8, 2020

Contextual Bayesian Optimization, Input Warping, TuRBO, sampling from polytopes.

Compatibility

• Require PyTorch >=1.7 (#614).
• Require GPyTorch >=1.3 (#614).

New Features

• Models (LCE-A, LCE-M and SAC ) for Contextual Bayesian Optimziation (#581).
  • Implements core models from: High-Dimensional Contextual Policy Search with Unknown Context Rewards using Bayesian Optimization. Q. Feng, B. Letham, H. Mao, E. Bakshy. NeurIPS 2020.
  • See Ax for usage of these models.
• Hit and run sampler for uniform sampling from a polytope (#592).
• Input warping:
  • Core functionality (#607).
  • Kumaraswamy Distribution (#606).
  • Tutorial (8f34871652042219c57b799669a679aab5eed7e3).
• TuRBO-1 tutorial (#598).
  • Implements the method from Scalable Global Optimization via Local Bayesian Optimization. D. Eriksson, M. Pearce, J. Gardner, R. D. Turner, M. Poloczek. NeurIPS 2019.

Bug fixes

• Fix bounds of HolderTable synthetic function (#596).
• Fix device issue in MOO tutorial (#621).

Other changes

• Add train_inputs option to qMaxValueEntropy (#593).
• Enable gpytorch settings to override BoTorch defaults for fast_pred_var and debug (#595).
• Rename set_train_data_transform -> preprocess_transform (#575).
• Modify _expand_bounds() shape checks to work with >2-dim bounds (#604).
• Add batch_shape property to models (#588).
• Modify qMultiFidelityKnowledgeGradient.evaluate() to work with project, expand and cost_aware_utility (#594).
• Add list of papers using BoTorch to website docs (#617).

[0.3.2] - Oct 23, 2020

Maintenance Release

New Features

• Add PenalizedAcquisitionFunction wrapper (#585)
• Input transforms
  • Reversible input transform (#550)
  • Rounding input transform (#562)
  • Log input transform (#563)
• Differentiable approximate rounding for integers (#561)

Bug fixes

• Fix sign error in UCB when maximize=False (a4bfacbfb2109d3b89107d171d2101e1995822bb)
• Fix batch_range sample shape logic (#574)

Other changes

• Better support for two stage sampling in preference learning (0cd13d0cb49b1ac8d0971e42f1f0e9dd6126fd9a)
• Remove noise term in PairwiseGP and add ScaleKernel by default (#571)
• Rename prior to task_covar_prior in MultiTaskGP and FixedNoiseMultiTaskGP (8e42ea82856b165a7df9db2a9b6f43ebd7328fc4)
• Support only transforming inputs on training or evaluation (#551)
• Add equals method for InputTransform (#552)

[0.3.1] - Sep 15, 2020

Maintenance Release

New Features

• Constrained Multi-Objective tutorial (#493)
• Multi-fidelity Knowledge Gradient tutorial (#509)
• Support for batch qMC sampling (#510)
• New evaluate method for qKnowledgeGradient (#515)

Compatibility

• Require PyTorch >=1.6 (#535)
• Require GPyTorch >=1.2 (#535)
• Remove deprecated botorch.gen module (#532)

Bug fixes

• Fix bad backward-indexing of task_feature in MultiTaskGP (#485)
• Fix bounds in constrained Branin-Currin test function (#491)
• Fix max_hv for C2DTLZ2 and make Hypervolume always return a float (#494)
• Fix bug in draw_sobol_samples that did not use the proper effective dimension (#505)
• Fix constraints for q>1 in qExpectedHypervolumeImprovement (c80c4fdb0f83f0e4f12e4ec4090d0478b1a8b532)
• Only use feasible observations in partitioning for qExpectedHypervolumeImprovement in get_acquisition_function (#523)
• Improved GPU compatibility for PairwiseGP (#537)

Performance Improvements

• Reduce memory footprint in qExpectedHypervolumeImprovement (#522)
• Add (q)ExpectedHypervolumeImprovement to nonnegative functions [for better initialization] (#496)

Other changes

• Support batched best_f in qExpectedImprovement (#487)
• Allow to return full tree of solutions in OneShotAcquisitionFunction (#488)
• Added construct_inputs class method to models to programmatically construct the inputs to the constructor from a standardized TrainingData representation (#477, #482, 3621198d02195b723195b043e86738cd5c3b8e40)
• Acquisition function constructors now accept catch-all **kwargs options (#478, e5b69352954bb10df19a59efe9221a72932bfe6c)
• Use psd_safe_cholesky in qMaxValueEntropy for better numerical stabilty (#518)
• Added WeightedMCMultiOutputObjective (81d91fd2e115774e561c8282b724457233b6d49f)
• Add ability to specify outcomes to all multi-output objectives (#524)
• Return optimization output in info_dict for fit_gpytorch_scipy (#534)
• Use setuptools_scm for versioning (#539)

[0.3.0] - July 6, 2020

Multi-Objective Bayesian Optimization

New Features

• Multi-Objective Acquisition Functions (#466)
  • q-Expected Hypervolume Improvement
  • q-ParEGO
  • Analytic Expected Hypervolume Improvement with auto-differentiation
• Multi-Objective Utilities (#466)
  • Pareto Computation
  • Hypervolume Calculation
  • Box Decomposition algorithm
• Multi-Objective Test Functions (#466)
  • Suite of synthetic test functions for multi-objective, constrained optimization
• Multi-Objective Tutorial (#468)
• Abstract ConstrainedBaseTestProblem (#454)
• Add optimize_acqf_list method for sequentially, greedily optimizing 1 candidate from each provided acquisition function (d10aec911b241b208c59c192beb9e4d572a092cd)

Bug fixes

• Fixed re-arranging mean in MultiTask MO models (#450).

Other changes

• Move gpt_posterior_settings into models.utils (#449)
• Allow specifications of batch dims to collapse in samplers (#457)
• Remove outcome transform before model-fitting for sequential model fitting in MO models (#458)

[0.2.5] - May 14, 2020

Bugfix Release

Bug fixes

• Fixed issue with broken wheel build (#444).

Other changes

• Changed code style to use absolute imports throughout (#443).

[0.2.4] - May 12, 2020

Bugfix Release

Bug fixes

• There was a mysterious issue with the 0.2.3 wheel on pypi, where part of the botorch/optim/utils.py file was not included, which resulted in an ImportError for many central components of the code. Interestingly, the source dist (built with the same command) did not have this issue.
• Preserve order in ChainedOutcomeTransform (#440).

New Features

• Utilities for estimating the feasible volume under outcome constraints (#437).

[0.2.3] - Apr 27, 2020

Pairwise GP for Preference Learning, Sampling Strategies.

Compatibility

• Require PyTorch >=1.5 (#423).
• Require GPyTorch >=1.1.1 (#425).

New Features

• Add PairwiseGP for preference learning with pair-wise comparison data (#388).
• Add SamplingStrategy abstraction for sampling-based generation strategies, including MaxPosteriorSampling (i.e. Thompson Sampling) and BoltzmannSampling (#218, #407).

Deprecations

• The existing botorch.gen module is moved to botorch.generation.gen and imports from botorch.gen will raise a warning (an error in the next release) (#218).

Bug fixes

• Fix & update a number of tutorials (#394, #398, #393, #399, #403).
• Fix CUDA tests (#404).
• Fix sobol maxdim limitation in prune_baseline (#419).

Other changes

• Better stopping criteria for stochastic optimization (#392).
• Improve numerical stability of LinearTruncatedFidelityKernel (#409).
• Allow batched best_f in qExpectedImprovement and qProbabilityOfImprovement (#411).
• Introduce new logger framework (#412).
• Faster indexing in some situations (#414).
• More generic BaseTestProblem (9e604fe2188ac85294c143d249872415c4d95823).

[0.2.2] - Mar 6, 2020

Require PyTorch 1.4, Python 3.7 and new features for active learning, multi-fidelity optimization, and a number of bug fixes.

Compatibility

• Require PyTorch >=1.4 (#379).
• Require Python >=3.7 (#378).

New Features

• Add qNegIntegratedPosteriorVariance for Bayesian active learning (#377).
• Add FixedNoiseMultiFidelityGP, analogous to SingleTaskMultiFidelityGP (#386).
• Support scalarize_posterior for m>1 and q>1 posteriors (#374).
• Support subset_output method on multi-fidelity models (#372).
• Add utilities for sampling from simplex and hypersphere (#369).

Bug fixes

• Fix TestLoader local test discovery (#376).
• Fix batch-list conversion of SingleTaskMultiFidelityGP (#370).
• Validate tensor args before checking input scaling for more informative error messaages (#368).
• Fix flaky qNoisyExpectedImprovement test (#362).
• Fix test function in closed-loop tutorial (#360).
• Fix num_output attribute in BoTorch/Ax tutorial (#355).

Other changes

• Require output dimension in MultiTaskGP (#383).
• Update code of conduct (#380).
• Remove deprecated joint_optimize and sequential_optimize (#363).

[0.2.1] - Jan 15, 2020

Minor bug fix release.

New Features

• Add a static method for getting batch shapes for batched MO models (#346).

Bug fixes

• Revamp qKG constructor to avoid issue with missing objective (#351).
• Make sure MVES can support sampled costs like KG (#352).

Other changes

• Allow custom module-to-array handling in fit_gpytorch_scipy (#341).

[0.2.0] - Dec 20, 2019

Max-value entropy acquisition function, cost-aware / multi-fidelity optimization, subsetting models, outcome transforms.

Compatibility

• Require PyTorch >=1.3.1 (#313).
• Require GPyTorch >=1.0 (#342).

New Features

• Add cost-aware KnowledgeGradient (qMultiFidelityKnowledgeGradient) for multi-fidelity optimization (#292).
• Add qMaxValueEntropy and qMultiFidelityMaxValueEntropy max-value entropy search acquisition functions (#298).
• Add subset_output functionality to (most) models (#324).
• Add outcome transforms and input transforms (#321).
• Add outcome_transform kwarg to model constructors for automatic outcome transformation and un-transformation (#327).
• Add cost-aware utilities for cost-sensitive acquisiiton functions (#289).
• Add DeterminsticModel and DetermisticPosterior abstractions (#288).
• Add AffineFidelityCostModel (f838eacb4258f570c3086d7cbd9aa3cf9ce67904).
• Add project_to_target_fidelity and expand_trace_observations utilties for use in multi-fidelity optimization (1ca12ac0736e39939fff650cae617680c1a16933).

Performance Improvements

• New prune_baseline option for pruning X_baseline in qNoisyExpectedImprovement (#287).
• Do not use approximate MLL computation for deterministic fitting (#314).
• Avoid re-evaluating the acquisition function in gen_candidates_torch (#319).
• Use CPU where possible in gen_batch_initial_conditions to avoid memory issues on the GPU (#323).

Bug fixes

• Properly register NoiseModelAddedLossTerm in HeteroskedasticSingleTaskGP (671c93a203b03ef03592ce322209fc5e71f23a74).
• Fix batch mode for MultiTaskGPyTorchModel (#316).
• Honor propagate_grads argument in fantasize of FixedNoiseGP (#303).
• Properly handle diag arg in LinearTruncatedFidelityKernel (#320).

Other changes

• Consolidate and simplify multi-fidelity models (#308).
• New license header style (#309).
• Validate shape of best_f in qExpectedImprovement (#299).
• Support specifying observation noise explicitly for all models (#256).
• Add num_outputs property to the Model API (#330).
• Validate output shape of models upon instantiating acquisition functions (#331).

Tests

• Silence warnings outside of explicit tests (#290).
• Enforce full sphinx docs coverage in CI (#294).

[0.1.4] - Oct 1, 2019

Knowledge Gradient acquisition function (one-shot), various maintenance

Breaking Changes

• Require explicit output dimensions in BoTorch models (#238)
• Make joint_optimize / sequential_optimize return acquisition function values (#149) [note deprecation notice below]
• standardize now works on the second to last dimension (#263)
• Refactor synthetic test functions (#273)

New Features

• Add qKnowledgeGradient acquisition function (#272, #276)
• Add input scaling check to standard models (#267)
• Add cyclic_optimize, convergence criterion class (#269)
• Add settings.debug context manager (#242)

Deprecations

• Consolidate sequential_optimize and joint_optimize into optimize_acqf (#150)

Bug fixes

• Properly pass noise levels to GPs using a FixedNoiseGaussianLikelihood (#241) [requires gpytorch > 0.3.5]
• Fix q-batch dimension issue in ConstrainedExpectedImprovement (6c067185f56d3a244c4093393b8a97388fb1c0b3)
• Fix parameter constraint issues on GPU (#260)

Minor changes

• Add decorator for concatenating pending points (#240)
• Draw independent sample from prior for each hyperparameter (#244)
• Allow dim > 1111 for gen_batch_initial_conditions (#249)
• Allow optimize_acqf to use q>1 for AnalyticAcquisitionFunction (#257)
• Allow excluding parameters in fit functions (#259)
• Track the final iteration objective value in fit_gpytorch_scipy (#258)
• Error out on unexpected dims in parameter constraint generation (#270)
• Compute acquisition values in gen_ functions w/o grad (#274)

Tests

• Introduce BotorchTestCase to simplify test code (#243)
• Refactor tests to have monolithic cuda tests (#261)

[0.1.3] - Aug 9, 2019

Compatibility & maintenance release

Compatibility

• Updates to support breaking changes in PyTorch to boolean masks and tensor comparisons (#224).
• Require PyTorch >=1.2 (#225).
• Require GPyTorch >=0.3.5 (itself a compatibility release).

New Features

• Add FixedFeatureAcquisitionFunction wrapper that simplifies optimizing acquisition functions over a subset of input features (#219).
• Add ScalarizedObjective for scalarizing posteriors (#210).
• Change default optimization behavior to use L-BFGS-B by for box constraints (#207).

Bug fixes

• Add validation to candidate generation (#213), making sure constraints are strictly satisfied (rater than just up to numerical accuracy of the optimizer).

Minor changes

• Introduce AcquisitionObjective base class (#220).
• Add propagate_grads context manager, replacing the propagate_grads kwarg in model posterior() calls (#221)
• Add batch_initial_conditions argument to joint_optimize() for warm-starting the optimization (ec3365a37ed02319e0d2bb9bea03aee89b7d9caa).
• Add return_best_only argument to joint_optimize() (#216). Useful for implementing advanced warm-starting procedures.

[0.1.2] - July 9, 2019

Maintenance release

Bug fixes

• Avoid [PyTorch bug]((pytorch/pytorch#22353) resulting in bad gradients on GPU by requiring GPyTorch >= 0.3.4
• Fixes to resampling behavior in MCSamplers (#204)

Experimental Features

• Linear truncated kernel for multi-fidelity bayesian optimization (#192)
• SingleTaskMultiFidelityGP for GP models that have fidelity parameters (#181)

[0.1.1] - June 27, 2019

API updates, more robust model fitting

Breaking changes

• rename botorch.qmc to botorch.sampling, move MC samplers from acquisition.sampler to botorch.sampling.samplers (#172)

New Features

• Add condition_on_observations and fantasize to the Model level API (#173)
• Support pending observations generically for all MCAcqusitionFunctions (#176)
• Add fidelity kernel for training iterations/training data points (#178)
• Support for optimization constraints across q-batches (to support things like sample budget constraints) (2a95a6c3f80e751d5cf8bc7240ca9f5b1529ec5b)
• Add ModelList <-> Batched Model converter (#187)
• New test functions
  • basic: neg_ackley, cosine8, neg_levy, neg_rosenbrock, neg_shekel (e26dc7576c7bf5fa2ba4cb8fbcf45849b95d324b)
  • for multi-fidelity BO: neg_aug_branin, neg_aug_hartmann6, neg_aug_rosenbrock (ec4aca744f65ca19847dc368f9fee4cc297533da)

Improved functionality:

• More robust model fitting
  • Catch gpytorch numerical issues and return NaN to the optimizer (#184)
  • Restart optimization upon failure by sampling hyperparameters from their prior (#188)
  • Sequentially fit batched and ModelListGP models by default (#189)
  • Change minimum inferred noise level (e2c64fef1e76d526a33951c5eb75ac38d5581257)
• Introduce optional batch limit in joint_optimize to increases scalability of parallel optimization (baab5786e8eaec02d37a511df04442471c632f8a)
• Change constructor of ModelListGP to comply with GPyTorch’s IndependentModelList constructor (a6cf739e769c75319a67c7525a023ece8806b15d)
• Use torch.random to set default seed for samplers (rather than random) to making sampling reproducible when setting torch.manual_seed (ae507ad97255d35f02c878f50ba68a2e27017815)

Performance Improvements

• Use einsum in LinearMCObjective (22ca29535717cda0fcf7493a43bdf3dda324c22d)
• Change default Sobol sample size for MCAquisitionFunctions to be base-2 for better MC integration performance (5d8e81866a23d6bfe4158f8c9b30ea14dd82e032)
• Add ability to fit models in SumMarginalLogLikelihood sequentially (and make that the default setting) (#183)
• Do not construct the full covariance matrix when computing posterior of single-output BatchedMultiOutputGPyTorchModel (#185)

Bug fixes

• Properly handle observation_noise kwarg for BatchedMultiOutputGPyTorchModels (#182)
• Fix a issue where f_best was always max for NoisyExpectedImprovement (de8544a75b58873c449b41840a335f6732754c77)
• Fix bug and numerical issues in initialize_q_batch (844dcd1dc8f418ae42639e211c6bb8e31a75d8bf)
• Fix numerical issues with inv_transform for qMC sampling (#162)

Other

• Bump GPyTorch minimum requirement to 0.3.3

[0.1.0] - April 30, 2019

First public beta release.