Document how to implement custom models

[jakee417]

Motivation

Issue #2306

Have you read the Contributing Guidelines on pull requests?

Yes. Added a tutorial which can be used for smoke tests.

Test Plan

Probabilistic linear regression, bayesian linear regression, and ensemble linear regression all yield optimization results close to (0, 0) which is groundtruth answer.

Random Forest doesn't seem to achieve groundtruth answer, likely due to its inability to incorporate gradient information into the optimization of the acquisition function.

Related PRs

N/A

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.98%. Comparing base (dd6448b) to head (0087151).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2474   +/-   ##
=======================================
  Coverage   99.98%   99.98%           
=======================================
  Files         191      191           
  Lines       16856    16864    +8     
=======================================
+ Hits        16854    16862    +8     
  Misses          2        2           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[saitcakmak]

This is a great tutorial, thank you very much for the contribution! I just have minor point about the sampler registration. Once it is ready, you can import the PR to fbcode. Landing it in fbcode will sync the changes to GitHub and close the PR.

[Balandat]

This is an awesome tutorial, thanks a lot for the contribution!

Some additional linear algebraic comments:

        # Inverse of the gram matrix.
        self.V = torch.linalg.inv(x.T @ x)

It's generally inadvisable to compute the inverse of a matrix. Rather than doing that, you'd typically compute a matrix decomposition of x.T @ x and use that for solves down the line using forward-backward substitutions. Could you modify the code to that end? Happy to provide more specifics / code changes if that would be useful.

Also, some of this can get hairy in 32bit precision if the gram matrix is ill-conditioned. In general we recommend folks use botorch with 64bit precision (except when running on larger data sets on a GPU where perf really counts). Could you modify the tutorial to use torch.float64? Either by choosing the dtype explicitly or by setting the default torch dtype at the beginning of the tutorial.

[jakee417]

This is an awesome tutorial, thanks a lot for the contribution!

Some additional linear algebraic comments:

        # Inverse of the gram matrix.
        self.V = torch.linalg.inv(x.T @ x)

It's generally inadvisable to compute the inverse of a matrix. Rather than doing that, you'd typically compute a matrix decomposition of x.T @ x and use that for solves down the line using forward-backward substitutions. Could you modify the code to that end? Happy to provide more specifics / code changes if that would be useful.

Also, some of this can get hairy in 32bit precision if the gram matrix is ill-conditioned. In general we recommend folks use botorch with 64bit precision (except when running on larger data sets on a GPU where perf really counts). Could you modify the tutorial to use torch.float64? Either by choosing the dtype explicitly or by setting the default torch dtype at the beginning of the tutorial.

torch.Double seems to improve the results, I see why this is recommended now. Switched the torch.linalg.inv out for torch.linalg.cholesky followed by torch.cholesky_inverse. I am not sure if torch.cholesky_inverse does forwards-backwards under the hood, but it gives the same result as torch.linalg.inv.

[Balandat]

torch.Double seems to improve the results, I see why this is recommended now.

Great

Switched the torch.linalg.inv out for torch.linalg.cholesky followed by torch.cholesky_inverse. I am not sure if torch.cholesky_inverse does forwards-backwards under the hood, but it gives the same result as torch.linalg.inv.

torch.cholesky_inverse still computes the inverse explicitly. It's better than torch.linalg.inv since you're using the knowledge that the matrix is pd, but instead of computing the inverse explicitly and then doing matmuls you should use torch.cholesky_solve (which does forward-backward substitution). So if you have A x = b you want to do x = torch.cholesky_solve(b, torch.linalg.cholesky(A)) instead of x = torch.cholesky_inverse(torch.linalg.cholesky(A)) @ x. You can assign self.L = torch.linalg.cholesky(A) instead of self.V = torch.cholesky_inverse(L) and use that both to compute the Least squares estimate as well as in the posterior call (swapping out self.V @ X.squeeze().T for the respective cholesky_solve() call).

[jakee417]

okay, now I got it. Replaced the V @ x with torch.cholesky_solve(b, L) in the appropriate places.

[facebook-github-bot]

@jakee417 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[esantorella]

What happened here, a whitespace change?

[esantorella]

Awesome, this is a great tutorial and I may refer to it myself in the future. I left a couple minor suggestions, but this looks pretty good to me as-is.

[facebook-github-bot]

@jakee417 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jakee417 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jakee417 merged this pull request in 81f2a88.