-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Ethan Weinberger
authored and
Ethan Weinberger
committed
Oct 7, 2024
1 parent
e57ffe4
commit c3f3132
Showing
193 changed files
with
15,608 additions
and
0 deletions.
There are no files selected for viewing
Binary file not shown.
Binary file added
BIN
+19.4 KB
src/_build/.doctrees/content/gaussian_processes/gp_regression.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
# Sphinx build info version 1 | ||
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. | ||
config: e328333ee53f8889ac0e122766c16991 | ||
tags: 645f666f9bcd5a90fca523b33c5a78b7 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Content in Jupyter Book | ||
======================= | ||
|
||
There are many ways to write content in Jupyter Book. This short section | ||
covers a few tips for how to do so. |
81 changes: 81 additions & 0 deletions
81
src/_build/html/_sources/content/gaussian_processes/gp_regression.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
# Gaussian Processes 1: Gaussian Process Regression | ||
|
||
We begin with a definition. | ||
|
||
````{prf:definition} | ||
:label: stochastic-process | ||
A stochastic process $f$ is a collection of random variables indexed by $x \in \mathcal{X}$. I.e., | ||
$$ f = \{ f(x) : x \in \mathcal{X}\} $$ | ||
```` | ||
|
||
In the above definition, $\mathcal{X}$ is an arbitrary indexing set. When modeling real-world | ||
phenomena, we typically assume that $\mathcal{X}$ corresponds to some semantically meaningful concept. For example, | ||
if we want to model some phenomenon over _time_, we could use $\mathcal{X} = \mathbb{R}$ with $x \in \mathcal{X}$ | ||
corresponding to an individual point in time. Similarly, if we want to model some phenomenon that varies across | ||
two-dimensional _space_ we could let $\mathcal{X} = \mathbb{R}^2$ with $x \in \mathcal{X}$ now corresponding | ||
to spatial coordinates. | ||
|
||
If $\mathcal{X} = \mathbb{R}^n$, then we say that $f$ is an infinite-dimensional process. Based on the examples above, | ||
it's clear that we'd like to be able to model infinite-dimensional processes! However, dealing with infite collections | ||
of random variables presents some technical mathematical difficulties. For example, can we define the law of | ||
law of $f$? I.e., can we compute statements like $\mathbb{P}(f(x_1) \in [a_1, b_1], f(x_2) \in [a_2, b_2], \ldots)$?. Is | ||
this law guaranteed to be unique? Etc. | ||
|
||
|
||
Fortunately for us, Kolmogorov [showed](https://en.wikipedia.org/wiki/Kolmogorov_extension_theorem) that we can get | ||
away with only considering finite-dimensional distributions. | ||
|
||
````{prf:definition} | ||
:label: fdds | ||
For a stochastic process $f$ we define $f$'s finite-dimensional distributions (FDDs) as the collection of distributions | ||
$$ | ||
\mathbb{P}(f(x_1) \leq y_1, \ldots, f(x_n) \leq y_n) | ||
$$ | ||
for all finite sets $(x_1, \ldots, x_n)$ of indices in $\mathcal{X}$, | ||
```` | ||
|
||
In particular, for a given process $f$, the FDDs uniquely determine the law of $f$. This brings us to our central object of study, | ||
the _Gaussian process_. | ||
|
||
````{prf:definition} | ||
:label: gaussian-process | ||
A Gaussian process (GP) is a stochastic process with Gaussian finite dimensional distributions. I.e., | ||
$$ (f(x_1), \ldots, f(x_n)) \sim \mathcal{N}(\mu, \Sigma)$$ | ||
A GP is completely specified by its mean and covariance, which specify as functions of the index set. For a GP $f$ with | ||
mean function $m(x)$ and covariance function $k(x, x')$, we write | ||
$$f \sim \mathcal{GP}(m(x), k(x, x'))$$ | ||
```` | ||
|
||
For computational convenience we'll typyically take $m(x) = 0$. For the covariance, we can choose any positive | ||
semidefinite function, and we'll typically choose $k$ to reflect some prior knowledge. For example, if we expect output values | ||
to vary smoothly across time, we'll choose $k$ to reflect this fact. We defer a detailed discussion on kernel functions until later. | ||
|
||
Now, _why_ are we considering the Gaussian process specifically? In short, the answer lies | ||
in the many convenient properties of multivariate Gaussians. For example, sums of Gaussians are Gaussian, and the marginal | ||
distributions of a multivariate Gaussian are Gaussian. In particular, one useful property of Gaussians is that they're closed | ||
under conditioning. | ||
|
||
```{prf:proposition} | ||
:label: gp-conditioning | ||
Let $\mathbf{f}$ denote the output of $f \sim \mathcal{GP}(\mathbf{0}, k(x, x'))$ at a set of training inputs $X$, and define $\mathbf{f_*}$ correspondingly for a set of test inputs whose values we don't observe. We then have the joint distribution | ||
$$ \begin{pmatrix} \mathbf{f} \\ \mathbf{f_*} \end{pmatrix} \sim \mathcal{N}\left(\mathbf{0},\begin{bmatrix} K(X,X), K(X, X_*) \\ K(X_*, X), K(X_*, X_*)\end{bmatrix}\right)$$ | ||
Conditioning on the observed training points then gives us | ||
$$\mathbf{f_*} \mid X_*, X, \mathbf{f} \sim \mathcal{N}(K(X_*, X)K(X, X)^{-1}\mathbf{f}, K(X_*, X_*) - K(X_*, X)K(X, X)^{-1}K(X, X_*))$$ | ||
``` | ||
|
||
The above proposition is _extremely_ useful. By specifying some prior on how our function's outputs should be have with respect to the inputs (i.e., the covariance function $k$), we can leverage any observed data points to make predictions on the distributions for points at unobserved inputs. |
6 changes: 6 additions & 0 deletions
6
src/_build/html/_sources/content/gaussian_processes/overview.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Gaussian Processes | ||
|
||
These notes contain explanations on various topics in Gaussian Processes. | ||
|
||
```{tableofcontents} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# Machine Learning Notes | ||
|
||
This book contains various machine learning notes that I've compiled over the course of my PhD. | ||
|
||
```{tableofcontents} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
--- | ||
jupytext: | ||
cell_metadata_filter: -all | ||
formats: md:myst | ||
text_representation: | ||
extension: .md | ||
format_name: myst | ||
format_version: 0.13 | ||
jupytext_version: 1.11.5 | ||
kernelspec: | ||
display_name: Python 3 | ||
language: python | ||
name: python3 | ||
--- | ||
|
||
# Notebooks with MyST Markdown | ||
|
||
Jupyter Book also lets you write text-based notebooks using MyST Markdown. | ||
See [the Notebooks with MyST Markdown documentation](https://jupyterbook.org/file-types/myst-notebooks.html) for more detailed instructions. | ||
This page shows off a notebook written in MyST Markdown. | ||
|
||
## An example cell | ||
|
||
With MyST Markdown, you can define code cells with a directive like so: | ||
|
||
```{code-cell} | ||
print(2 + 2) | ||
``` | ||
|
||
When your book is built, the contents of any `{code-cell}` blocks will be | ||
executed with your default Jupyter kernel, and their outputs will be displayed | ||
in-line with the rest of your content. | ||
|
||
```{seealso} | ||
Jupyter Book uses [Jupytext](https://jupytext.readthedocs.io/en/latest/) to convert text-based files to notebooks, and can support [many other text-based notebook files](https://jupyterbook.org/file-types/jupytext.html). | ||
``` | ||
|
||
## Create a notebook with MyST Markdown | ||
|
||
MyST Markdown notebooks are defined by two things: | ||
|
||
1. YAML metadata that is needed to understand if / how it should convert text files to notebooks (including information about the kernel needed). | ||
See the YAML at the top of this page for example. | ||
2. The presence of `{code-cell}` directives, which will be executed with your book. | ||
|
||
That's all that is needed to get started! | ||
|
||
## Quickly add YAML metadata for MyST Notebooks | ||
|
||
If you have a markdown file and you'd like to quickly add YAML metadata to it, so that Jupyter Book will treat it as a MyST Markdown Notebook, run the following command: | ||
|
||
``` | ||
jupyter-book myst init path/to/markdownfile.md | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
# Markdown Files | ||
|
||
Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or | ||
in regular markdown files (`.md`), you'll write in the same flavor of markdown | ||
called **MyST Markdown**. | ||
This is a simple file to help you get started and show off some syntax. | ||
|
||
## What is MyST? | ||
|
||
MyST stands for "Markedly Structured Text". It | ||
is a slight variation on a flavor of markdown called "CommonMark" markdown, | ||
with small syntax extensions to allow you to write **roles** and **directives** | ||
in the Sphinx ecosystem. | ||
|
||
For more about MyST, see [the MyST Markdown Overview](https://jupyterbook.org/content/myst.html). | ||
|
||
## Sample Roles and Directives | ||
|
||
Roles and directives are two of the most powerful tools in Jupyter Book. They | ||
are kind of like functions, but written in a markup language. They both | ||
serve a similar purpose, but **roles are written in one line**, whereas | ||
**directives span many lines**. They both accept different kinds of inputs, | ||
and what they do with those inputs depends on the specific role or directive | ||
that is being called. | ||
|
||
Here is a "note" directive: | ||
|
||
```{note} | ||
Here is a note | ||
``` | ||
|
||
It will be rendered in a special box when you build your book. | ||
|
||
Here is an inline directive to refer to a document: {doc}`markdown-notebooks`. | ||
|
||
|
||
## Citations | ||
|
||
You can also cite references that are stored in a `bibtex` file. For example, | ||
the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like | ||
this: {cite}`holdgraf_evidence_2014`. | ||
|
||
Moreover, you can insert a bibliography into your page with this syntax: | ||
The `{bibliography}` directive must be used for all the `{cite}` roles to | ||
render properly. | ||
For example, if the references for your book are stored in `references.bib`, | ||
then the bibliography is inserted with: | ||
|
||
```{bibliography} | ||
``` | ||
|
||
## Learn more | ||
|
||
This is just a simple starter to get you started. | ||
You can learn a lot more at [jupyterbook.org](https://jupyterbook.org). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,122 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"# Content with notebooks\n", | ||
"\n", | ||
"You can also create content with Jupyter Notebooks. This means that you can include\n", | ||
"code blocks and their outputs in your book.\n", | ||
"\n", | ||
"## Markdown + notebooks\n", | ||
"\n", | ||
"As it is markdown, you can embed images, HTML, etc into your posts!\n", | ||
"\n", | ||
"![](https://myst-parser.readthedocs.io/en/latest/_static/logo-wide.svg)\n", | ||
"\n", | ||
"You can also $add_{math}$ and\n", | ||
"\n", | ||
"$$\n", | ||
"math^{blocks}\n", | ||
"$$\n", | ||
"\n", | ||
"or\n", | ||
"\n", | ||
"$$\n", | ||
"\\begin{aligned}\n", | ||
"\\mbox{mean} la_{tex} \\\\ \\\\\n", | ||
"math blocks\n", | ||
"\\end{aligned}\n", | ||
"$$\n", | ||
"\n", | ||
"But make sure you \\$Escape \\$your \\$dollar signs \\$you want to keep!\n", | ||
"\n", | ||
"## MyST markdown\n", | ||
"\n", | ||
"MyST markdown works in Jupyter Notebooks as well. For more information about MyST markdown, check\n", | ||
"out [the MyST guide in Jupyter Book](https://jupyterbook.org/content/myst.html),\n", | ||
"or see [the MyST markdown documentation](https://myst-parser.readthedocs.io/en/latest/).\n", | ||
"\n", | ||
"## Code blocks and outputs\n", | ||
"\n", | ||
"Jupyter Book will also embed your code blocks and output in your book.\n", | ||
"For example, here's some sample Matplotlib code:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"from matplotlib import rcParams, cycler\n", | ||
"import matplotlib.pyplot as plt\n", | ||
"import numpy as np\n", | ||
"plt.ion()" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"metadata": {}, | ||
"outputs": [], | ||
"source": [ | ||
"# Fixing random state for reproducibility\n", | ||
"np.random.seed(19680801)\n", | ||
"\n", | ||
"N = 10\n", | ||
"data = [np.logspace(0, 1, 100) + np.random.randn(100) + ii for ii in range(N)]\n", | ||
"data = np.array(data).T\n", | ||
"cmap = plt.cm.coolwarm\n", | ||
"rcParams['axes.prop_cycle'] = cycler(color=cmap(np.linspace(0, 1, N)))\n", | ||
"\n", | ||
"\n", | ||
"from matplotlib.lines import Line2D\n", | ||
"custom_lines = [Line2D([0], [0], color=cmap(0.), lw=4),\n", | ||
" Line2D([0], [0], color=cmap(.5), lw=4),\n", | ||
" Line2D([0], [0], color=cmap(1.), lw=4)]\n", | ||
"\n", | ||
"fig, ax = plt.subplots(figsize=(10, 5))\n", | ||
"lines = ax.plot(data)\n", | ||
"ax.legend(custom_lines, ['Cold', 'Medium', 'Hot']);" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"metadata": {}, | ||
"source": [ | ||
"There is a lot more that you can do with outputs (such as including interactive outputs)\n", | ||
"with your book. For more information about this, see [the Jupyter Book documentation](https://jupyterbook.org)" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.8.0" | ||
}, | ||
"widgets": { | ||
"application/vnd.jupyter.widget-state+json": { | ||
"state": {}, | ||
"version_major": 2, | ||
"version_minor": 0 | ||
} | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 4 | ||
} |
Oops, something went wrong.