notebooks/machine_learning_101.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML 101 Recap\n",
    "\n",
    "**ML = model + loss + optimizer**\n",
    "\n",
    "\n",
    "## Linear regression example\n",
    "\n",
    "1. Model:\n",
    "  - $f(X) = X \\beta = \\hat y$\n",
    "\n",
    "2. Loss / criterion:\n",
    "  - $ err_i = y_i - f(X_i)$\n",
    "  - $MSE = \\frac{1}{n} \\sum_{i=1}^{N} err_i^2$\n",
    "  \n",
    "3. Optimize:\n",
    "  - minimize the MSE yields the optimal $\\hat\\beta$ (after doing some math)\n",
    "  - $\\hat\\beta = (X^TX)^{-1}X^Ty$\n",
    "  - (or, more generally, use gradient descent to optimize the parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import inv\n",
    "from numpy.linalg import multi_dot as mdot\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LinReg with numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.random.random((5, 3))\n",
    "y = np.random.random(5)\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the optimal parameter:\n",
    "$\\hat\\beta = (X^T X)^{-1} X^T y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XT = X.T  # transpose\n",
    "\n",
    "beta_ = mdot([inv(XT @ X), XT, y])\n",
    "beta_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XT = X.T  # transpose\n",
    "\n",
    "beta_ = inv(XT @ X) @ XT @ y\n",
    "beta_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model $f$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(X, beta):\n",
    "    return X @ beta\n",
    "\n",
    "f(X, beta_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LinReg with PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# X = torch.rand((5, 3))\n",
    "# y = torch.rand(5)\n",
    "X = torch.from_numpy(X)\n",
    "y = torch.from_numpy(y)\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\hat\\beta = (X^T X)^{-1} X^T y$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XT = X.t()\n",
    "\n",
    "beta__ = (XT @ X).inverse() @ XT @ y\n",
    "beta__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beta__.numpy() - beta_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LinReg with PyTorch and Gradent Descent\n",
    "\n",
    "Previously, we had to do some math to calculate the optimal $\\hat\\beta$.\n",
    "PyTorch calculates the gradients for us automatically (more on that later)\n",
    "and we can use some version of gradient desctent to find our $\\hat\\beta$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_regression\n",
    "\n",
    "n_features = 1\n",
    "n_samples = 100\n",
    "\n",
    "X, y = make_regression(\n",
    "    n_samples=n_samples,\n",
    "    n_features=n_features,\n",
    "    noise=10,\n",
    ")\n",
    "\n",
    "fix, ax = plt.subplots()\n",
    "ax.plot(X, y, \".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.from_numpy(X).float()\n",
    "y = torch.from_numpy(y.reshape((n_samples, n_features))).float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "\n",
    "class LinReg(nn.Module):\n",
    "    def __init__(self, input_dim):\n",
    "        super().__init__()\n",
    "        self.beta = nn.Linear(input_dim, 1)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        return self.beta(X)\n",
    "\n",
    "\n",
    "model = LinReg(n_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import optim\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.00001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train step\n",
    "model.train()\n",
    "optimizer.zero_grad()\n",
    "\n",
    "y_ = model(X)\n",
    "\n",
    "loss = criterion(y_, y)\n",
    "loss.backward()\n",
    "optimizer.step()\n",
    "\n",
    "# Eval\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    y_ = model(X)\n",
    "    \n",
    "\n",
    "# Vis\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(X.numpy(), y_.numpy(), \".\", label=\"pred\")\n",
    "ax.plot(X.numpy(), y.numpy(), \".\", label=\"data\")\n",
    "ax.set_title(f\"MSE: {loss.item():0.1f}\")\n",
    "ax.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.beta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.beta.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.beta.weight.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.beta.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LinReg with GPU\n",
    "\n",
    "Simply move the data and the model to the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = LinReg(n_features).to(device)  # <-- here\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.0001)\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "X, y = X.to(device), y.to(device)  # <-- here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rest stays the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train step\n",
    "model.train()\n",
    "optimizer.zero_grad()\n",
    "\n",
    "y_ = model(X)\n",
    "loss = criterion(y_, y)\n",
    "\n",
    "loss.backward()\n",
    "optimizer.step()\n",
    "\n",
    "# Eval\n",
    "model.eval()\n",
    "with torch.no_grad():\n",
    "    y_ = model(X)    \n",
    "\n",
    "# Vis\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(X.cpu().numpy(), y_.cpu().numpy(), \".\", label=\"pred\")\n",
    "ax.plot(X.cpu().numpy(), y.cpu().numpy(), \".\", label=\"data\")\n",
    "ax.set_title(f\"MSE: {loss.item():0.1f}\")\n",
    "ax.legend();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}