Chapter 03 - Linear Neural Networks for Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4fc7730a",
   "metadata": {},
   "source": [
    "# Chapter 3: Linear Neural Networks for Regression\n",
    "\n",
    "### Dhuvi Karthikeyan\n",
    "\n",
    "##### 1/11/2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b55935",
   "metadata": {},
   "source": [
    "Minimal notes just on the things I had forgotten/wish to review for retention."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca43b952",
   "metadata": {},
   "source": [
    "## 3.1 Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb02e983",
   "metadata": {},
   "source": [
    "### 3.1.1 Basics\n",
    "\n",
    "#### The Model \n",
    "$$ \\hat{y} = \\mathbf{w}^T\\mathbf{x} + b $$\n",
    "$$ \\hat{\\mathbf{y}} = \\mathbf{X}\\mathbf{w} + b $$\n",
    "\n",
    "Design matrix is the matrix X s.t. rows are examples and columns are features. All the data\n",
    "\n",
    "\n",
    "Affine transformation that decomposes into a linear transformation by weighted sum fo input features and then translation by the addition of a bias term. \n",
    "\n",
    "Conditional mean: $E[Y|X=x]$ -> MLE Estimates\n",
    "\n",
    "#### Loss Function\n",
    "\n",
    "$$l^{(i)}(\\mathbf{w}, b) = \\frac{1}{2}(\\hat{y^{(i)}}-y^{(i)})^2$$\n",
    "$$ L(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i \\in Data}l^{(i)}(\\mathbf{w}, b)$$\n",
    "\n",
    "**Analytic Solution to Linear Regression:** \n",
    "\n",
    "$$ 0 = \\partial ||y-Xw||^2 = -2X^T(y-Xw)$$\n",
    "$$ 0 = -2X^Ty+2X^TXw$$\n",
    "$$ X^Ty = X^TXw$$\n",
    "$$ (X^TX)^{-1}X^Ty = w^*$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e259f96f",
   "metadata": {},
   "source": [
    "### 3.1.3 Normal Distribution and the Squared Loss\n",
    "\n",
    "Take the following: For a given x, we observe y with a value decomposed as the following:\n",
    "$$ y = \\textbf{w}^Tx + b + \\epsilon ~ N(0, \\sigma^2)$$\n",
    "\n",
    "$$ P(y | x) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{(-\\frac{1}{2\\sigma^2}(y - \\textbf{w}^Tx -b)^2)}$$ \n",
    "\n",
    "$$ P(y | X) = \\prod_{i=1}^np(y^{(i)}|x^{(i)}) $$\n",
    "\n",
    "$$ -logP(y|X) = \\sum_{i=1}^n \\frac{1}{2}log(2\\pi\\sigma^2) + \\frac{1}{2\\sigma^2}(y^{(i)} - \\textbf{w}^Tx^{(i)} - b)$$\n",
    "\n",
    "Since the solution does not depend on $\\sigma$ we can ignore the terms involving $\\sigma$:\n",
    "\n",
    "$$ -logP(y|X) = \\sum_{i=1}^n  (y^{(i)} - \\textbf{w}^Tx^{(i)} - b)$$\n",
    "\n",
    "This is the same loss function that is being minimized using the L2 loss."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d7b621e",
   "metadata": {},
   "source": [
    "## 3.2 Object-Oriented Design for Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89723a21",
   "metadata": {},
   "source": [
    "### 3.2.1 Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4c7ec4d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Useful function for adding attributes to objects after instantiation\n",
    "\n",
    "def add_to_class(Class):\n",
    "    def wrapper(obj):\n",
    "        setattr(Class, obj.__name__, obj)\n",
    "    return wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1d88c976",
   "metadata": {},
   "outputs": [],
   "source": [
    "class C:\n",
    "    def __init__(self):\n",
    "        self.instantiated = True\n",
    "        \n",
    "c = C()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "70e06bb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "@add_to_class(C)                  # Limits the scope of adding attributes to just the class\n",
    "def add_name(self, name):\n",
    "    self.name = name\n",
    "    \n",
    "def add_directive(obj, directive): # Global scope to all objects [Not best practices]\n",
    "    obj.directive = directive\n",
    "    \n",
    "@add_to_class(C)\n",
    "def add_subdirective(self, subdirective):\n",
    "    self.subdirective = subdirective"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e02ad3b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "c.add_name('ClassC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "07f1478f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ClassC'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6b08d89b",
   "metadata": {},
   "outputs": [],
   "source": [
    "add_directive(c, \"Exist\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "39167296",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Exist'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.directive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6166147e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pass the butter.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.add_subdirective(\"Pass the butter.\")\n",
    "c.subdirective"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d76fd757",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGiCAYAAADEJZ3cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLAklEQVR4nO3deXhU9b0/8PcsyUwSsu/7RtghbErjRq2puOFDe1vBeoVSa28t/ATT+ii2yK0brVVKrVQqleq9XaC2xaVYLDcKlAKyBKjsITvBbJBMFpLMdn5/4BkmkEAmmZnvWd6v55k/GGb5wMw58/6uxyBJkgQiIiIiBTOKLoCIiIjoWhhYiIiISPEYWIiIiEjxGFiIiIhI8RhYiIiISPEYWIiIiEjxGFiIiIhI8RhYiIiISPEYWIiIiEjxGFiIiIhI8XwOLDt27MDs2bORlpYGg8GAd95555rP2bZtG6ZOnQqLxYKRI0fizTffHEKpREREpFc+B5auri4UFhZizZo1g3p8VVUV7r77btx66604dOgQli5dim9/+9v48MMPfS6WiIiI9MkwnIsfGgwGbNq0CXPmzBnwMU888QQ2b96MI0eOeO6bN28e2trasGXLlqG+NREREemIOdBvsHv3bhQXF/e5b9asWVi6dOmAz+nt7UVvb6/nz263G+fPn0d8fDwMBkOgSiUiIiI/kiQJHR0dSEtLg9E4vGmzAQ8sDQ0NSE5O7nNfcnIy2tvb0d3djbCwsCues3LlSvz4xz8OdGlEREQUBHV1dcjIyBjWawQ8sAzFsmXLUFJS4vmzzWZDVlYW6urqEBUVJbAy5ZEkCevWrcOTTz4Jl8uFMWPG4JlnnkF0dDTCwsIQHh6O7u5u/OAHP8Ann3wCAPj617+Ol19+GdHR0YKrJyIamM1mw3333Yc9e/YgLCwMX/3qVzFp0iQUFhZi/PjxcLvdeOmll7B27Vo4HA4AwIMPPohnn30WsbGxgqsnAGhvb0dmZiYiIyOH/2LSMACQNm3adNXH3HzzzdKSJUv63Ld+/XopKipq0O9js9kkAJLNZhtCldp14cIFaf78+RIACYB03333SR0dHf0+1uFwSM8884xkMpkkAFJ2drb0r3/9K8gVExENTlNTkzR16lQJgBQTEyPt2rVrwMdWVFRI9913n+dceOutt0oOhyOI1dJA/Pn7HfB9WIqKilBaWtrnvq1bt6KoqCjQb61pHR0duPnmm/E///M/MJlMeOmll7BhwwaMGDGi38ebzWYsX74c//znP5Gbm4uamhrMmjUL1dXVwS2ciOga6uvrccstt6CsrAxJSUnYtm3bVX8z8vLysHHjRuzYsQMjRozAxx9/jBUrVgSxYgoKXxNOR0eHdPDgQengwYMSAGnVqlXSwYMHpZqaGkmSJOnJJ5+UHnzwQc/jKysrpfDwcOnxxx+Xjh8/Lq1Zs0YymUzSli1bBv2e7GG50qJFiyQAUkJCgvTRRx/59FybzSbdcMMNEgBp5syZksvlClCVRES+qaqqknJyciQAUmZmpnTy5Emfnr9hwwZPT8vf/va3AFVJg+XP32+fA8vHH3/s+TJ43xYsWCBJkiQtWLBAmjlz5hXPmTx5shQaGirl5eVJv/3tb316TwaWvnbs2OH5f9+6deuQXuP06dNSRESEJ3QSEYnmdrulL37xixIAqaCgwNMQ9tXixYslAFJsbKxUVVXl3yLJJ/78/R7WPizB0t7ejujoaNhsNt1Puu3u7kZhYSHKy8vx0EMP4Te/+c2QX+vXv/41vvvd78JisaCsrAzjxo3zY6VERL7ZuHEj5s2bB6vVimPHjiE3N3dIr9Pb24tbbrkFe/fuxfTp07Fz505YLBY/V0uD4c/fb15LSGX++7//G+Xl5UhLS8NLL700rNf6zne+gzvuuAO9vb2YP3++Z5Y9EVGwdXZ24vvf/z4A4KmnnhpyWAEAi8WCP/3pT4iLi8P+/fv7rDol9WJgUZF9+/Z5Qsprr72GmJiYYb2ewWDAG2+8gdjYWBw4cADPP/+8H6okIvLd888/j/r6euTl5eHxxx8f9utlZ2fjd7/7HQDgV7/6Ff785z8P+zVJLAYWlbDb7XjooYfgdrsxb9483HvvvX553bS0NLz22msAgOeeew779+/3y+sSEQ3WqVOn8PLLLwMAVq9eDavV6pfXvfPOO7Fs2TIAF3ttXC6XX16XxGBgUYmVK1fi008/RUJCAl555RW/vvbcuXMxd+5cuFwufPOb34Tb7fbr6xMRDUSSJDz66KNwOBy46667cM899/j19Z966inEx8ejvLwcGzdu9OtrU3AxsKhAVVWVZ7jmlVdeQWJiot/fY82aNYiJicHRo0fx3nvv+f31iYj68+677+LDDz9EaGgoVq9e7ffrxY0YMcIzh+W5555jL4uKMbCowC9+8Qs4HA7cdtttmDdvXkDeIz4+Ht/97ncBAD/72c8C8h5ERN66u7s9F8L9wQ9+gIKCgoC8z+LFixETE4Pjx4/jL3/5S0DegwKPgUXhbDYb3njjDQDA448/HtCrVT/66KMIDQ3Frl27sGvXroC9DxERcHG+Sk1NDTIzM/HUU08F7H2ioqI8wei5557jsLdKMbAo3G9+8xt0dnZi3LhxuP322wP6XqmpqXjwwQcBsJeFiALL6XRizZo1AC6GiIiIiIC+36OPPoqoqCh8+umnePfddwP6XhQYDCwK5nQ6PRNsH3vssYD2rsjkfRDeffddnDp1KuDvR0T69N5776G+vh6JiYmYO3duwN8vNjYW/+///T8AwLPPPgsV7JlKl2FgUbBNmzahtrYWCQkJeOCBB4LynmPHjsXs2bMhSZJnmSERkb/96le/AgB8+9vfDtoutI899hgiIiJw8OBBbN68OSjvSf7DwKJgq1atAgA88sgjCAsLC9r7yps2vfXWW2hsbAza+xKRPpw4cQKlpaUwGo34r//6r6C9b3x8PBYtWgQAeOaZZ9jLojIMLAq1Z88e7NmzB6Ghofje974X1Pe+6aabMGPGDPT29uLVV18N6nsTkfbJm1Xec889yM7ODup7f//730dYWBj27duHDz/8MKjvTcPDwKJQP//5zwEA3/jGN5CSkhLU9zYYDJ5eljVr1qCzszOo709E2tXV1YU333wTADy9HcGUlJTk2cJBnvRL6sDAokA1NTWe617IS/GCbc6cORg5ciRaW1uxfv16ITUQkfb8/ve/R3t7O0aOHIni4mIhNXznO98BAGzZsgXNzc1CaiDfMbAo0C9/+Uu43W7cdtttKCwsFFKDyWTy7A65evVq7ltARMMmSZKnV+N73/sejEYxP0FjxozB9OnT4XQ6sWHDBiE1kO8YWBSmo6MD69atA3BxRrtICxYsQGRkJKqqqrBnzx6htRCR+u3atQv//ve/ERYWhm9+85tCa5H3nPrf//1foXXQ4DGwKMw777zj6S698847hdYSHh6OOXPmAAD++Mc/Cq2FiNRPXsp8//33IzY2Vmgt8+bNg8lkwr59+3Dy5EmhtdDgMLAozJ/+9CcAFyfbiuou9Xb//fcDuFiX0+kUXA0RqVVjYyPefvttAGIm214uKSkJd9xxBwD2sqiF+F9E8mhra/Mss7vvvvsEV3NRcXEx4uPj0dTUhG3btokuh4hU6re//S0cDgdmzJiBqVOnii4HwKVhod/97necp6cCDCwK8s4778DhcGD8+PEYP3686HIAACEhIfja174GAJycRkRDtnHjRgAXd7ZVinvvvRdRUVGoqanBzp07RZdD18DAoiDycJBSeldk8+bNAwD85S9/QW9vr+BqiEhtKisrcejQIZhMJs+8OCUICwvzNMg4LKR8DCwKcf78eWzduhWA8gLLzTffjLS0NLS1teEf//iH6HKISGX++te/AgBmzpyJhIQEwdX0JQ8Lvf322+ju7hZcDV0NA4tCbNq0CU6nE5MmTcKYMWNEl9OHyWTyhCiuFiIiX8mB5atf/argSq50yy23ICsrCzabDe+//77ocugqGFgUQh4OCsZl1odCXi307rvvoqurS3A1RKQW9fX12L17NwDgK1/5iuBqrmQ0GvHAAw8A4LCQ0jGwKEBzczNKS0sBAF//+tcFV9O/6667Dnl5ebhw4QL+9re/iS6HiFTinXfeAQAUFRUhLS1NbDEDkIeFuFW/sjGwKMCmTZvgcrkwZcoUFBQUiC6nXwaDwTP5lquFiGiw/vKXvwAA/uM//kNwJQMbO3Yspk2bxq36FY6BRQGUPhwkkwPLBx98gLa2NrHFEJHitbS0YPv27QCUOX/FmzwsJPcIkfIwsAjW2NiIjz/+GIByh4NkEydOxPjx42G323lQE9E1vfvuu3C73ZgyZQpyc3NFl3NV99xzDwBgx44daG9vF1wN9YeBRbC//vWvcLvdnjkiSif3snC1EBFdi5JXB12uoKAABQUFcDqd+L//+z/R5VA/GFgEU+pmcQORA0tpaSlsNpvgaohIqWw2m2dvKSXPX/F21113AQA2b94suBLqDwOLQA0NDZ7xXaUPB8lGjhyJ0aNHw+Vy4aOPPhJdDhEp1ObNm+FwODBmzBiMHTtWdDmDcvfddwO4OE9PkiTB1dDlGFgE2rJlCyRJwvTp05GdnS26nEGbNWsWgIv1ExH1Rw2rgy53yy23ICIiAg0NDTh06JDocugyDCwCydvcy5c4Vwu5XjlwERF5u3DhAv7+978DUMf8FZnFYkFxcTEADgspEQOLIG632zO+e/vttwuuxjczZ86ExWJBbW0tTpw4IbocIlKYLVu2oLu7Gzk5OZgyZYrocnwiz2P54IMPBFdCl2NgEeTgwYNoaWlBZGQkvvCFL4guxyfh4eG45ZZbAAAffvih4GqISGnka/J85StfgcFgEFyNb+TAsmfPHrS0tAiuhrwxsAgi/9B/6UtfQkhIiOBqfOc9LEREJJMkybMs+M477xRcje8yMjIwadIkSJLEBpnCMLAIIs9fkSewqo0cWLZv385LshORx6lTp3DmzBlYLBbcdNNNossZEnm1EOexKAsDiwAdHR3YtWsXAPXNX5GNHTsWGRkZ6OnpwY4dO0SXQ0QKIfeu3HjjjQgLCxNczdDIw0JbtmyBy+USXA3JGFgE2LZtGxwOB/Ly8pCfny+6nCExGAwcFiKiK8iBRV5to0Zf+MIXEBsbi9bWVuzZs0d0OfQ5BhYB1D4cJGNgISJvTqfTc200NQcWs9nsOT9ztZByMLAIIE/kUutwkOy2226DyWTCiRMnUFNTI7ocIhLswIEDsNlsiImJwdSpU0WXMyycx6I8DCxBVlVVhfLycphMJtx6662iyxmWmJgYz5JszqYnInk46Etf+hJMJpPgaobnjjvugMFgwOHDh1FfXy+6HAIDS9DJm8UVFRUhOjpacDXDx2EhIpJpYf6KLCEhATNmzADAYSGlYGAJMq0MB8nkwFJaWgqHwyG4GiISpaury7P6UQuBBbi0Wkied0hiMbAEkdPpRGlpKQD1T7iVTZ06FQkJCWhvb+dseiId27lzJ+x2O7KysjBy5EjR5fiFPGy/Y8cOXjdNARhYgmjv3r2w2WyIjY3FtGnTRJfjF0aj0dNbxGEhIv3yHg5S23b8A7nuuutgtVrR1NSEkydPii5H9xhYgkjuViwuLlb9hDRvnMdCRHJgue222wRX4j8Wi8WzsGD79u2CqyEGliDSyv4rl/vyl78MACgrK0Nra6vgaogo2FpaWnDo0CEA2goswMWr0wPgjt4KwMASJK2trfjkk08AXPqB14qUlBSMGjUKAPCvf/1LcDVEFGwfffQRAGDixIlITk4WXI1/yVem3759O+exCMbAEiT//Oc/4Xa7MXr0aGRlZYkux+9uvvlmABf/nUSkL1pazny5L3zhCwgJCUF9fT0qKytFl6NrDCxBsnPnTgCX0rrWyIFF/ncSkX5oObCEh4fj+uuvB8BhIdEYWIJE/iFX6+XWr0UOLPv27UN3d7fgaogoWCorK1FVVQWz2azZBpk8j4UTb8ViYAmC7u5u7N+/H8DFS65rUW5uLlJTU+FwOLB3717R5RBRkMi9K0VFRRgxYoTgagLDex4LicPAEgT79++Hw+FASkoK8vLyRJcTEAaDgfNYiHRo27ZtALS3OsjbDTfcAJPJhOrqatTW1oouR7cYWILAezhIKxsq9YeBhUh/5JWBWh3uBoDIyEjPZp+cxyIOA0sQaH3+ikwOLLt27YLT6RRcDREF2pkzZ1BbWwuj0ei5UKBWcVhIPAaWAHO73bpogQDAhAkTEBUVhc7OTvz73/8WXQ4RBdju3bsBAIWFhZqdvyLjxFvxGFgC7OjRo7DZbIiIiEBhYaHocgLKZDJ5JhVzWIhI++SrM99www2CKwk8eUi/vLwcn332mehydImBJcDk4aCioiKYzWbB1QQe57EQ6YeeAktMTIyn0cl5LGIwsASYXoaDZN6BhdtYE2lXd3c3ysrKAOgjsAAcFhKNgSXA5B4Wre6/crnp06cjNDQUTU1NKC8vF10OEQXIvn374HQ6kZaWhuzsbNHlBAUvhCgWA0sA1dXVoaamBiaTSfMz6GVWq9WzjTW36SfSLu/hIC1v1+BN7kE+evQoWlpaBFejPwwsASQPB02ePBmRkZGCqwkezmMh0j49zV+RJSQkYPz48QDYyyLCkALLmjVrkJOTA6vVihkzZlxzK/bVq1dj9OjRCAsLQ2ZmJh577DH09PQMqWA10cv+K5djYCHSNkmSdBlYAA4LieRzYNm4cSNKSkqwYsUKlJWVobCwELNmzUJTU1O/j//DH/6AJ598EitWrMDx48fxxhtvYOPGjXjqqaeGXbzS6TWwyF3EFRUVXP5HpEHl5eU4d+4crFYrpkyZIrqcoJLnI+7Zs0dwJfrjc2BZtWoVHn74YSxcuBDjxo3D2rVrER4ejvXr1/f7+F27duHGG2/EN77xDeTk5OD222/H/fffr/kL5NlsNs/maXqZcCuLjo7GpEmTALCXhUiL5OHu6667DqGhoYKrCS55jt7Bgwdht9sFV6MvPgUWu92OAwcOoLi4+NILGI0oLi727Hh4uRtuuAEHDhzwBJTKykp88MEHuOuuuwZ8n97eXrS3t/e5qc2ePXsgSRLy8/ORmpoqupygk4eFOPGWSHv0OhwEAPn5+YiLi4Pdbsfhw4dFl6MrPgWWlpYWuFwuJCcn97k/OTkZDQ0N/T7nG9/4Bp555hncdNNNCAkJQX5+Pr74xS9edUho5cqViI6O9twyMzN9KVMR9Lac+XKcx0KkXXoOLAaDwdPLovWRAqUJ+Cqhbdu24YUXXsCvfvUrlJWV4a9//Ss2b96MZ599dsDnLFu2DDabzXOrq6sLdJl+p9f5KzL533348GF0dnYKroaI/KW1tRXHjh0DcHEHbz2St6n45JNPBFeiLz7tFZ+QkACTyYTGxsY+9zc2NiIlJaXf5yxfvhwPPvggvv3tbwMAJk6ciK6uLnznO9/BD3/4QxiNV2Ymi8UCi8XiS2mKYrfbPV9kvQaWtLQ0ZGRk4MyZMygrK/Nc6ZSI1E2ebFpQUIDExETB1YghBxb2sASXTz0soaGhmDZtGkpLSz33ud1ulJaWDpi0L1y4cEUoMZlMAKDZrdsPHTqE7u5uxMXFYfTo0aLLEea6664DwIOaSEvk4SC9DncDl85tJ0+eRGtrq+Bq9MPnIaGSkhKsW7cOb731Fo4fP45HHnkEXV1dWLhwIQBg/vz5WLZsmefxs2fPxmuvvYYNGzagqqoKW7duxfLlyzF79mxPcNGaffv2AbiYwvvrQdIL+aCW/z+ISP3kFUJ6nL8iS0hIQH5+PgCe34LJ58sHz507F83NzXj66afR0NCAyZMnY8uWLZ6JuLW1tX1+pH/0ox/BYDDgRz/6Eerr65GYmIjZs2fj+eef99+/QmHkL7D8g61XDCxE2uJ0Oj3D3XoOLMDF5c0VFRXYu3cvbr/9dtHl6IJBUsG4THt7O6Kjo2Gz2RAVFSW6nGsaP348jh07hvfffx/33HOP6HKEaWtrQ2xsLACgubkZCQkJgisiouEoKyvDtGnTEB0djfPnz+u6B/kXv/gFli5dinvuuQfvv/++6HIUy5+/3/r9tgVIZ2cnjh8/DuDilYv1LCYmBqNGjQLAXhYiLZDnrxQVFek6rADos7RZBe1+TdD3Ny4AysrKIEkSMjIyBlw5pSccFiLSDnmFkN6HgwBgypQpMJvNaGpqQk1NjehydIGBxc/2798PgPNXZAwsRNrB89slVqsVhYWFALgSMlgYWPxM/mHW+3CQTO423bdvH7tNiVSsvb0dp06dAgBMmzZNcDXKwA3kgouBxc+4QqivyZMnw2w2o7GxUZU7FhPRRQcPHoQkScjMzNTthnGXY2AJLgYWP2ptbUVFRQUAtkBkYWFhmDBhAgAOCxGp2YEDBwCw99ib3INcVlYGh8MhuBrtY2DxI3l8V76aJ13EeSxE6icHFjbGLhk1ahSio6PR3d2NI0eOiC5H8xhY/IgT0vrnPY+FiNRJPr+xh+USo9HIS5AEEQOLH3HCbf/kA3r//v1wu92CqyEiX3HC7cA4jyV4GFj8iBNu+zd+/HiEhYX1OekRkXqUlZUBALKysrhj9WW8N5CjwGJg8ZOGhgacOXMGBoMBU6ZMEV2OopjNZkydOhUAh4WI1IgTbgcmB5Zjx46hvb1dcDXaxsDiJ/L47tixYxEZGSm4GuXhxFsi9eKE24GlpKQgKysLkiR5fgcoMBhY/IQTbq+OgYVIveTzGwNL/+R5LBwWCiwGFj/hhNurkwPLwYMHYbfbBVdDRINls9lQXl4OgIFlIFwJGRwMLH4gSRIn3F7DyJEjERMTg97eXu5XQKQi8oTb7OxsTrgdgDxv8dChQ2IL0TgGFj+oq6tDc3MzzGaz52JY1JfBYOCwEJEKccLttcmBpbKyEjabTXA12sXA4gfyD/DEiRNhtVoFV6NcDCxE6sMJt9cWFxeHrKwsAOxlCSQGFj/ghNvB4Y6QROrDCbeDI/eyHDx4UHAl2sXA4geccDs4cmA5evQouru7BVdDRNfS1taG06dPA2BguZbJkycDYA9LIDGwDJPb7WYPyyClpaUhMTERbrebE2+JVECecJuTk4P4+HjB1Sgbe1gCj4FlmCoqKmCz2WC1WjF+/HjR5SiawWBgK4RIRTh/ZfDkwHLs2DH09vYKrkabGFiGSW6BTJo0CSEhIYKrUT45sBw+fFhsIUR0TVwhNHiZmZmIi4uD0+lkD3KAMLAMk/zDK/8Q09Wxh4VIPTjhdvDYgxx4DCzDxMDiG3mfmsOHD8PtdguuhogG0tbWhoqKCgDwXLyUro7zWAKLgWWY5CTNDeMGZ/To0bBYLOjs7ERlZaXocohoAPJwd25uLifcDhIDS2AxsAxDS0sLzp49C+DipnF0bWaz2fN/xXksRMrF4SDfyYHl8OHDcLlcgqvRHgaWYZB/cPPz8xEZGSm4GvXgOC+R8sk9LAwsgzd69GiEhYWhq6vLs38N+Q8DyzDIP7icv+IbefiMgYVIuf79738D4HC3L0wmk6cHmec3/2NgGQa5h4UHtG/Yw0KkbD09PTh58iQAnt98xXksgcPAMgwMLEMzadIkAMCZM2dw7tw5wdUQ0eWOHj0Kt9uN+Ph4pKamii5HVRhYAoeBZYh6e3tx7NgxABwS8lVUVBTy8/MBcOItkRJ5DwcZDAbB1aiLd2CRJElwNdrCwDJEx48fh9PpRExMDDIzM0WXozqcx0KkXHJDQu4NpcGbMGECjEYjmpub8dlnn4kuR1MYWIbIeziILRDfcR4LkXJxwu3QhYeHY8yYMQA4LORvDCxDxA3jhofXFCJSJkmSPIGFPSxDw3ksgcHAMkTckn945P83XtmUSFnOnj2Lc+fOwWQyYdy4caLLUSUGlsBgYBkCSZK4QmiYMjIyEBsbC6fT6Zm8TETiyb0ro0ePhtVqFVyNOnHIOzAYWIbgzJkzOH/+PFsgw8ArmxIpEyfcDp/cw1JZWQmbzSa4Gu1gYBkC+YAeO3YsWyDDwMBCpDyccDt8cXFxyMrKAsDzmz8xsAwBh4P8gxNviZSHE279g/NY/I+BZQi4Qsg/vPdi4QZLROL19PTgxIkTABhYhos9yP7HwDIE7GHxj7FjxyIkJAQ2mw01NTWiyyHSvePHj8PlciEuLg7p6emiy1E1OfB9+umngivRDgYWH3V2dnouG87AMjyhoaEYP348ALZCiJTAe8ItN8QcHvmqzceOHYPL5RJcjTYwsPjo008/hSRJSElJQXJysuhyVI/zWIiUgxNu/ScvLw9hYWHo6elBRUWF6HI0gYHFRxwO8i9eU4hIOTjh1n+8t704cuSI4Gq0gYHFRwws/sWJaUTKwA0x/W/ChAkAGFj8hYHFR/IPK7fk9w/5xFhdXc0NlogEamhoQEtLC4xGIzfE9BN5Hgsn3voHA4sP3G6354vHFoh/xMbGelYjcIt+InHk3pVRo0YhLCxMcDXawB4W/2Jg8UFFRQW6urpgsVgwatQo0eVohrxSiAc1kTiccOt/cg9LeXk5enp6BFejfgwsPpB7V8aPHw+z2Sy4Gu2QWyFHjx4VXAmRfnHCrf+lpqYiNjYWLpfLsyEfDR0Diw/kHgA5NZN/sNuUSDxOuPU/g8HAeSx+xMDiA7kHQB7CIP9gYCESq7e3l1vyBwjPb/7DwOID+QvHwOJfY8eOBQA0NjaipaVFcDVE+nPixAk4nU7ExsYiIyNDdDmawh4W/2FgGSS73Y5Tp04BuJSYyT9GjBiB3NxcAJzHQiQCt+QPHPaw+A8DyyCVl5fD6XQiMjISmZmZosvRHB7UROJwwm3gyOe2uro6tLW1iS1G5RhYBkn+IR03bhxbIAHAwEIkDhcUBE5MTIxnmI09yMPDwDJI8heNw0GBwb1YiMThgoLA4jwW/2BgGSROuA0s771YJEkSXA2RfthsNpw5cwYAuCV/gLAH2T8YWAaJPSyBNXr0aJhMJrS2tuKzzz4TXQ6RbsiXxEhPT0dMTIzYYjSKgcU/GFgGoaenB6dPnwbAHpZAsVqtGDlyJAAe1ETBJAcW9q4EjveQEHuQh46BZRBOnDgBt9uNmJgYpKamii5Hs9gKIQo+zl8JvDFjxsBoNOL8+fNoaGgQXY5qMbAMgvdwEFcIBQ4DC1HwMbAEXlhYGAoKCgBw4u1wMLAMAg/o4OBFEImCj+e34GCDbPgYWAZB/oJxwm1gySfMo0ePwu12C66GSPtsNhvq6+sBXLpEBgUGlzYP35ACy5o1a5CTkwOr1YoZM2Zg7969V318W1sbFi1ahNTUVFgsFowaNQoffPDBkAoWgS2Q4Bg5ciRCQ0PR1dWFmpoa0eUQaR5XCAUPe1iGz+fAsnHjRpSUlGDFihUoKytDYWEhZs2ahaampn4fb7fb8eUvfxnV1dX485//jJMnT2LdunVIT08fdvHB0NXVhcrKSgAMLIEWEhKCMWPGAOBBTRQMbIwFj9zDcvToUbhcLsHVqJPPgWXVqlV4+OGHsXDhQowbNw5r165FeHg41q9f3+/j169fj/Pnz+Odd97BjTfeiJycHMycOROFhYXDLj4Yjh8/DgBITExEUlKS4Gq0j/NYiIJHPs64pDnw8vPzYbVa0d3djaqqKtHlqJJPgcVut+PAgQMoLi6+9AJGI4qLi7F79+5+n/Pee++hqKgIixYtQnJyMiZMmIAXXnjhqgmzt7cX7e3tfW6isAUSXNyinyh4eH4LHpPJ5AmGnMcyND4FlpaWFrhcLiQnJ/e5Pzk5ecC15ZWVlfjzn/8Ml8uFDz74AMuXL8fLL7+M5557bsD3WblyJaKjoz03kVdH5oTb4OI4L1HwyHNYGFiCg+e34Qn4KiG3242kpCS8/vrrmDZtGubOnYsf/vCHWLt27YDPWbZsGWw2m+dWV1cX6DIHxBZIcMkH9PHjx+F0OgVXQ6RdbW1tnhVCHBIKDnkeCwPL0Jh9eXBCQgJMJhMaGxv73N/Y2IiUlJR+n5OamoqQkBCYTCbPfWPHjkVDQwPsdjtCQ0OveI7FYoHFYvGltIDhRQ+DKycnB+Hh4bhw4QIqKiowevRo0SURaZL3CqHo6GjB1eiDvHRcnhtJvvGphyU0NBTTpk1DaWmp5z63243S0lIUFRX1+5wbb7wRp0+f7rOvxqlTp5CamtpvWFGS9vZ2T+8OA0twGI1GT2uPrRCiwGHvcfDJ57aTJ0+yB3kIfB4SKikpwbp16/DWW2/h+PHjeOSRR9DV1YWFCxcCAObPn49ly5Z5Hv/II4/g/PnzWLJkCU6dOoXNmzfjhRdewKJFi/z3rwgQuQWSmpqKuLg4wdXoB8d5iQKP81eCLzs7G2FhYbDb7VwpNAQ+DQkBwNy5c9Hc3Iynn34aDQ0NmDx5MrZs2eKZiFtbWwuj8VIOyszMxIcffojHHnsMkyZNQnp6OpYsWYInnnjCf/+KAOFwkBgMLESBxyXNwWc0GjF69GgcOnQIx48f91xfiAbH58ACAIsXL8bixYv7/btt27ZdcV9RURH27NkzlLcSyvuihxQ8XNpMFHgcEhJj7NixnsBy7733ii5HVXgtoavgAS2GHBDLy8vR29sruBoi7Wlra8PZs2cBsIcl2OT/b3lIjgaPgeUquAeLGOnp6YiMjITL5cLp06dFl0OkOfKPZUZGBlcIBRlXCg0dA8sAzp8/j88++wwAWyDBZjAYPNcU4kFN5H+cvyKOHFhOnDgBSZIEV6MuDCwDkFsgmZmZiIqKElyN/rAVQhQ4HO4WZ+TIkTCZTOjo6PBs3EeDw8AyAPmHki0QMbxbIUTkXwws4oSGhnpWB3Eei28YWAYgBxb5h5OCiz0sRIHDPVjE4vltaBhYBiC37OW5FBRc8v/7iRMn+uySTETD471CiA0yMRhYhoaBZQDsYRErPz8fISEh6O7uFnrxSyKtkYeDuEJIHAaWoWFg6ceFCxdQU1MDgD0sopjNZs84Lw9qIv/h/BXxuBfL0DCw9KO8vBySJCEuLg6JiYmiy9EttkKI/E/+keSCAnHkq9C3tLSgpaVFcDXqwcDSD/kHcsyYMTAYDIKr0S/uxULkf1wBKV5ERASys7MB8PzmCwaWfnDCrTJwaTOR/8nHE+fnicUeZN8xsPSDE26VgQc0kX91dXWhtrYWABtkonEei+8YWPrBHhZl4DgvkX+dOnUKABAfH4/4+HjB1egbG2S+Y2C5jMvlwsmTJwGwh0W0iIgIZGVlAeCwEJE/sDGmHAwsvmNguUxNTQ16e3thsViQk5Mjuhzd40FN5D8MLMohn9vq6urQ0dEhuBp1YGC5jPzDOGrUKJhMJsHVEAMLkf/IvccMLOLFxcUhOTkZAHuQB4uB5TJsgSgLAwuR//D8piw8v/mGgeUyXCGkLN7XFCKioXO73exhURgGFt8wsFyGLRBlkQ/ompoaXLhwQXA1ROpVW1uLnp4ehIaGcn6eQnBps28YWC7DTZWUJTExEfHx8ZAkydM6JCLfyee2goICmM1mwdUQwB4WXzGweGlubsa5c+dgMBgwatQo0eXQ57jjLdHwsfdYeeRzW0VFBXp7ewVXo3wMLF7kAzo7Oxvh4eGCqyEZrylENHwMLMqTmpqKqKgouN1ulJeXiy5H8RhYvHhf9JCUg92mRMPHwKI8BoOB81h8wMDihfNXlImBhWj4GFiUiee3wWNg8cIeFmWSD+jy8nI4nU7B1RCpT2trKxobGwFcukYXKQMDy+AxsHhhD4syZWVlISwsDHa7HVVVVaLLIVIdeYVdeno6IiMjBVdD3uQAyVWQ18bA8rkLFy6gpqYGAHtYlMZoNHoOarZCiHzH4SDlks9tp06dgtvtFlyNsjGwfO7UqVOQJAnx8fFITEwUXQ5dhkubiYaOgUW58vLyYDabceHCBZw9e1Z0OYrGwPI5zl9RNi5tJho6bsmvXCEhIcjLywPAYaFrYWD5HOevKBsnphENHXtYlI3zWAaHgeVzPKCVzTuwSJIkuBoi9XA4HDh9+jQArhBSKgaWwWFg+Ryv0qxsBQUFMBqNaG9v9yzPJKJrq6yshNPpREREBNLT00WXQ/1gYBkcBhYALpcLp06dAsAeFqWyWCyeK8zKnxURXZvcezx69GgYjTzlK5F87ToGlqvjtxdAdXU1ent7YbVakZ2dLbocGgAPaiLfcbhb+eQelpqaGnR3dwuuRrkYWND3susmk0lwNTQQ7/0KiGhwGFiULykpCdHR0ZAkCRUVFaLLUSwGFoDDQSrBHhYi3zGwKJ/BYOA8lkFgYMGlwCL/IJIysYeFyDeSJDGwqAQDy7UxsICBRS3kz6eiogIOh0NwNUTK19TUhLa2NhgMBhQUFIguh66CgeXaGFhw6QvCwKJs6enpCAsLg9PpRHV1tehyiBRP7l3Jzc2F1WoVXA1dDYe8r033gaWzsxP19fUAGFiUzmg08qAm8gGHg9TDu4eFm2P2T/eBRd4BMiEhAXFxcYKroWuRAwvnsRBdGwOLehQUFMBgMKCtrQ3Nzc2iy1Ek3QcWDgepC8d5iQZPDvbckl/5wsLCkJWVBYDnt4HoPrBwwq26sIeFaPB4flMXroS8OgYWtkBUhQc00eDY7XZUVVUBYGBRC/YgXx0DC1sgqiJ/TmfPnkVHR4fgaoiUq6qqCi6XCxEREUhNTRVdDg0CA8vV6TqwSJLEOSwqExMTg6SkJABAeXm54GqIlMu7MWYwGARXQ4PBVZBXp+vA0tzcDJvNBoPBgPz8fNHl0CDxoCa6NvYeq4/cw8LNMfun68AiH9DZ2dkICwsTXA0NFifeEl0bA4v6ZGRkeDbHlOcf0SUMLOABrTYc5yW6Np7f1IebY16drgML56+oE3tYiK5NnuPF85u6cCXkwHQdWNgCUSduYU10dd6XHOFFD9WFPcgDY2AB92BRm7y8PBiNRnR2dqKhoUF0OUSK433JkdjYWMHVkC84JDQw3QYWl8vlOajZw6IuFosFubm5AHhQE/WHvcfqxR6Wgek2sNTU1MBut8NisSAzM1N0OeQjzmMhGhgDi3rJgaWxsRE2m01wNcqi28AiH9AjR46EyWQSXA35ioGFaGAMLOoVFRWFlJQUAOxluZzuAwvnr6gTu02JBsbAom48v/VP94GFB7Q6sYeFqH+85Ij6cWlz/3QbWHhAq5t8QFdWVnILayIv586dQ1tbG4CLQ96kPuxh6Z9uAwt7WNQtLS0N4eHh3MKa6DLyhnFZWVm85IhKyXvn8AKvfekysHR3d6O2thYA57CoFbewJuofG2Pq5x1YuDnmJboMLPL+K7GxsYiPjxdcDQ0V57EQXYmBRf3kzTG7urq4OaYXXQYW7/krBoNBcDU0VBznJbqSHFi4Jb96hYaGIjs7GwCHhbwNKbCsWbMGOTk5sFqtmDFjBvbu3Tuo523YsAEGgwFz5swZytv6DVsg2sAeFqIr8fymDXLg5PntEp8Dy8aNG1FSUoIVK1agrKwMhYWFmDVrFpqamq76vOrqavzgBz/AzTffPORi/YUHtDZwDgtRX263m1dp1gj582MPyyU+B5ZVq1bh4YcfxsKFCzFu3DisXbsW4eHhWL9+/YDPcblceOCBB/DjH/8YeXl513yP3t5etLe397n5EzeN0wb582toaPD7d4RIjerr69Hd3Q2z2YycnBzR5dAwcKXQlXwKLHa7HQcOHEBxcfGlFzAaUVxcjN27dw/4vGeeeQZJSUl46KGHBvU+K1euRHR0tOfm72v9cA8WbYiOjkZiYiKASxOpifRMbozl5+fDbDYLroaGg4HlSj4FlpaWFrhcLiQnJ/e5Pzk5ecCZzDt37sQbb7yBdevWDfp9li1bBpvN5rnV1dX5UuZVnTt3DufPnwfATZW0QP4MGViIwOEgDZEDy+nTp+F2uwVXowwBXSXU0dGBBx98EOvWrUNCQsKgn2exWBAVFdXn5i9yCyQjIwMRERF+e10Sg60Qoks4P087cnJyYDab0dPTg/r6etHlKIJPfYYJCQkwmUxobGzsc39jY6Pn6pLeKioqUF1djdmzZ3vuk5Oi2WzGyZMnkZ+fP5S6h4wHtLYwsBBdwvObdpjNZuTm5qK8vBzl5eV+nxqhRj71sISGhmLatGkoLS313Od2u1FaWoqioqIrHj9mzBh8+umnOHTokOd277334tZbb8WhQ4eEfADy0AH3KNAGBhaiSxhYtIXnt758npVVUlKCBQsWYPr06bj++uuxevVqdHV1YeHChQCA+fPnIz09HStXroTVasWECRP6PD8mJgYArrg/WOQPnvNXtMF7nJdIzxwOByorKwEwsGgFA0tfPgeWuXPnorm5GU8//TQaGhowefJkbNmyxTMRt7a2FkajcjfQZQ+LtsjBs6mpCe3t7X6d70SkJlVVVXC5XIiIiEBqaqrocsgPGFj6GtK6t8WLF2Px4sX9/t22bduu+tw333xzKG/pF5IksYdFY6KiopCUlISmpiaUl5dj2rRpoksiEsJ7S35eckQbGFj6Um5XSAA0Nzejvb0dBoMh6JN9KXB4UBNx/ooWyee2iooKuFwuwdWIp6vAIg8HZWZmwmq1Cq6G/IXzWIgYWLQoKysLoaGhsNvtqK2tFV2OcLoKLBwO0ib582QPC+mZ/P3n/DztMJlMntEAnt90Flg44VabOCRExPObVvH8domuAgtbINrEA5r0rqenx3MJE/YgawvPb5foMrDwgNYW+fNsaWlBW1ub2GKIBKisrIQkSYiKivLpMiikfAwsl+gmsEiSxC5TjYqMjPTsA8SJt6RH3uc2LmnWFgaWS3QTWLyXNOfl5Ykuh/yMBzXpmRxY2HusPfK5raqqCk6nU3A1YukmsMg/ZFzSrE0MLKRnnJ+nXenp6bBarXA6naiurhZdjlC6CSwcDtI27sVCesYeFu0yGo3cuuFzugksnHCrbexhIT3j+U3beH67SDeBhT0s2sYWCOlVb2+vZxdUnt+0Sd69WO/nN90EFo7xapscWM6dO4fW1lbB1RAFT1VVFSRJQmRkJBITE0WXQwHAHpaLdBFYeJVm7RsxYgRSU1MBcB4L6Yv3uY1LmrVJDizy9aL0SheBpbm5GR0dHVzSrHFshZAecbhb++TPtqamBna7XXA14ugisHBJsz5wHgvpEVcIaV9KSgpGjBgBt9uNyspK0eUIo4vAwhaIPrCHhfSIw93aZzAY2CCDTgILJ9zqA/diIT1ig0wf2CDTWWBhC0TbeECT3tjtdtTU1ADg+U3reH7TSWBhC0Qf8vPzAQDnz5/H+fPnBVdDFHhVVVVwu90YMWKE5wKgpE0MLDoILN5LmhlYtC0iIgJpaWkA9H1Qk354T7jlkmZtk3vQKioqBFcijuYDi/eS5tzcXNHlUIBxHgvpCYe79UP+jGtra3W7tFnzgUU+oLOysrikWQfYbUp6wuFu/UhOTkZERATcbrdur9qsm8DCFog+MLCQnnAPFv3wXtqs1x5kzQcWtkD0hXsVkJ6wQaYv8sICBhaN4oRbfeEcFtILu93uGRrg+U0f2MOicWyB6IvcAmltbcW5c+cEV0MUONXV1XC73QgPD0dKSorocigI9L5SSNOBRZIkDgnpTHh4ONLT0wHo96AmfeCSZv1hD4uGNTU18SrNOqT3cV7SBzbG9Ec+t1VVVcHpdAquJvg0HVjkAzorKwsWi0VwNRQsem+FkD5wuFt/MjIyYLFY4HA4UFdXJ7qcoNNFYOEBrS9yK4RDQqRlPL/pj9Fo9IwW6LFBxsBCmsMeFtIDDgnpk54n3mo6sMgfqNziJn3Q8wFN+uBwOFBVVQWADTK90XODTNOBhT0s+iQH1MbGRnR0dAiuhsj/ampq4HK5EBYWhtTUVNHlUBDpeVGBpgMLe1j0KTo6GgkJCQDYy0La5D3h1mjU9GmcLsMeFg1qbW3F+fPnATCw6BEn3pKWsfdYv+TPvLKyEm63W3A1waXZwCL/UKWmpiIiIkJwNRRsem6FkPZxwq1+ZWdnw2w2o7u7G5999pnocoJKs4FFPqDZu6JPnHhLWsbzm36ZzWZkZ2cD0F+DTLOBRf6hYpepPul5YhppH4eE9E2vPciaDSxsgeibXg9o0j6Xy+VZ0szzmz7p9fym2cDCHhZ9k0/kZ86cQU9Pj+BqiPznzJkzcDgcCA0NRUZGhuhySAC9DnlrNrCwh0XfEhMTERkZCUmSPK1RIi2Qz225ubkwmUyCqyER2MOiIV1dXZ7Z0+xh0SeDwaDbVghpG/eXIu85epIkCa4meDQZWCorKwEAsbGxiI2NFVwNicKJt6RFDCyUm5sLg8GAjo4ONDc3iy4naDQZWDh/hQD9dpuStnGFEFmtVmRmZgLQVw+yJgML568QoN+JaaRt7GEhQJ8NMk0GFvawEMAhIdIeSZIYWAiAPs9vmgws7GEh4FJgra6uhtPpFFwN0fA1NTWhs7MTBoMBubm5osshgdjDohHsYSEASEtLg8VigdPpRG1trehyiIZNPrdlZmbCYrEIroZEYmDRALvdjpqaGgDsYdE7o9Goy25T0i4OB5FMj3P0NBdYampq4Ha7ER4ejpSUFNHlkGB6PKhJu7hCiGRyaD137hxaW1sFVxMcmgss3vNXDAaD4GpINPawkJawh4VkERERnka5XhpkmgssnL9C3tjDQlrCwELe9DaPRXOBhSuEyBt7WEhLeH4jb3prkGkusLCHhbx5H9But1twNURD197ejpaWFgAMLHSRfH4rLy8XXElwaC6wsAVC3rKzs2E2m9HT0+O5ICaRGsmNscTERERFRQmuhpRA/p1jD4sKuVwuz4UP2cNCAGA2m5GdnQ2Aw0KkbmyM0eUYWFSsvr4edrsdISEhngtDEeltnJe0icPddDk5sHz22We4cOGC4GoCT1OBRT6gc3NzYTKZBFdDSsGJt6QFXCFEl4uLi0NMTAwAeEYXtExTgYVdptQfvS39I23i+Y36o6dhIU0FFnaZUn84JERawPMb9YeBRaXYAqH+eA8JSZIkuBoi3/X09ODMmTMAeH6jvhhYVIotEOpPXl4egIv7WJw7d05wNUS+q6qqgiRJGDFiBBITE0WXQwrCwHINa9asQU5ODqxWK2bMmIG9e/cO+Nh169bh5ptvRmxsLGJjY1FcXHzVxw+VJEnsYaF+Wa1WpKenA9DHQU3a4z3hltdII2/y7x0n3fZj48aNKCkpwYoVK1BWVobCwkLMmjULTU1N/T5+27ZtuP/++/Hxxx9j9+7dyMzMxO233476+vphF++tubkZnZ2dMBgMyM3N9etrk/rpqRVC2sPeYxqIfG6rrq6Gy+USXE1g+RxYVq1ahYcffhgLFy7EuHHjsHbtWoSHh2P9+vX9Pv73v/89vve972Hy5MkYM2YMfvOb38DtdqO0tHTA9+jt7UV7e3uf27XIvSuZmZmwWCy+/rNI4xhYSM24pJkGkpaWhtDQUDgcDtTV1YkuJ6B8Cix2ux0HDhxAcXHxpRcwGlFcXIzdu3cP6jUuXLgAh8OBuLi4AR+zcuVKREdHe26D2QSOBzRdDQMLqRmHu2kgJpPJM6qg9fObT4GlpaUFLpcLycnJfe5PTk5GQ0PDoF7jiSeeQFpaWp/Qc7lly5bBZrN5boNJjQwsdDUMLKRmHBKiq9HL+c0czDf7yU9+gg0bNmDbtm2wWq0DPs5isfg8rMPAQlcjrxTSw8Q00haXy4WqqioAPL9R//QSWHzqYUlISIDJZEJjY2Of+xsbG5GSknLV57700kv4yU9+gn/84x+YNGmS75Veg/xDxAOa+iN/L86ePYvu7m7B1RANXl1dHRwOB0JCQpCRkSG6HFIgBpZ+hIaGYtq0aX0mzMoTaIuKigZ83osvvohnn30WW7ZswfTp04de7VWwh4WuJi4uDtHR0QDYy0LqIp/b8vLyeI006hcDywBKSkqwbt06vPXWWzh+/DgeeeQRdHV1YeHChQCA+fPnY9myZZ7H//SnP8Xy5cuxfv165OTkoKGhAQ0NDejs7PTbP6Kzs9PT6yN3/RN5MxgMujmoSVvYGKNr8T63aXk3b5/nsMydOxfNzc14+umn0dDQgMmTJ2PLli2eibi1tbUwGi/loNdeew12ux1f+9rX+rzOihUr8N///d/Dq/5zcovZ+8qVRJfLz89HWVkZAwupClcI0bXk5ubCYDCgo6MDLS0tmt0NeUiTbhcvXozFixf3+3fbtm3r8+fq6uqhvIVP2AKhwWAPC6kRz290LfJu3mfOnEFFRYVmA4smriXECbc0GFwpRGrEwEKDoYcGmSYCi/ekNKKB6OGAJm2RJIkNMhoUPZzfNBVYeEDT1cjfj6qqKs1fc4O0oaWlBR0dHQDAa6TRVTGwqAQDCw1GRkYGQkJC4HA4cObMGdHlEF2TfG5LT0+/6mabRAwsKuB0OlFTUwOAgYWuTk/X3CBt4HAQDRYDiwrU1dXB6XTCYrEgLS1NdDmkcPI8Jy0f1KQd7D2mwZK/Iw0NDejq6hJcTWCoPrDIB3Rubm6f/V+I+iMf1FwpRGrABQU0WLGxsYiNjQWg3fOb6n/h2QIhX+ih25S0g0NC5Autn98YWEhXtH5Ak7bw/Ea+0Pr5TfWBRW6BsMuUBkMv19wg9evu7sbZs2cB8PxGg8PAonBsgZAv5FVCNpsN58+fF1wN0cCqqqoAAFFRUYiPjxdcDakBA4uCSZLEwEI+CQ8PR2pqKgDtHtSkDd7nNoPBILgaUgMGFgXjLpA0FFwpRGrA4W7ylXxuq6mpgdPpFFyN/6k6sMgHNHeBJF9ovRVC2sDeY/JVWloaLBYLnE4namtrRZfjd6oOLDygaSgYWEgNeH4jXxmNRk1flZ6BhXSHu92SGnBIiIZCyw0yBhbSHS0f0KQNbrfbs0qI5zfyhZbPb5oILGyBkC/kA7q+vh7d3d2CqyG6Un19PXp7e2E2m5GZmSm6HFIRBhaF4rbVNBQJCQmIjIwEAFRXV4sthqgf8rktOzsbZrNZcDWkJloe8lZtYPHeBZKBhXxhMBg03Qoh9eNwNw2V97YNWtvNW7WBRW6BREdHIy4uTnA1pDYMLKRkDCw0VDk5OTAYDOjo6EBLS4vocvxKtYHFe/4Kd4EkX2m525TUj8PdNFRWqxXp6ekAtLe0WbWBhQc0DQd7WEjJuKCAhkOrDTLVBhZ2mdJwMLCQkvH8RsOh1cuPMLCQLsnfm6qqKrjdbsHVEF3S1tbmuZI4e1hoKNjDojDsMqXhyMzMhNlsht1uR319vehyiDzkVnFSUhJGjBghuBpSI/awKIjL5eIukDQsZrMZ2dnZALR3UJO6sfeYhos9LApSX18Ph8OBkJAQ7gJJQ8Z5LKREXFBAw+W9m3dPT4/gavxHlYFF/oHJycmByWQSXA2plZavakrqxeFuGq74+HjPbt7yaIQWqDKw8Cqm5A9aHecldWMPCw2X927eWjq/qTKwcIyX/EGr47ykbjy/kT9o8fymysDCHhbyBw4JkdLY7XbU1tYC4PmNhoc9LArBFgj5g/yD0NLSgvb2dsHVEAG1tbVwu90IDw9HSkqK6HJIxdjDohDsYSF/iIqKQkJCAgBttUJIvXiNNPIXLfYgqy6weO8CmZubK7gaUjsubSYl4Qoh8hfvISGt7OatusDivQukvGyLaKi02Aoh9eIKIfKXrKwsmEwm9PT0oKGhQXQ5fqHawMIWCPmDFiemkXqxh4X8JSQkBFlZWQC004OsusDCCbfkT1qcmEbqxR4W8iet9SCrLrCwh4X8iT0spBSSJLFBRn6ltfMbAwvpmvw9qqmpgdPpFFwN6VlzczO6urpgMBg8F+YkGg6t9SCrLrCwBUL+lJaWBovFAqfTibq6OtHlkI7JjbGMjAxYLBbB1ZAWsIdFIIfDwV0gya+MRqNnebxWDmpSJzbGyN/YwyJQXV0dXC4XrFYrUlNTRZdDGqG1g5rUicPd5G9y+G1qakJnZ6fgaoZPVYGluroawMUN44xGVZVOCqa1blNSJ/awkL9FR0cjLi4OgDbOb6r61a+qqgLAA5r8iz0spATsYaFA0FKDTFWBRe5h4QFN/qSlA5rUi4GFAkFLDTIGFtI97wNakiTB1ZAedXd3o76+HgB7kMm/tNQgU1Vg4ZAQBYK8Sshms6G1tVVwNaRHcmMsKirKM+eAyB/YwyIIe1goEMLDwz2rzrTQCiH18Z5wazAYBFdDWsIeFkE6OjoAXGoRE/mLllohpD6cv0KBIn+nqqur4XK5BFczPKoKLMDFnUnDwsJEl0Eao6VWCKkPAwsFSnp6OkJCQuBwOHDmzBnR5QyL6gILD2gKBPawkEjcg4UCxWQyeUYl1H5+U11g4QFNgcAeFhKJPSwUSPL3Su3nN9UFFh7QFAhaOaBJfdxuNwMLBZTcIGMPS5Cxh4UCQf5e1dXVwW63C66G9KShoQE9PT0wmUzIysoSXQ5pkFaGvFUXWNgCoUBISkpCREQE3G43ampqRJdDOiL3rmRlZSEkJERwNaRFWhnyZmAhAmAwGDTTCiF14YRbCjStDHmrKrCEh4cjKSlJdBmkUVo5qEldOH+FAk3+brW2tqp6N29VBZacnBzuAkkBwx4WEkH+vjGwUKBEREQgOTkZgLobZKoKLNzhlgJJK+O8pC7y941DQhRIWji/qSqw5OTkiC6BNIxDQiQCh4QoGLTQg8zAQvQ5770KJEkSXA3pQWdnJxobGwGwh4UCSwsNMlUFFg4JUSBlZ2fDYDCgq6sLTU1NosshHaiqqgIAxMXFITo6WnA1pGW6HRJas2YNcnJyYLVaMWPGDOzdu/eqj3/77bcxZswYWK1WTJw4ER988MGQimVgoUCyWCzIzMwEoO6DmtSDE24pWHQ5JLRx40aUlJRgxYoVKCsrQ2FhIWbNmjVgi3TXrl24//778dBDD+HgwYOYM2cO5syZgyNHjvhcLHeBpEDTQrcpqQcn3FKwyN+x2tpaOBwOwdUMjdnXJ6xatQoPP/wwFi5cCABYu3YtNm/ejPXr1+PJJ5+84vG/+MUvcMcdd+Dxxx8HADz77LPYunUrXn31Vaxdu7bf9+jt7UVvb6/nzzabDQDQ09OD9vZ2X0smGrSMjAwAwNGjR/ldo4A7ceIEACA9PZ3fNwqo8PBwWK1W9PT04MiRI0ELyfL32i/zAiUf9Pb2SiaTSdq0aVOf++fPny/de++9/T4nMzNT+vnPf97nvqefflqaNGnSgO+zYsUKCQBvvPHGG2+88aaBW0VFhS9xo18+9bC0tLTA5XJ5NqCRJScne1oKl2toaOj38Q0NDQO+z7Jly1BSUuL5c1tbG7Kzs1FbW8uJaYK1t7cjMzMTdXV1iIqKEl2OrvGzUA5+FsrCz0M5bDYbsrKyEBcXN+zX8nlIKBgsFgssFssV90dHR/PLpxBRUVH8LBSCn4Vy8LNQFn4eymE0Dn9Rsk+vkJCQAJPJ5Nk3QNbY2IiUlJR+n5OSkuLT44mIiIgu51NgCQ0NxbRp01BaWuq5z+12o7S0FEVFRf0+p6ioqM/jAWDr1q0DPp6IiIjocj4PCZWUlGDBggWYPn06rr/+eqxevRpdXV2eVUPz589Heno6Vq5cCQBYsmQJZs6ciZdffhl33303NmzYgP379+P1118f9HtaLBasWLGi32EiCi5+FsrBz0I5+FkoCz8P5fDnZ2GQJN/XGr366qv42c9+hoaGBkyePBmvvPIKZsyYAQD44he/iJycHLz55puex7/99tv40Y9+hOrqahQUFODFF1/EXXfdNeziiYiISB+GFFiIiIiIgklV1xIiIiIifWJgISIiIsVjYCEiIiLFY2AhIiIixVN8YFmzZg1ycnJgtVoxY8YM7N27V3RJurRy5Upcd911iIyMRFJSEubMmYOTJ0+KLkv3fvKTn8BgMGDp0qWiS9Gt+vp6/Od//ifi4+MRFhaGiRMnYv/+/aLL0h2Xy4Xly5cjNzcXYWFhyM/Px7PPPuufi+7RVe3YsQOzZ89GWloaDAYD3nnnnT5/L0kSnn76aaSmpiIsLAzFxcUoLy/3+X0UHVg2btyIkpISrFixAmVlZSgsLMSsWbPQ1NQkujTd2b59OxYtWoQ9e/Zg69atcDgcuP3229HV1SW6NN3at28ffv3rX2PSpEmiS9Gt1tZW3HjjjQgJCcHf//53HDt2DC+//DJiY2NFl6Y7P/3pT/Haa6/h1VdfxfHjx/HTn/4UL774In75y1+KLk3zurq6UFhYiDVr1vT79y+++CJeeeUVrF27Fp988gkiIiIwa9Ys9PT0+PZGw758YgBdf/310qJFizx/drlcUlpamrRy5UqBVZEkSVJTU5MEQNq+fbvoUnSpo6NDKigokLZu3SrNnDlTWrJkieiSdOmJJ56QbrrpJtFlkCRJd999t/Stb32rz31f/epXpQceeEBQRfoEQNq0aZPnz263W0pJSZF+9rOfee5ra2uTLBaL9Mc//tGn11ZsD4vdbseBAwdQXFzsuc9oNKK4uBi7d+8WWBkBF6/ACcAvV+Ak3y1atAh33313n+ODgu+9997D9OnT8fWvfx1JSUmYMmUK1q1bJ7osXbrhhhtQWlqKU6dOAQAOHz6MnTt34s477xRcmb5VVVWhoaGhz7kqOjoaM2bM8Pm3XJFXawaAlpYWuFwuJCcn97k/OTkZJ06cEFQVARevH7V06VLceOONmDBhguhydGfDhg0oKyvDvn37RJeie5WVlXjttddQUlKCp556Cvv27cOjjz6K0NBQLFiwQHR5uvLkk0+ivb0dY8aMgclkgsvlwvPPP48HHnhAdGm61tDQAAD9/pbLfzdYig0spFyLFi3CkSNHsHPnTtGl6E5dXR2WLFmCrVu3wmq1ii5H99xuN6ZPn44XXngBADBlyhQcOXIEa9euZWAJsj/96U/4/e9/jz/84Q8YP348Dh06hKVLlyItLY2fhUYodkgoISEBJpMJjY2Nfe5vbGxESkqKoKpo8eLF+Nvf/oaPP/4YGRkZosvRnQMHDqCpqQlTp06F2WyG2WzG9u3b8corr8BsNsPlcokuUVdSU1Mxbty4PveNHTsWtbW1girSr8cffxxPPvkk5s2bh4kTJ+LBBx/EY4895rkQL4kh/17747dcsYElNDQU06ZNQ2lpqec+t9uN0tJSFBUVCaxMnyRJwuLFi7Fp0yZ89NFHyM3NFV2SLt1222349NNPcejQIc9t+vTpeOCBB3Do0CGYTCbRJerKjTfeeMXy/lOnTiE7O1tQRfp14cIFGI19f9JMJhPcbregiggAcnNzkZKS0ue3vL29HZ988onPv+WKHhIqKSnBggULMH36dFx//fVYvXo1urq6sHDhQtGl6c6iRYvwhz/8Ae+++y4iIyM9Y4/R0dEICwsTXJ1+REZGXjFvKCIiAvHx8ZxPJMBjjz2GG264AS+88ALuu+8+7N27F6+//jpef/110aXpzuzZs/H8888jKysL48ePx8GDB7Fq1Sp861vfEl2a5nV2duL06dOeP1dVVeHQoUOIi4tDVlYWli5diueeew4FBQXIzc3F8uXLkZaWhjlz5vj2Rn5ayRQwv/zlL6WsrCwpNDRUuv7666U9e/aILkmXAPR7++1vfyu6NN3jsmax3n//fWnChAmSxWKRxowZI73++uuiS9Kl9vZ2acmSJVJWVpZktVqlvLw86Yc//KHU29srujTN+/jjj/v9fViwYIEkSReXNi9fvlxKTk6WLBaLdNttt0knT570+X0MksRtAImIiEjZFDuHhYiIiEjGwEJERESKx8BCREREisfAQkRERIrHwEJERESKx8BCREREisfAQkRERIrHwEJERESKx8BCREREisfAQkRERIrHwEJERESK9/8BZPixlgPlmUUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "\n",
    "x = np.linspace(0, 10, 100)\n",
    "y = np.sin(x)\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "line, = ax.plot(x, y, color='k')\n",
    "\n",
    "for n in range(len(x)):\n",
    "    time.sleep(1)\n",
    "    line.set_data(x[:n], y[:n])\n",
    "    ax.axis([0, 10, 0, 1])\n",
    "    fig.canvas.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b6078e6",
   "metadata": {},
   "source": [
    "## 3.4 Linear Regression from Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c01ebd9",
   "metadata": {},
   "source": [
    "This is the main problem from HW1 and is pulled from my implementation in HW1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "baf11f9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the DataLoader Class\n",
    "\n",
    "class DataLoader:\n",
    "    \"\"\"\n",
    "    Implements a Data Loading Class for passing mini-batches to model after\n",
    "    minor processing to ensure that matrix multiplication works. Assumes shuffle\n",
    "    to be true.\n",
    "    \"\"\"\n",
    "    def __init__(\n",
    "          self, \n",
    "          inputs,\n",
    "          labels,\n",
    "          batch_size\n",
    "    ):\n",
    "        self.__len__ = len(labels)\n",
    "        self.data = self.gen_tensor(inputs)\n",
    "        self.labels = labels\n",
    "        self.batch_size = batch_size\n",
    "        self.called_idx = np.array([], dtype=int)\n",
    "        self.prefetch = self.__get_idx__()\n",
    "\n",
    "  \n",
    "    def reshape_input(self, inputs):\n",
    "        '''Reshapes input of dims >= 1 to a matrix of n x n_features'''\n",
    "        n_examples = inputs.shape[0] #Assumes first dim is n_examples\n",
    "        assert n_examples == self.__len__\n",
    "        return inputs.reshape(n_examples, int(inputs.size/n_examples))\n",
    "\n",
    "        def gen_tensor(self, inputs):\n",
    "            '''Checks for iterable inputs and runs reshape above''' \n",
    "            try:\n",
    "                iter(inputs)\n",
    "            except TypeError:\n",
    "                print(\"Inputs is not an iterable.\")\n",
    "            return self.reshape_input(inputs)\n",
    "\n",
    "    def __get_idx__(self):\n",
    "        '''Check that we have enough examples for another batch_size'''\n",
    "        if self.__len__ - len(self.called_idx) < self.batch_size:\n",
    "          # On epoch end reset the called indices\n",
    "          self.called_idx = np.array([], dtype=int)\n",
    "        remaining_idx = np.delete(np.arange(self.__len__), list(self.called_idx))\n",
    "        idx = np.random.choice(remaining_idx, self.batch_size, replace=False)\n",
    "        self.called_idx = np.append(self.called_idx, idx)\n",
    "        return idx\n",
    "\n",
    "    def _get_item_(self):\n",
    "        '''Get a batch of the data when called with batch_size'''\n",
    "        sampled_idx = self.__get_idx__()\n",
    "        return (self.data[sampled_idx, :], self.labels[sampled_idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "81041d5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the LinearRegression Model Class\n",
    "\n",
    "class LinearRegression:\n",
    "    \"\"\"\n",
    "    Implements a Linear Regression Model Class for performing linear regression\n",
    "    without the closed form solution. Randomly initializes weights and biases \n",
    "    before using mini-batch SGD to optimize weights\n",
    "    \"\"\"\n",
    "    def __init__(\n",
    "      self,\n",
    "      input_dim\n",
    "    ):\n",
    "        self.weights = np.random.randn(input_dim)\n",
    "        self.bias = 0\n",
    "    \n",
    "    def forward(self, X):\n",
    "        #Vectorized implementation of Forward Call w// broadcasting for bias\n",
    "        return X @ self.weights + self.bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b084f60c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the Gradient Descent Class\n",
    "\n",
    "class SGD:\n",
    "    \"\"\"\n",
    "    Implements gradient descent algorithm. Works with arbitrary models\n",
    "    and loss functions (whose gradients are manually implemented).\n",
    "    \"\"\"\n",
    "    def __init__(\n",
    "        self,\n",
    "        model, \n",
    "        learning_rate,\n",
    "        grad_fxn\n",
    "    ):\n",
    "        self.eta = learning_rate\n",
    "        self.grad = grad_fxn\n",
    "        self.model = model\n",
    "\n",
    "    def step(self, inputs, preds, labels):\n",
    "        grads = self.grad(inputs, preds, labels)\n",
    "        self.update_params(grads)\n",
    "\n",
    "    def update_params(self, grads):\n",
    "        params = self.model.params.keys()\n",
    "        for i,p in enumerate(params):\n",
    "            self.model.params[p] -= self.eta*grads[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "841df9d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the Train_Iters Function\n",
    "def train_iters(iterations, data_loader, model, lossfn, optimizer):\n",
    "    running_loss = 0\n",
    "    loss = []\n",
    "    for i in tqdm(range(iterations)):\n",
    "        x,y = data_loader._get_item_()\n",
    "        preds = model.forward(x)\n",
    "        running_loss += lossfn(preds, y)\n",
    "        if  (i + 1) % 1000 == 0:\n",
    "            loss += [running_loss/1000]\n",
    "            running_loss = 0\n",
    "        optimizer.step(x, preds, y)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e27d3237",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the L2 Loss Function and Gradients\n",
    "\n",
    "def l2_loss(preds, targets):\n",
    "    return 1/2*np.sum(np.square(preds-targets))\n",
    "\n",
    "def l2_grad(inputs, preds, targets):\n",
    "    weights_grad = np.dot(inputs.T, preds-targets)/len(preds)\n",
    "    bias_grad = np.dot(np.ones(len(targets)).T, preds-targets)/len(preds) \n",
    "    return weights_grad, bias_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c2eae34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the L1 Loss Function and Gradients\n",
    "\n",
    "def l1_loss(preds, targets):\n",
    "    return np.sum(np.abs(preds-targets))\n",
    "\n",
    "def l1_grad(inputs, preds, targets):\n",
    "    # Taken by piecewise derivative calculation\n",
    "    weights_grad = np.dot(inputs.T, np.sign(preds-targets))/len(preds)\n",
    "    bias_grad = np.dot(np.ones(len(targets)).T, np.sign(preds-targets))/len(preds)\n",
    "    return weights_grad, bias_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f796629",
   "metadata": {},
   "source": [
    "## 3.6 Generalization\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742f8f09",
   "metadata": {},
   "source": [
    "### 3.6.1 Training Error and Generalization Error\n",
    "\n",
    "* Strong IID assumption for training data and test data\n",
    "* $R_{emp}$ is the empirical risk which is also known as the training error and is a statistic that is computed on the training data and also a function of the model\n",
    " $$ R_{emp}[X, y,f] = \\frac{1}{n}\\sum_{i\\in \\text{Data}}^nl(x_i, y_i, f(x_i))$$\n",
    " \n",
    " * The generalization error on the other hand is an expectation over all the data re: a certain models predictions and the true values:\n",
    " \n",
    " $$ R[p, f] = E_{(x,y)~P}[l(x, y, f(x)] = \\int \\int l(x, y, f(x))p(x,y)dxdy $$\n",
    " \n",
    " * Since we can never observe the population of data we instead holdout some of the training data which we take to be IID with the rest of the non-heldout training data and then use that as teh generalization set. \n",
    " \n",
    " * Pointing a fixed classifier on the test set to estimate the TRUE error is a problem of mean estimation. (How off is the sample mean from the population mean or true parameter)\n",
    " \n",
    " #### Model Complexity \n",
    " \n",
    " * When we start working with more complicated models, we actually expect the training loss and the validation looss to begin to diverge.\n",
    " \n",
    "     * The problem of overfitting the training data: with that number of free parameters it makes perfect sense that an overly expressive model would be able to predict the training labels even if the labels were generated at random\n",
    "     \n",
    " * Model complexity can describe both model architecture as well as the parameter ranges that the free params can take\n",
    " \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e09f721",
   "metadata": {},
   "source": [
    "### 3.6.2 Underfitting or Overfitting?\n",
    "\n",
    "* Overfitting is when the training loss is close to zero and the validation loss is high and often can appear to be diverging. \n",
    "* Underfitting is when the training loss and validation loss are both high and there is barely any difference between the two\n",
    "\n",
    "#### Dataset Size\n",
    "\n",
    "* Probability of overfitting is inversely proportional to the dataset size\n",
    "* For a fixed task and data distribution, the model complexity increasing **shouldnt outpace** data generation capability.\n",
    "    * Deep learning can only outperform linear models given that there are many thousands of data examples to use"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e00105",
   "metadata": {},
   "source": [
    "### 3.6.3 Model Selection\n",
    "\n",
    "* Test set shouldn't be touched until the hyperparameters and model features are chosen beforehand\n",
    "* Three way split to the data:\n",
    "    * Training Data: Useful for figuring out which hyper-params to use\n",
    "    * Test Data: This is during inference time\n",
    "    * Validation Data: This can be used for model gets the lowest validation error\n",
    "\n",
    "#### Cross-Validation \n",
    "\n",
    "* K-fold cross validation is a strategy that is used when there is a sparse amount of training data such that the experimentalist cannot afford to sequester data for validation. \n",
    "    * The data needs to be split into K non-overlapping subsets and model training + validation is used on the data K times where each time 1 of the subsets is used as validation and K-1 is used as the training data.\n",
    "    * Average over the k-runs of doing this to figure out the overall training error and the test error.\n",
    "        * Is there any way to even go over the different dynamics of the loss curves over the different validations?\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4858a2",
   "metadata": {},
   "source": [
    "## 3.7 Weight Decay (Regularization)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8609abbd",
   "metadata": {},
   "source": [
    "### 3.7.1 Norms and Weight Decay\n",
    "\n",
    "* Instead of varying the number of parameters, we instead vary the range of values it can take (dampening it) which is essentially the same as feature selection because setting these weights to zero is essentially the same thing.\n",
    "\n",
    "* Weight decay is equivalent to be the $l-2$ regularization for stochastic gradient descent but maybe not for other optimization algorithms\n",
    "     * Might be the most widely used regularization method for parameteric models\n",
    "     \n",
    " * Adding the norm of the weight vector to the loss function as a penalty term penalizes large values of the weights\n",
    " \n",
    " $$ L(w,b) + \\frac{\\lambda}{2}||w||^2$$\n",
    " \n",
    " * The one half is there as a convenient feature for differentiation of the loss function and teh square is included as a means of getting rid of the square root introduced by the l-2 loss. This is just chosen because of simplicity because the L1 has the absolute value which is finicky to use.\n",
    " \n",
    "$$ w <- (1 - \\eta \\lambda)w - \\frac{\\eta}{|\\beta|}\\sum_{i \\in B}\\nabla f^{(i)}(x^{(i)}, y^{(i)})$$\n",
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}