Ch05_Linear_Neural_Networks/Linear_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression\n",
    "\n",
    "\n",
    "To start off, we will introduce the problem of regression.\n",
    "This is the task of predicting a *real valued target* $y$\n",
    "given a data point $\\mathbf{x}$.\n",
    "Regression problems are common in practice, arising\n",
    "whenever we want to predict a continuous numerical value.\n",
    "Some examples of regression problems include\n",
    "predicting house prices, stock prices,\n",
    "length of stay (for patients in the hospital),\n",
    "tomorrow's temperature, demand forecasting (for retail sales), and many more.\n",
    "Note that not every prediction problem is a regression problem.\n",
    "In subsequent sections we will discuss classification problems,\n",
    "where our predictions are discrete categories.\n",
    "\n",
    "## Basic Elements of Linear Regression\n",
    "\n",
    "Linear regression, which dates to Gauss and Legendre,\n",
    "is perhaps the simplest, and by far the most popular approach\n",
    "to solving regression problems.\n",
    "What makes linear regression *linear* is that\n",
    "we assume that the output truly can be expressed\n",
    "as a *linear* combination of the input features.\n",
    "\n",
    "\n",
    "### Linear Model\n",
    "\n",
    "To keep things simple, we will start with running example\n",
    "in which we consider the problem\n",
    "of estimating the price of a house (e.g. in dollars)\n",
    "based on area (e.g. in square feet) and age (e.g. in years).\n",
    "More formally, the assumption of linearity suggests\n",
    "that our model can be expressed in the following form:\n",
    "\n",
    "$$\\mathrm{price} = w_{\\mathrm{area}} \\cdot \\mathrm{area} + w_{\\mathrm{age}} \\cdot \\mathrm{age} + b$$\n",
    "\n",
    "In economics papers, it is common for authors to write out linear models in this format with a gigantic equation that spans multiple lines containing terms for every single feature.\n",
    "For the high-dimensional data that we often address in machine learning,\n",
    "writing out the entire model can be tedious.\n",
    "In these cases, we will find it more convenient to use linear algebra notation.\n",
    "In the case of $d$ variables, we could express our prediction $\\hat{y}$ as follows:\n",
    "\n",
    "$$\\hat{y} = w_1 \\cdot x_1 + ... + w_d \\cdot x_d + b$$\n",
    "\n",
    "or alternatively, collecting all features into a single vector $\\mathbf{x}$ and all parameters into a vector $\\mathbf{w}$, we can express our linear model as $\\hat{y} = \\mathbf{w}^T \\mathbf{x} + b$.\n",
    "\n",
    "Above, the vector $\\mathbf{x}$ corresponds to a single data point.\n",
    "Commonly, we will want notation to refer to\n",
    "the entire dataset of all input data points.\n",
    "This matrix, often denoted using a capital letter $X$,\n",
    "is called the *design matrix* and contains one row for every example,\n",
    "and one column for every feature.\n",
    "\n",
    "Given a collection of data points $X$ and a vector\n",
    "containing the corresponding target values $\\mathbf{y}$,\n",
    "the goal of linear regression is to find\n",
    "the *weight* vector $w$ and bias term $b$\n",
    "(also called an *offset* or *intercept*)\n",
    "that associates each data point $\\mathbf{x}_i$\n",
    "with an approximation $\\hat{y}_i$ of its corresponding label $y_i$.\n",
    "\n",
    "Expressed in terms of a single data point,\n",
    "this gives us the expression (same as above)\n",
    "$\\hat{y} = \\mathbf{w}^\\top \\mathbf{x} + b$.\n",
    "\n",
    "Finally, for a collection of data points $\\mathbf{X}$,\n",
    "the predictions $\\hat{\\mathbf{y}}$ can be expressed via the matrix-vector product:\n",
    "\n",
    "$${\\hat{\\mathbf{y}}} = X \\mathbf{w} + b$$\n",
    "\n",
    "Even if we believe that the best model\n",
    "to relate $\\mathbf{x}$ and $y$ is linear,\n",
    "it's unlikely that we'd find data where $y$\n",
    "lines up exactly as a linear function of $\\mathbf{x}$.\n",
    "For example, both the target values $y$ and the features $X$\n",
    "might be subject to some amount of measurement error.\n",
    "Thus even when we believe that the linearity assumption holds,\n",
    "we will typically incorporate a noise term to account for such errors.\n",
    "\n",
    "Before we can go about solving for the best setting of the parameters $w$ and $b$, we will need two more things:\n",
    "(i) some way to measure the quality of the current model\n",
    "and (ii) some way to manipulate the model to improve its quality.\n",
    "\n",
    "### Training Data\n",
    "\n",
    "The first thing that we need is training data.\n",
    "Sticking with our running example, we'll need some collection of examples\n",
    "for which we know both the actual selling price of each house\n",
    "as well as their corresponding area and age.\n",
    "Our goal is to identify model parameters\n",
    "that minimize the error between the predicted price and the real price.\n",
    "In the terminology of machine learning, the data set is called a ‘training data’ or ‘training set’, a house (often a house and its price) here comprises one ‘sample’, and its actual selling price is called a ‘label’.\n",
    "The two factors used to predict the label\n",
    "are called ‘features’ or 'covariates'.\n",
    "\n",
    "Typically, we will use $n$ to denote the number of samples in our dataset.\n",
    "We index the samples by $i$, denoting each input data point as $x^{(i)} = [x_1^{(i)}, x_2^{(i)}]$ and the corresponding label as $y^{(i)}$.\n",
    "\n",
    "### Loss Function\n",
    "\n",
    "In model training, we need to measure the error\n",
    "between the predicted value and the real value of the price.\n",
    "Usually, we will choose a non-negative number as the error.\n",
    "The smaller the value, the smaller the error.\n",
    "A common choice is the square function.\n",
    "For given parameters $\\mathbf{w}$ and $b$,\n",
    "we can express the error of our prediction on a given a sample as follows:\n",
    "\n",
    "$$l^{(i)}(\\mathbf{w}, b) = \\frac{1}{2} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2,$$\n",
    "\n",
    "The constant $1/2$ is just for mathematical convenience,\n",
    "ensuring that after we take the derivative of the loss,\n",
    "the constant coefficient will be $1$.\n",
    "The smaller the error, the closer the predicted price is to the actual price, and when the two are equal, the error will be zero.\n",
    "\n",
    "Since the training dataset is given to us, and thus out of our control,\n",
    "the error is only a function of the model parameters.\n",
    "In machine learning, we call the function that measures the error the ‘loss function’.\n",
    "The squared error function used here is commonly referred to as ‘square loss’.\n",
    "\n",
    "To make things a bit more concrete, consider the example below where we plot a regression problem for a one-dimensional case, e.g. for a model where house prices depend only on area.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"202pt\" version=\"1.1\" viewBox=\"0 0 302 202\" width=\"302pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 5.8125 -4.546875 C 5.859375 -4.71875 5.859375 -4.75 5.859375 -4.828125 C 5.859375 -5.046875 5.6875 -5.15625 5.515625 -5.15625 C 5.390625 -5.15625 5.203125 -5.078125 5.09375 -4.90625 C 5.0625 -4.84375 4.96875 -4.46875 4.921875 -4.25 C 4.84375 -3.9375 4.75 -3.625 4.6875 -3.296875 L 4.140625 -1.140625 C 4.09375 -0.96875 3.578125 -0.125 2.796875 -0.125 C 2.1875 -0.125 2.0625 -0.65625 2.0625 -1.09375 C 2.0625 -1.65625 2.265625 -2.390625 2.671875 -3.4375 C 2.859375 -3.9375 2.90625 -4.0625 2.90625 -4.296875 C 2.90625 -4.84375 2.515625 -5.28125 1.921875 -5.28125 C 0.78125 -5.28125 0.34375 -3.546875 0.34375 -3.4375 C 0.34375 -3.328125 0.46875 -3.328125 0.484375 -3.328125 C 0.609375 -3.328125 0.625 -3.34375 0.6875 -3.53125 C 1 -4.65625 1.484375 -5.015625 1.890625 -5.015625 C 1.984375 -5.015625 2.1875 -5.015625 2.1875 -4.640625 C 2.1875 -4.34375 2.0625 -4.03125 1.984375 -3.796875 C 1.5 -2.53125 1.296875 -1.859375 1.296875 -1.296875 C 1.296875 -0.234375 2.046875 0.125 2.75 0.125 C 3.21875 0.125 3.625 -0.078125 3.953125 -0.40625 C 3.796875 0.21875 3.65625 0.796875 3.171875 1.4375 C 2.875 1.84375 2.421875 2.1875 1.859375 2.1875 C 1.703125 2.1875 1.15625 2.15625 0.953125 1.6875 C 1.140625 1.6875 1.296875 1.6875 1.46875 1.546875 C 1.59375 1.4375 1.703125 1.28125 1.703125 1.046875 C 1.703125 0.6875 1.390625 0.640625 1.265625 0.640625 C 0.984375 0.640625 0.59375 0.828125 0.59375 1.40625 C 0.59375 2.015625 1.125 2.453125 1.859375 2.453125 C 3.09375 2.453125 4.328125 1.359375 4.65625 0.015625 Z M 5.8125 -4.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 5.515625 -4.046875 C 5.578125 -4.3125 5.703125 -4.75 5.703125 -4.828125 C 5.703125 -5.046875 5.53125 -5.15625 5.359375 -5.15625 C 5.21875 -5.15625 5 -5.0625 4.90625 -4.8125 C 4.875 -4.734375 4.3125 -2.453125 4.234375 -2.140625 C 4.140625 -1.78125 4.125 -1.5625 4.125 -1.34375 C 4.125 -1.21875 4.125 -1.1875 4.140625 -1.140625 C 3.859375 -0.5 3.5 -0.125 3.03125 -0.125 C 2.078125 -0.125 2.078125 -1.015625 2.078125 -1.21875 C 2.078125 -1.609375 2.140625 -2.0625 2.703125 -3.53125 C 2.828125 -3.890625 2.90625 -4.0625 2.90625 -4.296875 C 2.90625 -4.84375 2.515625 -5.28125 1.921875 -5.28125 C 0.78125 -5.28125 0.34375 -3.546875 0.34375 -3.4375 C 0.34375 -3.328125 0.46875 -3.328125 0.484375 -3.328125 C 0.609375 -3.328125 0.625 -3.34375 0.6875 -3.53125 C 1 -4.65625 1.46875 -5.015625 1.890625 -5.015625 C 2 -5.015625 2.1875 -5.015625 2.1875 -4.625 C 2.1875 -4.328125 2.0625 -4 1.96875 -3.796875 C 1.453125 -2.375 1.296875 -1.828125 1.296875 -1.375 C 1.296875 -0.28125 2.109375 0.125 3 0.125 C 3.203125 0.125 3.765625 0.125 4.25 -0.703125 C 4.546875 0.0625 5.375 0.125 5.734375 0.125 C 6.640625 0.125 7.15625 -0.625 7.46875 -1.34375 C 7.875 -2.265625 8.265625 -3.875 8.265625 -4.453125 C 8.265625 -5.109375 7.9375 -5.28125 7.734375 -5.28125 C 7.4375 -5.28125 7.140625 -4.96875 7.140625 -4.703125 C 7.140625 -4.546875 7.203125 -4.484375 7.3125 -4.390625 C 7.453125 -4.25 7.75 -3.9375 7.75 -3.375 C 7.75 -2.96875 7.40625 -1.796875 7.09375 -1.1875 C 6.78125 -0.546875 6.359375 -0.125 5.765625 -0.125 C 5.21875 -0.125 4.890625 -0.484375 4.890625 -1.171875 C 4.890625 -1.5 4.96875 -1.875 5.015625 -2.046875 Z M 5.515625 -4.046875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 4 -3.609375 C 4.0625 -3.921875 4.34375 -5.015625 5.171875 -5.015625 C 5.234375 -5.015625 5.515625 -5.015625 5.765625 -4.859375 C 5.4375 -4.8125 5.203125 -4.5 5.203125 -4.21875 C 5.203125 -4.03125 5.328125 -3.796875 5.65625 -3.796875 C 5.921875 -3.796875 6.296875 -4.015625 6.296875 -4.5 C 6.296875 -5.109375 5.59375 -5.28125 5.1875 -5.28125 C 4.5 -5.28125 4.078125 -4.65625 3.9375 -4.375 C 3.640625 -5.15625 2.984375 -5.28125 2.640625 -5.28125 C 1.40625 -5.28125 0.71875 -3.734375 0.71875 -3.4375 C 0.71875 -3.328125 0.84375 -3.328125 0.859375 -3.328125 C 0.953125 -3.328125 0.984375 -3.34375 1.015625 -3.453125 C 1.421875 -4.71875 2.21875 -5.015625 2.625 -5.015625 C 2.84375 -5.015625 3.265625 -4.90625 3.265625 -4.21875 C 3.265625 -3.84375 3.0625 -3.046875 2.625 -1.375 C 2.421875 -0.640625 2.015625 -0.125 1.484375 -0.125 C 1.40625 -0.125 1.140625 -0.125 0.890625 -0.28125 C 1.1875 -0.34375 1.453125 -0.59375 1.453125 -0.9375 C 1.453125 -1.25 1.1875 -1.34375 1 -1.34375 C 0.640625 -1.34375 0.34375 -1.046875 0.34375 -0.65625 C 0.34375 -0.109375 0.9375 0.125 1.46875 0.125 C 2.265625 0.125 2.6875 -0.703125 2.71875 -0.78125 C 2.875 -0.328125 3.296875 0.125 4.015625 0.125 C 5.25 0.125 5.921875 -1.40625 5.921875 -1.703125 C 5.921875 -1.828125 5.828125 -1.828125 5.78125 -1.828125 C 5.671875 -1.828125 5.65625 -1.78125 5.625 -1.703125 C 5.234375 -0.421875 4.421875 -0.125 4.046875 -0.125 C 3.578125 -0.125 3.375 -0.515625 3.375 -0.921875 C 3.375 -1.1875 3.453125 -1.453125 3.578125 -1.96875 Z M 4 -3.609375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.859375 -8.15625 C 2.859375 -8.171875 2.859375 -8.296875 2.703125 -8.296875 C 2.421875 -8.296875 1.546875 -8.203125 1.25 -8.171875 C 1.140625 -8.15625 1.015625 -8.15625 1.015625 -7.9375 C 1.015625 -7.796875 1.125 -7.796875 1.296875 -7.796875 C 1.875 -7.796875 1.90625 -7.703125 1.90625 -7.59375 C 1.90625 -7.5 1.796875 -7.09375 1.734375 -6.84375 L 0.75 -2.953125 C 0.609375 -2.359375 0.5625 -2.15625 0.5625 -1.75 C 0.5625 -0.609375 1.1875 0.125 2.078125 0.125 C 3.484375 0.125 4.953125 -1.65625 4.953125 -3.375 C 4.953125 -4.453125 4.328125 -5.28125 3.375 -5.28125 C 2.828125 -5.28125 2.328125 -4.9375 1.96875 -4.5625 Z M 1.734375 -3.640625 C 1.8125 -3.90625 1.8125 -3.9375 1.90625 -4.0625 C 2.5 -4.84375 3.03125 -5.015625 3.34375 -5.015625 C 3.78125 -5.015625 4.09375 -4.65625 4.09375 -3.890625 C 4.09375 -3.1875 3.703125 -1.8125 3.484375 -1.359375 C 3.09375 -0.5625 2.546875 -0.125 2.078125 -0.125 C 1.671875 -0.125 1.28125 -0.453125 1.28125 -1.34375 C 1.28125 -1.5625 1.28125 -1.796875 1.46875 -2.546875 Z M 1.734375 -3.640625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 8.21875 -3.90625 C 8.390625 -3.90625 8.625 -3.90625 8.625 -4.140625 C 8.625 -4.390625 8.390625 -4.390625 8.21875 -4.390625 L 1.0625 -4.390625 C 0.890625 -4.390625 0.671875 -4.390625 0.671875 -4.140625 C 0.671875 -3.90625 0.890625 -3.90625 1.078125 -3.90625 Z M 8.21875 -1.59375 C 8.390625 -1.59375 8.625 -1.59375 8.625 -1.828125 C 8.625 -2.0625 8.390625 -2.0625 8.21875 -2.0625 L 1.078125 -2.0625 C 0.890625 -2.0625 0.671875 -2.0625 0.671875 -1.828125 C 0.671875 -1.59375 0.890625 -1.59375 1.0625 -1.59375 Z M 8.21875 -1.59375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 4.890625 -2.75 L 8.21875 -2.75 C 8.390625 -2.75 8.625 -2.75 8.625 -2.984375 C 8.625 -3.234375 8.390625 -3.234375 8.21875 -3.234375 L 4.890625 -3.234375 L 4.890625 -6.578125 C 4.890625 -6.734375 4.890625 -6.96875 4.65625 -6.96875 C 4.40625 -6.96875 4.40625 -6.734375 4.40625 -6.578125 L 4.40625 -3.234375 L 1.0625 -3.234375 C 0.890625 -3.234375 0.671875 -3.234375 0.671875 -2.984375 C 0.671875 -2.75 0.890625 -2.75 1.0625 -2.75 L 4.40625 -2.75 L 4.40625 0.59375 C 4.40625 0.765625 4.40625 0.984375 4.65625 0.984375 C 4.890625 0.984375 4.890625 0.765625 4.890625 0.59375 Z M 4.890625 -2.75 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 3.9375 -5.40625 L 6.5 -5.40625 C 6.640625 -5.40625 6.828125 -5.40625 6.828125 -5.609375 C 6.828125 -5.8125 6.640625 -5.8125 6.5 -5.8125 L 0.984375 -5.8125 C 0.859375 -5.8125 0.640625 -5.8125 0.640625 -5.609375 C 0.640625 -5.40625 0.84375 -5.40625 0.984375 -5.40625 L 3.53125 -5.40625 L 3.53125 -0.328125 C 3.53125 -0.1875 3.53125 0 3.75 0 C 3.9375 0 3.9375 -0.203125 3.9375 -0.328125 Z M 3.9375 -5.40625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-1\" overflow=\"visible\">\n",
       "<path d=\"M 4 -3.609375 C 4.0625 -3.921875 4.34375 -5.015625 5.171875 -5.015625 C 5.234375 -5.015625 5.515625 -5.015625 5.765625 -4.859375 C 5.4375 -4.8125 5.203125 -4.5 5.203125 -4.21875 C 5.203125 -4.03125 5.328125 -3.796875 5.65625 -3.796875 C 5.921875 -3.796875 6.296875 -4.015625 6.296875 -4.5 C 6.296875 -5.109375 5.59375 -5.28125 5.1875 -5.28125 C 4.5 -5.28125 4.078125 -4.65625 3.9375 -4.375 C 3.640625 -5.15625 2.984375 -5.28125 2.640625 -5.28125 C 1.40625 -5.28125 0.71875 -3.734375 0.71875 -3.4375 C 0.71875 -3.328125 0.84375 -3.328125 0.859375 -3.328125 C 0.953125 -3.328125 0.984375 -3.34375 1.015625 -3.453125 C 1.421875 -4.71875 2.21875 -5.015625 2.625 -5.015625 C 2.84375 -5.015625 3.265625 -4.90625 3.265625 -4.21875 C 3.265625 -3.84375 3.0625 -3.046875 2.625 -1.375 C 2.421875 -0.640625 2.015625 -0.125 1.484375 -0.125 C 1.40625 -0.125 1.140625 -0.125 0.890625 -0.28125 C 1.1875 -0.34375 1.453125 -0.59375 1.453125 -0.9375 C 1.453125 -1.25 1.1875 -1.34375 1 -1.34375 C 0.640625 -1.34375 0.34375 -1.046875 0.34375 -0.65625 C 0.34375 -0.109375 0.9375 0.125 1.46875 0.125 C 2.265625 0.125 2.6875 -0.703125 2.71875 -0.78125 C 2.875 -0.328125 3.296875 0.125 4.015625 0.125 C 5.25 0.125 5.921875 -1.40625 5.921875 -1.703125 C 5.921875 -1.828125 5.828125 -1.828125 5.78125 -1.828125 C 5.671875 -1.828125 5.65625 -1.78125 5.625 -1.703125 C 5.234375 -0.421875 4.421875 -0.125 4.046875 -0.125 C 3.578125 -0.125 3.375 -0.515625 3.375 -0.921875 C 3.375 -1.1875 3.453125 -1.453125 3.578125 -1.96875 Z M 4 -3.609375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph4-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph4-1\" overflow=\"visible\">\n",
       "<path d=\"M 5.8125 -4.546875 C 5.859375 -4.71875 5.859375 -4.75 5.859375 -4.828125 C 5.859375 -5.046875 5.6875 -5.15625 5.515625 -5.15625 C 5.390625 -5.15625 5.203125 -5.078125 5.09375 -4.90625 C 5.0625 -4.84375 4.96875 -4.46875 4.921875 -4.25 C 4.84375 -3.9375 4.75 -3.625 4.6875 -3.296875 L 4.140625 -1.140625 C 4.09375 -0.96875 3.578125 -0.125 2.796875 -0.125 C 2.1875 -0.125 2.0625 -0.65625 2.0625 -1.09375 C 2.0625 -1.65625 2.265625 -2.390625 2.671875 -3.4375 C 2.859375 -3.9375 2.90625 -4.0625 2.90625 -4.296875 C 2.90625 -4.84375 2.515625 -5.28125 1.921875 -5.28125 C 0.78125 -5.28125 0.34375 -3.546875 0.34375 -3.4375 C 0.34375 -3.328125 0.46875 -3.328125 0.484375 -3.328125 C 0.609375 -3.328125 0.625 -3.34375 0.6875 -3.53125 C 1 -4.65625 1.484375 -5.015625 1.890625 -5.015625 C 1.984375 -5.015625 2.1875 -5.015625 2.1875 -4.640625 C 2.1875 -4.34375 2.0625 -4.03125 1.984375 -3.796875 C 1.5 -2.53125 1.296875 -1.859375 1.296875 -1.296875 C 1.296875 -0.234375 2.046875 0.125 2.75 0.125 C 3.21875 0.125 3.625 -0.078125 3.953125 -0.40625 C 3.796875 0.21875 3.65625 0.796875 3.171875 1.4375 C 2.875 1.84375 2.421875 2.1875 1.859375 2.1875 C 1.703125 2.1875 1.15625 2.15625 0.953125 1.6875 C 1.140625 1.6875 1.296875 1.6875 1.46875 1.546875 C 1.59375 1.4375 1.703125 1.28125 1.703125 1.046875 C 1.703125 0.6875 1.390625 0.640625 1.265625 0.640625 C 0.984375 0.640625 0.59375 0.828125 0.59375 1.40625 C 0.59375 2.015625 1.125 2.453125 1.859375 2.453125 C 3.09375 2.453125 4.328125 1.359375 4.65625 0.015625 Z M 5.8125 -4.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "<clipPath id=\"clip1\">\n",
       "  <path d=\"M 214.199219 103 L 221 103 L 221 111.398438 L 214.199219 111.398438 Z M 214.199219 103 \"/>\n",
       "</clipPath>\n",
       "<clipPath id=\"clip2\">\n",
       "  <path d=\"M 280 165.203125 L 286 165.203125 L 286 170.964844 L 280 170.964844 Z M 280 165.203125 \"/>\n",
       "</clipPath>\n",
       "<clipPath id=\"clip3\">\n",
       "  <path d=\"M 30 13.199219 L 35.800781 13.199219 L 35.800781 21.964844 L 30 21.964844 Z M 30 13.199219 \"/>\n",
       "</clipPath>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<rect height=\"202\" style=\"fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;\" width=\"302\" x=\"0\" y=\"0\"/>\n",
       "<rect height=\"202\" style=\"fill:rgb(100%,100%,100%);fill-opacity:1;stroke:none;\" width=\"302\" x=\"0\" y=\"0\"/>\n",
       "<path d=\"M 75 134.898438 L 75 325 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 75 126.898438 L 72 134.898438 L 78 134.898438 Z M 75 126.898438 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 340.101562 300 L 50 300 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 348.101562 300 L 340.101562 297 L 340.101562 303 Z M 348.101562 300 \" style=\"fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 339 171 L 58.0625 284.402344 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 80.269531 151.730469 C 81.242188 152.707031 81.242188 154.292969 80.269531 155.269531 C 79.292969 156.242188 77.707031 156.242188 76.730469 155.269531 C 75.757812 154.292969 75.757812 152.707031 76.730469 151.730469 C 77.707031 150.757812 79.292969 150.757812 80.269531 151.730469 \" style=\" stroke:none;fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;\"/>\n",
       "<path d=\"M 110.269531 102.730469 C 111.242188 103.707031 111.242188 105.292969 110.269531 106.269531 C 109.292969 107.242188 107.707031 107.242188 106.730469 106.269531 C 105.757812 105.292969 105.757812 103.707031 106.730469 102.730469 C 107.707031 101.757812 109.292969 101.757812 110.269531 102.730469 \" style=\" stroke:none;fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;\"/>\n",
       "<path d=\"M 137.269531 34.730469 C 138.242188 35.707031 138.242188 37.292969 137.269531 38.269531 C 136.292969 39.242188 134.707031 39.242188 133.730469 38.269531 C 132.757812 37.292969 132.757812 35.707031 133.730469 34.730469 C 134.707031 33.757812 136.292969 33.757812 137.269531 34.730469 \" style=\" stroke:none;fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;\"/>\n",
       "<path d=\"M 173.269531 124.730469 C 174.242188 125.707031 174.242188 127.292969 173.269531 128.269531 C 172.292969 129.242188 170.707031 129.242188 169.730469 128.269531 C 168.757812 127.292969 168.757812 125.707031 169.730469 124.730469 C 170.707031 123.757812 172.292969 123.757812 173.269531 124.730469 \" style=\" stroke:none;fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;\"/>\n",
       "<path d=\"M 240.269531 42.730469 C 241.242188 43.707031 241.242188 45.292969 240.269531 46.269531 C 239.292969 47.242188 237.707031 47.242188 236.730469 46.269531 C 235.757812 45.292969 235.757812 43.707031 236.730469 42.730469 C 237.707031 41.757812 239.292969 41.757812 240.269531 42.730469 \" style=\" stroke:none;fill-rule:nonzero;fill:rgb(0%,0%,0%);fill-opacity:1;\"/>\n",
       "<path d=\"M 127.507812 275 L 127.5625 256.347656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 184.5 163 L 184.453125 233.386719 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 157.605469 230.996094 L 158.152344 244 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 220.488281 248 L 220.332031 218.902344 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<path d=\"M 287.507812 171 L 287.574219 191.757812 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-49,-124)\"/>\n",
       "<g clip-path=\"url(#clip1)\" clip-rule=\"nonzero\">\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"214.2164\" xlink:href=\"#glyph0-1\" y=\"108.616\"/>\n",
       "</g>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"223.8272\" xlink:href=\"#glyph1-1\" y=\"108.616\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"236.4464\" xlink:href=\"#glyph0-2\" y=\"108.616\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"245.3264\" xlink:href=\"#glyph2-1\" y=\"103.6804\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"253.3964\" xlink:href=\"#glyph0-3\" y=\"108.616\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"262.886\" xlink:href=\"#glyph1-2\" y=\"108.616\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"274.8416\" xlink:href=\"#glyph0-4\" y=\"108.616\"/>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip2)\" clip-rule=\"nonzero\">\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"279.6776\" xlink:href=\"#glyph3-1\" y=\"170.579\"/>\n",
       "</g>\n",
       "</g>\n",
       "<g clip-path=\"url(#clip3)\" clip-rule=\"nonzero\">\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"29.9488\" xlink:href=\"#glyph4-1\" y=\"19.179\"/>\n",
       "</g>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename='../img/linearregression.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear regression is a single-layer neural network. \n",
    "\n",
    "As you can see, large differences between estimates $\\hat{y}^{(i)}$ and observations $y^{(i)}$ lead to even larger contributions in terms of the loss, due to the quadratic dependence. To measure the quality of a model on the entire dataset, we can simply average the losses on the training set.\n",
    "\n",
    "$$L(\\mathbf{w}, b) =\\frac{1}{n}\\sum_{i=1}^n l^{(i)}(\\mathbf{w}, b) =\\frac{1}{n} \\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n",
    "\n",
    "When training the model, we want to find parameters ($\\mathbf{w}^*, b^*$) that minimize the average loss across all training samples:\n",
    "\n",
    "$$\\mathbf{w}^*, b^* = \\operatorname*{argmin}_{\\mathbf{w}, b}\\  L(\\mathbf{w}, b).$$\n",
    "\n",
    "\n",
    "### Analytic Solution\n",
    "\n",
    "Linear regression happens to be an unusually simple optimization problem.\n",
    "Unlike nearly every other model that we will encounter in this book,\n",
    "linear regression can be solved easily with a simple formula,\n",
    "yielding a global optimum.\n",
    "To start we can subsume the bias $b$ into the parameter $\\mathbf{w}$\n",
    "by appending a column to the design matrix consisting of all $1s$.\n",
    "Then our prediction problem is to minimize $||\\mathbf{y} - X\\mathbf{w}||$.\n",
    "Because this expression has a quadratic form it is clearly convex,\n",
    "and so long as the problem is not degenerate\n",
    "(our features are linearly independent), it is strictly convex.\n",
    "\n",
    "Thus there is just one global critical point on the loss surface\n",
    "corresponding to the global minimum.\n",
    "Taking the derivative of the loss with respect to $\\mathbf{w}$\n",
    "and setting it equal to 0 gives the analytic solution:\n",
    "\n",
    "$$\\mathbf{w}^* = (X^T X)^{-1}X^T y$$\n",
    "\n",
    "While simple problems like linear regression may admit analytic solutions,\n",
    "you should not get used to such good fortune.\n",
    "Although analytic solutions allow for nice mathematical analysis,\n",
    "the requirement of an analytic solution confines one to\n",
    "an restrictive set of models that would exclude all of deep learning.\n",
    "\n",
    "### Gradient descent\n",
    "\n",
    "Even in cases where we cannot solve the models analytically,\n",
    "and even when the loss surfaces are high-dimensional and nonconvex,\n",
    "it turns out that we can still make progress.\n",
    "Moreover, when those difficult-to-optimize models are sufficiently superior for the task at hand, figuring out how to train them is well worth the trouble.\n",
    "\n",
    "The key trick behind nearly all of deep learning\n",
    "and that we will repeatedly throughout this book\n",
    "is to reduce the error gradually by iteratively updating the parameters,\n",
    "each step moving the parameters in the direction\n",
    "that incrementally lowers the loss function.\n",
    "This algorithm is called gradient descent.\n",
    "On convex loss surfaces it will eventually converge to a global minimum,\n",
    "and while the same can't be said for nonconvex surfaces,\n",
    "it will at least lead towards a (hopefully good) local minimum.\n",
    "\n",
    "The most naive application of gradient descent consists of taking the derivative of the true loss, which is an average of the losses computed on every single example in the dataset. In practice, this can be extremely slow. We must pass over the entire dataset before making a single update.\n",
    "Thus, we'll often settle for sampling a random mini-batch\n",
    "of examples every time we need to computer the update,\n",
    "a variant called *stochastic gradient descent*.\n",
    "\n",
    "In each iteration, we first randomly and uniformly sample a mini-batch $\\mathcal{B}$ consisting of a fixed number of training data examples.\n",
    "We then compute the derivative (gradient) of the average loss on the mini batch with regard to the model parameters.\n",
    "Finally, the product of this result and a predetermined step size $\\eta > 0$ are used to update the parameters in the direction that lowers the loss.\n",
    "\n",
    "We can express the update mathematically as follows ($\\partial$ denotes the partial derivative):\n",
    "\n",
    "$$(\\mathbf{w},b) \\leftarrow (\\mathbf{w},b) - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\partial_{(\\mathbf{w},b)} l^{(i)}(\\mathbf{w},b)$$\n",
    "\n",
    "\n",
    "To summarize, steps of the algorithm are the following:\n",
    "(i) we initialize the values of the model parameters, typically at random;\n",
    "(ii) we iterate over the data many times,\n",
    "updating the parameters in each by moving the parameters in the direction of the negative gradient, as calculated on a random minibatch of data.\n",
    "\n",
    "\n",
    "For quadratic losses and linear functions we can write this out explicitly as follows. Note that $\\mathbf{w}$ and $\\mathbf{x}$ are vectors. Here the more elegant vector notation makes the math much more readable than expressing things in terms of coefficients, say $w_1, w_2, \\ldots w_d$.\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\mathbf{w} &\\leftarrow \\mathbf{w} -   \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\partial_{\\mathbf{w}} l^{(i)}(\\mathbf{w}, b) && =\n",
    "w - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right),\\\\\n",
    "b &\\leftarrow b -  \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\partial_b l^{(i)}(\\mathbf{w}, b)  && =\n",
    "b - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right).\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "In the above equation $|\\mathcal{B}|$ represents the number of samples (batch size) in each mini-batch, $\\eta$ is referred to as ‘learning rate’ and takes a positive number. It should be emphasized that the values of the batch size and learning rate are set somewhat manually and are typically not learned through model training. Therefore, they are referred to as *hyper-parameters*. What we usually call *tuning hyper-parameters* refers to the adjustment of these terms. In the worst case this is performed through repeated trial and error until the appropriate hyper-parameters are found. A better approach is to learn these as parts of model training. This is an advanced topic and we do not cover them here for the sake of simplicity.\n",
    "\n",
    "### Model Prediction\n",
    "\n",
    "After completing the training process, we record the estimated model parameters, denoted $\\hat{\\mathbf{w}}, \\hat{b}$\n",
    "(in general the \"hat\" symbol denotes estimates).\n",
    "Note that the parameters that we learn via gradient descent\n",
    "are not exactly equal to the true minimizers of the loss on the training set,\n",
    "that's because gradient descent converges slowly to a local minimum but does not achieve it exactly.\n",
    "Moreover if the problem has multiple local minimum, we may not necessarily achieve the lowest minimum.\n",
    "Fortunately, for deep neural networks, finding parameters that minimize the loss *on training data* is seldom a significant problem. The more formidable task is to find parameters that will achieve low loss on data that we have not seen before, a challenge called *generalization*. We return to these topics throughout the book.\n",
    "\n",
    "Given the learned linear regression model $\\hat{\\mathbf{w}}^\\top x + \\hat{b}$, we can now estimate the price of any house outside the training data set with area (square feet) as $x_1$ and house age (year) as $x_2$. Here, estimation also referred to as ‘model prediction’ or ‘model inference’.\n",
    "\n",
    "Note that calling this step 'inference' is a misnomer,\n",
    "but has become standard jargon in deep learning.\n",
    "In statistics, 'inference' means estimating parameters\n",
    "and outcomes based on other data.\n",
    "This misuse of terminology in deep learning\n",
    "can be a source of confusion when talking to statisticians.\n",
    "\n",
    "\n",
    "## From Linear Regression to Deep Networks\n",
    "\n",
    "So far we only talked about linear functions. While neural networks cover a much richer family of models, we can begin thinking of the linear model as a neural network by expressing it the language of neural networks. To begin, let's start by rewriting things in a 'layer' notation.\n",
    "\n",
    "### Neural Network Diagram\n",
    "\n",
    "Commonly, deep learning practitioners represent models visually using neural network diagrams. In Figure 3.1, we represent linear regression with a neural network diagram. The diagram shows the connectivity among the inputs and output, but does not depict the weights or biases (which are given implicitly).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"88pt\" version=\"1.1\" viewBox=\"0 0 263 88\" width=\"263pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M -0.015625 0 L 2.015625 -2.375 L 0.859375 -4.671875 L 1.734375 -4.671875 L 2.125 -3.84375 C 2.269531 -3.53125 2.398438 -3.226562 2.515625 -2.9375 L 3.875 -4.671875 L 4.84375 -4.671875 L 2.875 -2.3125 L 4.0625 0 L 3.171875 0 L 2.71875 -0.953125 C 2.613281 -1.148438 2.5 -1.398438 2.375 -1.703125 L 0.984375 0 Z M -0.015625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -1.765625 C 0.4375 -2.679688 0.707031 -3.4375 1.25 -4.03125 C 1.6875 -4.519531 2.265625 -4.765625 2.984375 -4.765625 C 3.546875 -4.765625 4 -4.585938 4.34375 -4.234375 C 4.6875 -3.890625 4.859375 -3.421875 4.859375 -2.828125 C 4.859375 -2.285156 4.75 -1.785156 4.53125 -1.328125 C 4.3125 -0.867188 4.003906 -0.515625 3.609375 -0.265625 C 3.210938 -0.015625 2.789062 0.109375 2.34375 0.109375 C 1.976562 0.109375 1.644531 0.03125 1.34375 -0.125 C 1.050781 -0.28125 0.828125 -0.5 0.671875 -0.78125 C 0.515625 -1.070312 0.4375 -1.398438 0.4375 -1.765625 Z M 1.234375 -1.84375 C 1.234375 -1.40625 1.335938 -1.070312 1.546875 -0.84375 C 1.765625 -0.625 2.035156 -0.515625 2.359375 -0.515625 C 2.523438 -0.515625 2.691406 -0.546875 2.859375 -0.609375 C 3.023438 -0.679688 3.179688 -0.785156 3.328125 -0.921875 C 3.472656 -1.066406 3.59375 -1.226562 3.6875 -1.40625 C 3.789062 -1.582031 3.875 -1.773438 3.9375 -1.984375 C 4.03125 -2.273438 4.078125 -2.554688 4.078125 -2.828125 C 4.078125 -3.242188 3.96875 -3.566406 3.75 -3.796875 C 3.539062 -4.035156 3.273438 -4.15625 2.953125 -4.15625 C 2.703125 -4.15625 2.472656 -4.09375 2.265625 -3.96875 C 2.066406 -3.851562 1.882812 -3.679688 1.71875 -3.453125 C 1.550781 -3.222656 1.425781 -2.957031 1.34375 -2.65625 C 1.269531 -2.351562 1.234375 -2.082031 1.234375 -1.84375 Z M 1.234375 -1.84375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.875 0 L 0.875 -4.375 L 4.375 -4.375 L 4.375 0 Z M 0.984375 -0.109375 L 4.265625 -0.109375 L 4.265625 -4.265625 L 0.984375 -4.265625 Z M 0.984375 -0.109375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.6875 0 L 2.484375 -3.78125 C 2.140625 -3.507812 1.65625 -3.296875 1.03125 -3.140625 L 1.15625 -3.703125 C 1.457031 -3.828125 1.757812 -3.988281 2.0625 -4.1875 C 2.363281 -4.382812 2.585938 -4.554688 2.734375 -4.703125 C 2.828125 -4.796875 2.914062 -4.90625 3 -5.03125 L 3.359375 -5.03125 L 2.3125 0 Z M 1.6875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.40625 0 C 0.46875 -0.300781 0.554688 -0.550781 0.671875 -0.75 C 0.785156 -0.945312 0.9375 -1.132812 1.125 -1.3125 C 1.3125 -1.5 1.671875 -1.804688 2.203125 -2.234375 C 2.523438 -2.492188 2.75 -2.6875 2.875 -2.8125 C 3.039062 -2.988281 3.160156 -3.160156 3.234375 -3.328125 C 3.285156 -3.441406 3.3125 -3.566406 3.3125 -3.703125 C 3.3125 -3.929688 3.226562 -4.125 3.0625 -4.28125 C 2.90625 -4.445312 2.707031 -4.53125 2.46875 -4.53125 C 2.238281 -4.53125 2.035156 -4.445312 1.859375 -4.28125 C 1.679688 -4.125 1.554688 -3.863281 1.484375 -3.5 L 0.875 -3.59375 C 0.9375 -4.039062 1.109375 -4.390625 1.390625 -4.640625 C 1.679688 -4.898438 2.039062 -5.03125 2.46875 -5.03125 C 2.75 -5.03125 3.003906 -4.96875 3.234375 -4.84375 C 3.460938 -4.726562 3.632812 -4.5625 3.75 -4.34375 C 3.875 -4.132812 3.9375 -3.914062 3.9375 -3.6875 C 3.9375 -3.351562 3.816406 -3.035156 3.578125 -2.734375 C 3.429688 -2.535156 3.003906 -2.15625 2.296875 -1.59375 C 1.984375 -1.351562 1.753906 -1.15625 1.609375 -1 C 1.460938 -0.84375 1.351562 -0.695312 1.28125 -0.5625 L 3.515625 -0.5625 L 3.390625 0 Z M 0.40625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 2.6875 -0.53125 C 2.332031 -0.125 1.960938 0.078125 1.578125 0.078125 C 1.234375 0.078125 0.945312 -0.046875 0.71875 -0.296875 C 0.488281 -0.554688 0.375 -0.925781 0.375 -1.40625 C 0.375 -1.84375 0.460938 -2.242188 0.640625 -2.609375 C 0.816406 -2.984375 1.039062 -3.257812 1.3125 -3.4375 C 1.59375 -3.625 1.867188 -3.71875 2.140625 -3.71875 C 2.585938 -3.71875 2.925781 -3.5 3.15625 -3.0625 L 3.578125 -5.015625 L 4.1875 -5.015625 L 3.140625 0 L 2.578125 0 Z M 0.984375 -1.515625 C 0.984375 -1.265625 1.007812 -1.066406 1.0625 -0.921875 C 1.113281 -0.773438 1.195312 -0.648438 1.3125 -0.546875 C 1.4375 -0.453125 1.582031 -0.40625 1.75 -0.40625 C 2.03125 -0.40625 2.285156 -0.550781 2.515625 -0.84375 C 2.816406 -1.238281 2.96875 -1.71875 2.96875 -2.28125 C 2.96875 -2.570312 2.890625 -2.796875 2.734375 -2.953125 C 2.585938 -3.117188 2.398438 -3.203125 2.171875 -3.203125 C 2.023438 -3.203125 1.890625 -3.164062 1.765625 -3.09375 C 1.648438 -3.03125 1.53125 -2.921875 1.40625 -2.765625 C 1.289062 -2.609375 1.191406 -2.40625 1.109375 -2.15625 C 1.023438 -1.914062 0.984375 -1.703125 0.984375 -1.515625 Z M 0.984375 -1.515625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.84375 0 L 0.84375 -6.4375 L 1.6875 -6.4375 L 1.6875 0 Z M 0.84375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 0 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4 C 1.644531 -4.507812 2.140625 -4.765625 2.796875 -4.765625 C 3.078125 -4.765625 3.332031 -4.710938 3.5625 -4.609375 C 3.800781 -4.515625 3.976562 -4.382812 4.09375 -4.21875 C 4.207031 -4.0625 4.289062 -3.867188 4.34375 -3.640625 C 4.375 -3.492188 4.390625 -3.238281 4.390625 -2.875 L 4.390625 0 L 3.59375 0 L 3.59375 -2.84375 C 3.59375 -3.164062 3.5625 -3.40625 3.5 -3.5625 C 3.4375 -3.71875 3.328125 -3.84375 3.171875 -3.9375 C 3.015625 -4.039062 2.832031 -4.09375 2.625 -4.09375 C 2.289062 -4.09375 2 -3.984375 1.75 -3.765625 C 1.507812 -3.554688 1.390625 -3.148438 1.390625 -2.546875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 1.78125 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4.0625 C 1.476562 -4.300781 1.664062 -4.476562 1.875 -4.59375 C 2.09375 -4.707031 2.359375 -4.765625 2.671875 -4.765625 C 3.066406 -4.765625 3.414062 -4.660156 3.71875 -4.453125 C 4.019531 -4.253906 4.25 -3.96875 4.40625 -3.59375 C 4.5625 -3.21875 4.640625 -2.8125 4.640625 -2.375 C 4.640625 -1.894531 4.550781 -1.460938 4.375 -1.078125 C 4.207031 -0.691406 3.960938 -0.394531 3.640625 -0.1875 C 3.316406 0.0078125 2.972656 0.109375 2.609375 0.109375 C 2.347656 0.109375 2.113281 0.0507812 1.90625 -0.0625 C 1.695312 -0.175781 1.523438 -0.316406 1.390625 -0.484375 L 1.390625 1.78125 Z M 1.3125 -2.3125 C 1.3125 -1.707031 1.429688 -1.257812 1.671875 -0.96875 C 1.921875 -0.6875 2.21875 -0.546875 2.5625 -0.546875 C 2.90625 -0.546875 3.203125 -0.691406 3.453125 -0.984375 C 3.710938 -1.285156 3.84375 -1.75 3.84375 -2.375 C 3.84375 -2.96875 3.71875 -3.410156 3.46875 -3.703125 C 3.226562 -4.003906 2.9375 -4.15625 2.59375 -4.15625 C 2.257812 -4.15625 1.960938 -3.992188 1.703125 -3.671875 C 1.441406 -3.359375 1.3125 -2.90625 1.3125 -2.3125 Z M 1.3125 -2.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 3.65625 0 L 3.65625 -0.6875 C 3.289062 -0.15625 2.796875 0.109375 2.171875 0.109375 C 1.898438 0.109375 1.644531 0.0546875 1.40625 -0.046875 C 1.164062 -0.160156 0.984375 -0.296875 0.859375 -0.453125 C 0.742188 -0.609375 0.664062 -0.800781 0.625 -1.03125 C 0.59375 -1.1875 0.578125 -1.4375 0.578125 -1.78125 L 0.578125 -4.671875 L 1.359375 -4.671875 L 1.359375 -2.078125 C 1.359375 -1.660156 1.378906 -1.382812 1.421875 -1.25 C 1.460938 -1.039062 1.5625 -0.875 1.71875 -0.75 C 1.882812 -0.632812 2.085938 -0.578125 2.328125 -0.578125 C 2.566406 -0.578125 2.789062 -0.632812 3 -0.75 C 3.207031 -0.875 3.351562 -1.039062 3.4375 -1.25 C 3.519531 -1.457031 3.5625 -1.765625 3.5625 -2.171875 L 3.5625 -4.671875 L 4.359375 -4.671875 L 4.359375 0 Z M 3.65625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-5\" overflow=\"visible\">\n",
       "<path d=\"M 2.328125 -0.703125 L 2.4375 -0.015625 C 2.207031 0.0351562 2.007812 0.0625 1.84375 0.0625 C 1.550781 0.0625 1.328125 0.015625 1.171875 -0.078125 C 1.015625 -0.171875 0.898438 -0.289062 0.828125 -0.4375 C 0.765625 -0.582031 0.734375 -0.890625 0.734375 -1.359375 L 0.734375 -4.046875 L 0.15625 -4.046875 L 0.15625 -4.671875 L 0.734375 -4.671875 L 0.734375 -5.828125 L 1.53125 -6.296875 L 1.53125 -4.671875 L 2.328125 -4.671875 L 2.328125 -4.046875 L 1.53125 -4.046875 L 1.53125 -1.328125 C 1.53125 -1.097656 1.539062 -0.953125 1.5625 -0.890625 C 1.59375 -0.828125 1.640625 -0.773438 1.703125 -0.734375 C 1.765625 -0.691406 1.851562 -0.671875 1.96875 -0.671875 C 2.0625 -0.671875 2.179688 -0.679688 2.328125 -0.703125 Z M 2.328125 -0.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-6\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-7\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -6.4375 L 1.359375 -6.4375 L 1.359375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-8\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -0.578125 C 3.347656 -0.328125 3.066406 -0.148438 2.796875 -0.046875 C 2.523438 0.0546875 2.234375 0.109375 1.921875 0.109375 C 1.410156 0.109375 1.015625 -0.015625 0.734375 -0.265625 C 0.460938 -0.515625 0.328125 -0.835938 0.328125 -1.234375 C 0.328125 -1.460938 0.378906 -1.671875 0.484375 -1.859375 C 0.585938 -2.046875 0.722656 -2.195312 0.890625 -2.3125 C 1.054688 -2.425781 1.242188 -2.515625 1.453125 -2.578125 C 1.609375 -2.609375 1.84375 -2.644531 2.15625 -2.6875 C 2.800781 -2.757812 3.273438 -2.851562 3.578125 -2.96875 C 3.578125 -3.070312 3.578125 -3.140625 3.578125 -3.171875 C 3.578125 -3.492188 3.503906 -3.71875 3.359375 -3.84375 C 3.148438 -4.03125 2.847656 -4.125 2.453125 -4.125 C 2.078125 -4.125 1.800781 -4.054688 1.625 -3.921875 C 1.445312 -3.796875 1.316406 -3.566406 1.234375 -3.234375 L 0.46875 -3.328125 C 0.53125 -3.660156 0.640625 -3.925781 0.796875 -4.125 C 0.960938 -4.332031 1.195312 -4.488281 1.5 -4.59375 C 1.8125 -4.707031 2.164062 -4.765625 2.5625 -4.765625 C 2.96875 -4.765625 3.289062 -4.71875 3.53125 -4.625 C 3.78125 -4.53125 3.960938 -4.410156 4.078125 -4.265625 C 4.203125 -4.128906 4.285156 -3.953125 4.328125 -3.734375 C 4.359375 -3.597656 4.375 -3.359375 4.375 -3.015625 L 4.375 -1.953125 C 4.375 -1.222656 4.390625 -0.757812 4.421875 -0.5625 C 4.453125 -0.363281 4.519531 -0.175781 4.625 0 L 3.796875 0 C 3.710938 -0.164062 3.660156 -0.359375 3.640625 -0.578125 Z M 3.578125 -2.34375 C 3.285156 -2.226562 2.851562 -2.128906 2.28125 -2.046875 C 1.957031 -1.992188 1.726562 -1.9375 1.59375 -1.875 C 1.457031 -1.820312 1.351562 -1.738281 1.28125 -1.625 C 1.207031 -1.507812 1.171875 -1.382812 1.171875 -1.25 C 1.171875 -1.039062 1.25 -0.863281 1.40625 -0.71875 C 1.5625 -0.582031 1.796875 -0.515625 2.109375 -0.515625 C 2.410156 -0.515625 2.679688 -0.582031 2.921875 -0.71875 C 3.160156 -0.851562 3.335938 -1.035156 3.453125 -1.265625 C 3.535156 -1.441406 3.578125 -1.703125 3.578125 -2.046875 Z M 3.578125 -2.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 1.796875 L 0.46875 1.0625 C 0.644531 1.101562 0.796875 1.125 0.921875 1.125 C 1.097656 1.125 1.238281 1.09375 1.34375 1.03125 C 1.445312 0.976562 1.535156 0.898438 1.609375 0.796875 C 1.648438 0.710938 1.726562 0.515625 1.84375 0.203125 C 1.863281 0.160156 1.890625 0.0976562 1.921875 0.015625 L 0.140625 -4.671875 L 1 -4.671875 L 1.96875 -1.96875 C 2.09375 -1.625 2.207031 -1.265625 2.3125 -0.890625 C 2.394531 -1.242188 2.5 -1.597656 2.625 -1.953125 L 3.625 -4.671875 L 4.421875 -4.671875 L 2.640625 0.078125 C 2.453125 0.585938 2.304688 0.941406 2.203125 1.140625 C 2.054688 1.398438 1.894531 1.585938 1.71875 1.703125 C 1.539062 1.828125 1.320312 1.890625 1.0625 1.890625 C 0.914062 1.890625 0.75 1.859375 0.5625 1.796875 Z M 0.5625 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-10\" overflow=\"visible\">\n",
       "<path d=\"M 3.78125 -1.5 L 4.609375 -1.40625 C 4.472656 -0.925781 4.226562 -0.550781 3.875 -0.28125 C 3.53125 -0.0195312 3.085938 0.109375 2.546875 0.109375 C 1.867188 0.109375 1.328125 -0.0976562 0.921875 -0.515625 C 0.523438 -0.941406 0.328125 -1.535156 0.328125 -2.296875 C 0.328125 -3.078125 0.53125 -3.679688 0.9375 -4.109375 C 1.34375 -4.546875 1.867188 -4.765625 2.515625 -4.765625 C 3.140625 -4.765625 3.644531 -4.550781 4.03125 -4.125 C 4.425781 -3.707031 4.625 -3.113281 4.625 -2.34375 C 4.625 -2.289062 4.625 -2.21875 4.625 -2.125 L 1.140625 -2.125 C 1.171875 -1.613281 1.316406 -1.222656 1.578125 -0.953125 C 1.835938 -0.679688 2.164062 -0.546875 2.5625 -0.546875 C 2.851562 -0.546875 3.097656 -0.617188 3.296875 -0.765625 C 3.503906 -0.921875 3.664062 -1.164062 3.78125 -1.5 Z M 1.1875 -2.78125 L 3.796875 -2.78125 C 3.765625 -3.175781 3.664062 -3.472656 3.5 -3.671875 C 3.25 -3.972656 2.921875 -4.125 2.515625 -4.125 C 2.148438 -4.125 1.84375 -4 1.59375 -3.75 C 1.351562 -3.507812 1.21875 -3.1875 1.1875 -2.78125 Z M 1.1875 -2.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-11\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -4.671875 L 1.296875 -4.671875 L 1.296875 -3.953125 C 1.472656 -4.285156 1.640625 -4.503906 1.796875 -4.609375 C 1.953125 -4.710938 2.125 -4.765625 2.3125 -4.765625 C 2.570312 -4.765625 2.84375 -4.679688 3.125 -4.515625 L 2.84375 -3.78125 C 2.65625 -3.894531 2.460938 -3.953125 2.265625 -3.953125 C 2.097656 -3.953125 1.941406 -3.898438 1.796875 -3.796875 C 1.660156 -3.691406 1.5625 -3.546875 1.5 -3.359375 C 1.414062 -3.078125 1.375 -2.769531 1.375 -2.4375 L 1.375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-12\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -3.140625 C 0.4375 -4.203125 0.722656 -5.035156 1.296875 -5.640625 C 1.867188 -6.253906 2.609375 -6.5625 3.515625 -6.5625 C 4.109375 -6.5625 4.644531 -6.414062 5.125 -6.125 C 5.601562 -5.84375 5.96875 -5.445312 6.21875 -4.9375 C 6.46875 -4.425781 6.59375 -3.851562 6.59375 -3.21875 C 6.59375 -2.5625 6.460938 -1.972656 6.203125 -1.453125 C 5.941406 -0.941406 5.566406 -0.550781 5.078125 -0.28125 C 4.597656 -0.0195312 4.078125 0.109375 3.515625 0.109375 C 2.910156 0.109375 2.367188 -0.0351562 1.890625 -0.328125 C 1.410156 -0.617188 1.046875 -1.019531 0.796875 -1.53125 C 0.554688 -2.039062 0.4375 -2.578125 0.4375 -3.140625 Z M 1.3125 -3.125 C 1.3125 -2.34375 1.519531 -1.726562 1.9375 -1.28125 C 2.351562 -0.84375 2.878906 -0.625 3.515625 -0.625 C 4.148438 -0.625 4.675781 -0.847656 5.09375 -1.296875 C 5.507812 -1.742188 5.71875 -2.382812 5.71875 -3.21875 C 5.71875 -3.738281 5.628906 -4.191406 5.453125 -4.578125 C 5.273438 -4.972656 5.015625 -5.28125 4.671875 -5.5 C 4.328125 -5.71875 3.945312 -5.828125 3.53125 -5.828125 C 2.925781 -5.828125 2.40625 -5.617188 1.96875 -5.203125 C 1.53125 -4.785156 1.3125 -4.09375 1.3125 -3.125 Z M 1.3125 -3.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-0\" overflow=\"visible\">\n",
       "<path d=\"M 6.546875 -10.65625 L 1.625 -10.65625 L 1.625 -0.75 L 6.546875 -0.75 Z M 7.359375 -11.390625 L 7.359375 -0.015625 L 0.8125 -0.015625 L 0.8125 -11.390625 Z M 7.359375 -11.390625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-1\" overflow=\"visible\">\n",
       "<path d=\"M 12.4375 -1.78125 L 12.4375 0 L 14.203125 0 L 14.203125 -1.78125 Z M 7.109375 -1.78125 L 7.109375 0 L 8.875 0 L 8.875 -1.78125 Z M 1.78125 -1.78125 L 1.78125 0 L 3.546875 0 L 3.546875 -1.78125 Z M 1.78125 -1.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 124.015625 139.234375 C 128.996094 144.214844 128.996094 152.285156 124.015625 157.265625 C 119.035156 162.246094 110.964844 162.246094 105.984375 157.265625 C 101.003906 152.285156 101.003906 144.214844 105.984375 139.234375 C 110.964844 134.253906 119.035156 134.253906 124.015625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"86.803467\" xlink:href=\"#glyph0-1\" y=\"76.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"91.303467\" xlink:href=\"#glyph1-1\" y=\"78.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 176.890625 139.234375 C 181.871094 144.214844 181.871094 152.285156 176.890625 157.265625 C 171.910156 162.246094 163.839844 162.246094 158.859375 157.265625 C 153.878906 152.285156 153.878906 144.214844 158.859375 139.234375 C 163.839844 134.253906 171.910156 134.253906 176.890625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"139.678467\" xlink:href=\"#glyph0-1\" y=\"76.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.178467\" xlink:href=\"#glyph1-2\" y=\"78.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.98755\" xlink:href=\"#glyph2-1\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"11.48775\" xlink:href=\"#glyph2-2\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"16.49355\" xlink:href=\"#glyph2-3\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"21.49935\" xlink:href=\"#glyph2-4\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"26.50515\" xlink:href=\"#glyph2-5\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"29.00535\" xlink:href=\"#glyph2-6\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"31.50555\" xlink:href=\"#glyph2-7\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"33.50535\" xlink:href=\"#glyph2-8\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"38.51115\" xlink:href=\"#glyph2-9\" y=\"76.352783\"/>\n",
       "  <use x=\"43.01115\" xlink:href=\"#glyph2-10\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"48.01695\" xlink:href=\"#glyph2-11\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"5.4873\" xlink:href=\"#glyph2-12\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.4875\" xlink:href=\"#glyph2-4\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"17.4933\" xlink:href=\"#glyph2-5\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"19.9935\" xlink:href=\"#glyph2-3\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"24.9993\" xlink:href=\"#glyph2-4\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"30.0051\" xlink:href=\"#glyph2-5\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"32.5053\" xlink:href=\"#glyph2-6\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"35.0055\" xlink:href=\"#glyph2-7\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"37.0053\" xlink:href=\"#glyph2-8\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"42.0111\" xlink:href=\"#glyph2-9\" y=\"16.102783\"/>\n",
       "  <use x=\"46.5111\" xlink:href=\"#glyph2-10\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"51.5169\" xlink:href=\"#glyph2-11\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<path d=\"M 282.640625 138.484375 C 287.621094 143.464844 287.621094 151.535156 282.640625 156.515625 C 277.660156 161.496094 269.589844 161.496094 264.609375 156.515625 C 259.628906 151.535156 259.628906 143.464844 264.609375 138.484375 C 269.589844 133.503906 277.660156 133.503906 282.640625 138.484375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"245.428467\" xlink:href=\"#glyph0-1\" y=\"75.885498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"249.928467\" xlink:href=\"#glyph1-3\" y=\"77.885498\"/>\n",
       "</g>\n",
       "<path d=\"M 202.640625 78.234375 C 207.621094 83.214844 207.621094 91.285156 202.640625 96.265625 C 197.660156 101.246094 189.589844 101.246094 184.609375 96.265625 C 179.628906 91.285156 179.628906 83.214844 184.609375 78.234375 C 189.589844 73.253906 197.660156 73.253906 202.640625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"165.175781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"170.181152\" xlink:href=\"#glyph1-1\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 125.074219 140.433594 L 178.890625 98.683594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 182.050781 96.230469 L 178.890625 98.683594 M 177.96875 97.496094 L 182.050781 96.230469 L 179.808594 99.867188 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 172.835938 136.5 L 186.371094 104.433594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 187.925781 100.75 L 186.371094 104.433594 M 184.988281 103.851562 L 187.925781 100.75 L 187.753906 105.019531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 263.4375 139.828125 L 208.523438 98.472656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 205.328125 96.0625 L 208.523438 98.472656 M 207.621094 99.667969 L 205.328125 96.0625 L 209.425781 97.273438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"188.75\" xlink:href=\"#glyph3-1\" y=\"74.776001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename='../img/singleneuron.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear regression is a single-layer neural network.\n",
    "\n",
    "In the above network, the inputs are $x_1, x_2, \\ldots x_d$.\n",
    "Sometimes the number of inputs are referred to as the feature dimension.\n",
    "For linear regression models, we act upon $d$ inputs and output $1$ value.\n",
    "Because there is just a single computed neuron (node) in the graph,\n",
    "we can think of linear models as neural networks consisting of just a single neuron. Since all inputs are connected to all outputs (in this case it's just one), this layer can also be regarded as an instance of a *fully-connected layer*, also commonly called a *dense layer*.\n",
    "\n",
    "### Biology\n",
    "\n",
    "Neural networks derive their name from their inspirations in neuroscience.\n",
    "Although linear regression predates computation neuroscience,\n",
    "many of the models we subsequently discuss truly owe to neural inspiration.\n",
    "To understand the neural inspiration for artificial neural networks\n",
    "it is worth while considering the basic structure of a neuron.\n",
    "For the purpose of the analogy it is sufficient to consider the *dendrites*\n",
    "(input terminals), the *nucleus* (CPU), the *axon* (output wire),\n",
    "and the *axon terminals* (output terminals)\n",
    "which connect to other neurons via *synapses*.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg enable-background=\"new 0 0 399.821 214.98\" height=\"214.98\" id=\"Layer_1\" inkscape:version=\"0.44.1\" overflow=\"visible\" sodipodi:docbase=\"c:\\temp\" sodipodi:docname=\"Neuron.svg\" sodipodi:version=\"0.32\" version=\"1.1\" viewBox=\"0 0 399.821 214.98\" width=\"399.821\" xml:space=\"preserve\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:cc=\"http://web.resource.org/cc/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:inkscape=\"http://www.inkscape.org/namespaces/inkscape\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\" xmlns:sodipodi=\"http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd\" xmlns:svg=\"http://www.w3.org/2000/svg\"><metadata id=\"metadata78\"><rdf:RDF><cc:Work rdf:about=\"\"><dc:format>image/svg+xml</dc:format><dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/></cc:Work></rdf:RDF></metadata><defs id=\"defs76\"/><sodipodi:namedview bordercolor=\"#666666\" borderopacity=\"1.0\" gridtolerance=\"10.0\" guidetolerance=\"10.0\" id=\"base\" inkscape:current-layer=\"Layer_1\" inkscape:cx=\"199.91051\" inkscape:cy=\"151.14076\" inkscape:pageopacity=\"0.0\" inkscape:pageshadow=\"2\" inkscape:window-height=\"1031\" inkscape:window-width=\"1920\" inkscape:window-x=\"-4\" inkscape:window-y=\"-4\" inkscape:zoom=\"1.8327287\" objecttolerance=\"10.0\" pagecolor=\"#ffffff\"/>\n",
       "<path d=\"M126.833,132.089c3.606-1.491,1-6.333,1-6.333s-26.333-5-21-24s5.333-19,5.333-19  s6.333-4.668,11.667-8.334s3.667-4.999-0.667-3.666c-0.667-1.334,2.333-4.667,2.667-5.333c0.333-0.668,6-13.001-5.333,3.666  c0,1-9.667,12-4-7.666c0,0,7.333-35.334-1-9.668c0,0-2,6.334-4,2c0,0,4,13.335-8.333,46.001c-3-2.667-5-5.334-5-5.334  c-1-1.666-1.667-5.666,3-14.333s7.333-14,7.333-14s4.333-8.334-1-2.334c0,0,0.333-18.999-3,4.334c0,0-2.666,4.001-4.333,5.667  c0,0,4.667-11-1-8.333c0,0-3-7-2.667,1c0.333,7.999,2,9.667,1,14.333s-2.667,2.666-3-1.334s-10,0.334,5.333,24.667  c0,0,5.333,25.333-14.333,7.333c0,0-24.667-19.667,7-43c-4,3-5.667,6.335-5,2.334s0-3.667,0-3.667s-3.667,7.001-6,8.667  s-11.667,4.667,9-18.667c-4,3-11.333,8.667-4-7.667c-7,11-11.333,23.667-15.333,12.667c0,0,4.333,46.332-6,16.333  c0,0-3.667-6,5-35.667c-3,6.667-5.667,16-5-3.333c0,0-3,1-2.333,22.333c0,0-9.333,21.334-7.333-7c0,0,3-12.666,3.333-15  s-1.667,0.667-2.333,2s-0.667-5-0.667-5s-2,3-2.333,7s-3.333-3-4.333-4.667s6.666,11,1.333,21.666c0,0-7.666-12.666-8.333-9  c-0.667,3.667,16,23.668,16.333,25.001s10,11-2,9.333c-4-0.333-6.333-2.667-8.333-6s-6.333-10-6.333-10l-9.667-11L31.5,43.755  l-1.333,3.667l-2.333-2.333l2,4.667l0.333,6.333l-2.333,1c0,0,28,28.334,5.333,17.667c0,0-8.334,2.334,10.333,7  c0,0,1.333,7.666,17.333,8.666c0,0,12,13.001,15,25.334c0,0,3.333,26.667-12,6c0,0,3,7.333,1,7.333s-11.667-5.667-13-5.667  s-5.333,2-5.333,2s27,6.666,1.667,7c0,0-18,6.333-11-36c0,0-2.333,12.334-4.333,0.667c0,0-5.333,26.334,0,36.667  c0,0,5,6.332-2.333,6.666c0,0,9-1.334,18-4c0,0-6.667,13.334-10,13.334c0,0,3,0.666-0.333,2.666c0,0,1.333,2.666,5-2  s9-15.999,22-15.666c4.333,1.333,20.333,14.666-4,19.333c-7.333,3-35,13.001-32.667,14.667s16.333-8.334,14-3.334  s-10,19.334-10,19.334s9.333-21.333,7.333-5c1,1.333,25.333-45.333,8,0c0.333,1,4.333-6.001,5,3.333c0,0,0.333-35.332,25.333-24.666  c0.667,9-4.667,21.332-18.667,29.666c14.333-8.333,16.667-12,10,2c3.333-3.333,5-2.999,6,1.334c1.333-15,4-19.666,6.667-22  c2.667,7,3.667,9.999,4,11.666s-1.667-23.667-2-25s13-0.666,13-0.666s10,19.334,5.667,28c2.333-7.334-1.667-22.001,16,3.666  c0,0-12-13.333,7.333-8.333c-6.667-5.333-19-12.666-22.333-18.333s-4-13.333,6.333-6s16.667,10.332,17.667,9.666s-19.667-10,3.333-6  C116.5,151.756,92.167,146.422,126.833,132.089z\" fill=\"#AFA3CF\" id=\"path3\" stroke=\"#000000\"/>\n",
       "<path d=\"M85.125,136.589c-0.125-4.5,1.375-5.875,4.125-5.5s5.5,1.625,6.625,2.875s2.75,3.5,3,4.375  s0.375,1.75,0.5,2.625s0.5,1.75,0.125,2.5s-0.875,2.375-1.875,2.5s-1.625,0.375-2.75-0.25s-2.875-1.75-3.625-2.375  s-3.75-3.625-4.375-4.5S85.125,136.589,85.125,136.589z\" fill=\"#5B9974\" id=\"path5\" stroke=\"#000000\"/>\n",
       "<path d=\"M127.5,133.339c2.25-4.5,1.625-6,1.125-7.375s17.625-2,30,5c0,0,11.75,4,1.25,11.5  C159.875,142.464,140.375,145.339,127.5,133.339z\" fill=\"#F9D566\" id=\"path7\" stroke=\"#000000\"/>\n",
       "<path d=\"M168.375,145.839c0,0-3.625-1.75-2.5-5s2.125-3.25,3.125-3.625s6.75-0.25,7.25-0.25  s19.75,2,22.375,4.25s3.5,5,3.5,6.125s-0.25,4.75-3.75,4.75s-18.375-0.25-19-0.875S170.875,148.464,168.375,145.839z\" fill=\"#F9D566\" id=\"path9\" stroke=\"#000000\"/>\n",
       "<path d=\"M214.375,155.089c-6.625-2.125-9.5-4.625-9.25-7.25s0.75-3.625,1.875-4.25  s3.875-1.375,4.625-1.625s22.375-3.375,24.5-2.875s4.75,3,5,4.5s2.625,5-2.75,7.25s-9.375,3.75-13,4  S214.375,155.089,214.375,155.089z\" fill=\"#F9D566\" id=\"path11\" stroke=\"#000000\"/>\n",
       "<path d=\"M252.125,145.464c-6.009,0.376-7.875,1.125-7.75-3.375s-0.375-6.125,3.25-7.875  s13.875-8.5,15.5-9.25s6.875-2.375,9.25,0.75s2.125,4.5-0.375,7.25S256.125,145.214,252.125,145.464z\" fill=\"#F9D566\" id=\"path13\" stroke=\"#000000\"/>\n",
       "<path d=\"M274,122.839c-2.44-4.532-3.375-7-1.125-10.25s13.125-18.75,13.5-19.25  s4.625-2.375,8.375,0.875s4.5,5.5,1,10.25s-11.625,17.5-14.5,19S275.75,126.089,274,122.839z\" fill=\"#F9D566\" id=\"path15\" stroke=\"#000000\"/>\n",
       "<ellipse cx=\"140\" cy=\"130.964\" fill=\"#E68B60\" id=\"ellipse17\" rx=\"3.375\" ry=\"2.125\" stroke=\"#B76328\"/>\n",
       "<ellipse cx=\"181.875\" cy=\"144.839\" fill=\"#E68B60\" id=\"ellipse19\" rx=\"3.375\" ry=\"2.125\" stroke=\"#B76328\"/>\n",
       "<ellipse cx=\"224.125\" cy=\"147.839\" fill=\"#E68B60\" id=\"ellipse21\" rx=\"3.375\" ry=\"2.125\" stroke=\"#B76328\"/>\n",
       "<ellipse cx=\"258.5\" cy=\"134.214\" fill=\"#E68B60\" id=\"ellipse23\" rx=\"3.375\" ry=\"2.125\" stroke=\"#B76328\" transform=\"matrix(0.912 -0.4103 0.4103 0.912 -32.3066 117.8699)\"/>\n",
       "<ellipse cx=\"283.625\" cy=\"107.464\" fill=\"#E68B60\" id=\"ellipse25\" rx=\"3.375\" ry=\"2.125\" stroke=\"#B76328\" transform=\"matrix(0.5422 -0.8402 0.8402 0.5422 39.543 287.5071)\"/>\n",
       "<path d=\"M164.25,137.089c2.25,0.5,3.75,0.5,3.75,0.5l-2.125,3.25l-4.125-0.125L164.25,137.089z\" fill=\"#AFA3CF\" id=\"path27\" stroke=\"#000000\"/>\n",
       "<path d=\"M201.75,145.214l4.125-0.25L205,149.089h-3.125  C201.875,149.089,202.125,145.714,201.75,145.214z\" fill=\"#AFA3CF\" id=\"path29\" stroke=\"#000000\"/>\n",
       "<path d=\"M240.25,141.589c0,0,3.375-2.5,3.625-3.125s0.375,5,0.5,5.375s-2.25,0.875-2.25,0.875  L240.25,141.589z\" fill=\"#AFA3CF\" id=\"path31\" stroke=\"#000000\"/>\n",
       "<path d=\"M269.625,123.714l2.75-3.875l2.875,4.375l-2,1.875  C273.25,126.089,271.5,123.964,269.625,123.714z\" fill=\"#AFA3CF\" id=\"path33\" stroke=\"#000000\"/>\n",
       "<path d=\"M291.167,92.256c0.041-0.09,0.082-0.18,0.123-0.27c3.275-7.251,5.894-14.772,7.21-18.064  c1.333-3.333,1.833-11.166,1.333-12s-2-6.5-5.833-5.333s-2.501,0-3.667,1.833s-3.999,2-4.166,0.333s0.5-2.167,3.333-2.667  s1.333-2.5-0.667-4.333s-7.334-1.833-3.5-4.333s4-2.834,5.5-1.667s-3.167,2,1,4.5s7.166,4,8.5,4.167s2.5,0.333,2.5-1.167  s1.667-3,2.667-3.833s1.5-4.167,1.5-4.167s-3.5-3.166-0.5-2.833s4.5,0.5,3,2.333c0.333,0.5-0.166,2,0.167,2.5S315,47.089,315,47.089  s4.5-0.333,3,2.5c-0.333,0.833-1.5,2-3.5,0.667c-1.833,0.333-3.667-0.333-4.167-0.167s-7.167,3.167-6.5,18.167  c4.5-2.5,17.668-10.334,19.334-11s9.666-9.833,9.833-10.667s-1.666-3.333-1.333-4s3.499-2,4.166-2.167s1.501-2,2.334,0.167  s0.667,2.167,0,2.667s-1.166,0.833-1,1.833s0.333,1.667,0.833,1.5s0.833,0.833,0.833,0.833l-6.666,5.333c0,0,6.167-1.167,7-2  s2.5-1.667,2.833-0.5s0.667,0.834,0,1.667s-1.833,1.666-4,2.333s-3.667,1.333-5,1.5s-3.833,1-3.833,1s-1.333,0.333-4,2.5  s-4.5,3.167-6.167,4.167s-2.833,1.333-4.5,2.5s-4.5,2.5-4.5,2.5s-2.501,0.834-3.167,1.667s-1.832,2.167-2.166,2.667  s-1,2.166-1,2.166l-0.667,3l-2.167,2.5l-0.5,2.334c0,0,4.166,0.5,5,0.5s1.5-0.001,2.667-0.834s1-0.334,2-1.5  s2.167-2.333,2.667-3.333s1.666-2.167,2.666-3s2.333-2.001,3-2.167s3-0.833,3.5-1.333s1.001,0.001,1.334-0.833s1.334-2.5,1.5-3.333  s0.166-1.167,0.833-1.167s1.167-0.333,1.5,0.5s1.001,0.667,0.667,1.833S327,69.922,327,69.922l0.167,0.334c0,0,1.5,0.499,2-0.334  s1.333-1.333,1.333-1.333l1-1.333l0.833-1.5l0.834-0.667l1.5-0.333c0,0,0.5,0,0.833,1.167s0.5,1.667,0.5,2.167s-2,2-2,2  l-0.667,0.667c0,0-1.5,0.332-2,0.666s-2.166,0.834-2.166,0.834l-2.334,0.166l-1.666,0.834l-0.667,0.5v1c0,0,0.667,0.834,1.167,1  s3.5,0,3.5,0l0.666-0.167l-1,1.333l-3.333,0.667c0,0-0.834,0.166-1.667,0.333s-3,0.5-3,0.5s-1.834,0.834-2.5,1.334s-2,0.167-0.833,1  s0.333,1.334,4,1s1.334-0.667,4.667,0.333s4,0.667,6,1.667s-1.334,0.332,4.333,2.166s5.499,1.666,9.333,3s3.333-0.5,4.5,2  s6.667,2.833,1.167,2.5s-4.499-0.333-7.333-1.333s-2.5-1.167-6.667-2.667s-6.667-3.667-2.167,1.5s5.5,5.667,6.167,10.167  s1,14.167,5,14.5s7.501,1.333,5.167,3.333s-6.168,0.668-7.334-1.166s-3.666-3.5-4.166-6s0.334-4.501-0.5-4.834  s-0.834-0.332-1.667,0.334s-0.333-0.834-1.5,1.833s-1.501,3.167-2.167,3.5s-4.166,0.834-4.166-0.833s0.5-2.168,1.333-2.334  s-0.667,1.333,2-1s3.834-4.166,4.167-5.166s2.166-5.5-5.667-11.167s-13.666-9.166-20.833-5.833c2,3,5.667,5.333,5.5,11.333  c0,0.667,2.832,3-0.334,4c-0.5,0.167-3.5,0.333-1.833-4c0-2.167,0.501-3.499-3.833-7.333c0.166,3.666-0.168,7.332,0.166,8.666  s3.168,4.501-0.666,5.834c-2.167-0.5-3.834-1.833-1.334-6c0-3.834-3.999-20.333-14.166-5  C292.333,93.089,291.167,92.256,291.167,92.256z\" fill=\"#AFA3CF\" id=\"path35\" stroke=\"#000000\"/>\n",
       "<path d=\"M367.333,131.756\" fill=\"#AFA3CF\" id=\"path37\" stroke=\"#000000\"/>\n",
       "<line id=\"line39\" stroke=\"#000000\" x1=\"142.5\" x2=\"111.5\" y1=\"78.089\" y2=\"118.589\"/>\n",
       "<line id=\"line41\" stroke=\"#000000\" x1=\"74\" x2=\"75.5\" y1=\"18.589\" y2=\"65.089\"/>\n",
       "<line fill=\"none\" id=\"line43\" stroke=\"#000000\" x1=\"89.5\" x2=\"43.5\" y1=\"142.089\" y2=\"200.089\"/>\n",
       "<line fill=\"none\" id=\"line45\" stroke=\"#000000\" x1=\"127.5\" x2=\"143\" y1=\"133.339\" y2=\"162.089\"/>\n",
       "<line fill=\"none\" id=\"line47\" stroke=\"#000000\" x1=\"227\" x2=\"227.5\" y1=\"155.089\" y2=\"183.589\"/>\n",
       "<line fill=\"none\" id=\"line49\" stroke=\"#000000\" x1=\"259.372\" x2=\"304\" y1=\"136.151\" y2=\"159.589\"/>\n",
       "<line fill=\"none\" id=\"line51\" stroke=\"#000000\" x1=\"271\" x2=\"255\" y1=\"121.089\" y2=\"69.089\"/>\n",
       "<line fill=\"none\" id=\"line53\" stroke=\"#000000\" x1=\"350.333\" x2=\"349\" y1=\"90.922\" y2=\"35.589\"/>\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text55\" transform=\"matrix(1 0 0 1 42 12.0889)\">Dendrite</text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text57\" transform=\"matrix(1 0 0 1 138 76.5889)\">Cell body</text>\n",
       "\n",
       "<text id=\"text59\" transform=\"matrix(1 0 0 1 224.8135 49.5889)\"><tspan font-family=\"'Verdana'\" font-size=\"14\" id=\"tspan61\" x=\"0\" y=\"0\">Node of</tspan><tspan font-family=\"'Verdana'\" font-size=\"14\" id=\"tspan63\" x=\"0.465\" y=\"16.8\">Ranvier</tspan></text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text65\" transform=\"matrix(1 0 0 1 298.1777 26.5889)\">Axon Terminal</text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text67\" transform=\"matrix(1 0 0 1 308 171.5889)\">Schwann cell</text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text69\" transform=\"matrix(1 0 0 1 219.5 198.0889)\">Myelin sheath</text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text71\" transform=\"matrix(1 0 0 1 140 176.0889)\">Axon</text>\n",
       "\n",
       "<text font-family=\"'Verdana'\" font-size=\"14\" id=\"text73\" transform=\"matrix(1 0 0 1 0 212.0889)\">Nucleus</text>\n",
       "\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename='../img/Neuron.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The real neuron\n",
    "\n",
    "Information $x_i$ arriving from other neurons (or environmental sensors such as the retina) is received in the dendrites. In particular, that information is weighted by *synaptic weights* $w_i$ which determine how to respond to the inputs (e.g. activation or inhibition via $x_i w_i$). All this is aggregated in the nucleus $y = \\sum_i x_i w_i + b$, and this information is then sent for further processing in the axon $y$, typically after some nonlinear processing via $\\sigma(y)$. From there it either reaches its destination (e.g. a muscle) or is fed into another neuron via its dendrites.\n",
    "\n",
    "Brain *structures* vary significantly. Some look (to us) rather arbitrary whereas others have a regular structure. For example, the visual system of many insects is consistent across members of a species. The analysis of such structures has often inspired neuroscientists to propose new architectures, and in some cases, this has been successful. However, much research in artificial neural networks has little to do with any direct inspiration in neuroscience, just as although airplanes are *inspired* by birds, the study of orninthology hasn't been the primary driver of aeronautics innovaton in the last century. Equal amounts of inspiration these days comes from mathematics, statistics, and computer science.\n",
    "\n",
    "### Vectorization for Speed\n",
    "\n",
    "In model training or prediction, we often use vector calculations and process multiple observations at the same time. To illustrate why this matters, consider two methods of adding vectors. We begin by creating two 10000 dimensional ones first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "1"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from time import time\n",
    "\n",
    "a = torch.ones(10000)\n",
    "b = torch.ones(10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to add vectors is to add them one coordinate at a time using a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11870288848876953"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = time()\n",
    "c = torch.zeros(10000)\n",
    "for i in range(10000):\n",
    "    c[i] = a[i] + b[i]\n",
    "time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to add vectors is to add the vectors directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "3"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0002841949462890625"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start = time()\n",
    "d = a + b\n",
    "time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, the latter is vastly faster than the former. Vectorizing code is a good way of getting order of magnitude speedups. Likewise, as we saw above, it also greatly simplifies the mathematics and with it, it reduces the potential for errors in the notation.\n",
    "\n",
    "## The Normal Distribution and Squared Loss\n",
    "\n",
    "The following is optional and can be skipped but it will greatly help with understanding some of the design choices in building deep learning models. As we saw above, using the squared loss $l(y, \\hat{y}) = \\frac{1}{2} (y - \\hat{y})^2$ has many nice properties, such as having a particularly simple derivative $\\partial_{\\hat{y}} l(y, \\hat{y}) = (\\hat{y} - y)$. That is, the gradient is given by the difference between estimate and observation. You might reasonably point out that linear regression is a [classical](https://en.wikipedia.org/wiki/Regression_analysis#History) statistical model. Legendre first developed the method of least squares regression in 1805, which was shortly thereafter rediscovered by Gauss in 1809. To understand this a bit better, recall the normal distribution with mean $\\mu$ and variance $\\sigma^2$.\n",
    "\n",
    "$$p(x) = \\frac{1}{\\sqrt{2 \\pi \\sigma^2}} \\exp\\left(-\\frac{1}{2 \\sigma^2} (x - \\mu)^2\\right)$$\n",
    "\n",
    "It can be visualized as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "attributes": {
     "classes": [],
     "id": "",
     "n": "2"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"360.738125pt\" version=\"1.1\" viewBox=\"0 0 605.165625 360.738125\" width=\"605.165625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 360.738125 \n",
       "L 605.165625 360.738125 \n",
       "L 605.165625 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 36.465625 336.86 \n",
       "L 594.465625 336.86 \n",
       "L 594.465625 10.7 \n",
       "L 36.465625 10.7 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"m26bf400bc0\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"98.088928\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-2212\"/>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(90.717834 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"170.608261\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(163.237167 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"243.127594\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(235.756501 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-2212\"/>\n",
       "       <use x=\"83.789062\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"315.646927\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(312.465677 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"388.166261\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(384.985011 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"460.685594\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(457.504344 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"533.204927\" xlink:href=\"#m26bf400bc0\" y=\"336.86\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(530.023677 351.458437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"m8cba4c20be\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"322.034545\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.00 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(7.2 325.833764)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"284.872643\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.05 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"DejaVuSans-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(7.2 288.671862)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"247.71074\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(7.2 251.509959)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"210.548838\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.15 -->\n",
       "      <g transform=\"translate(7.2 214.348057)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"173.386935\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0.20 -->\n",
       "      <g transform=\"translate(7.2 177.186154)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"136.225033\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0.25 -->\n",
       "      <g transform=\"translate(7.2 140.024252)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"99.063131\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0.30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "z\n",
       "\" id=\"DejaVuSans-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(7.2 102.862349)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"61.901228\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 0.35 -->\n",
       "      <g transform=\"translate(7.2 65.700447)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"36.465625\" xlink:href=\"#m8cba4c20be\" y=\"24.739326\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 0.40 -->\n",
       "      <g transform=\"translate(7.2 28.538544)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-2e\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use x=\"159.033203\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_17\">\n",
       "    <path clip-path=\"url(#p456b25a518)\" d=\"M 61.829261 322.034545 \n",
       "L 171.333454 321.926815 \n",
       "L 182.573948 321.681928 \n",
       "L 189.825883 321.314451 \n",
       "L 195.264837 320.836277 \n",
       "L 199.978589 320.205073 \n",
       "L 203.967157 319.451735 \n",
       "L 207.59312 318.537641 \n",
       "L 210.856487 317.480446 \n",
       "L 213.757266 316.314206 \n",
       "L 216.658037 314.895152 \n",
       "L 219.196211 313.412887 \n",
       "L 221.734394 311.673759 \n",
       "L 224.272568 309.644691 \n",
       "L 226.810743 307.29067 \n",
       "L 228.986321 304.986636 \n",
       "L 231.161907 302.393374 \n",
       "L 233.337485 299.48696 \n",
       "L 235.513063 296.243486 \n",
       "L 237.688641 292.639454 \n",
       "L 239.864227 288.65217 \n",
       "L 242.402402 283.487561 \n",
       "L 244.940576 277.741495 \n",
       "L 247.478754 271.387652 \n",
       "L 250.016933 264.405421 \n",
       "L 252.555107 256.781166 \n",
       "L 255.093286 248.509366 \n",
       "L 257.994056 238.268494 \n",
       "L 260.894831 227.21003 \n",
       "L 264.158202 213.845728 \n",
       "L 267.784166 197.960587 \n",
       "L 271.77273 179.43679 \n",
       "L 277.211683 152.970444 \n",
       "L 287.364389 103.295829 \n",
       "L 290.990354 86.730654 \n",
       "L 293.891127 74.369325 \n",
       "L 296.429305 64.377893 \n",
       "L 298.967481 55.293622 \n",
       "L 301.143061 48.322173 \n",
       "L 302.956044 43.141635 \n",
       "L 304.769027 38.572599 \n",
       "L 306.582011 34.648089 \n",
       "L 308.032398 31.991923 \n",
       "L 309.482784 29.779205 \n",
       "L 310.933171 28.020389 \n",
       "L 312.020961 27.004314 \n",
       "L 313.108751 26.251007 \n",
       "L 314.196541 25.762551 \n",
       "L 315.284331 25.540273 \n",
       "L 316.372121 25.584751 \n",
       "L 317.459911 25.895851 \n",
       "L 318.547701 26.472753 \n",
       "L 319.635491 27.31393 \n",
       "L 320.723281 28.417055 \n",
       "L 321.811071 29.779205 \n",
       "L 323.261457 31.991923 \n",
       "L 324.711844 34.648089 \n",
       "L 326.16223 37.735165 \n",
       "L 327.975214 42.177791 \n",
       "L 329.788197 47.23885 \n",
       "L 331.963777 54.077177 \n",
       "L 334.139357 61.684293 \n",
       "L 336.677533 71.42957 \n",
       "L 339.578308 83.556373 \n",
       "L 342.841677 98.217736 \n",
       "L 346.830241 117.184571 \n",
       "L 354.807369 156.549997 \n",
       "L 360.608914 184.58212 \n",
       "L 364.597478 202.831564 \n",
       "L 368.223446 218.404453 \n",
       "L 371.486813 231.450512 \n",
       "L 374.387588 242.20626 \n",
       "L 377.288362 252.13369 \n",
       "L 380.189133 261.217024 \n",
       "L 382.727312 268.473193 \n",
       "L 385.265486 275.094181 \n",
       "L 387.803664 281.097805 \n",
       "L 390.341839 286.508079 \n",
       "L 392.880021 291.354035 \n",
       "L 395.418196 295.66848 \n",
       "L 397.95637 299.48696 \n",
       "L 400.494544 302.846681 \n",
       "L 403.032727 305.785604 \n",
       "L 405.570901 308.341639 \n",
       "L 408.109076 310.552 \n",
       "L 410.64725 312.452624 \n",
       "L 413.185433 314.077739 \n",
       "L 415.723607 315.459524 \n",
       "L 418.624377 316.77903 \n",
       "L 421.525157 317.860534 \n",
       "L 424.788524 318.838136 \n",
       "L 428.414487 319.680752 \n",
       "L 432.403055 320.372785 \n",
       "L 437.116807 320.950361 \n",
       "L 442.555761 321.385934 \n",
       "L 449.445099 321.706949 \n",
       "L 459.235209 321.917912 \n",
       "L 475.914657 322.017572 \n",
       "L 539.006468 322.034544 \n",
       "L 569.101989 322.034545 \n",
       "L 569.101989 322.034545 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_18\">\n",
       "    <path clip-path=\"url(#p456b25a518)\" d=\"M 61.829261 321.71024 \n",
       "L 77.783517 321.350868 \n",
       "L 90.111809 320.857649 \n",
       "L 100.264506 320.233296 \n",
       "L 108.966835 319.480494 \n",
       "L 116.581366 318.608459 \n",
       "L 123.470688 317.60773 \n",
       "L 129.997439 316.438105 \n",
       "L 136.161585 315.102753 \n",
       "L 141.963126 313.611948 \n",
       "L 147.402079 311.982955 \n",
       "L 152.478428 310.239487 \n",
       "L 157.554776 308.261265 \n",
       "L 162.268537 306.198034 \n",
       "L 166.982298 303.902501 \n",
       "L 171.69605 301.361759 \n",
       "L 176.409811 298.564363 \n",
       "L 181.123563 295.500778 \n",
       "L 185.837324 292.163764 \n",
       "L 190.551076 288.548854 \n",
       "L 195.264837 284.654737 \n",
       "L 199.978589 280.48369 \n",
       "L 205.054946 275.689325 \n",
       "L 210.131295 270.594442 \n",
       "L 215.570248 264.824431 \n",
       "L 221.371798 258.350708 \n",
       "L 227.898532 250.735668 \n",
       "L 235.875659 241.076533 \n",
       "L 260.169639 211.390498 \n",
       "L 265.971184 204.783397 \n",
       "L 271.047537 199.32524 \n",
       "L 275.761293 194.590087 \n",
       "L 280.112454 190.551128 \n",
       "L 284.101017 187.163628 \n",
       "L 287.726985 184.370231 \n",
       "L 291.35295 181.869836 \n",
       "L 294.616321 179.884855 \n",
       "L 297.879691 178.163385 \n",
       "L 301.143061 176.715635 \n",
       "L 304.043834 175.665571 \n",
       "L 306.944608 174.843598 \n",
       "L 309.845381 174.253649 \n",
       "L 312.746154 173.898548 \n",
       "L 315.646927 173.78 \n",
       "L 318.547701 173.898548 \n",
       "L 321.448474 174.253649 \n",
       "L 324.349247 174.843598 \n",
       "L 327.250021 175.665571 \n",
       "L 330.150794 176.715635 \n",
       "L 333.051567 177.98883 \n",
       "L 336.314937 179.680365 \n",
       "L 339.578308 181.636617 \n",
       "L 342.841677 183.846112 \n",
       "L 346.467645 186.582428 \n",
       "L 350.09361 189.596066 \n",
       "L 354.082172 193.206219 \n",
       "L 358.433332 197.462938 \n",
       "L 363.147089 202.402802 \n",
       "L 368.223446 208.043699 \n",
       "L 374.387588 215.242786 \n",
       "L 382.364715 224.935579 \n",
       "L 404.483112 252.02594 \n",
       "L 411.372451 259.997566 \n",
       "L 417.536589 266.781785 \n",
       "L 422.975542 272.447452 \n",
       "L 428.05189 277.436913 \n",
       "L 433.128248 282.120403 \n",
       "L 438.204605 286.486662 \n",
       "L 442.918357 290.251967 \n",
       "L 447.632118 293.73826 \n",
       "L 452.34587 296.948339 \n",
       "L 457.059631 299.88802 \n",
       "L 461.773391 302.565653 \n",
       "L 466.487135 304.991694 \n",
       "L 471.200896 307.178332 \n",
       "L 476.277244 309.280883 \n",
       "L 481.35361 311.139391 \n",
       "L 486.792546 312.881095 \n",
       "L 492.2315 314.387507 \n",
       "L 498.033058 315.761148 \n",
       "L 504.197187 316.986854 \n",
       "L 510.723938 318.056142 \n",
       "L 517.975864 319.009637 \n",
       "L 525.953001 319.822472 \n",
       "L 534.655312 320.483807 \n",
       "L 544.808027 321.028396 \n",
       "L 557.136301 321.455005 \n",
       "L 569.101989 321.704519 \n",
       "L 569.101989 321.704519 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_19\">\n",
       "    <path clip-path=\"url(#p456b25a518)\" d=\"M 61.829261 322.034545 \n",
       "L 280.112454 321.926815 \n",
       "L 291.35295 321.681928 \n",
       "L 298.604884 321.314451 \n",
       "L 304.043834 320.836277 \n",
       "L 308.757591 320.205073 \n",
       "L 312.746154 319.451735 \n",
       "L 316.372121 318.537641 \n",
       "L 319.635491 317.480446 \n",
       "L 322.536264 316.314206 \n",
       "L 325.437038 314.895152 \n",
       "L 327.975214 313.412887 \n",
       "L 330.513391 311.673759 \n",
       "L 333.051567 309.644691 \n",
       "L 335.589745 307.29067 \n",
       "L 337.765325 304.986627 \n",
       "L 339.940905 302.393374 \n",
       "L 342.116485 299.48696 \n",
       "L 344.292065 296.243486 \n",
       "L 346.467645 292.639454 \n",
       "L 348.643225 288.65217 \n",
       "L 351.181401 283.487561 \n",
       "L 353.719576 277.741495 \n",
       "L 356.257754 271.387652 \n",
       "L 358.795933 264.405421 \n",
       "L 361.334107 256.781166 \n",
       "L 363.872286 248.509366 \n",
       "L 366.773056 238.268494 \n",
       "L 369.673831 227.21003 \n",
       "L 372.937202 213.845728 \n",
       "L 376.563165 197.960587 \n",
       "L 380.551729 179.43679 \n",
       "L 385.990683 152.970444 \n",
       "L 396.143388 103.295829 \n",
       "L 399.769352 86.730654 \n",
       "L 402.670131 74.369303 \n",
       "L 405.208305 64.377893 \n",
       "L 407.746479 55.293622 \n",
       "L 409.922057 48.322173 \n",
       "L 411.735047 43.141613 \n",
       "L 413.548029 38.572599 \n",
       "L 415.361011 34.648089 \n",
       "L 416.811396 31.991923 \n",
       "L 418.261781 29.779205 \n",
       "L 419.712166 28.020389 \n",
       "L 420.799964 27.004314 \n",
       "L 421.887753 26.251007 \n",
       "L 422.975542 25.762551 \n",
       "L 424.063331 25.540273 \n",
       "L 425.15112 25.584751 \n",
       "L 426.238909 25.895851 \n",
       "L 427.326698 26.472753 \n",
       "L 428.414487 27.31393 \n",
       "L 429.502284 28.417055 \n",
       "L 430.590073 29.779205 \n",
       "L 432.040459 31.991923 \n",
       "L 433.490844 34.648089 \n",
       "L 434.941229 37.735165 \n",
       "L 436.754211 42.177791 \n",
       "L 438.567201 47.23885 \n",
       "L 440.742779 54.077199 \n",
       "L 442.918357 61.684293 \n",
       "L 445.456531 71.42957 \n",
       "L 448.35731 83.556373 \n",
       "L 451.620677 98.217736 \n",
       "L 455.609237 117.184549 \n",
       "L 463.586364 156.549986 \n",
       "L 469.387906 184.582086 \n",
       "L 473.376474 202.831531 \n",
       "L 477.002437 218.40442 \n",
       "L 480.265812 231.450512 \n",
       "L 483.166583 242.206249 \n",
       "L 486.067354 252.133668 \n",
       "L 488.968141 261.217052 \n",
       "L 491.506307 268.473176 \n",
       "L 494.04449 275.094187 \n",
       "L 496.582655 281.097788 \n",
       "L 499.120838 286.508079 \n",
       "L 501.659021 291.354035 \n",
       "L 504.197187 295.668469 \n",
       "L 506.73537 299.48696 \n",
       "L 509.273553 302.846692 \n",
       "L 511.811718 305.785597 \n",
       "L 514.349901 308.341639 \n",
       "L 516.888084 310.552009 \n",
       "L 519.42625 312.452624 \n",
       "L 521.964432 314.077739 \n",
       "L 524.502615 315.459528 \n",
       "L 527.403386 316.779032 \n",
       "L 530.304156 317.860534 \n",
       "L 533.567532 318.838139 \n",
       "L 537.193495 319.680754 \n",
       "L 541.182046 320.372783 \n",
       "L 545.895807 320.950361 \n",
       "L 551.33476 321.385934 \n",
       "L 558.224099 321.706949 \n",
       "L 568.014208 321.917912 \n",
       "L 569.101989 321.931023 \n",
       "L 569.101989 321.931023 \n",
       "\" style=\"fill:none;stroke:#2ca02c;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 36.465625 336.86 \n",
       "L 36.465625 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 594.465625 336.86 \n",
       "L 594.465625 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 36.465625 336.86 \n",
       "L 594.465625 336.86 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 36.465625 10.7 \n",
       "L 594.465625 10.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 458.615625 62.734375 \n",
       "L 587.465625 62.734375 \n",
       "Q 589.465625 62.734375 589.465625 60.734375 \n",
       "L 589.465625 17.7 \n",
       "Q 589.465625 15.7 587.465625 15.7 \n",
       "L 458.615625 15.7 \n",
       "Q 456.615625 15.7 456.615625 17.7 \n",
       "L 456.615625 60.734375 \n",
       "Q 456.615625 62.734375 458.615625 62.734375 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_20\">\n",
       "     <path d=\"M 460.615625 23.798437 \n",
       "L 480.615625 23.798437 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_21\"/>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- mean 0, variance 1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 52 44.1875 \n",
       "Q 55.375 50.25 60.0625 53.125 \n",
       "Q 64.75 56 71.09375 56 \n",
       "Q 79.640625 56 84.28125 50.015625 \n",
       "Q 88.921875 44.046875 88.921875 33.015625 \n",
       "L 88.921875 0 \n",
       "L 79.890625 0 \n",
       "L 79.890625 32.71875 \n",
       "Q 79.890625 40.578125 77.09375 44.375 \n",
       "Q 74.3125 48.1875 68.609375 48.1875 \n",
       "Q 61.625 48.1875 57.5625 43.546875 \n",
       "Q 53.515625 38.921875 53.515625 30.90625 \n",
       "L 53.515625 0 \n",
       "L 44.484375 0 \n",
       "L 44.484375 32.71875 \n",
       "Q 44.484375 40.625 41.703125 44.40625 \n",
       "Q 38.921875 48.1875 33.109375 48.1875 \n",
       "Q 26.21875 48.1875 22.15625 43.53125 \n",
       "Q 18.109375 38.875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.1875 51.21875 25.484375 53.609375 \n",
       "Q 29.78125 56 35.6875 56 \n",
       "Q 41.65625 56 45.828125 52.96875 \n",
       "Q 50 49.953125 52 44.1875 \n",
       "z\n",
       "\" id=\"DejaVuSans-6d\"/>\n",
       "      <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-65\"/>\n",
       "      <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-61\"/>\n",
       "      <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-6e\"/>\n",
       "      <path id=\"DejaVuSans-20\"/>\n",
       "      <path d=\"M 11.71875 12.40625 \n",
       "L 22.015625 12.40625 \n",
       "L 22.015625 4 \n",
       "L 14.015625 -11.625 \n",
       "L 7.71875 -11.625 \n",
       "L 11.71875 4 \n",
       "z\n",
       "\" id=\"DejaVuSans-2c\"/>\n",
       "      <path d=\"M 2.984375 54.6875 \n",
       "L 12.5 54.6875 \n",
       "L 29.59375 8.796875 \n",
       "L 46.6875 54.6875 \n",
       "L 56.203125 54.6875 \n",
       "L 35.6875 0 \n",
       "L 23.484375 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-76\"/>\n",
       "      <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-72\"/>\n",
       "      <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-69\"/>\n",
       "      <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-63\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(488.615625 27.298437)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
       "      <use x=\"97.412109\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"158.935547\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"220.214844\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"283.59375\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"315.380859\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      <use x=\"379.003906\" xlink:href=\"#DejaVuSans-2c\"/>\n",
       "      <use x=\"410.791016\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"442.578125\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "      <use x=\"501.757812\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"563.037109\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"604.150391\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"631.933594\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"693.212891\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"756.591797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"811.572266\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"873.095703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"904.882812\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_22\">\n",
       "     <path d=\"M 460.615625 38.476562 \n",
       "L 480.615625 38.476562 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_23\"/>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- mean 0, variance 2 -->\n",
       "     <g transform=\"translate(488.615625 41.976562)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
       "      <use x=\"97.412109\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"158.935547\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"220.214844\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"283.59375\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"315.380859\" xlink:href=\"#DejaVuSans-30\"/>\n",
       "      <use x=\"379.003906\" xlink:href=\"#DejaVuSans-2c\"/>\n",
       "      <use x=\"410.791016\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"442.578125\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "      <use x=\"501.757812\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"563.037109\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"604.150391\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"631.933594\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"693.212891\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"756.591797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"811.572266\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"873.095703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"904.882812\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_24\">\n",
       "     <path d=\"M 460.615625 53.154687 \n",
       "L 480.615625 53.154687 \n",
       "\" style=\"fill:none;stroke:#2ca02c;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_25\"/>\n",
       "    <g id=\"text_19\">\n",
       "     <!-- mean 3, variance 1 -->\n",
       "     <g transform=\"translate(488.615625 56.654687)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-6d\"/>\n",
       "      <use x=\"97.412109\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"158.935547\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"220.214844\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"283.59375\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"315.380859\" xlink:href=\"#DejaVuSans-33\"/>\n",
       "      <use x=\"379.003906\" xlink:href=\"#DejaVuSans-2c\"/>\n",
       "      <use x=\"410.791016\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"442.578125\" xlink:href=\"#DejaVuSans-76\"/>\n",
       "      <use x=\"501.757812\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"563.037109\" xlink:href=\"#DejaVuSans-72\"/>\n",
       "      <use x=\"604.150391\" xlink:href=\"#DejaVuSans-69\"/>\n",
       "      <use x=\"631.933594\" xlink:href=\"#DejaVuSans-61\"/>\n",
       "      <use x=\"693.212891\" xlink:href=\"#DejaVuSans-6e\"/>\n",
       "      <use x=\"756.591797\" xlink:href=\"#DejaVuSans-63\"/>\n",
       "      <use x=\"811.572266\" xlink:href=\"#DejaVuSans-65\"/>\n",
       "      <use x=\"873.095703\" xlink:href=\"#DejaVuSans-20\"/>\n",
       "      <use x=\"904.882812\" xlink:href=\"#DejaVuSans-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p456b25a518\">\n",
       "   <rect height=\"326.16\" width=\"558\" x=\"36.465625\" y=\"10.7\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display\n",
    "import torch\n",
    "import math\n",
    "\n",
    "x = torch.arange(-7, 7, 0.01)\n",
    "# Mean and variance pairs\n",
    "parameters = [(0,1), (0,2), (3,1)]\n",
    "\n",
    "# Display SVG rather than JPG\n",
    "display.set_matplotlib_formats('svg')\n",
    "plt.figure(figsize=(10, 6))\n",
    "for (mu, sigma) in parameters:\n",
    "    p = (1/math.sqrt(2 * math.pi * sigma**2)) * torch.exp(-(0.5/sigma**2) * (x-mu)**2)\n",
    "    plt.plot(x.numpy(), p.numpy(), label='mean ' + str(mu) + ', variance ' + str(sigma))\n",
    "\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen in the figure above, changing the mean shifts the function, increasing the variance makes it more spread-out with a lower peak. The key assumption in linear regression with least mean squares loss is that the observations actually arise from noisy observations, where noise is added to the data, e.g. as part of the observations process.\n",
    "\n",
    "$$y = \\mathbf{w}^\\top \\mathbf{x} + b + \\epsilon \\text{ where } \\epsilon \\sim \\mathcal{N}(0, \\sigma^2)$$\n",
    "\n",
    "This allows us to write out the *likelihood* of seeing a particular $y$ for a given $\\mathbf{x}$ via\n",
    "\n",
    "$$p(y|\\mathbf{x}) = \\frac{1}{\\sqrt{2 \\pi \\sigma^2}} \\exp\\left(-\\frac{1}{2 \\sigma^2} (y - \\mathbf{w}^\\top \\mathbf{x} - b)^2\\right)$$\n",
    "\n",
    "A good way of finding the most likely values of $b$ and $\\mathbf{w}$ is to maximize the *likelihood* of the entire dataset\n",
    "\n",
    "$$p(Y|X) = \\prod_{i=1}^{n} p(y^{(i)}|\\mathbf{x}^{(i)})$$\n",
    "\n",
    "The notion of maximizing the likelihood of the data subject to the parameters is well known as the *Maximum Likelihood Principle* and its estimators are usually called *Maximum Likelihood Estimators* (MLE). Unfortunately, maximizing the product of many exponential functions is pretty awkward, both in terms of implementation and in terms of writing it out on paper. Instead, a much better way is to minimize the *Negative Log-Likelihood* $-\\log P(Y|X)$. In the above case this works out to be\n",
    "\n",
    "$$-\\log P(Y|X) = \\sum_{i=1}^n \\frac{1}{2} \\log(2 \\pi \\sigma^2) + \\frac{1}{2 \\sigma^2} \\left(y^{(i)} - \\mathbf{w}^\\top \\mathbf{x}^{(i)} - b\\right)^2$$\n",
    "\n",
    "A closer inspection reveals that for the purpose of minimizing $-\\log P(Y|X)$ we can skip the first term since it doesn't depend on $\\mathbf{w}, b$ or even the data. The second term is identical to the objective we initially introduced, but for the multiplicative constant $\\frac{1}{\\sigma^2}$. Again, this can be skipped if we just want to get the most likely solution. It follows that maximum likelihood in a linear model with additive Gaussian noise is equivalent to linear regression with squared loss.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* Key ingredients in a machine learning model are training data, a loss function, an optimization algorithm, and quite obviously, the model itself.\n",
    "* Vectorizing makes everything better (mostly math) and faster (mostly code).\n",
    "* Minimizing an objective function and performing maximum likelihood can mean the same thing.\n",
    "* Linear models are neural networks, too.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Assume that we have some data x1,...xn ∈ R. Our goal is to find a constant b such that ∑ (xi −b)2 is minimized.\n",
    "  * Find the optimal form solution.\n",
    "  * What does this mean in terms of Normal distribution?\n",
    "2. Assume that we want to solve the optimization problem for linear regression with quadratic loss explicitly in closed form. To keep things simple, you can omit the bias b from the problem.\n",
    "  * Rewrite the problem in matrix and vector notation (hint - treat all the data as a single matrix).\n",
    "  * Compute the gradient of the optimization problem with respect to w.\n",
    "  * Find the closed form solution by solving a matrix equation.\n",
    "  * When might this be better than using stochastic gradient descent (i.e. the incremental opti- mization approach that we discussed above)? When will this break (hint - what happens for high-dimensional x, what if many observations are very similar)?.\n",
    "3. Assume that the noise model governing the additive noise ε is the exponential distribution. That is, p(ε) = 1 exp(−|ε|).\n",
    "2\n",
    "  * Write out the negative log-likelihood of the data under the model − log p(Y |X ).\n",
    "  * Can you find a closed form solution?\n",
    "  * Suggest a stochastic gradient descent algorithm to solve this problem. What could possibly go wrong (hint - what happens near the stationary point as we keep on updating the parameters). Can you fix this?\n",
    "4. Compare the runtime of the two methods of adding two vectors using other packages (such as NumPy) or other programming languages (such as MATLAB).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}