from_logistic_regression_to_deep_nets.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:3f33955fbd6bcf994cda85867b1dded295c96c605278a8fb23cf54fa9d1d2d36"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Diclaimer/Context:_ This is the material for a practical given at LSCP (ENS, Paris). Some stuff can be missing from my whiteboard explanations there. Tell me on Github, at https://github.com/SnippyHolloW/DL4H/, or at gabriel.synnaeve@gmail.com or @syhw on Twitter, I'll try and update it accordingly. Pull requests welcomed."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_How to read this (for students):_ go ahead and run things, change things, break things, and fix them!"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "From Logistic Regression to Deep Nets, a Crash Course [Practical 1/2]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Important concepts (I'll explain them on demand during this practical):\n",
      "- [loss functions](http://en.wikipedia.org/wiki/Loss_function) (e.g. log-loss vs hinge loss)\n",
      "- [regularization](http://bit.ly/1A4HuLo) (e.g. L1 vs L2)\n",
      "- [purely stochastic gradient](http://en.wikipedia.org/wiki/Stochastic_gradient_descent) vs. mini-batches averaging\n",
      "- [early stopping](http://en.wikipedia.org/wiki/Early_stopping) (one part of the solution to [overfitting](http://en.wikipedia.org/wiki/Overfitting#Machine_learning))\n",
      "- [momentum-based SGD](http://www.cs.toronto.edu/~fritz/absps/momentum.pdf)\n",
      "- [back-propagation](http://en.wikipedia.org/wiki/Backpropagation) (the chain rule with partial derivatives)\n",
      "\n",
      "Central concepts in machine learning that are not covered here:\n",
      "- ***[Cross-validation](http://scikit-learn.org/stable/modules/cross_validation.html)*** (for (hyper-)parameters optimization)\n",
      "- [Feature selection](http://scikit-learn.org/stable/modules/feature_selection.html)\n",
      "- ...\n",
      "\n",
      "_Practical 2/2_ will be more about how to work with Theano and how to extend my own code  \n",
      "(in particular https://github.com/SnippyHolloW/abnet or https://github.com/bootphon/crossitlearn)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In general:\n",
      "$X$ is for data/samples (n_samples, n_features), $x_i$ is one data point (n_features,),\n",
      "$y$ is for labels (n_samples,), $y_i$ is the label of $x_i$,\n",
      "$W$ is for weights,\n",
      "$\\alpha$ is for regularization,\n",
      "$\\lambda$ is for learning rates"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "\n",
      "0. The Small Digits Dataset"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from __future__ import print_function\n",
      "import numpy as np\n",
      "from sklearn.datasets import load_digits\n",
      "\n",
      "digits = load_digits()\n",
      "import matplotlib.pyplot as plt\n",
      "plt.rcParams['figure.figsize'] = (8, 5)\n",
      "plt.gray()\n",
      "plt.figure(1, figsize=(3, 3))\n",
      "plt.imshow(digits.images[0], cmap=plt.cm.gray_r, interpolation='nearest')\n",
      "data = np.asarray(digits.data, dtype='float32')\n",
      "target = np.asarray(digits.target, dtype='int32')\n",
      "x = data\n",
      "y = target\n",
      "from sklearn import preprocessing\n",
      "x = preprocessing.scale(x)\n",
      "from sklearn import cross_validation\n",
      "x_train, x_test, y_train, y_test = cross_validation.train_test_split(x, y, test_size=0.15, random_state=42)\n",
      "print(x_train.shape)\n",
      "print(y_train.shape)\n",
      "print(x_test.shape)\n",
      "print(y_test.shape)\n",
      "assert (len(set(y_train)) == len(set(y_test)))  # should work with random_state=42 ;-)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(1527, 64)\n",
        "(1527,)\n",
        "(270, 64)\n",
        "(270,)\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAEzCAYAAACPCO4GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADsJJREFUeJzt3VGMXPV5hvH3tY1l7KTBlCoJwRJYClVrVQLbdaIAyknF\nRg5KSS9qJ5aiVFzkqiSoVSNTLqKZO9+FSlUvnAAChZLYTokSCRoSlYNAlSi7sRuwjUmMkWwE2FK2\ntASQcP31YofpsOzsnN2Z2f98+PlJiPH6aHiB5eGc3dkzjggBQCarSg8AgKUiXADSIVwA0iFcANIh\nXADSIVwA0lkz7BPY5vUUAMYiIrzQx4cOV+fJh36OVqulVqs1/JgRGMWWgwcPjmTLgQMHtHv37qGe\nY+/evSPZMjs7q40bNw71HFNTU0PvmJ6e1vbt24d+nn379o3kOe68886hn2fYf67SB++/IXvBZkni\nUhFAQoQLQDoTE66qqkpP6JqkLVu2bCk9oWvdunWlJ0iSrrzyytITum688cbSE7om6fN23FsI1wIm\nacskhevSSy8tPUES4epnkj5vL5pwAUBThAtAOoQLQDqEC0A6hAtAOoQLQDqEC0A6A8Nle6ft523/\n2vZofugNAIawaLhsr5b0j5J2SvpjSXts/9FKDAOAfgadce2Q9JuIeCki3pH0A0lfGv8sAOhvULg+\nIel0z6/PdD4GAMUMuh9Xoxtt9d53p6qqifqZKQA51HWtuq4bHTsoXC9L2tTz602aO+t6j0m5eRmA\nvOaf9LTb7b7HDrpUnJb0SdtX214r6cuSfjKCjQCwbIuecUXEedu3S/qZpNWS7omI4yuyDAD6GHjP\n+Yh4VNKjK7AFABrhlfMA0iFcANIhXADSIVwA0iFcANIhXADSIVwA0iFcANIhXADSIVwA0iFcANIh\nXADSGfhD1lievXsn531FTp06VXpC1+zsbOkJXZdffnnpCV0HDhwoPaFr165dpScMxBkXgHQIF4B0\nCBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQGhsv2vbZfs/3sSgwC\ngEGanHHdJ2nnuIcAQFMDwxURT0qanJsoAbjo8TUuAOmM5A6orVar+7iqKlVVNYqnBXARqetadV03\nOnbk4QKA5Zh/0tNut/sey6UigHSavBziIUn/Lula26dt3zb+WQDQ38BLxYjYsxJDAKApLhUBpEO4\nAKRDuACkQ7gApEO4AKRDuACkQ7gApEO4AKRDuACkQ7gApEO4AKRDuACkM5L7cU2KmZmZ0hO6Tp06\nVXpC18mTJ0tP6Nq8eXPpCV1TU1OlJ3RN0ufurl27Sk8YiDMuAOkQLgDpEC4A6RAuAOkQLgDpEC4A\n6RAuAOkQLgDpEC4A6RAuAOkQLgDpEC4A6QwMl+1Nth+3fdT2c7a/uRLDAKCfJneHeEfS30TEEdsf\nkjRj++cRcXzM2wBgQQPPuCLi1Yg40nn8hqTjkq4c9zAA6GdJX+OyfbWk6yU9PY4xANBE43B1LhMP\nSbqjc+YFAEU0ugOq7Usk/UjS9yPix/N/v9VqdR9XVaWqqkY0D8DFoq5r1XXd6NiB4bJtSfdIOhYR\ndy90TG+4AGA55p/0tNvtvsc2uVS8QdJXJX3O9uHOHzuHHQkAyzXwjCsinhIvVAUwQQgSgHQIF4B0\nCBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQa3Ugwi9nZ2dITurZu\n3Vp6QtfmzZtLT5hI27ZtKz0By8QZF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQI\nF4B0CBeAdAgXgHQGhsv2OttP2z5i+znbrRXYBQB9Dbw7RES8bftzEfGm7TWSnrL9aEQ8vQL7AOB9\nGl0qRsSbnYdrJV0i6cLYFgHAAI3CZXuV7SOSXpP0WEQ8M95ZANBf0zOuCxFxnaSrJH3K9pbxzgKA\n/pZ0B9SIeN3245J2Sjr67sdbrVb3mKqqVFXViOYBuFjUda26rhsdOzBctq+QdD4i/sv2pZKmJO3r\nPaY3XACwHPNPetrtdt9jm5xxfVzS/bZXa+7S8ocR8ciQGwFg2Zq8HOJZSZPzzg8ALnq8ch5AOoQL\nQDqEC0A6hAtAOoQLQDqEC0A6hAtAOoQLQDqEC0A6hAtAOoQLQDqEC0A6hAtAOku6keCkm52dLT2h\na2pqqvQEDDBJny8bN24sPSEVzrgApEO4AKRDuACkQ7gApEO4AKRDuACkQ7gApEO4AKRDuACkQ7gA\npEO4AKRDuACk0yhctlfbPmz7p+MeBACDND3jukPSMUkxxi0A0MjAcNm+StItkr4nyWNfBAADNDnj\n+o6kb0m6MOYtANDIouGy/UVJZyPisDjbAjAhBt0B9TOSbrV9i6R1kn7P9gMR8bXeg1qtVvdxVVWq\nqmrEMwF80NV1rbquGx27aLgi4i5Jd0mS7c9K+rv50ZLeGy4AWI75Jz3tdrvvsUt9HRffVQRQXOM3\ny4iIJyQ9McYtANAIr5wHkA7hApAO4QKQDuECkA7hApAO4QKQDuECkA7hApAO4QKQDuECkA7hApAO\n4QKQDuECkE7ju0NksHHjxtITumZmZkpPmEizs7OlJ3RNT0+XntC1e/fu0hNS4YwLQDqEC0A6hAtA\nOoQLQDqEC0A6hAtAOoQLQDqEC0A6hAtAOoQLQDqEC0A6hAtAOo1+yNr2S5L+W9L/SnonInaMcxQA\nLKbp3SFCUhURvx3nGABoYimXih7bCgBYgqbhCkm/sD1t++vjHAQAgzS9VLwhIl6x/QeSfm77+Yh4\ncpzDAKCfRuGKiFc6fz5n+2FJOyR1w9VqtbrHVlWlqqpGOhLAB19d16rrutGxA8Nle72k1RHxP7Y3\nSPq8pHbvMb3hAoDlmH/S0263+x7b5Izro5Ietv3u8Q9GxGPDTQSA5RsYrog4Jem6FdgCAI3wynkA\n6RAuAOkQLgDpEC4A6RAuAOkQLgDpEC4A6RAuAOkQLgDpEC4A6RAuAOkQLgDpEC4A6TgihnsCO4Z9\njlF58cUXS0/o2rZtW+kJXfv37y89oevgwYOlJ3SdPHmy9ISumZmZ0hMmjm1FxILvdcEZF4B0CBeA\ndAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQIF4B0CBeAdAgXgHQGhsv2ZbYP2T5u+5jtT6/E\nMADoZ02DY/5B0iMR8Ze210jaMOZNALCoRcNl+yOSboqIv5KkiDgv6fWVGAYA/Qy6VLxG0jnb99n+\npe3v2l6/EsMAoJ9Bl4prJG2VdHtEPGP7bkl3Svp270GtVqv7uKoqVVU12pUAPvDqulZd142OHRSu\nM5LORMQznV8f0ly43qM3XACwHPNPetrtdt9jF71UjIhXJZ22fW3nQzdLOjr8RABYvibfVfyGpAdt\nr5V0UtJt450EAIsbGK6I+E9Jf7oCWwCgEV45DyAdwgUgHcIFIB3CBSAdwgUgHcIFIB3CBSAdwgUg\nHcIFIB3CBSAdwgUgHcIFIB3CBSAdR8RwT2DHsM/xQbR///7SE7r27dtXekLX9u3bS0/oOnDgQOkJ\nWIRtRYQX+j3OuACkQ7gApEO4AKRDuACkQ7gApEO4AKRDuACkQ7gApEO4AKRDuACkQ7gApEO4AKQz\nMFy2/9D24Z4/Xrf9zZUYBwALWTPogIg4Iel6SbK9StLLkh4e8y4A6Gupl4o3SzoZEafHMQYAmlhq\nuL4i6Z/HMQQAmhp4qfgu22sl/bmkvfN/r9VqdR9XVaWqqkYwDcDFpK5r1XXd6NjG4ZL0BUkzEXFu\n/m/0hgsAlmP+SU+73e577FIuFfdIemjZqwBgRBqFy/YGzX1h/l/GOwcABmt0qRgRv5N0xZi3AEAj\nvHIeQDqEC0A6hAtAOoQLQDqEC0A6hAtAOoQLQDqEC0A6hAtAOhMTrqY/Fb4SJmnLiRMnSk/oeuut\nt0pPkCSdPXu29ISuSfpcuZi2EK4FTNKWF154ofSErrfffrv0BEnSuXPvu0FJMZP0uXIxbZmYcAFA\nU4QLQDqOiOGewB7uCQCgj4jwQh8fOlwAsNK4VASQDuECkE7xcNneaft527+2/b53EFrhLffafs32\ns4V3bLL9uO2jtp8r+c7httfZftr2kc6WVqktPZtWd95V/aeFd7xk+1edLf9ReMtltg/ZPm77mO1P\nF9qxIu98X/RrXLZXSzqhufvZvyzpGUl7IuJ4oT03SXpD0gMR8SclNnR2fEzSxyLiiO0PSZqR9BcF\n/7msj4g3ba+R9JSkOyLi6RJbOnv+VtI2SR+OiFsL7jglaVtE/LbUhp4t90t6IiLu7fx72hARrxfe\ntEpz/13vGPWbSJc+49oh6TcR8VJEvCPpB5K+VGpMRDwpabbUX79nx6sRcaTz+A1JxyVdWXDPm52H\nayVdIulCqS22r5J0i6TvSVrwO04rrPgG2x+RdFNE3CtJEXG+dLQ6btaY3vm+dLg+Ian3b+pM52Po\nsH21pOsllTzDWWX7iKTXJD0WEc+U2iLpO5K+pYLx7BGSfmF72vbXC+64RtI52/fZ/qXt79peX3DP\nu76iMb3zfelw8VqMRXQuEw9p7tLsjVI7IuJCRFwn6SpJn7K9pcQO21+UdDYiDmsCznQk3RAR12vu\nzZL/uvOlhhLWSNoq6Z8iYquk30m6s9AWSZL//53vD47j+UuH62VJm3p+vUlzZ10XPduXSPqRpO9H\nxI9L75GkzuXH45J2FprwGUm3dr629JCkP7P9QKEtiohXOn8+J+lhzX3po4Qzks70nAkf0lzISvqC\ntPA7349C6XBNS/qk7as7hf6ypJ8U3lScbUu6R9KxiLi78JYrbF/WeXyppCnNfc1txUXEXRGxKSKu\n0dxlyL9FxNdKbLG93vaHO483SPq8pCLfjY6IVyWdtn1t50M3SzpaYkuPPZr7n8tYNHpD2HGJiPO2\nb5f0M0mrJd1T6jtnkmT7IUmflfT7tk9L+nZE3Fdgyg2SvirpV7YPdz729xHxrwW2fFzS/Z3vAK+S\n9MOIeKTAjoWU/FLDRyU9PPf/GK2R9GBEPFZwzzckPdg5ATgp6bZSQzohv1nS2L7ux4/8AEin9KUi\nACwZ4QKQDuECkA7hApAO4QKQDuECkA7hApAO4QKQzv8BhGdfWc6sVBEAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11256d4d0>"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "1. Dumb-Simple Logistic Regression"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "A minimal amount of theory"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "[Logistic regression](http://en.wikipedia.org/wiki/Logistic_regression) is linear regression ($y = X^T\\cdot W + \\epsilon$) to which we apply a logistic function ($\\frac{1}{1+e^{-z}}$). Why? To transform the $y$ into something more categorical (binary if only two classes). In particular, when we are doing logistic regression, we are making the assumption that the $\\log$ of the odds (of e.g. a $X \\sim Bernouilli(P)$) is a linear regression: \n",
      "\n",
      "$$\\log ( \\frac{P}{1-P} ) = X^T \\cdot W  \\Longleftrightarrow P = \\frac{\\exp(X^T \\cdot W)}{1 + \\exp(X^T \\cdot W)}$$\n",
      "\n",
      "We want to _maximize_ the likelihood of our model (actually, if we have a prior on the parameters, we want the maximum a posteriori, an in this case we need to add a term to the negative log-likelihood below, if that prior is Gaussian, that's like doing L2 regularization).\n",
      "\n",
      "If we think in term of loss function, we want to _minimize_ the _negative_ log-likelihood ($\\log$ is monotonically increasing). I'm writing it in the general case where $y$ is not just binary:\n",
      "\n",
      "$$NLL(W) = -\\frac{1}{n} \\sum_{i=1}^n \\sum_{j=1}^m \\mathbb{1}_{y_i=j} \\log P(y_i=j|x_i; W)$$\n",
      "with\n",
      "$$P(y_i=j|x_i; W) = softmax(W, x_i)[j]$$\n",
      "and\n",
      "$$softmax(W, x)[j] = \\frac{\\exp(x \\cdot W_j)}{\\sum_{k} \\exp(x \\cdot W_k)}$$\n",
      "We can derive the gradient:\n",
      "$$\\nabla_{W_j} = -\\frac{1}{n} \\sum_{i=1}^n x_i (\\mathbb{1}_{y_i=j} - P(y_i=j|x_i, W))$$"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "A not-so-minimal amount of code"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "By no mean this is minimal nor \"smart\" nor efficient, this aims at being _easy_ to understand, it fills only a pedagogical role. That said, it can always be improved. All pull requests making it easier to grasp are welcomed!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def softmax(W, x):\n",
      "    \"\"\" P(y=j | X) = exp(<X,W_j>) / (sum_k(exp(<X,W_k>)))\n",
      "    W: (n_features, n_outputs)\n",
      "    X: (n_features,)\n",
      "    \"\"\"\n",
      "    tmp = np.exp(np.dot(x, W))\n",
      "    return tmp / np.sum(tmp)\n",
      "\n",
      "from scipy.optimize.optimize import check_grad\n",
      "from scipy.optimize import fmin_l_bfgs_b\n",
      "\n",
      "class LogisticRegression:\n",
      "    def __init__(self):\n",
      "        self.name = \"Logistic Regression\"\n",
      "        \n",
      "    def loss(self, W):\n",
      "        \"\"\" Negative log-likelihood as described above \"\"\"\n",
      "        l = 0.\n",
      "        tmp_W = W.reshape((self.x.shape[1], self.n_classes))\n",
      "        for i, xi in enumerate(self.x):\n",
      "            p_i = softmax(tmp_W, xi)\n",
      "            l += np.log(p_i[self.y[i]])\n",
      "        return - l / self.x.shape[0]\n",
      "                   \n",
      "    def loss_derivative(self, W):\n",
      "        \"\"\" flattened (1-D) gradient vector \"\"\"\n",
      "        d = np.zeros((self.x.shape[1], self.n_classes)) # that's the Jacobian\n",
      "        tmp_W = W.reshape((self.x.shape[1], self.n_classes))\n",
      "        for i, xi in enumerate(self.x):\n",
      "            p_i = softmax(tmp_W, xi)\n",
      "            for j, v in enumerate(p_i):\n",
      "                if j == self.y[i]:  # because we don't have a \"one-hot\" encoding\n",
      "                    d[:,j] += (1 - p_i[j]) * xi\n",
      "                else:\n",
      "                    d[:,j] += - p_i[j] * xi\n",
      "        d = d.reshape(W.shape) # and we flatten it\n",
      "        return - d / self.x.shape[0]\n",
      "    \n",
      "    def train(self, x_train, y_train, debug=False):\n",
      "        self.x = np.hstack((x_train, np.ones((x_train.shape[0], 1))))\n",
      "        self.y = y_train\n",
      "        self.n_classes = len(set(y_train))\n",
      "        self.W = np.ones((x_train.shape[1] + 1, self.n_classes))\n",
      "        print(\"W shape:\", self.W.shape)\n",
      "        print(\"x shape:\", self.x.shape)\n",
      "        print(\"loss before training:\", self.loss(self.W))\n",
      "        print(\"prediction on x[0] before training:\", softmax(self.W, self.x[0]))\n",
      "        if debug == True:\n",
      "            tmp = self.loss_derivative(self.W.reshape((self.W.shape[0]*self.W.shape[1],)))\n",
      "            print(\"loss derivative shape:\", tmp.shape)\n",
      "            print(\"loss derivative value:\", tmp)\n",
      "            print(\"checking the gradient with finite differences, show be low (something E-6):\",\n",
      "                  check_grad(self.loss, self.loss_derivative, self.W.reshape((self.W.shape[0]*self.W.shape[1],))))\n",
      "            # that's nothing more than checking that f'(x) ~= (f(x+h)-f(x))/h with h->0\n",
      "        tmp = fmin_l_bfgs_b(self.loss, self.W.reshape((self.W.shape[0]*self.W.shape[1],)), fprime=self.loss_derivative)\n",
      "        self.W = tmp[0].reshape((self.x.shape[1], self.n_classes))\n",
      "        print(\"loss after training:\", self.loss(self.W))\n",
      "        print(\"prediction on x[0] after training:\", softmax(self.W, self.x[0]))\n",
      "        print(\"true x[0] label:\", self.y[0])\n",
      "        \n",
      "    def score(self, x_test, y_test):\n",
      "        x = np.hstack((x_test, np.ones((x_test.shape[0], 1))))\n",
      "        y = y_test\n",
      "        c = 0\n",
      "        for i, xi in enumerate(x):\n",
      "            a = np.argmax(softmax(self.W, xi))\n",
      "            if a == y[i]:\n",
      "                c += 1\n",
      "        return c * 1. / y.shape[0]\n",
      "\n",
      "lr = LogisticRegression()\n",
      "lr.train(x_train, y_train)\n",
      "plt.imshow(lr.W.transpose(), interpolation='none')\n",
      "plt.show()\n",
      "print(\"weights for 0:\")\n",
      "plt.imshow(lr.W.transpose()[0][:64].reshape((8,8)), interpolation='none')\n",
      "plt.show()\n",
      "print(\"weights for 5:\")\n",
      "plt.imshow(lr.W.transpose()[5][:64].reshape((8,8)), interpolation='none')\n",
      "plt.show()\n",
      "print(\"weights for 8 minus weights for 3:\")\n",
      "plt.imshow(lr.W.transpose()[8][:64].reshape((8,8)) - lr.W.transpose()[3][:64].reshape((8,8)), interpolation='none')\n",
      "plt.show()\n",
      "print(\"training accuracy:\", lr.score(x_train, y_train))\n",
      "print(\"test accuracy:\", lr.score(x_test, y_test))\n",
      "print(\"We should overfit, i.e. test score << train score, you can add an L2 regularization\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "W shape: (65, 10)\n",
        "x shape: (1527, 65)\n",
        "loss before training: 2.30258509299\n",
        "prediction on x[0] before training: [ 0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1]\n",
        "loss after training:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 1.84669284921e-05\n",
        "prediction on x[0] after training: [  1.08822355e-31   9.98300553e-01   1.77810355e-28   3.23091472e-47\n",
        "   9.64927736e-20   2.79499404e-36   1.13639647e-16   6.26997256e-36\n",
        "   1.69944705e-03   1.81750303e-29]\n",
        "true x[0] label: 1\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAdYAAABiCAYAAADtNvKNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFaBJREFUeJzt3VlwFNUXBvAvbAIhCYIhgYTNkD0hGQggqH+LTREFRGKJ\noljI8oBW4fJgWWVR5cYi5YLog+VWuBRY+iAUQhQ0aCAsQoIgRNkSGSJBUYOBgEDo/wOVISFzvkBs\nydLf72mmz/Rw53ZPX9Jzzz0hjuM4EBEREVe0auwGiIiItCQaWEVERFykgVVERMRFGlhFRERcpIFV\nRETERRpYRUREXFTvwJqTk4OkpCTEx8dj4cKFV6NNIiIizVYIy2OtqqpCYmIi1q1bh5iYGAwaNAjL\nli1DcnLyxTcICbkqDRUREWlKrOGzDdtp69at6NevH/r06QMAmDx5MlasWFFrYAWAZ599NvA4NzcX\nw4cPB8AHXbYuRWhoqBn77bffzNgff/xhxsLCwsxYRERE0O2tWtl/0P/9999mrG3btrWe5+Xl4eab\nbwYAxMfHm/uVl5ebsaqqKjN25MgRM2b1ZZs29qFnn5sdm8rKSjOWl5cXeLx//37069cv8Hzy5Mnm\nfkePHg26/Z9//jH3YaKjo80YOye/++47M7Zp0yYzNmvWLDPWrVu3wOMVK1ZgwoQJAIBff/21QW3s\n0aOHGWPnz9dff23G/H6/Gbv99tuDbmfftYqKCjN2qZrXklOnTpmvGzJkiBlbs2aNGWvdurUZu/76\n64Nuv5LvfU3suB0+fNiMpaenBx6vXr0aY8eODTxn18LU1NSg2/fs2WPu89dff5mxrl27mjF2vWBj\nADumVv8DwO7duwOPN23ahKFDhwLg14TOnTubsWuuucaM/f7770G3h4eHY/78+eZ+9FZwaWkpevbs\nGXgeGxuL0tJStouIiIin0YFVt3lFRESuDL0VHBMTU+tWkN/vR2xsbJ3X5ebmBh6zP6u9qlevXo3d\nhCalS5cujd2EJicxMbGxm9DkVP8EJRewn5G8Kth49F8pLS0N3LGtb5yjA2tWVhb27duHkpIS9OjR\nA5988gmWLVtW53XVv4NIcL17927sJjQpGljrSkpKauwmNDl9+/Zt7CY0KRpY66r5U+V/LSYmBjEx\nMQAu/Ma6YcMG87V0YG3Tpg3eeOMN3HbbbaiqqsL06dPrTFwSERGRi+jAClyY+WfN/qsPm4l47Ngx\nM5aRkWHG3n33XTO2atUqM/b555+bMWs2HJvJ19DZdWzGJ+tn9u+xWYXW7LrIyEhzH/YXZVlZmRmb\nM2eOGWOfbcmSJWYsLi4u6Pbz58+b+7BZimzmOFM9qzuYGTNmmDF2Lmzfvj3o9o4dO5r7sHOSHZua\ns7AvZfUxwO9GWedQfn6+uQ+bncmOG/vesNhDDz1kxthfHLt27Qq6/cCBA+Y+7Bxhx5Sdy8XFxWbM\nmvkL8OurhX22wYMHmzHWxnPnzpkxNhG2Q4cOZswaH6Kiosx9WOYEmx1eUlISdDubAQ5o5SURERFX\naWAVERFxkQZWERERF2lgFRERcZEGVhERERfVO7D6/X4MHz4cqampSEtLw+uvv3412iUiItIs0eo2\nwIUp/GVlZcjMzMSJEycwcOBAfP7554F81pCQkFqL8NfEFuNmCyazJrE0EZZ2cubMGTNmTe1m0+Bv\nuOEGM7Zz504ztm/fPjN21113mTGWwsA+W6dOnYJuZ33cvn17M/bzzz+bsdOnT5uxW2+91YwVFhaa\nsf79+wfdzqbPszQvlhrw5ZdfmjG2vOeYMWPMWHVCeTA1F+GviRVjYMeNLRDPigikpaWZMbawv5UC\nxlJq2GpKLN3G6iuAX0tqLlp/qZkzZ5oxq0gFS3FhfcW+G+3atTNj7Bp69913mzHrGsSunzt27DBj\nVtoJwAt6sGtJeHi4GWP9ZV3T2EI8LO3t+PHjZmzv3r1Bt4eFhWHu3Lnm97Hev1ijo6ORmZkJ4MIH\nSk5OpnljIiIiXnZFv7GWlJSgsLCQlmkSERHxsnpXXqp24sQJZGdnY/HixXX+FK+5CH+fPn20xqeI\niLQohw4dChSlYbfvgcscWM+ePYtJkybhgQceCPo7oBbhFxGRlqxXr16BSmVhYWFYv369+dp6bwU7\njoPp06cjJSUFjz32mGuNFBERaYnqHVg3btyIjz76CLm5ufD5fPD5fMjJybkabRMREWl26r0VfNNN\nN9G0E4ZNn2eVDVjRZ1a3klVSYFOqrXSVU6dOmfvs3r3bjFkpIgCvisDayKbJs5qEVpUOlrbBYixt\ngx3vkSNHmjFWVcMqKGxNuQd4agxLTWLtHzVqlBljqVeLFi0yY1aR5ob+tMLSTqwqR/XF2LllVQVp\n6PWC7bd//34zxtJE5s2bZ8bmzp1rxqzUDXb+sLQxVkGFHTcWY+0fMGBA0O0sbYZdW1maIPvesEpZ\n7HizmDV2sMpV3bt3N2PsWl596/dSoaGh5j6AVl4SERFxlQZWERERF2lgFRERcZEGVhERERdpYBUR\nEXHRZQ2sVVVV8Pl8GDdu3H/dHhERkWat3uo2APDKK69g+/btqKiowMqVK2u/AaluU1FRYb4nS2XZ\nvn27GevatasZY9Vt0tPTzdjgwYODbmcVEa677jozxirAsNhvv/1mxthUeDal3aquwiqosKnpViUg\ngKdQsUoWERERZswq+MDOg7Nnz5oxlqbDKq+wyjcsJYVVnLEq/mzdutXch52TLG2GFc5gKRjDhg0z\nY9alg1VkYZWf2HGbNGmSGVu6dKkZYxVgWOrMtm3bgm5nqSWsIgv7bNVFToL59NNPzRhL67PS1Nas\nWWPuwyoIsesPS405ceKEGWMpSD/99JMZs77Dx44dM/dJSUkxY9dee60Zs1J4wsPD8fzzzze8us3h\nw4exevVqzJgxg+Y3ioiIyGUMrI8//jgWLVpE/6cmIiIiF9CVl1atWoVu3brB5/PRBYdV3UZERFqy\nw4cPB35utG61V6MDa35+PlauXInVq1fj9OnT+PvvvzF16lR88MEHtV6n6jYiItKSxcbGBpYhDQ8P\n58u7sjeaN28e/H4/iouLsXz5cowYMaLOoCoiIiIXXXahc4Avbh6MtZA1YM+IBICXX37ZjHXu3NmM\nPfXUU2aMFab96KOPgm6Piooy92ELfw8cONCM/e9//zNjGzZsMGNs4hiboWwt4s1uZfzyyy9mzFrc\nG+DHhs3cZLMprc89efLkK94H4DN427dvb8asxbgB/rnZ4vFvvfVW0O1sdjJz8OBBMxYdHW3G2MxN\nNkPZmoXM/i2GFah45513zBj73GwR/ri4ODNmXbvYrPLKykoz1qaNfall1xLWJ3v27DFj1kxjdv1h\ni9GzAidsdm9CQoIZszIWAN7O77//Puh2NhudzTJOS0szY9YYwM4D4AoG1ltuuQW33HLL5b5cRETE\nkzTVV0RExEUaWEVERFykgVVERMRFGlhFRERcVO/AWl5ejuzsbCQnJyMlJQWbN2++Gu0SERFpluqd\nFTxnzhyMHTsWn332Gc6dO4eTJ09e9puzBcNffPFFM8bSF9iU6i+++MKMsenR1qLzVVVV5j5sivm3\n335rxthC6SxtgKVgsDQRK5WFfTZr4WkAOH78uBlji5qz9rP3tAoksPQplhpw6NAhM8YWo2fHOy8v\nz4yxdAmrnSytjaUSscXcWdELNtufnQtW0Qh2HjQU68e5c+eaMfadslLRALugBEtta+g5efToUTM2\nYsQIM8YKURQWFgbdztLlqhc/CIYthsBW2isrKzNjP/zwgxljx8a6ho4fP97ch6U0sXPc7/cH3c4K\nhwD1DKzHjx9HXl5eIAexTZs29b6hiIiIl9FbwcXFxYiMjMS0adMwYMAAzJw5kyZBi4iIeB0dWM+d\nO4eCggLMnj0bBQUFCA0NxYIFC65W20RERJodeiu4etHhQYMGAQCys7ODDqyqbiMiIi1ZaWkpfv31\nVwD/srpNdHQ0evbsib179yIhIQHr1q1Dampqndepuo2IiLRkMTExgfWNIyIisHHjRvO19c4KXrJk\nCaZMmYIzZ84gLi4O77//vnstFRERaWFCHDZ//3LeICQEzz77bNBYly5dzP3YlHy2H6uEwqZ2s6nk\nv//+e9DtZ8+eNfdhtwLYfqGhoWbshhtuMGNr165t0L8XHh4edDtLvyguLjZj1YV+g4mMjDRjLN2G\npVdZVWVYZQ9WSaSoqMiMWWlXAK/8YaUEAUBycrIZsyYCsn/r2muvNWOs/SylgKWCsAokVoz1P6vg\nxNKM2PeGnf+smtGRI0fMmFXxh6X9sBQ2hl0LKyoqzFhWVpYZsyonscs9O4/Zecfes1UrexoPO6as\nGo2V3sYq6bA0R3b9sb6jYWFheOaZZ8zPrpWXREREXKSBVURExEUaWEVERFykgVVERMRFGlhFRERc\nVO/AOn/+fKSmpiI9PR33338/XRxZRETE62i6TUlJCUaMGIGioiJcc801uPfeezF27Fg89NBDF9+A\npNtY1S8AO40CALZt22bG0tPTzdgvv/xixn7++WczVr2y1KWio6PNfdiayWw/K7UH4FPMp0yZYsZY\nCoz1nmz6PKuaERcXZ8ZYmghz4403mjHrs23ZssXch6V7sOn/LDZ79mwz9sILL5gx1s9WpSiWhsDS\nPfbv32/GWFWWqVOnmrH8/HwztmzZsqDb9+7da+7zyCOPmDGWNsOODavOw9KyWJqF1RYrDQdAYFWe\nYNhxY+1gaWrsvOvRo0fQ7RMmTDD3YZXDrGo5AK+uxaqhsWPDKjVlZGQE3b5p0yZzH5ZeyCqfzZkz\nJ+j21q1bIzY21ky3oQtEhIeHo23btqisrETr1q1RWVlJ89pERES8jt4K7tKlC5588kn06tULPXr0\nQOfOnTFq1Kir1TYREZFmh/7FeuDAAbz22msoKSlBREQE7rnnHnz88cd1bktqEX4REWnJNm3ahM2b\nNwPgq4QB9Qys27Ztw7BhwwK/ud19993Iz8+vM7BqEX4REWnJhg4diqFDhwK48Bvrq6++ar6W3gpO\nSkrC5s2bcerUKTiOg3Xr1iElJcXd1oqIiLQgdGDNyMjA1KlTkZWVhf79+wMAZs2adVUaJiIi0hz9\np9Vt2PR/ltKRnZ1txkpLS80YqwATHx9vxqz0jOPHj5v7sKoZDa1yUVBQYMbY79YdO3Y0Y+fPnw+6\nvby8/PIbVkPv3r3NGKuSEhERYcbuueceM/bcc88F3c76mKXbsNQAVl2IpT2wVAR2bHbt2hV0Ozs2\n7LOx85+lNCUmJpqxFStWmDGrYhTLdWffG/a71dGjR83Yl19+acYOHDhgxsaMGWPGrD5h7WCVjNhl\n1kqNAYCDBw+asW+//daMWd9T9h0dOHCgGWMpQaxPWAoh+/dYOqN1LWcplex7yFKorO9bp06d8MQT\nT6i6jYiIyNWggVVERMRFrg+srEi2V7FbFF7U0BWaWjL2s4lX6VpSG1tZzqvY6k2NyfWBtaSkxO23\nbPbYEm9e1FS/DI2J/RboVRpYazt06FBjN6HJaarXEt0KFhERcRFdIOJyde/ePfC4U6dOgecNnSXK\nFiFni5qzhf2joqLMWOvWrYNuZzPJ2KzmS2fiduzYkS6CXq2h7WczXa1Zax06dKi3PcGwz81m6tZc\n6Lp9+/b0ONZU89y63H+LzZxt166dGWOLcbNF1Nm5zI5Nt27dAo9DQ0MDz9mxYZ8tISHBjLE1vtkx\nvf76682YdU6eOXPG3IcVqLhUp06dAq9nxy01NdWMsWPTr18/M2bNeGbH5ko+W02s/0+fPh14HBYW\nVmsGMZvNbbWFfW/YNYYt0M8KJNRs/6Vqnv9Xsl/Na0fNa4l1rah+nYV9toaMDYBL6TYiIiJe06Dq\nNv/mjUVERLxIv7GKiIi4SAOriIiIi1wbWHNycpCUlIT4+HgsXLjQrbdtVh5++GFERUUhPT09sO3P\nP//E6NGjkZCQgFtvvbXBSwk2V36/H8OHD0dqairS0tLw+uuvA/Buv5w+fRpDhgxBZmYmUlJS8PTT\nTwPwbn/UVFVVBZ/Ph3HjxgFQn/Tp0wf9+/eHz+fD4MGDAXi7T8rLy5GdnY3k5GSkpKRgy5YtTbY/\nXBlYq6qq8OijjyInJwd79uzBsmXLUFRU5MZbNyvTpk1DTk5OrW0LFizA6NGjsXfvXowcORILFixo\npNY1jrZt2+LVV1/F7t27sXnzZrz55psoKirybL+0b98eubm52LFjB3bu3Inc3Fxs2LDBs/1R0+LF\ni5GSkhKYEOn1PgkJCcH69etRWFiIrVu3AvB2n8yZMwdjx45FUVERdu7ciaSkpKbbH44L8vPzndtu\nuy3wfP78+c78+fPdeOtmp7i42ElLSws8T0xMdMrKyhzHcZwjR444iYmJjdW0JmHChAnO2rVr1S+O\n45w8edLJyspyfvzxR8/3h9/vd0aOHOl88803zp133uk4jr47ffr0cY4dO1Zrm1f7pLy83Onbt2+d\n7U21P1z5i7W0tLRW1YrY2FhahcZLjh49GsgPi4qKopUgWrqSkhIUFhZiyJAhnu6X8+fPIzMzE1FR\nUYHb5F7uDwB4/PHHsWjRolo5kV7vk5CQEIwaNQpZWVl4++23AXi3T4qLixEZGYlp06ZhwIABmDlz\nJk6ePNlk+8OVgVW5rJcnJCTEs3114sQJTJo0CYsXL66TkO21fmnVqhV27NiBw4cP47vvvkNubm6t\nuNf6Y9WqVejWrRt8Pp+Zvue1PgGAjRs3orCwEGvWrMGbb76JvLy8WnEv9cm5c+dQUFCA2bNno6Cg\nAKGhoXVu+zal/nBlYI2JiYHf7w889/v95solXhMVFYWysjIAwJEjR+hqIy3V2bNnMWnSJDz44IO4\n6667AKhfgAurAt1xxx3Yvn27p/sjPz8fK1euRN++fXHffffhm2++wYMPPujpPgEuriQUGRmJiRMn\nYuvWrZ7tk9jYWMTGxmLQoEEALtTsLigoQHR0dJPsD1cG1qysLOzbtw8lJSU4c+YMPvnkE4wfP96N\nt272xo8fj6VLlwIAli5dGhhYvMJxHEyfPh0pKSl47LHHAtu92i/Hjh0LzFw8deoU1q5dC5/P59n+\nAIB58+bB7/ejuLgYy5cvx4gRI/Dhhx96uk8qKytRUVEBADh58iS++uorpKene7ZPoqOj0bNnz0BB\nk3Xr1iE1NRXjxo1rmv3h1o+1q1evdhISEpy4uDhn3rx5br1tszJ58mSne/fuTtu2bZ3Y2Fjnvffe\nc/744w9n5MiRTnx8vDN69Gjnr7/+auxmXlV5eXlOSEiIk5GR4WRmZjqZmZnOmjVrPNsvO3fudHw+\nn5ORkeGkp6c7L730kuM4jmf741Lr1693xo0b5ziOt/vk4MGDTkZGhpORkeGkpqYGrqle7pMdO3Y4\nWVlZTv/+/Z2JEyc65eXlTbY//vVawSIiInKRVl4SERFxkQZWERERF2lgFRERcZEGVhERERdpYBUR\nEXGRBlYREREXaWAVERFxkQZWERERF/0fbUaFRy9a7DsAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x10f5f30d0>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "weights for 0:\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAEzCAYAAACPCO4GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEmJJREFUeJzt3X9s1PUdx/HXAZ382qq4QhnXrQTE3n0PeydlTTaIY1M3\nNTV2VlvUsQH6z7Jk+Kd/mewPgnPLgnN/mUFmtsCBxvgjpVEyGBXjOteitjBhW9laMMtYggKFtL1+\n9wfzuyH07lvue/e593g+EpNWvvnyEvWZ7/3o92K+7/sCAEOmuR4AAFNFuACYQ7gAmEO4AJhDuACY\nQ7gAmDOj2BPEYrEodgDAZSZ7t1bR4ZKkF154oehzZLNZtbe3F3WO6dOnF71Dknbu3KmOjo6izpHL\n5SLZEsWfy+c///lItmzfvl3r168v6hwffPBB0TteffVVtbS0FH2ev/zlL0Wf480339SqVauKPk9t\nbW3R5+jq6tK3vvWtos/zhS98oehz7N69Ww888EBR58j3/yAPFQGYQ7gAmFMx4fI8z/WEQCqVcj0h\nUEl/Lul02vUESdKyZctcTwh88YtfdD0hsHTpUtcTAslksqTnjxX7s4qxWCyS57iiENVzXFGI6jmu\nKET1HFcUoniOKypRPMcVlSie44pKFM9xRaGjo2PSJ+cr5ooLAMIiXADMIVwAzCFcAMwhXADMIVwA\nzCFcAMwpGK6uri41NDTopptu0lNPPVWOTQCQV95w5XI5/eAHP1BXV5cOHz6sHTt26MiRI+XaBgBX\nlDdcPT09Wrp0qerr61VVVaWOjg69/PLL5doGAFeUN1wnTpxQXV1d8H08HteJEydKPgoA8sl7P66w\nNwnMZrPB157nVdQPKQOwYWBgQIcPHw51bN5wLVq0SENDQ8H3Q0NDisfjlx1X7I3uAMDzvEvuhvLi\niy9Oemzeh4pNTU06duyYjh8/rtHRUWWzWd17773RLQWAq5D3imvGjBl69tln9c1vflO5XE4bN25U\nIpEo1zYAuKKC95y/6667dNddd5VjCwCEwjvnAZhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJhDuACY\nQ7gAmEO4AJhDuACYQ7gAmEO4AJhT8O4QYYS9U2qpDQ8Pu54QmDVrlusJgRUrVrieEJg+fbrrCYGl\nS5e6nhB4//33XU8IjI2NuZ5QEFdcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFc\nAMwhXADMIVwAzCkYrg0bNmjBggVavnx5OfYAQEEFw7V+/Xp1dXWVYwsAhFIwXKtXr9YNN9xQji0A\nEArPcQEwJ5IbCWaz2eBrz/OUSqWiOC2Aa8iRI0d05MiRUMdGEq729vYoTgPgGpZIJJRIJILvX3rp\npUmP5aEiAHMKhmvt2rX6yle+oqNHj6qurk7bt28vxy4AmFTBh4o7duwoxw4ACI2HigDMIVwAzCFc\nAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMyJ5H5cExMTUZymaPF43PWEwLx5\n81xPCLz77ruuJwSeeeYZ1xMC27Ztcz0hMDY25npC4MyZM64nFMQVFwBzCBcAcwgXAHMIFwBzCBcA\ncwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMKhmtoaEhr1qyR53lKpVIV9UOyAK5NBe8OUVVV\npZ/97GdKp9M6e/asVqxYoTvuuEOJRKIc+wDgMgWvuGpra5VOpyVJc+fOVSKR0MmTJ0s+DAAmM6Xn\nuI4fP66+vj41NzeXag8AFBQ6XGfPnlVbW5u2bt2quXPnlnITAOQV6g6oY2Njuv/++/XII4/ovvvu\nu+zXd+3aFXzteZ48z4tuIYBrwsDAgAYGBkIdWzBcvu9r48aNSiaT2rRp0xWPefDBB6e2EAA+5dMX\nPbt375702IIPFQ8ePKhf//rX2rdvnzKZjDKZjLq6uqJZCgBXoeAV16pVqyrmwzAAQOKd8wAMIlwA\nzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMCXUjwUJ834/iNEW7\n+eabXU8I5HI51xMCv/nNb1xPCOzbt8/1hEDYm9aVQ0NDg+sJgXfffdf1hIK44gJgDuECYA7hAmAO\n4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmBOwXBduHBBzc3NSqfTSiaTeuKJJ8qx\nCwAmVfDuEDNnztS+ffs0e/ZsjY+Pa9WqVXrzzTe1atWqcuwDgMuEeqg4e/ZsSdLo6KhyuZzmzZtX\n0lEAkE+ocE1MTCidTmvBggVas2aNkslkqXcBwKRChWvatGk6dOiQhoeHdeDAAe3fv7/EswBgclO6\nA2p1dbXuuecevfPOO/ra174W/P1du3YFX3ueJ8/zIhsI4NrQ39+v/v7+UMcWDNepU6c0Y8YMXX/9\n9Tp//rzeeOMNPfnkk5cc8+CDD17dUgD4j1QqpVQqFXz/vxdEn1YwXB9++KG++93vamJiQhMTE/rO\nd76jb3zjG9EsBYCrUDBcy5cvV29vbzm2AEAovHMegDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmE\nC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDlTupHgZKZNq4z+VVVVuZ5QkUZGRlxPCCQSCdcTAufPn3c9\nIXDy5EnXEwITExOuJxRUGcUBgCkgXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwh\nXADMIVwAzAkVrlwup0wmo5aWllLvAYCCQoVr69atSiaTisVipd4DAAUVDNfw8LA6Ozv16KOPyvf9\ncmwCgLwKhuvxxx/X008/XTH33AKAvDV67bXXNH/+fGUyGa62AFSMvHdAfeutt/TKK6+os7NTFy5c\n0Mcff6x169bp+eefv+S4bDYbfO15nlKpVGnWAvi/1d/fr4GBgVDH5g3X5s2btXnzZknS7373O/3k\nJz+5LFqS1N7efhUzAeC/UqnUJRc9u3fvnvTYKT1xxauKACpB6A/LuO2223TbbbeVcgsAhMJLhQDM\nIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMCX13iHwq5XY3\n3d3dricEvve977meEPjSl77kekJg2bJlricE5s2b53pCYGhoyPWEwMyZM11PKIgrLgDmEC4A5hAu\nAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOaE+iHr+vp6fe5zn9P06dNVVVWl\nnp6eUu8CgEmFClcsFtP+/fsr6qfpAVy7Qj9U9H2/lDsAILRQ4YrFYrr99tvV1NSk5557rtSbACCv\nUA8VDx48qIULF+qf//yn7rjjDjU0NGj16tWl3gYAVxQqXAsXLpQk1dTUqLW1VT09PZeEK5vNBl97\nnqdUKhXxTAD/7/r7+9Xf3x/q2ILhGhkZUS6X02c/+1mdO3dOr7/+up588slLjmlvb7+6pQDwH6lU\n6pKLnl27dk16bMFw/eMf/1Bra6skaXx8XA8//LDuvPPOCGYCwNUpGK7Fixfr0KFD5dgCAKHwznkA\n5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5oS6kWAhlXI/\n+mPHjrmeEPjjH//oekKgoaHB9YRAS0uL6wmBzs5O1xMCf/vb31xPCCxevNj1hIK44gJgDuECYA7h\nAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmBOwXCdPn1abW1tSiQSSiaTevvt\nt8uxCwAmVfDuED/84Q91991364UXXtD4+LjOnTtXjl0AMKm84froo4/U3d2tX/3qVxcPnjFD1dXV\nZRkGAJPJ+1BxcHBQNTU1Wr9+vW699VY99thjGhkZKdc2ALiivFdc4+Pj6u3t1bPPPquVK1dq06ZN\n2rJli370ox9dclw2mw2+9jxPqVSqNGsB/N8aGBjQ4cOHQx2bN1zxeFzxeFwrV66UJLW1tWnLli2X\nHdfe3n4VMwHgvzzPk+d5wfcvvvjipMfmfahYW1ururo6HT16VJK0d+/eS04MAC4UfFXx5z//uR5+\n+GGNjo5qyZIl2r59ezl2AcCkCoarsbFRf/jDH8qxBQBC4Z3zAMwhXADMIVwAzCFcAMwhXADMIVwA\nzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwpeHeIMGKxWBSnKVpTU5PrCYH+/n7XEwJf/epX\nXU8IbNu2zfWEQCV98EtdXZ3rCYHp06e7nlAQV1wAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFc\nAMwhXADMIVwAzCFcAMwhXADMKRiuDz74QJlMJvirurpazzzzTDm2AcAVFbw7xM0336y+vj5J0sTE\nhBYtWqTW1taSDwOAyUzpoeLevXu1ZMmSiroFB4Brz5TCtXPnTj300EOl2gIAoYS+keDo6KheffVV\nPfXUU5f9WjabDb72PE+pVCqadQCuGQMDAxoYGAh1bOhw7dmzRytWrFBNTc1lv9be3h5+HQBcged5\n8jwv+H737t2THhv6oeKOHTu0du3a4pYBQARChevcuXPau3evvv3tb5d6DwAUFOqh4pw5c3Tq1KlS\nbwGAUHjnPABzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMqJlz9/f2uJwQqacuf/vQn\n1xMCv//9711PkCT9+c9/dj0hMDg46HpC4MiRI64nBMLe5eFqVUy4Sv0POhWVtIVwXY5wXVkl/bdy\nzYQLAMIKfT+ufK6//vqizzFz5sxIzhOFKLZUVVVFsmXWrFm68cYbizrHddddF8mWGTNmFH2u+fPn\nF71jzpw5kZzn/PnzRZ9j7ty5qq2tLfo88+bNK/ocs2bNiuw8xSr1/88x3/f9ok4Qi0W1BQAuMVme\nir7iKrJ7ADBlPMcFwBzCBcAc5+Hq6upSQ0ODbrrppit+glA5bdiwQQsWLNDy5cud7hgaGtKaNWuC\nT0xy+cnhFy5cUHNzs9LptJLJpJ544glnWz6Ry+WUyWTU0tLidEd9fb1uueUWZTIZffnLX3a65fTp\n02pra1MikVAymdTbb7/tZEfZPvned2h8fNxfsmSJPzg46I+OjvqNjY3+4cOHne05cOCA39vb66dS\nKWcbfN/3P/zwQ7+vr8/3fd8/c+aMv2zZMqd/LufOnfN93/fHxsb85uZmv7u729kW3/f9n/70p/5D\nDz3kt7S0ON1RX1/v/+tf/3K64RPr1q3zf/nLX/q+f/Hf0+nTpx0v8v1cLufX1tb6f//73yM/t9Mr\nrp6eHi1dulT19fWqqqpSR0eHXn75ZWd7Vq9erRtuuMHZ7/+J2tpapdNpSRdfbk8kEjp58qSzPbNn\nz5Z08bM1c7lcJC+5X63h4WF1dnbq0UcfrYgXhiphw0cffaTu7m5t2LBB0sW3rVRXVzteVdpPvnca\nrhMnTlzyDxWPx3XixAmHiyrP8ePH1dfXp+bmZmcbJiYmlE6ntWDBAq1Zs0bJZNLZlscff1xPP/20\npk1z/iyHYrGYbr/9djU1Nem5555ztmNwcFA1NTVav369br31Vj322GMaGRlxtucTpfzke6f/9nkP\nWH5nz55VW1ubtm7dqrlz5zrbMW3aNB06dEjDw8M6cOCA9u/f72THa6+9pvnz5yuTyVTElc7BgwfV\n19enPXv26Be/+IW6u7ud7BgfH1dvb6++//3vq7e3V3PmzNGWLVucbPnE6OjFT75/4IEHSnJ+p+Fa\ntGiRhoaGgu+HhoYUj8cdLqocY2Njuv/++/XII4/ovvvucz1HklRdXa177rlH77zzjpPf/6233tIr\nr7yixYsXa+3atfrtb3+rdevWOdkiSQsXLpQk1dTUqLW1VT09PU52xONxxeNxrVy5UpLU1tam3t5e\nJ1s+sWfP5J98HwWn4WpqatKxY8d0/PhxjY6OKpvN6t5773U5qSL4vq+NGzcqmUxq06ZNTrecOnVK\np0+flnTxR2TeeOMNZTIZJ1s2b96soaEhDQ4OaufOnfr617+u559/3smWkZERnTlzRtLFD0x+/fXX\nnb0aXVtbq7q6Oh09elTSxeeW/vej7F0o+SffR/50/xR1dnb6y5Yt85csWeJv3rzZ6ZaOjg5/4cKF\n/mc+8xk/Ho/727Ztc7Kju7vbj8VifmNjo59Op/10Ou3v2bPHyZb33nvPz2QyfmNjo798+XL/xz/+\nsZMdn7Z//36nryr+9a9/9RsbG/3Gxkbf8zzn/+0eOnTIb2pq8m+55Ra/tbXV6auKZ8+e9W+88Ub/\n448/LtnvUfTPKgJAubl/aQYApohwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATDn315kycsrKKOX\nAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11056ae10>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "weights for 5:\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAEzCAYAAACPCO4GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEmRJREFUeJzt3VtslHX+x/FPkaJBpIopbWUaZgS1nQKdgWI9gEg9xTUo\nXboJeMBw8MYYxcQb4wXEC4KnKB6uiBqNCZJ4IR5oVbaUVtR0tWU1wSwmzqxtQSNGBApYOn3+F/55\ndpHOAebwm+/yfiUmrTx5/Cjk7TPTmWdKPM/zBACGjHE9AADOFOECYA7hAmAO4QJgDuECYA7hAmDO\n2GxPUFJSkosdAHCaZK/WyjpckvTkk09mfY729nY1NTVldY4pU6ZkvUOStm7dqrvuuiurc6xYsSIn\nW9atW6d169ZldY7ly5fnZMs///lP1dfXZ3WO8vLyrHd8/vnnuvbaa7M+z8MPP5z1OV544QWtWbMm\n6/Ocf/75WZ/j2Wef1WOPPZb1edauXZv1Ob766ivNmTMnq3Ns2rQp6a/xUBGAOYQLgDlFE65QKOR6\ngu+qq65yPcF34403up7gq6iocD1BkhQIBFxP8F1zzTWuJ/iuu+461xN8VVVVeT0/4RpFTU2N6wm+\nYgpXZWWl6wmSpOrqatcTfIRrdJdddllez1804QKATBEuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5qQN\nV1tbm2pqanTFFVfoqaeeKsQmAEgpZbgSiYQeeughtbW1ac+ePdq8ebO+/fbbQm0DgFGlDFd3d7em\nT5+uYDCo0tJSLV26VFu3bi3UNgAYVcpwDQwMnPL2ikAgoIGBgbyPAoBUUt6PK9ObBLa3t/tfh0Kh\nonrfIQAb9u3bp/3792d0bMpwTZkyRX19ff73fX19o74zP9sbAALAZZdddsqbs3t6epIem/KhYkND\ng7777jvF43ENDQ1py5YtuvPOO3O3FADOQsorrrFjx+rll1/WbbfdpkQioVWrVqm2trZQ2wBgVGnv\nOX/77bfr9ttvL8QWAMgIr5wHYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7h\nAmAO4QJgDuECYE7au0Nk4rzzzsvFabK2Y8cO1xN848aNcz3Bt3r1atcTfFu2bHE9wff000+7nuAb\n7Qadrvz3zfyKFVdcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwA\nzEkbrpUrV6qiokIzZ84sxB4ASCttuFasWKG2trZCbAGAjKQN1/z583XJJZcUYgsAZITnuACYk5Mb\nCf7973/3vw6FQrr88stzcVoA55B4PK54PJ7RsTkJ10033ZSL0wA4hwWDQQWDQf/7nTt3Jj2Wh4oA\nzEkbrmXLlum6667T3r17VV1drddff70QuwAgqbQPFTdv3lyIHQCQMR4qAjCHcAEwh3ABMIdwATCH\ncAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMCcn9+M6duxYLk6TtbKyMtcTfBMnTnQ9wffN\nN9+4nuB76623XE/wDQ4Oup7g8zzP9QTfE0884XpCWlxxATCHcAEwh3ABMIdwATCHcAEwh3ABMIdw\nATCHcAEwh3ABMIdwATCHcAEwh3ABMCdtuPr6+rRw4ULV1dVpxowZevHFFwuxCwCSSnt3iNLSUj3/\n/POKRCI6cuSI5syZo1tuuUW1tbWF2AcAp0l7xVVZWalIJCJJmjBhgmpra7Vv3768DwOAZM7oOa54\nPK7e3l41Njbmaw8ApJVxuI4cOaKWlhZt3LhREyZMyOcmAEgpozugnjhxQkuWLNG9996rxYsXn/br\nO3fu9L+eOnWqgsFgzgYCODfE43H9+9//zujYtOHyPE+rVq1SOBzWmjVrRj1mwYIFZ7YQAP4kGAye\nctHT1dWV9Ni0DxV37dqlt956Szt27FA0GlU0GlVbW1tOhgLA2Uh7xTVv3jyNjIwUYgsAZIRXzgMw\nh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwJ6MbCabjeV4u\nTpO1JUuWuJ7gK6abKf7444+uJ/jeeOMN1xN8ra2trif4fv31V9cTfOPGjXM9IS2uuACYQ7gAmEO4\nAJhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJiTNlzHjx9XY2OjIpGIwuGwHn/8\n8ULsAoCk0t4d4oILLtCOHTs0fvx4DQ8Pa968efr00081b968QuwDgNNk9FBx/PjxkqShoSElEglN\nmjQpr6MAIJWMwjUyMqJIJKKKigotXLhQ4XA437sAIKmMwjVmzBjt3r1b/f396uzsVEdHR55nAUBy\nZ3QH1LKyMt1xxx368ssvdeONN/p/v7Oz0/966tSpmjp1as4GAjg3xGIxxWKxjI5NG64DBw5o7Nix\nuvjii3Xs2DF98sknWrt27SnH3HDDDWe3FAD+XygUUigU8r9P9cgubbj279+v+++/XyMjIxoZGdF9\n992nm266KSdDAeBspA3XzJkz1dPTU4gtAJARXjkPwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzC\nBcAcwgXAHMIFwBzCBcAcwgXAHMIFwJwzupFgMueff34uTpO1Tz/91PUE38DAgOsJvtbWVtcTfJMn\nT3Y9wTd79mzXE3zXX3+96wm+d9991/WEtLjiAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJg\nDuECYA7hAmAO4QJgDuECYE5G4UokEopGo1q0aFG+9wBAWhmFa+PGjQqHwyopKcn3HgBIK224+vv7\ntW3bNq1evVqe5xViEwCklDZcjz76qJ555hmNGcPTYQCKQ8oaffDBB5o8ebKi0ShXWwCKRso7oH72\n2Wd67733tG3bNh0/flyHDh3S8uXL9eabb55yXHt7u/91KBRSKBTKz1oA/7NisZhisVhGx6YM1/r1\n67V+/XpJ0s6dO/Xss8+eFi1JampqOouZAPAff77o6ejoSHrsGT1xxU8VARSDjD8sY8GCBVqwYEE+\ntwBARvhRIQBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBz\nMr47RCrFcrubeDzueoJv69atrif4EomE6wm+Sy+91PUE3/Hjx11P8PX09Lie4LNwt2OuuACYQ7gA\nmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJiT0Zusg8GgJk6cqPPOO0+l\npaXq7u7O9y4ASCqjcJWUlKijo0OTJk3K9x4ASCvjh4oWbnUB4NyQUbhKSkp08803q6GhQZs2bcr3\nJgBIKaOHirt27VJVVZV+/vln3XLLLaqpqdH8+fPzvQ0ARpVRuKqqqiRJ5eXlam5uVnd39ynham9v\n978OhUIKhUI5ngngf10sFsv4LsZpw3X06FElEglddNFFGhwc1Mcff6y1a9eeckxTU9NZDQWAk/58\n0dPR0ZH02LTh+umnn9Tc3CxJGh4e1j333KNbb701+5UAcJbShisUCmn37t2F2AIAGeGV8wDMIVwA\nzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMyehGgukUy/3oT97w\nsBj88ssvrif4RkZGXE/wzZo1y/UE36FDh1xP8H300UeuJ/iuvvpq1xPS4ooLgDmEC4A5hAuAOYQL\ngDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOWnDdfDgQbW0tKi2tlbhcFhffPFFIXYB\nQFJp7w7xyCOP6C9/+YveeecdDQ8Pa3BwsBC7ACCplOH67bff1NXVpTfeeOOPg8eOVVlZWUGGAUAy\nKR8qxmIxlZeXa8WKFZo9e7YeeOABHT16tFDbAGBUKa+4hoeH1dPTo5dffllz587VmjVrtGHDBj35\n5JOnHNfe3u5/HQqFFAqF8rMWwP+seDyueDye0bEpwxUIBBQIBDR37lxJUktLizZs2HDacU1NTWe+\nEgD+SzAYVDAY9L/v7OxMemzKh4qVlZWqrq7W3r17JUnbt29XXV1dblYCwFlK+1PFl156Sffcc4+G\nhoY0bdo0vf7664XYBQBJpQ1XfX29/vGPfxRiCwBkhFfOAzCHcAEwh3ABMIdwATCHcAEwh3ABMIdw\nATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATAn7d0hLKmtrXU9wbd48WLXE3ye57me4Pvwww9dT/B9\n//33rif4Dhw44HqC7/Dhw64npMUVFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMI\nFwBzCBcAcwgXAHPShutf//qXotGo/1dZWZlefPHFQmwDgFGlvTvEVVddpd7eXknSyMiIpkyZoubm\n5rwPA4Bkzuih4vbt2zVt2jRVV1fnaw8ApHVG4Xr77bd1991352sLAGQk4xsJDg0N6f3339dTTz11\n2q+1t7f7X4dCIYVCodysA3DO6OvrU39/f0bHZhyu1tZWzZkzR+Xl5af9WlNTU+brAGAU1dXVpzwN\n9cUXXyQ9NuOHips3b9ayZcuyWwYAOZBRuAYHB7V9+3b99a9/zfceAEgro4eKF154YVHdzB/AuY1X\nzgMwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCnaMIVi8VcT/Dt2bPH9QTfl19+6XqC\n76uvvnI9QVJx/Vn58ccfXU/wHTp0yPUEX19fX17PT7hG8e2337qe4CuWWEjFsyUej7ue4CumcB0+\nfNj1BF+mt6c5W0UTLgDIVMb340qlsrIy63NMmDAh6/NccsklWe+QpAsuuCDrc40fPz4nW0pLS7M+\nl+d5RbOlqqoq6x0XXXRRTs6TSCSyPkc8HtfUqVOzPs/IyEjW5/j9999VU1OT9XkCgUDW55g4cWJO\nzpNMiZfln+qSkpJcbQGAUyTLU9ZXXLn6vzkAZIrnuACYQ7gAmOM8XG1tbaqpqdEVV1wx6icIFdLK\nlStVUVGhmTNnOt3R19enhQsXqq6uTjNmzHD6yeHHjx9XY2OjIpGIwuGwHn/8cWdbTkokEopGo1q0\naJHTHcFgULNmzVI0GtXVV1/tdMvBgwfV0tKi2tpahcPhlB80kU8F++R7z6Hh4WFv2rRpXiwW84aG\nhrz6+npvz549zvZ0dnZ6PT093owZM5xt8DzP279/v9fb2+t5nucdPnzYu/LKK53+dxkcHPQ8z/NO\nnDjhNTY2el1dXc62eJ7nPffcc97dd9/tLVq0yOmOYDDo/fLLL043nLR8+XLv1Vdf9Tzvj9+ngwcP\nOl7keYlEwqusrPR++OGHnJ/b6RVXd3e3pk+frmAwqNLSUi1dulRbt251tmf+/Pk5e0lFNiorKxWJ\nRCT98TKR2tpa7du3z9meky+BGBoaUiKR0KRJk5xt6e/v17Zt27R69eqi+MFQMWz47bff1NXVpZUr\nV0qSxo4dq7KyMser8vvJ907DNTAwcMq/VCAQ0MDAgMNFxScej6u3t1eNjY3ONoyMjCgSiaiiokIL\nFy5UOBx2tuXRRx/VM888ozFjnD/LoZKSEt18881qaGjQpk2bnO2IxWIqLy/XihUrNHv2bD3wwAM6\nevSosz0n5fOT753+7vMasNSOHDmilpYWbdy4URMmTHC2Y8yYMdq9e7f6+/vV2dmpjo4OJzs++OAD\nTZ48WdFotCiudHbt2qXe3l61trbqlVdeUVdXl5Mdw8PD6unp0YMPPqienh5deOGF2rBhg5MtJw0N\n/fHJ93/729/ycn6n4ZoyZcopb8bs6+vL66ttLTlx4oSWLFmie++9V4sXL3Y9R5JUVlamO+64w9kb\nvz/77DO99957CoVCWrZsmdrb27V8+XInW6T/vAugvLxczc3N6u7udrIjEAgoEAho7ty5kqSWlhb1\n9PQ42XJSa2vyT77PBafhamho0Hfffad4PK6hoSFt2bJFd955p8tJRcHzPK1atUrhcFhr1qxxuuXA\ngQM6ePCgJOnYsWP65JNPFI1GnWxZv369+vr6FIvF9Pbbb6upqUlvvvmmky1Hjx7139Q8ODiojz/+\n2NlPoysrK1VdXa29e/dK+uO5pbq6OidbTsr7J9/n/On+M7Rt2zbvyiuv9KZNm+atX7/e6ZalS5d6\nVVVV3rhx47xAIOC99tprTnZ0dXV5JSUlXn19vReJRLxIJOK1trY62fL111970WjUq6+v92bOnOk9\n/fTTTnb8WUdHh9OfKn7//fdefX29V19f79XV1Tn/s7t7926voaHBmzVrltfc3Oz0p4pHjhzxLr30\nUu/QoUN5+2dk/V5FACg09z+aAYAzRLgAmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmPN/E2rQDsyI\nW2gAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110df80d0>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "weights for 8 minus weights for 3:\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAS4AAAEzCAYAAACPCO4GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEllJREFUeJzt3VlsVHX/x/FPaxGlaAHTdaZaLCCdUjtDi01MiSKoQYKx\nWhNwwbAYl5iol155R1xjcLkyajQaJPEGVGgUoVJRU2RKXAOINJa2UjFBoBWGtud/wcP5PwidGZjl\nN9+H9yshaeXk8FHMO2emM2fyPM/zBACG5LseAADni3ABMIdwATCHcAEwh3ABMIdwATCnINUT5OXl\npWMHAJxlrFdrpRwuSXr00UdTPseOHTs0Z86clM6xd+/elHdI0m+//aZrr702pXPccsstadnyxRdf\naP78+Smd44MPPkjLloGBAZWUlKR0jlgslvKOv/76S1dddVXK55k0aVLK5+jt7VUgEEj5PP39/Smf\n48iRI7ryyitTPk9zc3PK5/jxxx81a9aslM6xbt26MX+Ph4oAzCFcAMzJmXBVVFS4nuCbPHmy6wm+\nqVOnup7gKywsdD1BknT55Ze7nuC74oorXE/wjR8/3vUEX6pPKSSSM+FKx/ME6ZJL4Ur1ubZ0ypVw\nTZgwwfUEXzqeU0oXwgUAOYxwATCHcAEwh3ABMIdwATCHcAEwh3ABMCdhuNra2jRz5kxNnz5dzz//\nfDY2AUBcccM1MjKiJ554Qm1tbfr555+1du1a/fLLL9naBgDnFDdcnZ2dmjZtmqqqqjRu3DgtWbJE\n69evz9Y2ADinuOHq7e1VZWWl/30wGFRvb2/GRwFAPHHvx5XsTQJ37Njhf11RUZFT7zsEYMPAwIAG\nBgaSOjZuuAKBgHp6evzve3p6FAwGzzou1RsAAkBJSckZb87+6aefxjw27kPFxsZG7d27V93d3YrF\nYlq3bp3uvPPO9C0FgAsQ94qroKBAr7/+um6//XaNjIxo5cqVqqmpydY2ADinhPecX7hwoRYuXJiN\nLQCQFF45D8AcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcCc\nhHeHSMbkyZPTcZqU/fdNyFz78ssvXU/wRSIR1xN87e3trif4HnvsMdcTfN98843rCb78/Ny/nsn9\nhQDwL4QLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5CcO1YsUKlZaW\nqq6uLht7ACChhOFavny52trasrEFAJKSMFxz587NmdvWAIDEc1wADErLjQQ7Ojr8r6+++mpdc801\n6TgtgIvIwYMHNTAwkNSxaQnX3Llz03EaABex0tJSlZaW+t//+OOPYx7LQ0UA5iQM19KlS3XjjTdq\nz549qqys1DvvvJONXQAwpoQPFdeuXZuNHQCQNB4qAjCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCH\ncAEwh3ABMIdwATCHcAEwh3ABMCct9+M6cuRIOk6TsmAw6HqC74MPPnA9wffDDz+4nuAbHR11PcG3\ndOlS1xN8U6ZMcT3Bt2XLFtcTEuKKC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmE\nC4A5hAuAOYQLgDkJw9XT06N58+aptrZWs2bN0quvvpqNXQAwpoR3hxg3bpxeeeUVhcNhHTt2TA0N\nDbr11ltVU1OTjX0AcJaEV1xlZWUKh8OSpIkTJ6qmpkZ9fX0ZHwYAYzmv57i6u7vV1dWlpqamTO0B\ngISSDtexY8fU2tqqNWvWaOLEiZncBABxJXUH1JMnT+qee+7RAw88oLvuuuus3+/s7PS/DgQCCgQC\n6VsI4KLwxx9/6ODBg0kdmzBcnudp5cqVCoVCeuqpp855zA033HB+CwHgX8rKylRWVuZ///333495\nbMKHitu3b9f777+vrVu3KhKJKBKJqK2tLT1LAeACJLziam5uzqkPOAAAXjkPwBzCBcAcwgXAHMIF\nwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwBzCBcAcwgXAHMIFwJykbiSYyNDQUDpOk7JrrrnG9QRf\nV1eX6wm+6dOnu57g6+7udj3Bt2fPHtcTfMPDw64n+MaPH+96QkJccQEwh3ABMIdwATCHcAEwh3AB\nMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATAnYbiOHz+upqYmhcNhhUIhPfPMM9nYBQBjSnh3\niMsuu0xbt27VhAkTNDw8rObmZn311Vdqbm7Oxj4AOEtSDxUnTJggSYrFYhoZGdGUKVMyOgoA4kkq\nXKOjowqHwyotLdW8efMUCoUyvQsAxpRUuPLz87Vr1y4dOHBA27ZtU3t7e4ZnAcDYzusOqEVFRVq0\naJG+++473Xzzzf4//++7fZaVlam8vDxtAwFcHPr6+tTf35/UsQnDdejQIRUUFGjSpEn6559/9Pnn\nn+vZZ58945hIJHJhSwHgPyoqKlRRUeF/H+/25wnD1d/fr4ceekijo6MaHR3Vgw8+qPnz56dnKQBc\ngIThqqurUzQazcYWAEgKr5wHYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7h\nAmAO4QJgDuECYM553UhwLOPHj0/HaVKWS3exOHHihOsJvoaGBtcTfNOmTXM9wbdq1SrXE3z79u1z\nPcH3yCOPuJ6QEFdcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwAzCFcAMwhXADMIVwA\nzEkqXCMjI4pEIlq8eHGm9wBAQkmFa82aNQqFQsrLy8v0HgBIKGG4Dhw4oI0bN2rVqlXyPC8bmwAg\nroThevrpp/Xiiy8qP5+nwwDkhrg1+uSTT1RSUqJIJMLVFoCcEfcOqF9//bU2bNigjRs36vjx4zpy\n5IiWLVum995774zjduzY4X9dUVGhQCCQmbUA/mf19vaqr68vqWPjhmv16tVavXq1JOnLL7/USy+9\ndFa0JGnOnDkXMBMA/l8gEDjjomfnzp1jHnteT1zxU0UAuSDpD8u46aabdNNNN2VyCwAkhR8VAjCH\ncAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATAn6btDxHPy5Ml0\nnCZl5eXlrif4li1b5nqCr6AgLX/NabF161bXE3x1dXWuJ/iuu+461xN8J06ccD0hIa64AJhDuACY\nQ7gAmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmEO4AJhDuACYQ7gAmJPUu2+rqqp05ZVX6pJLLtG4\ncePU2dmZ6V0AMKakwpWXl6f29nZNmTIl03sAIKGkHyp6npfJHQCQtKTClZeXpwULFqixsVFvvvlm\npjcBQFxJPVTcvn27ysvL9eeff+rWW2/VzJkzNXfu3ExvA4BzSipcp+8sWlxcrJaWFnV2dp4Rrmg0\nesaxuXQnUgA29Pf3648//kjq2IThGhoa0sjIiK644goNDg7qs88+07PPPnvGMbNnz76wpQDwH/++\n6Nm1a9eYxyYM18GDB9XS0iJJGh4e1v3336/bbrstDTMB4MIkDNfUqVPjlg8Aso1XzgMwh3ABMIdw\nATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwh3ABMIdwATCHcAEwJ6kbCSaSn58b/bvkkktc\nT/B9+umnrif43nnnHdcTfP39/a4n+BYvXux6gq+3t9f1BF9BQVqykFG5URwAOA+EC4A5hAuAOYQL\ngDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOQnDdfjwYbW2tqqmpkahUEjffvttNnYB\nwJgSvg38ySef1B133KGPPvpIw8PDGhwczMYuABhT3HD9/fff6ujo0Lvvvnvq4IICFRUVZWUYAIwl\n7kPF/fv3q7i4WMuXL9fs2bP18MMPa2hoKFvbAOCc4oZreHhY0WhUjz/+uKLRqAoLC/Xcc8+dddzO\nnTv9X319fRkbC+B/V19f3xktiSfuQ8VgMKhgMKg5c+ZIklpbW88ZroaGhhTmAoBUUVGhiooK//to\nNDrmsXGvuMrKylRZWak9e/ZIkjZv3qza2to0zQSAC5Pwp4qvvfaa7r//fsViMVVXV+fU/csBXJwS\nhqu+vl47duzIxhYASAqvnAdgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuECYA7hAmAO4QJgDuEC\nYA7hAmAO4QJgTsK7Q1iSS3exeP/9911P8DU3N7ue4LvvvvtcT/Bt3brV9QTfvn37XE/wBQIB1xMS\n4ooLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOQnDtXv3bkUi\nEf9XUVGRXn311WxsA4BzSnh3iOuuu05dXV2SpNHRUQUCAbW0tGR8GACM5bweKm7evFnV1dWqrKzM\n1B4ASOi8wvXhhx/m1P2UAFyckr6RYCwW08cff6znn3/+rN/buXOn/3V5ebkqKirSsw7ARaO3t1d9\nfX1JHZt0uDZt2qSGhgYVFxef9XsNDQ3JrwOAcwgEAmfcffW7774b89ikHyquXbtWS5cuTW0ZAKRB\nUuEaHBzU5s2bdffdd2d6DwAklNRDxcLCQh06dCjTWwAgKbxyHoA5hAuAOYQLgDmEC4A5hAuAOYQL\ngDmEC4A5hAuAOYQLgDk5E65k3xWeDYcPH3Y9wXfixAnXE3wDAwOuJ0iSfv31V9cTfLn0jpLBwUHX\nE3y9vb0ZPX/OhKu/v9/1BF8uhSsWi7me4Pvzzz9dT5BEuMaSS+HK9IVIzoQLAJKV9P244rn66qtT\nPsevv/6alvOkw9GjRzVjxoyUznHppZemZcu+fftUXV2d0jmmTp2ali19fX0pn6ukpCTlHYWFhWk5\nT6r/XaVTD5/TcZ7x48enfA7P8xQKhVI+Tzpuzb579+6M3uI9z/M8L6UT5OWlawsAnGGsPKV8xZVi\n9wDgvPEcFwBzCBcAc5yHq62tTTNnztT06dPP+QlC2bRixQqVlpaqrq7O6Y6enh7NmzdPtbW1mjVr\nltNPDj9+/LiampoUDocVCoX0zDPPONty2sjIiCKRiBYvXux0R1VVla6//npFIhHdcMMNTrccPnxY\nra2tqqmpUSgU0rfffutkx+7dWfrke8+h4eFhr7q62tu/f78Xi8W8+vp67+eff3a2Z9u2bV40GvVm\nzZrlbIPneV5/f7/X1dXleZ7nHT161JsxY4bT/y6Dg4Oe53neyZMnvaamJq+jo8PZFs/zvJdfftm7\n7777vMWLFzvdUVVV5f31119ON5y2bNky76233vI879Tf0+HDhx0v8ryRkRGvrKzM+/3339N+bqdX\nXJ2dnZo2bZqqqqo0btw4LVmyROvXr3e2Z+7cuZo8ebKzP/+0srIyhcNhSdLEiRNVU1Pj9J0FEyZM\nkHTqxbAjIyOaMmWKsy0HDhzQxo0btWrVqpz4wVAubPj777/V0dGhFStWSJIKCgpUVFTkeFVmP/ne\nabh6e3vP+JcKBoMZf6uANd3d3erq6lJTU5OzDaOjowqHwyotLdW8efPS8lqhC/X000/rxRdfVH6+\n82c5lJeXpwULFqixsVFvvvmmsx379+9XcXGxli9frtmzZ+vhhx/W0NCQsz2nZfKT753+7fMasPiO\nHTum1tZWrVmzRhMnTnS2Iz8/X7t27dKBAwe0bds2tbe3O9nxySefqKSkRJFIJCeudLZv366uri5t\n2rRJb7zxhjo6OpzsGB4eVjQa1eOPP65oNKrCwkI999xzTracFoud+uT7e++9NyPndxquQCCgnp4e\n//uenh4Fg0GHi3LHyZMndc899+iBBx7QXXfd5XqOJKmoqEiLFi2K+wnDmfT1119rw4YNmjp1qpYu\nXaotW7Zo2bJlTrZIUnl5uSSpuLhYLS0t6uzsdLIjGAwqGAxqzpw5kqTW1lZFo1EnW07btGnsT75P\nB6fhamxs1N69e9Xd3a1YLKZ169bpzjvvdDkpJ3iep5UrVyoUCumpp55yuuXQoUP+m87/+ecfff75\n54pEIk62rF69Wj09Pdq/f78+/PBD3XLLLXrvvfecbBkaGtLRo0clnXpz82effebsp9FlZWWqrKzU\nnj17JJ16bqm2ttbJltMy/sn3aX+6/zxt3LjRmzFjhlddXe2tXr3a6ZYlS5Z45eXl3qWXXuoFg0Hv\n7bffdrKjo6PDy8vL8+rr671wOOyFw2Fv06ZNTrZ8//33XiQS8err6726ujrvhRdecLLj39rb253+\nVPG3337z6uvrvfr6eq+2ttb5/7u7du3yGhsbveuvv95raWlx+lPFY8eOeVdddZV35MiRjP0ZKb9X\nEQCyzf2PZgDgPBEuAOYQLgDmEC4A5hAuAOYQLgDmEC4A5hAuAOb8H/sO10rDKlnxAAAAAElFTkSu\nQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110e42710>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "training accuracy: 1.0\n",
        "test accuracy: 0.955555555556\n",
        "We should overfit, i.e. test score << train score, you can add an L2 regularization\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Bonus:_ you can add regularization (order to try: L2, L1, ElasticNet), start with adding the square of the weights magnitudes to the loss function (weight this sum by an $\\alpha$ as \"L2 regularization parameter\") http://en.wikipedia.org/wiki/Tikhonov_regularization // http://en.wikipedia.org/wiki/Least_squares#Lasso_method"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Reality check: linear and RBF SVMs from scikit-learn"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn import svm\n",
      "clf = svm.SVC(kernel='linear')\n",
      "clf.fit(x_train, y_train)\n",
      "print(\"Linear SVM\")\n",
      "print(\"training accuracy:\", clf.score(x_train, y_train))\n",
      "print(\"test accuracy:\", clf.score(x_test, y_test))\n",
      "clf = svm.SVC(kernel='rbf')\n",
      "clf.fit(x_train, y_train)\n",
      "print(\"with and RBF kernel\")\n",
      "print(\"training accuracy:\", clf.score(x_train, y_train))\n",
      "print(\"test accuracy:\", clf.score(x_test, y_test))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Linear SVM\n",
        "training accuracy: 1.0\n",
        "test accuracy:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.97037037037\n",
        "with and RBF kernel"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "training accuracy:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.996725605763\n",
        "test accuracy: 0.981481481481\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Bonus:_ change the loss of our logistic regression for a [hinge loss](http://en.wikipedia.org/wiki/Hinge_loss) and get a linear SVM. With regularization, you should approach scores above."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "2. Intro to SGD"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We used an [L-BFGS](http://en.wikipedia.org/wiki/Limited-memory_BFGS) optimization procedure, which is quasi-Newton (Newton algorithm to find the 0s of f: $x_{n+1} = x_n - \\frac{f(x_n)}{f'(x_n)}$, and we search the 0s of $\\nabla(loss)$). It uses an approximation of the Hessian matrix (matrix of second-order partial derivatives), but it's still more costly (in CPU time, but particularly in RAM) than just using the Jacobian (matrix of the first-order partial derivatives). Let's optimize this $loss$ by just taking small steps in the direction of the gradient, each datapoint at a time."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def init(lr, x_train, y_train):\n",
      "    lr.x = np.hstack((x_train, np.ones((x_train.shape[0], 1))))\n",
      "    lr.y = y_train\n",
      "    lr.n_classes = len(set(y_train))\n",
      "    lr.W = np.zeros((x_train.shape[1] + 1, lr.n_classes))\n",
      "    \n",
      "    \n",
      "def one_epoch_of_SGD(lr, learning_rate):\n",
      "    \"\"\" an \"epoch\" is a full pass/iteration on the dataset \"\"\"\n",
      "    tmp_W = lr.W[:]\n",
      "    for i, xi in enumerate(lr.x):\n",
      "        p_i = softmax(tmp_W, xi)\n",
      "        for j, v in enumerate(p_i):\n",
      "            if j == lr.y[i]:\n",
      "                tmp_W[:,j] += learning_rate * (xi * (1 - v))\n",
      "            else:\n",
      "                tmp_W[:,j] += learning_rate * (- xi * v)\n",
      "    return tmp_W\n",
      "    \n",
      "    \n",
      "class LogisticRegression2:\n",
      "    def __init__(self):\n",
      "        self.name = \"Logistic Regression\"\n",
      "        \n",
      "    def loss(self):  # we redefine it so that we dont have to do all the reshaping, as we don't use scipy's L-BFGS anymore\n",
      "        \"\"\" Negative log-likelihood as described above \"\"\"\n",
      "        l = 0.\n",
      "        for i, xi in enumerate(self.x):\n",
      "            p_i = softmax(self.W, xi)\n",
      "            l += np.log(p_i[self.y[i]])\n",
      "        return - l / self.x.shape[0]\n",
      "    \n",
      "    def score(self, x_test, y_test):\n",
      "        x = np.hstack((x_test, np.ones((x_test.shape[0], 1))))\n",
      "        y = y_test\n",
      "        c = 0\n",
      "        for i, xi in enumerate(x):\n",
      "            a = np.argmax(softmax(self.W, xi))\n",
      "            if a == y[i]:\n",
      "                c += 1\n",
      "        return c * 1. / y.shape[0]\n",
      "\n",
      "    \n",
      "def stupid_train_with_SGD(lr, x_train, y_train):\n",
      "    init(lr, x_train, y_train)\n",
      "    print(\"loss before training:\", lr.loss())\n",
      "    lr.W = one_epoch_of_SGD(lr, learning_rate=0.01)\n",
      "    old_loss = 1.E31\n",
      "    new_loss = lr.loss()\n",
      "    losses = [new_loss]\n",
      "    print(\"loss after one epoch of training:\", new_loss)\n",
      "    lr.W = one_epoch_of_SGD(lr, learning_rate=0.01)\n",
      "    new_loss = lr.loss()\n",
      "    losses.append(new_loss)\n",
      "    print(\"loss after two epochs of training:\", new_loss)\n",
      "    while old_loss - new_loss > 1.E-5:  # we want to optimize up to decreasing in the loss of 1.E-5 ==> STUPIDELY BAD\n",
      "        old_loss = new_loss\n",
      "        lr.W = one_epoch_of_SGD(lr, learning_rate=0.01)\n",
      "        new_loss = lr.loss()\n",
      "        losses.append(new_loss)\n",
      "    print(\"best loss:\", new_loss)\n",
      "    plt.plot(np.arange(len(losses)), np.log(losses), label=\"stupid\", linewidth=2, linestyle=\"--\")\n",
      "    \n",
      "    \n",
      "def slightly_less_stupid_train_with_SGD(lr, x_train, y_train):\n",
      "    init(lr, x_train, y_train)\n",
      "    print(\"loss before training:\", lr.loss())\n",
      "    old_loss = 1.E31\n",
      "    losses = []\n",
      "    new_loss = 1.E30\n",
      "    while abs(old_loss - new_loss) > 1.E-5:  # we want to optimize up to changes in the loss of 1.E-5 ==> BAD\n",
      "        old_loss = new_loss\n",
      "        lr.W = one_epoch_of_SGD(lr, learning_rate=0.01)\n",
      "        new_loss = lr.loss()\n",
      "        losses.append(new_loss)\n",
      "    print(\"best loss:\", new_loss)\n",
      "    plt.plot(np.arange(len(losses)), np.log(losses), label=\"slightly less stupid\", linewidth=2, linestyle=\":\")\n",
      "    \n",
      "    \n",
      "def early_stopping_train_with_SGD(lr, x_train, y_train):\n",
      "    \"\"\" using the training set as validation set, \n",
      "    DO NOT DO THIS IN PRACTICE! USE A SEPARATE VALIDATION SET \"\"\"\n",
      "    init(lr, x_train, y_train)\n",
      "    print(\"loss before training:\", lr.loss())\n",
      "    old_score = -1.\n",
      "    new_score = lr.score(x_train, y_train)\n",
      "    losses = []\n",
      "    patience = 3  # number of epochs of patience\n",
      "    while new_score > old_score or patience:\n",
      "        if new_score > old_score:\n",
      "            patience = 3\n",
      "        else:\n",
      "            patience -= 1\n",
      "        old_score = new_score\n",
      "        lr.W = one_epoch_of_SGD(lr, learning_rate=0.01)\n",
      "        new_score = lr.score(x_train, y_train)\n",
      "        losses.append(lr.loss())\n",
      "    print(\"best loss:\", losses[-1])\n",
      "    plt.plot(np.arange(len(losses)), np.log(losses), label=\"crude early stopping\\n3 epochs of patience\", linewidth=1.5)\n",
      "\n",
      "    \n",
      "lr = LogisticRegression2()\n",
      "    \n",
      "print(\"stupid stopping condition [will stop at the first bump in the loss function]\")\n",
      "stupid_train_with_SGD(lr, x_train, y_train)\n",
      "print(\"training accuracy:\", lr.score(x_train, y_train))\n",
      "print(\"test accuracy:\", lr.score(x_test, y_test))\n",
      "print()\n",
      "\n",
      "print(\"slightly less stupid stopping condition [will stop when the loss function is flat]\")\n",
      "slightly_less_stupid_train_with_SGD(lr, x_train, y_train)\n",
      "print(\"training accuracy:\", lr.score(x_train, y_train))\n",
      "print(\"test accuracy:\", lr.score(x_test, y_test))\n",
      "print()\n",
      "\n",
      "print(\"(very) early stopping (crude, with 3 epochs of patience)\")\n",
      "print(\"[optimizes for the right thing, but do it on a _separate_ validation set!]\")\n",
      "early_stopping_train_with_SGD(lr, x_train, y_train)  # using the training set as validation set, \n",
      "                                                     # DO NOT DO THIS IN PRACTICE!\n",
      "                                                     # USE A SEPARATE VALIDATION SET\n",
      "print(\"training accuracy:\", lr.score(x_train, y_train))\n",
      "print(\"test accuracy:\", lr.score(x_test, y_test))\n",
      "plt.ylabel(\"log of the loss function\")\n",
      "plt.xlabel(\"#epochs\")\n",
      "plt.legend()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "stupid stopping condition [will stop at the first bump in the loss function]\n",
        "loss before training: 2.30258509299\n",
        "loss after one epoch of training:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.229635078515\n",
        "loss after two epochs of training:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.157913366763\n",
        "best loss:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.00421204401221\n",
        "training accuracy:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 1.0\n",
        "test accuracy: 0.962962962963\n",
        "\n",
        "slightly less stupid stopping condition [will stop when the loss function is flat]\n",
        "loss before training: 2.30258509299\n",
        "best loss:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.00421204401221\n",
        "training accuracy: 1.0\n",
        "test accuracy: 0.962962962963\n",
        "\n",
        "(very) early stopping (crude, with 3 epochs of patience)\n",
        "[optimizes for the right thing, but do it on a _separate_ validation set!]\n",
        "loss before training: 2.30258509299\n",
        "best loss:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.029649373213\n",
        "training accuracy: 0.998035363458\n",
        "test accuracy: 0.966666666667\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "<matplotlib.legend.Legend at 0x110dffa10>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAf8AAAFHCAYAAACvaEWmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8Tdf6+PHPSSIIJ5FJZB5MQUVUDDXGEPPsKlJjW9pe\ndVt6abVFzEOrVYq2WtWaiqJqCoqgKpQW10yIhCQaMwkZTtbvj3ydn8goyck5SZ7363VeN/vsddZ6\n9u65nrPXXnstjVJKIYQQQohSw8zYAQghhBCiaEnyF0IIIUoZSf5CCCFEKSPJXwghhChlJPkLIYQQ\npYwkfyGEEKKUMUryX7duHXXq1MHc3Jy//vor23KhoaH4+vpSvXp1Zs+eXYQRCiGEECWXUZJ/3bp1\n2bhxIy1btsy2jE6n4+233yY0NJQzZ86wevVqzp49W4RRCiGEECWThTEa9fX1zbXMkSNHqFatGl5e\nXgD079+fTZs2UatWLQNHJ4QQQpRsJnvP//r167i7u+u33dzcuH79uhEjEkIIIUoGg135BwUFERcX\nl+n9GTNm0K1bt1w/r9FoDBGWEEIIUeoZLPnv2rWrQJ93dXUlOjpavx0dHY2bm1uWZeWHghBCiNKm\nIEvzGL3bP7vgAwICuHjxIpGRkSQnJ7NmzRq6d++eYz0l9TVp0iSjxyDHJscnx1fyXiX5+ErysSlV\n8PX4jJL8N27ciLu7O+Hh4XTp0oVOnToBEBMTQ5cuXQCwsLDgyy+/pEOHDtSuXZt+/frJYD8hhBCi\nEBhltH+vXr3o1atXpvddXFzYunWrfrtTp076HwZCCCGEKBxG7/YXuQsMDDR2CAZTko8N5PiKOzm+\n4qskH1th0KjCuHlgZBqNplDugQghhBDFQUHznlG6/YUQwlTI00LC1Bni4laSvxCi1JOeQ2GqDPXj\nVO75CyGEEKWMJH8hhBCilJHkL4QQQpQykvyFEKIUmzlzJsOHD892v5eXF7t37y7CiERRkAF/QghR\njISEhBAREcHy5csLpb7x48fnuF+j0cgTESWQXPkLIYQQpYwkfyGEMFGzZ8/Gzc0Na2trfH192bZt\nGzNnzmTNmjVotVrq168PZO6aDwkJYdCgQQBERkZiZmbGkiVLcHV1xcXFhblz52ZZFmD58uV4enri\n4ODAjBkziuhIRVGTbn8hhMhBdj3e2U0NkFX5/EwjcP78eRYuXMjRo0epUqUKUVFRpKam8uGHHxIR\nEcGPP/74VJsZu+az6qYPCwvj0qVLRERE0KZNG/z9/Wnbtm2GsmfOnOHf//4327dvp1GjRowfP55r\n1649f/DC5MmVvxBCmCBzc3OSkpI4ffo0KSkpeHh44OPjk6clXbPaP2nSJMqXL88LL7zAsGHDWL16\ndaayP//8M926daN58+ZYWloydepUzMwkTZRE8l9VCCFyoFTWr+cpnx/VqlVj3rx5hISE4OTkxIAB\nA4iNjc1fZYC7u7v+bw8PD2JiYjKViYmJwc3NTb9tZWWFvb19vtsUpkuSvxBCmKgBAwZw4MABrl69\nikaj4f3338/ySrxChQokJCTot+Pi4jKViYqKyvC3q6trpjIuLi5ER0frtxMTE7l161ZBD0OYIEn+\nQghhgi5cuMCePXtISkqibNmylCtXDnNzc5ycnIiMjMzQXe/v789PP/1EamoqR48eZf369Znu+0+b\nNo1Hjx5x+vRpli1bRr9+/TK12adPH7Zs2cLBgwdJTk5m4sSJpKWlGfxYRdGT5C+EECYoKSmJ8ePH\n4+joiLOzMzdv3mTmzJn07dsXAHt7ewICAgCYOnUqERER2NraEhISwiuvvJKpvlatWlGtWjXatWvH\n2LFjadeuHZBxsGCdOnVYuHAhwcHBuLi4YGdnl+F2gSg5NKoELGdV0HWNhRClV0n/9yMyMhIfHx9S\nU1Nl8F4xlN33s6DfW/kmCCGEEKWMJH8hhCjhZHpe8Szp9hdClGry74cwZdLtL4QQQohCYZTkv27d\nOurUqYO5uTl//fVXtuW8vLzw8/Ojfv36NGrUqAgjFEIIIUouo8ztX7duXTZu3Mgbb7yRYzmNRkNY\nWBh2dnZFFJkQQghR8hkl+fv6+ua5rNyLE0IIIQqXSd/z12g0tGvXjoCAAJYsWWLscIQQQogSwWDJ\nPygoiLp162Z6bd68Oc91HDx4kL///pvt27ezcOFCDhw4YKhwhRCi2AgJCWHQoEFA+jz9Wq02T72k\nkZGRmJmZPdeUvYGBgXz33XfPHeOyZcto0aLFc3/O1L311ltMmzYt2/1mZmZcvny5CCPKH4N1++/a\ntavAdTg7OwPg6OhIr169OHLkSLZfppCQEP3fgYGBBAYGFrh9IYQwRU8/t+/h4cGDBw8Kpd6QkBAi\nIiJYvnx5hraK8zwBQ4cOxd3dnalTpxZKfYsXLy6Uep5XWFgYYWFhhVafUe75Py27X6uJiYnodDq0\nWi0JCQns3LmTSZMmZVvP08lfCCFKMhkLVfo8e1E7efLkAtVnlHv+GzduxN3dnfDwcLp06UKnTp2A\n9LWku3TpAqQvSdmiRQv8/f1p3LgxXbt2pX379sYIVwghjGL27Nm4ublhbW2Nr68ve/bsyVTm2a78\nK1eu0LJlS6ytrQkKCmLkyJH6WwRPrFixAk9PTxwdHZkxYwYAoaGhzJw5kzVr1qDVaqlfv36GzyQn\nJ2NnZ8epU6f07/3zzz9UqFAhT8v+njt3jqCgIOzt7fH19WXdunX6fdu2baNOnTpYW1vj5ubG3Llz\nAbh58yZdu3bF1tYWe3t7WrZsme0Pn9GjR+Pk5ISNjQ1+fn6cPn2ab775hlWrVjFnzhy0Wi09evQA\nMnfNDx06lAkTJgDpV9hubm7MnDkTR0dHvL29WbVqVZZlAT755BNcXFxwc3Nj6dKluZ4HU2GUK/9e\nvXrRq1evTO+7uLiwdetWAHx8fDh+/Hie61QKinHPlBDCRGkma1CTVKFt59X58+dZuHAhR48epUqV\nKkRFRZGamprr54KDg2nRogV79uzh8OHDdO7cWZ/0njh48CAXLlzg/PnzNGrUiD59+tCxY0c+/PBD\nIiIi+PHHHzPVa2lpyYABA1ixYgWzZs0CYPXq1bRr1w57e/scY0pISCAoKIhp06axY8cOTp48qR8X\n5uvry2uvvcbPP/9Ms2bNuHfvnj4xz507F3d3d27evAlAeHh4lrcgduzYwYEDB7h48SLW1tacP38e\nGxsbRowYwaFDh3B3d2fKlCnZxvfsrY0bN25w69YtYmJiOHToEJ07d6Zhw4ZUr149Q9nQ0FDmzp3L\nnj178PLy4vXXX8/xPJgSkx7t/zwk8QshShJzc3OSkpI4ffo0KSkpeHh44OPjk+NnoqKiOHr0KFOm\nTMHCwoJmzZrRvXv3TFfLkyZNomzZsvj5+VGvXj1OnDgBpN9OyOmWwuDBg1m9erV+e/ny5Zl6FbKy\nZcsWvL29GTJkCGZmZvj7+9O7d2/Wrl0LpP+wOH36NPfv38fGxkbf62BpaUlsbCyRkZGYm5vTrFmz\nLOu3tLTkwYMHnD17lrS0NGrWrEmVKlX0+/Nym+TZMlOnTqVMmTK0bNmSLl26sGbNmkyfWbt2La++\n+iq1a9fGysqqwF3xRanEJH+5ByaEMIRnr9oLup1X1apVY968eYSEhODk5MSAAQOIjY3N8TMxMTHY\n2dlRrlw5/Xvu7u6Zyj2dGK2srHj48GGeYmrcuDHly5cnLCyMc+fOERERQffu3XP93NWrVzl8+DC2\ntrb616pVq7hx4wYA69evZ9u2bXh5eREYGEh4eDgAY8eOpVq1arRv356qVasye/bsLOtv3bo1b7/9\nNiNHjsTJyYk33nijQIMgbW1tKV++vH7b09Mzy3MfGxub4fx6eHjku82iVmKSf9S9KGOHIIQQhWrA\ngAEcOHCAq1evotFoeP/993Ms7+zszO3bt3n06JH+vaiovP/bmJdR/UOGDGHFihUsX76cvn37Ymlp\nmetnPDw8aNWqFXfu3NG/Hjx4wMKFCwEICAjgl19+IT4+np49e/Lyyy8DULFiRT799FMiIiL49ddf\n+eyzz7Ic9wAwatQojh49ypkzZ7hw4QKffPJJtsdkZWVFYmKifjs2NjZDuTt37mTYf/XqVVxcXDLV\n4+zsnOH8Ps+5NrYSk/w9K3kaOwQhhCg0Fy5cYM+ePSQlJVG2bFnKlSuHubl5jp/x9PQkICCAkJAQ\nUlJSOHToEFu2bMnzo3pVqlQhMjIyU0/q09sDBw5kw4YNrFy5ksGDB+ep3i5dunDhwgVWrFhBSkoK\nKSkp/Pnnn5w7d46UlBRWrlzJvXv3MDc3R6vV6o9zy5YtXLp0CaUU1tbWmJubZ3kOjh49yuHDh0lJ\nScHKyirDuXJycsr03L2/vz8rV65Ep9MRGhrK/v37M9U5adIkUlJSOHDgAFu3bqVv3776c/HkfLz8\n8sssW7aMs2fPkpiYKN3+QgghCiYpKYnx48fj6OiIs7MzN2/eZObMmUDmAWpP/71y5UoOHTqEvb09\nEyZMoF+/fhmuznP6IfAkwdnb2xMQEJDlZ9zd3XnxxRcxMzOjefPm2db1dIxarZadO3fy008/4erq\nirOzM+PHjyc5ORlIf/rA29sbGxsbvvnmG1auXAnApUuXCAoKQqvV0rRpU0aOHEmrVq0ytXX//n1G\njBiBnZ0dXl5eODg4MHbsWABee+01zpw5g62tLb179wbgiy++YPPmzfrbD88OQK9SpQq2tra4uLgw\naNAgvv76a2rUqJHpuDp27Mi7775LmzZtqFGjBm3bti02cyJoVAm4Wa7RaEhKeISlVbncCwshxFMK\nui66qevXrx+1a9fOcZ6U5/Xaa6/h6uqa4wj64iosLIxBgwYRHR1t7FCA7L+fBf3elpgr/1nLNhg7\nBCGEMLqjR48SERFBWloa27dv59dff6Vnz56FVn9kZCQbNmzgtddeK7Q6RdErMcm/TnLJm0NaCCGe\nV1xcHK1bt0ar1TJ69Gi++uor6tWrVyh1T5gwgbp16zJu3Dg8PUvuOKvi0nVfECWm23/1hNP0n1Lb\n2KEIIYqZkt7tL4o36fbPRdKdu8YOQQghhCgWSkzyP35vibFDEEIIIYqFEpP8/R8W3oAWIYQQoiQr\nMcl/SJ+8TU8phBBClHYlJvmn3Ltj7BCEEEKIYqHEJP9Vh742dghCCFFshISE5GlFvqJgZmaWaQre\n4kSr1RIZGWnsMJ5LiUn+Q6r2MXYIQghRbJTkZ9nDwsKyXM3QUB48eICXl1eRtVcYSkzyJ49LUgoh\nREmj0+mMHUK+pKamGjuEUqvEJP+EuHhjhyCEEIUqOjqa3r17U7lyZRwcHBg1ahQAy5Yto1mzZowZ\nMwYHBwdCQkIydeNHRkZiZmZGWloaAFeuXKFVq1ZYW1vTvn17bt68maGt8PBwmjZtiq2tLf7+/uzb\nty/buGJiYujTpw+VK1fGx8eHBQsW6PcdOXKEl156Sb8wzqhRo0hJSdHvNzMzY9GiRVSvXp2aNWtm\nqPfPP/+kSpUqGSav2bBhA/7+/lnGsW3bNurUqYO1tTVubm589tlnJCYm0qlTJ2JiYtBqtVhbWxMX\nF0dSUhLvvvsurq6uuLq6Mnr0aP3CQmFhYbi5uTFz5kwcHR3x9vZm1apV+naGDh3Km2++Sfv27bG2\ntiYwMDDD8r1P37YYOnQoI0eOpGvXrlhbW9OkSZMMtzR27txJzZo1qVSpkn6hou+++y7bc20wqgQA\n1IZq5Y0dhhCiGDLVfwZTU1OVn5+fGjNmjEpMTFSPHz9WBw8eVEop9f333ysLCwv15ZdfKp1Opx49\neqRCQkLUwIED9Z+/cuWK0mg0SqfTKaWUatKkiXrvvfdUcnKy2r9/v9JqtWrQoEFKKaWuXbum7O3t\n1fbt25VSSu3atUvZ29ur+Pj4THHpdDr14osvqqlTp6qUlBR1+fJl5ePjo3bs2KGUUurYsWPq8OHD\nSqfTqcjISFWrVi01b948/ec1Go1q3769unPnjnr8+LH+vYiICKWUUrVr19bHoZRSPXv2VJ999lmW\n56hKlSrq999/V0opdffuXfXXX38ppZQKCwtTbm5uGcpOmDBBvfTSSyo+Pl7Fx8erpk2bqgkTJiil\nlNq7d6+ysLDQn599+/apChUqqPPnzyullBoyZIjSarXqwIEDKikpSb3zzjuqefPmGY7pSfxDhgxR\n9vb26s8//1SpqanqlVdeUf3791dKKRUfH6+sra3Vxo0blU6nU1988YUqU6aM+u6777I8PqWy/34W\n9Htrmt/65wSo/TRXKSnGjkQIUdyYavL/448/lKOjoz55P+37779XHh4eGd6bNGlStsn/6tWrysLC\nQiUmJur3BwcH65P/rFmz9H8/0aFDB/XDDz9kajs8PDxT2zNmzFDDhg3L8jg+//xz1atXL/22RqNR\ne/fuzVDm6eQ5a9Ys9corryillLp165aysrJScXFxWdbt4eGhvv76a3Xv3r0M7+/duzdT8q9atWqG\nHxU7duxQXl5e+vLPnp+XX35ZTZ06VSmVntAHDBig3/fw4UNlbm6url27lin+oUOHquHDh+vLbtu2\nTfn6+iqllPrhhx9U06ZNM8Tl7u5ulORfYrr9HbjJ7dvGjkIIIQpHdHQ0np6emJll/c/08wxoi4mJ\nwdbWlvLly+vf8/T01HevX716lXXr1mFra6t/HTx4kLi4uEx1Xb16VV/fk9fMmTP5559/ALhw4QJd\nu3bF2dkZGxsbPvroI27dupXn2F955RU2b95MYmIia9eupWXLljg5OWVZdv369Wzbtg0vLy8CAwMJ\nDw/P8Rw8vRiRh4cHMTEx+u2szk9sbCyQPjjSzc1Nv69ChQrY2dll+PzTno63fPnyPPy/MWkxMTEZ\n6gEybReVEpT844m5kZJ7QSGEKAbc3d2JiorKdjDfs6P1K1asSGJion776cTt7OzMnTt3Muy/evWq\nvg4PDw8GDRrEnTt39K8HDx4wbty4TO16eHjg7e2doez9+/fZsmULAG+99Ra1a9fm0qVL3Lt3j+nT\np+vHHWQX+9Pc3Nxo0qQJGzZsYMWKFTk+jhgQEMAvv/xCfHw8PXv25OWXX862fhcXlwyP40VFReHi\n4qLfzur8PNmvlCI6Olq/7+HDh9y+fTvD5/PCxcWFa9eu6beVUhm2i5JRkv/YsWOpVasW9erVo3fv\n3ty7dy/LcqGhofj6+lK9enVmz56dY512mlscunTYEOEKIUSRa9y4Mc7OznzwwQckJiby+PFj/vjj\nj2zL+/v7s3//fqKjo7l37x4zZ87U7/P09CQgIIBJkyaRkpLC77//rk/WAAMHDmTz5s3s3LkTnU7H\n48ePCQsL4/r165naadSoEVqtljlz5vDo0SN0Oh2nTp3i6NGjQHpi1Gq1WFlZce7cORYvXvzcxz54\n8GBmz57NqVOn6N27d5ZlUlJSWLlyJffu3cPc3BytVou5uTmQfuV969Yt7t+/ry8/YMAApk2bxs2b\nN7l58yZTpkzJ9MPiyfk5cOAAW7dupW/fvvp927Zt4+DBgyQnJzNhwgReeuklXF1dM8Wlclhpr3Pn\nzvzvf/9j06ZNpKamsnDhwix7V4qCUZJ/+/btOX36NCdOnKBGjRoZvqRP6HQ63n77bUJDQzlz5gyr\nV6/m7Nmz2dZprqDq41qGDFsIIYqMmZkZmzdv5tKlS3h4eODu7s7atWuB9CvbZ69u27VrR79+/fDz\n86Nhw4Z069YtQ5lVq1Zx+PBh7OzsmDJlCkOGDNHvc3NzY9OmTcyYMYPKlSvj4eHB3LlzM12xP4lr\ny5YtHD9+HB8fHxwdHRkxYoQ+0X766aesWrUKa2trRowYQf/+/TPEkdVV+bPv9e7dm6ioKHr16kW5\ncuWyPUcrVqzA29sbGxsbvvnmG1auXAmAr68vAwYMwMfHBzs7O+Li4vj4448JCAjAz88PPz8/AgIC\n+Pjjj/V1ValSRf+EwqBBg/j666+pUaOGPr7g4GAmT56Mvb09f//9NytWrMgy/qz+2zzZdnBwYN26\ndYwbNw4HBwfOnj1LQEAAZcuWzfYYDUWjcvqZUgQ2btzI+vXrM5xIgEOHDjF58mRCQ0MBmDVrFgAf\nfPBBpjo0Gg0K4Nw5eObRESGEyElB10UXhlG9enW+/vpr2rRpY/C2wsLCGDRoUIau/acNGzYMNzc3\npk6dWqjtpqWl4e7uzqpVq2jVqlWWZbL7fhb0e2v0e/5Lly6lc+fOmd6/fv16hkEhbm5uWXZBPS0x\nJirH/UIIIUzfhg0b0Gg0RZL486Iwfxzu3LmTu3fvkpSUxIwZMwBo0qRJodWfVxaGqjgoKCjLexkz\nZsygW7duAEyfPh1LS0uCg4MzlcvP1JO/HPyO4NZBzx+sEEIIkxAYGMi5c+dYvnx5kbabU87Jqis/\nvw4dOkRwcDDJycnUqVOHX375pXR1+y9btowlS5awe/fuLO/phIeHExISou/2nzlzJmZmZrz//vuZ\nymo0GiYBdOsGL75IYGAggYGBhj0AIUSJkGP36bvvwvHjBW/E3x/mzSt4PaLUefL9DAsLIywsTP/+\n5MmTC9QjYZTkHxoaynvvvce+fftwcHDIskxqaio1a9Zk9+7duLi40KhRI1avXk2tWpkH9env+c+a\nBVn8OBBCiOxI8hemzFD3/A3W7Z+TUaNGkZycTFBQehf9Sy+9xKJFi4iJiWH48OFs3boVCwsLvvzy\nSzp06IBOp+O1117LMvE/ocqV486VaOyK6iCEECWfJOwcDR06FHd390IfCPesYcOGsWnTJmrUqJHj\nRD6FYebMmVy+fJklS5YYtB1jM/po/8Kg0WiIsoY9Lg4MOSsL/Agh8s6UR/sPHDiQ3bt3k5CQgIOD\nA6+99hofffSRscPSGzZsGO7u7kyZMsVgbRw4cIDg4GAuXryY42N/+ZHbKH9TUGJH+xeWm/f9cbrY\nCBP9/7AQQjy38ePHc+XKFe7fv8/27dtZsGCBfhxUaXH16lW8vLwKPfGXdiUm+ceZu+Gsu8bdu8aO\nRAghCkedOnUyJD0LCwsqV66cbfmlS5dSu3Zt7Ozs6NixY6ZlZxcsWEDVqlVxdHRk3Lhx+itHpRTT\npk3Dy8sLJycnhgwZkmF2vN9//12/3K+Hhwc//vijft/t27ezXb529OjRODk5YWNjg5+fH6dPn84y\n7piYGLp37469vT3Vq1fn22+/BeC7775j+PDhHDp0CK1Wy+TJkzN99snyxqNGjaJSpUrUqlWLPXv2\n6Pd///331K5dG2tra6pWrco333wDQEJCQqalf2NjYzMtjZzTUseBgYFMnDiR5s2bY21tTYcOHTKs\nY/Dsefvhhx8ASEpK4r///S+enp5UqVKFt956i8ePH2f739UgCrQskIkA1KpKb6pbVFLhxxKMHY4Q\nohgx9X8G33rrLWVlZaXMzc3V4sWLsy33yy+/qGrVqqlz584pnU6npk2blmEFOY1Go9q0aaPu3Lmj\noqKiVI0aNdS3336rlFLqu+++U9WqVVNXrlxRDx8+VL1799av8hcZGam0Wq366aefVGpqqrp165Y6\nfvy4Uirn5WtDQ0NVgwYN9CvunTt3TsXGxmYZe4sWLdTIkSNVUlKSOn78uHJ0dFR79uxRSim1bNmy\nDMvnPuvJ8sbz5s1Tqampas2aNcrGxkbdvn1bKaXU1q1b1eXLl5VSSu3bt09ZWVnluPTv00sjZ7fU\n8c2bN5VSSrVq1UpVq1ZNXbx4UT169EgFBgaqDz74INfz9u6776oePXqoO3fuqAcPHqhu3bqp8ePH\nZ3l82X0/C/q9Ne1vfR4BakrTmkqBmvt9qLHDEUIUI6ae/JVSKi0tTe3du1fZ29urw4cPZ1mmY8eO\nGZaG1el0ysrKSkVFRSml0pP/jh079PsXLVqk2rZtq5RSqk2bNhl+WJw/f16VKVNGpaamqhkzZqje\nvXtn2WZOy9fu3r1b1ahRQ4WHh2e5LPETUVFRytzcXD18+FD/3vjx49XQoUOVUunJPbfk7+LikuG9\nRo0aqeXLl2dZvmfPnuqLL75QSmW99O/TSyPnttRxYGCgmj59un7fokWLVMeOHZVSKtvzlpaWpipU\nqKBfAlip9OWbvb29s4zXUMm/xHT7O5dNHwRT+46HkSMRQojCpdFoCAwMpG/fvqxevTrLMlevXuWd\nd97RL7Nrb28PkGFm1KdnTX16SdvY2NhMy92mpqZy48YNrl27ho+PT7axZbd8bZs2bXj77bcZOXIk\nTk5OvPHGGzx48CDT52NiYrCzs6NChQoZ2s9tRtenPbvAztPL8W7fvp0mTZpgb2+Pra0t27Zty7TE\ncHbystRxlSpVsjz+6OjoLM9bfHw8iYmJNGjQQF9np06duHnzZp6PtzCUmOT/6sT0L27HOqY7alMI\nIQoiJSUlQ5J8moeHB998802GpXYTEhIyTB379BiAqKgofdLMarlbCwsLqlSpgru7OxEREfmKd9So\nURw9epQzZ85w4cIFPvnkk0xlXFxcuH37tj5pPmn/eda5f/aHwpPleJOSkujTpw/jxo3jn3/+4c6d\nO3Tu3Fk/1iG3WfueZ6njrD6b1XlzcHCgfPnynDlzRl/n3bt3M4yxKAolJvlrPNN/0d46XwgTcggh\nhJHFx8fz008/kZCQgE6nY8eOHaxbt44ePXpkWf7NN99kxowZnDlzBoB79+6xbt26DGU+/fRT7t69\nS3R0NPPnz6dfv35A+nK3n3/+OZGRkTx8+JAPP/yQ/v37Y2ZmRnBwML/99hvr1q0jNTWVW7duceLE\nCSDnOe+PHj3K4cOHSUlJwcrKinLlyumX3H2au7s7TZs2Zfz48SQlJXHy5EmWLl3KwIED83yu/vnn\nH+bPn09KSgrr1q3j3LlzdO7cmeTkZJKTk3FwcMDMzIzt27ezc+dO/eeyWvr3aXlZ6ji7c5DdeTMz\nM2P48OG8++67xMenP5p+/fr1DHEVhRKT/PcmXyBNA78fzLpLTAghihONRsNXX32Fm5sb9vb2TJgw\ngeXLl9OwYcMsy/fs2ZP333+f/v37Y2NjQ926ddmxY0eGMj169KBBgwbUr1+frl278uqrrwLw6quv\nMmjQIFpUJIJ8AAAgAElEQVS2bImPjw9WVlYsWLAASL+C3bZtG3PnzsXe3p769etz8uRJfYzZLV97\n//59RowYgZ2dHV5eXjg4ODB27NgsY1+9ejWRkZG4uLjQu3dvpkyZol/UJy/z6jdu3JiLFy/i6OjI\nhAkTWL9+Pba2tmi1WubPn8/LL7+MnZ0dq1evzvDj6dmlf2NjYzO0l91Sx08n/OyW883pvM2ePZtq\n1arRpEkTbGxsCAoK4sKFCzkeY2ErMZP8pKWloXF1hY4dYelSY4ckhCgmTHmSn8JkZmbGpUuXcrx/\nXxwtW7aM7777jgMHDhg7FIOQSX5yodFowMMDTHimJiGEEMIUlJjkD5Dk4sbDcxf4506CsUMRQgiT\nUlhL0pqawlxutzQpMd3+Sik+aejF6GNXWb7hGMN6vmjssIQQxUBp6fYXxZN0++eBTdkQLBRY/a+i\nsUMRQgghTFaJSv5l69QA4OHfF40ciRBCCGG6SlTyt25YFYBHV/bkUlIIIYQovSyMHUBhSq0ex52y\nGsonbgPmGjscIUQxIQPGRGlTogb8JSTABdvGoNXif/M35P/PQgghSqKCDvjL05X/9evXiYyMRKfT\noZRCo9HQsmXLfDdqKBUqQP2Xq8P+/SCJXwghhMhSrsn//fffZ82aNdSuXTvDvMymmPwBHlX3ovzK\nlVy5egJvz3rGDkcIIYQwObkm/40bN3L+/HnKli1bFPEU2HrziwwEYsN3SfIXQgghspDraP+qVauS\nnJxcFLEUioEDZwPQ9K7WyJEIIYQQpinXK//y5cvj7+9P27Zt9Vf/Go2G+fPnGzy4fPH0BK0W/m/1\nJCGEEEJklGvy7969O927d9c/CvNkwF9BjB07li1btmBpaUnVqlX5/vvvsbGxyVTOy8sLa2trzM3N\nKVOmDEeOHMm9co2GW15Vub1zN5UfJmFTsXjcrhBCCCGKSp4e9UtKStKvNezr60uZMmUK1OiuXbto\n27YtZmZmfPDBBwDMmjUrUzlvb2+OHTuGnZ1djvU9+8jDt7U96XMplr/3XaPNS5ULFKsQQghhagw+\nt39YWBg1atRg5MiRjBw5kurVq7Nv3758NwgQFBSEmVl6040bN+batWvZls3Pwany47FNSeHmgcR8\nxyiEEEKUVLkm/zFjxrBz507279/P/v372blzJ6NHjy60AJYuXUrnzp2z3KfRaGjXrh0BAQEsWbIk\nz3WWbRYAwL3f/iyUGIUQQoiSJNd7/qmpqdSsWVO/XaNGDVJTU3OtOCgoiLi4uEzvz5gxg27dugEw\nffp0LC0tCQ4OzrKOgwcP4uzsTHx8PEFBQfj6+tKiRYtc23buUpOkLy3QRS4B+uZaXgghhChNck3+\nDRo04PXXX2fgwIEopVi5ciUBAQG5Vrxr164c9y9btoxt27axe/fubMs4OzsD4OjoSK9evThy5Ei2\nyT8kJET/9wv1G3HCrjw14y+QlATFZIoCIYQQIkthYWGEhYUVWn25Dvh7/PgxCxcu5ODBgwC0aNGC\nf//73wWa9Cc0NJT33nuPffv24eDgkGWZxMREdDodWq2WhIQE2rdvz6RJk2jfvn3mg8hi4MNvNUfS\n/MqPPIq5i62DeabPCCGEEMVVQQf8GWVhn+rVq5OcnKwfxf/SSy+xaNEiYmJiGD58OFu3buXy5cv0\n7t0bSL/18MorrzB+/Pgs68vyJPzwAwwdCqdOQZ06hjwcIYQQokgZLPn37duXdevW8cILL2R6rl+j\n0XDShCbRyeok3D55GLt6Tdj7QX9az1xtpMiEEEKIwmewVf2++OILALZu3ZqpgeKw9vUtF1vMK5XD\n+9R1Y4cihBBCmJRsH/VzcXEBYNGiRXh5eWV4LVq0qMgCzK/qDjWwadsFr1PRxg5FCCGEMCm5Pue/\nc+fOTO9t27bNIMEUuhYtIDIScphESAghhChtsk3+ixcvpm7dupw/f566devqX15eXvj5+RVljPn2\nt3NVAGa8+66RIxFCCCFMR7b3/IODg+nUqRMffPABs2fP1t/312q12NvbF1mABRHl5IOHRXm89iSj\nFBSDoQpCCCGEweX6qN+hQ4eoU6cO1tbWANy/f5+zZ8/SuHHjIgkwL7Ib9agUbCnfl3pJh7l34ip1\n/ST7CyGEKP4MvrDPW2+9RcWKFfXbFSpU4M0338x3g0VJo4Fb/m3xIJojKy8aOxwhhBDCJOSa/AH9\nCnwA5ubm6HQ6gwVU6Pqkr0sQcWiAkQMRQgghTEOuyd/b25v58+eTkpJCcnIyX3zxBT4+PkURW6Fo\nOsCPi2Ur0yK8Eomywq8QQgiRe/L/6quvOHjwIK6urri5uREeHs4333xTFLEVihpu9qS1HkAHsz8o\nryT7CyGEEEaZ27+w5TrwYedO6NCBlE0bKdO9Z9EFJoQQQhiAwRf2+eeff1iyZAmRkZGkpqbqG126\ndGm+Gy1suZ2Ea/ER2LrXYEdje3rv+6cIIxNCCCEKn8Hm9n+iR48etGzZkqCgIP3Av+Iwt//TnO29\niG/TjF5/X4S0NDDL0zhHIYQQokTK9crf39+f48ePF1U8+ZKnX0CrV0NwMBw8CE2bFk1gQgghhAEY\n/Dn/rl27snXr1nw3YDK6dEFZWnL6s4WkpRX7YQ5CCCFEvuV65V+xYkUSExOxtLSkTJky6R/SaLh/\n/36RBJgXef0FtM2nEi/EJxCx6xqtmzgVQWRCCCFE4TP4lf/Dhw9JS0vj8ePHPHjwgAcPHphU4n8e\nUT4z8HiYyqXFMtufEEKI0ivXAX/79+/P8v2WLVsWejCG5vvOIB7tfg+rTatRqrks9COEEKJUyrXb\nv2vXrvrR/Y8fP+bIkSM0aNCAPXv2FEmAeZHX7o/UVNii/RfNUnayZ8s++nWsXwTRCSGEEIXL4I/6\nbdmyJcN2dHQ077zzTr4bNCYLC9jfzZye6x5wecFukOQvhBCiFHruB97d3Nw4e/asIWIpEq99uJxb\nFpUJvvGHsUMRQgghjCLXK/9Ro0bp/05LS+P48eM0aNDAoEEZUh1/S/jPQOznz4cbN8BJRv0LIYQo\nXXK98g8ICNC/mjZtypw5c1ixYkWBGp0wYQL16tXD39+ftm3bEh0dnWW50NBQfH19qV69OrNnzy5Q\nm09LGDwAUlNZ/373QqtTCCGEKC6yHfDXtm1bdu/ezbhx45gzZ06hNvrgwQO0Wi0ACxYs4MSJE3z7\n7bcZyuh0OmrWrMlvv/2Gq6srDRs2ZPXq1dSqVSvzQTznwId7j+9xo0FN3B6VwerSVZnuVwghRLFi\nsOf8Y2Nj+eOPP/j111/566+/Mr0K4knih/R5BBwcHDKVOXLkCNWqVcPLy4syZcrQv39/Nm3aVKB2\nn7ApZ0ON8Z9ideUa7N5dKHUKIYQQxUW29/wnT57MlClTuH79Ou+9916m/Xv37i1Qwx999BHLly/H\nysqK8PDwTPuvX7+Ou7u7ftvNzY3Dhw8XqM0M+vaFMWNInvcFiS+1oFLFcoVXtxBCCGHCsr3y79u3\nL6GhoYwdO5a9e/dmeuUmKCiIunXrZnpt3rwZgOnTpxMVFcXQoUMZPXp0ps8bfOXAsmXZ7N8I8+1b\nmTPzR8O2JYQQQpiQXEf7T5w4MV8V79q1K0/lgoOD6dy5c6b3XV1dMwwEjI6Oxs3NLdt6QkJC9H8H\nBgYSGBiYa9tRnd8mbdcOaiw+S9pUufUvhBDCNIWFhREWFlZo9eU6w58hXLx4kerVqwPpA/6OHDnC\n8uXLM5RJTU2lZs2a7N69GxcXFxo1alRoA/7+fxuwyWYQ7RN/IXzdNYL+ZZO/AxJCCCGKkMEX9jGE\n8ePHU7duXfz9/QkLC2Pu3LkAxMTE0KVLFwAsLCz48ssv6dChA7Vr16Zfv35ZJv6CsLCAB6+PQctD\nTkx9rVDrFkIIIUxVrlf+ly5dws3NjXLlyrF3717+97//MXjwYCpVqlRUMeaqIL+A7tzV8Wc9LX43\nFOc2xxEYJFf/QgghTJvBr/z79OmDhYUFly5d4o033iA6Oprg4OB8N2hqbCuZE9nqZ6okPcZn90pj\nhyOEEEIYXK5X/vXr1+fvv/9mzpw5lC9fnlGjRunfMxUF/QWUplOYtWwO0dFw6RJYWhZidEIIIUTh\nMviVv6WlJatWreLHH3+ka9euKKVISUnJd4OmyMxcw/1x70J0ND+8G2jscIQQQgiDyjX5L126lPDw\ncD766CO8vb2JjIxk0KBBRRFbkUps05yIWlV45ddIePzY2OEIIYQQBvNcj/rdvn2ba9eu4efnZ8iY\nnltBuz/09u6FNm3g88/h3XcLXp8QQghhAAbv9m/VqhX379/n9u3bNGjQgNdffz3LGflKhNatoV07\nUiZPYVPoH8aORgghhDCIXJP/vXv3sLa2ZsOGDQwePJgjR47w22+/FUVsRrGq478oc/cOUe9MIS3N\n2NEIIYQQhS/X5K/T6YiNjWXt2rX6CXgMPu++EbV/fRAbKvbg9Qv7+PmzKGOHI4QQQhS6XJP/xIkT\n6dChA1WrVqVRo0ZERETop+YtiRxsrGDqfAAsJozn/v0in/1YCCGEMCijzO1f2AptwN//SUuDpd5j\neD3qc/49+DUW/fBtodUthBBCFJTBB/xFR0fTq1cvHB0dcXR0pE+fPly7di3fDRYHZmZQdkEPostW\n5I2VR7h/O9XYIQkhhBCFJtfkP2zYMLp3705MTAwxMTF069aNYcOGFUVsRjWoeyuO91tKPd3/sF65\n2NjhCCGEEIUm127/evXqceLEiVzfM6bC7vbXUwo6dUJ38Hf2bltIuxZDCr8NIYQQ4jkZvNvf3t6e\n5cuXo9PpSE1NZcWKFTg4OOS7wWJFo+Hc5FEkP06gesgCY0cjhBBCFIo8Te+7du1aqlSpgrOzM+vW\nreP7778vithMQs1GndFN/BjPPcfg55+NHY4QQghRYDLaPy9SUqBJE9S1a5zY9Bv+Teoari0hhBAi\nFwXNe9km/1GjRuXY6Pz58/PdaGEzePIHbh4IxzqwKb94Vqb14TgcHQ3anBBCCJGtguY9i+x2NGjQ\nIMuZ/JRSJXqGv+xYN2nIghe68N7JLczttJoxfw6gFJ4GIYQQJYB0+z+HqxGp3KjZghq6s2yfdYIB\n73savE0hhBDiWQYf7S/+P8+qFsTOWYG5JgXXL+oQfjzG2CEJIYQQz02S/3PqProqU4Oa0jI2gX9e\nW2LscIQQQojnJt3++XDntiKy1WD8T69Es3s3tG5dZG0LIYQQBu/2P3/+PG3btqVOnToAnDx5kmnT\npuW7QYAJEyZQr149/P39adu2LdHR0VmW8/Lyws/Pj/r169OoUaMCtVmYbO001D+0GGrUIOFfPfhl\nr0z/K4QQovjI9cq/ZcuWfPLJJ7z55pv8/fffKKV44YUXOH36dL4bffDgAVqtFoAFCxZw4sQJvv02\n88p53t7eHDt2DDs7u5wPooiv/J/4ffvXNOj5b3jxRcrv/wPKlCnyGIQQQpQ+Br/yT0xMpHHjxhka\nLFPAJPck8QM8fPgwx+mCTfmuRLOOIyjz3feUDz8K771n7HCEEEKIPMn2Of8nHB0duXTpkn77559/\nxtnZucANf/TRRyxfvhwrKyvCw8OzLKPRaGjXrh3m5ua88cYbDB8+vMDtFiaNRoPFwMHw13H4/HNm\nXb3KmJ83YFnG3NihCSGEENnKtds/IiKCESNG8Mcff2Bra4u3tzcrV67Ey8srx4qDgoKIi4vL9P6M\nGTPo1q2bfnvWrFmcP38+y/UCYmNjcXZ2Jj4+nqCgIBYsWECLFi0yH4SRuv2fSEtOYV+1yjS7dp/F\nfXfzzppAo8UihBCi5DPY9L7PSkhIIC0tLUOXfWGIioqic+fOnDp1KsdykydPpmLFiryXRfe6RqNh\n0qRJ+u3AwEACAwMLNc7cbFl/lpr/6kEl7rB7xhH6j/cu0vaFEEKUXGFhYYSFhem3J0+ebNjk//jx\nY9avX09kZCQ6nU4/ve/EiRPz3ejFixepXr06kD7g78iRIyxfvjxDmcTERHQ6HVqtloSEBNq3b8+k\nSZNo37595oMw8pX/E+tnXsD2wzd51e1tRs2z5b0+8gigEEKIwmewuf2f6NGjB5UqVaJBgwaUK1cu\n3w09bfz48Zw/fx5zc3OqVq3K4sXpj8rFxMQwfPhwtm7dSlxcHL179wYgNTWVV155JcvEb0r6jK/B\n4HtzuJrWiQmTfqJHPahWzdhRCSGEEBnleuX/wgsv5Nolb2ymcuUPkKpLo8fQy3jbVOOLL8Bcxv4J\nIYQoZAZ/1K9p06acPHky3w2UNhbmZmz6vhoLFoCZmeKTg5/wT8I/xg5LCCGE0Mv2yr9u3boA6HQ6\nLl68iLe3N2XLlk3/kEZjUj8ITOnK/2nT9k9j68WtbBmwBXsre2OHI4QQooQw2Gj/yMjIbBvQaDR4\neprOcrammvzjE+KxKmNFBcsKxg5FCCFECWKwbn8vLy+8vLz4+OOP9X8//Z7InWMFR33iP3XlBjXG\nvsrNO0lGjkoIIURpl+s9/2cH+6WmpnLs2DGDBVRSBc4fwsU/vejdoywJCcaORgghRGmWbfKfMWMG\nWq2W//3vf2i1Wv2rcuXKdO/evShjLBF2jfgJ10sTOXAAunaFhATTu00hhBCidMj1Ub8PPviAWbNm\nFVU8+WKq9/yfdfEitGoFsdrN2Lf9kYiZ67CxMXZUQgghipsim97XlBWX5A9w6mwKLy58iZSNi5n/\nQUNGjTJ2REIIIYobSf4Ur+QPcPmKjlUrzfnoI0hTOm49ukXlCpWNHZYQQohiwmDJ/8qVK3h7F4/F\naYpb8n9CKcXwzcNJTUtlWc9lxg5HCCFEMWGwR/3+9a9/AdCmTZt8Vy5ylpCSQFnzsnzZ+UtjhyKE\nEKIUyfbK39/fn759+7J48WLGjBmT4ReGRqNhzJgxRRZkborrlf+zwk5cZsfxk8wc0tPYoQghhDBh\nBrvy/+mnnzA3N0en0/HgwQMePHjAw4cP9X+LwhV3+yEdfuzMrIUxfP45lIDfMkIIIUxUrgP+tm3b\nRufOnYsqnnwpCVf+aWmK4TP3sPTjtgCMGgVz5yrKlNEYOTIhhBCmxuCj/e/evcvkyZPZv38/AIGB\ngUycOBEbE3pAvSQk/ydWr4ahQyG51o+4vnSQk9O/xs7O2FEJIYQwJQZf0vfVV1/F2tqadevWsXbt\nWrRaLcOGDct3gyJnAwbA1h2PMW85h+vr/8PWrcaOSAghREmT65V/vXr1OHHiRK7vGVNJuvJ/4nJk\nKj+vtWDcOEhMSeT3qN9pX7W9scMSQghhAgx+5V++fHkOHDig3/7999+xsrLKd4Mib3y80hO/UorX\nf32d5SeXGzskIYQQJYRFbgW++uorBg8ezL179wCwtbXlhx9+MHhg4v8L8gmi/wv99duPk1MpZ5nr\nfzohhBAiS3me3vdJ8jelgX5PlMRu/+xsPHiKfusGsLnLX3QIKmPscIQQQhiBwbv9n7CxsTHJxF/a\nfLhpASlhY+nUoQyTJ4NOZ+yIhBBCFDeysE8xk5KaxrSpZkydmj4eoNrgz9g4aQgv+DgYOzQhhBBF\npMiu/A1h7ty5mJmZcfv27Sz3h4aG4uvrS/Xq1Zk9e3YRR2eayliYMXky7NwJ2hY/cqniUroEVSQ5\n2diRCSGEKC5yHTW2fv16NJqMs8zZ2NhQt25dKlfO/zK00dHR7Nq1C09Pzyz363Q63n77bX777Tdc\nXV1p2LAh3bt3p1atWvlusyRp1w72e7dn1H8bMeyjclhawoVbF3Cq4IRNObk9I4QQInu5Jv+lS5dy\n6NAhWrduDUBYWBgvvvgiV65cYeLEiQwePDhfDY8ZM4Y5c+bQo0ePLPcfOXKEatWq4eXlBUD//v3Z\ntGmTJP+n+Fd1Zv8GZyB9LoDuq7szvc10+tTuY+TIhBBCmLJcu/1TUlI4e/Ys69evZ/369Zw5cwaN\nRsPhw4fz3RW/adMm3Nzc8PPzy7bM9evXcXd312+7ublx/fr1fLVXkmk06a/bj24TXDdYn/hTdWnc\nvv/IyNEJIYQwRble+UdHR+Pk5KTfrly5MtHR0djb22NpaZnt54KCgoiLi8v0/vTp05k5cyY7d+7U\nv5fVoIVnbzWInLlZuzGx1UT99uAvvmHzyf1sfXUVLVsaMTAhhBAmJ9fk37p1a7p06cLLL7+MUor1\n69cTGBhIQkIClSpVyvZzu3btyvL9U6dOceXKFerVqwfAtWvXaNCgAUeOHMkwhsDV1ZXo6Gj9dnR0\nNG5ubtm2FxISov87MDCQwMDA3A6txEpLU2y9upaH276k1Q/w1lswc6bCxkZ+UAkhRHEUFhZGWFhY\nodWX66N+aWlpbNiwgYMHDwLQrFkz+vTpU2hX5t7e3hw7dgy7Z5auS01NpWbNmuzevRsXFxcaNWrE\n6tWrs7znX5oe9curpCTFzJkaZsyAFLP7WA7rzIrOW+jbLfsfbEIIIYoHgz/qZ2ZmRvPmzWnTpg1t\n2rShZcuWhdol/3RdMTExdOnSBQALCwu+/PJLOnToQO3atenXr58M9nsOZctqCAmBY8fArdcikmN8\n+WxGJeQ3khBCiFyv/NeuXcvYsWNp1aoVAPv37+eTTz6hb9++RRJgXsiVf86SU3R8+sVjOrWtQP36\n8Nmhz6hmV43uNbsbOzQhhBD5UNC8l2vy9/Pz47ffftPfj4+Pj6dt27acPHky340WNkn+eXfj4Q3q\nLKrDsRHH8KyU9RwLQgghTJvBu/2VUjg6Ouq37e3tJdEWY04Vnfhz+J/6xH/qyg1eGDeKuDj5byqE\nEKVFrqP9O3bsSIcOHQgODkYpxZo1a+jUqVNRxCYMxNvWW/9370UfcvGEHb6+GiZMgLffhrJljRic\nEEIIg8u1218pxYYNG/j999/RaDS0aNGCXr16FVV8eSLd/vm3++9LzJ7gxK6tWgBsu89gfN+O/PeV\nF5GpFoQQwjQZ/J5/cSDJv+BCQ+HtkPNEtGyBZtFZLhy3p1o1Y0clhBAiKwZL/hUrVsz2kT6NRsP9\n+/fz3Whhk+RfOB4npzL16zMQ58f06XDx1kUWHFnA/E7zjR2aEEKIp8iVP5L8DeXldS/zovOLfND8\nA2OHIoQQ4imS/JHkbyiX71zGuaIz5cuUB6Dm2NdpXnEY88Y0Q6s1cnBCCFGKGfxRP1F6+dj66BP/\njqMXuJC2naXT6+PjA/PmwcPEVCNHKIQQIj8k+Ys86RBQg40d/+alACtu3oTRnxzF/r8t+eEH6XER\nQojiRpK/yLOeQZU5eBA2bwb7dktJ/nMw586lDwq9dv8aj1MfGzlCIYQQeSH3/EW+pOrSWLsujaC2\nFjg6Qrsf2zG43mAG1xts7NCEEKLEk3v+wigszM0I7p+e+O89vodVGSsGvDAAgLQ0xVuLV3D3QbKR\noxRCCJEVSf6iwGzK2fDrgF8pY14GgLkb9/DVmalUr2bO9Olw+7aRAxRCCJGBJH9R6CqWK0eNy19w\n8x9zPv4YXAK30vr9BURFGTsyIYQQIMlfGMBbXZpxbktHfvsN2reHpEbTCdvizKZN6fsfJj80boBC\nCFHKyYA/YXBr951kx4o6zJ9nTnmrNPwW+7Gs5zICXAKMHZoQQhRLMuBPmLyXW/nx3RJzKlSA8zfP\nY1velgbODQBIfJzCawuWkZAgP96EEKKoSPIXRaqWYy32D92vXzTqgx/Xs/TYD7i7a3j/fWRcgBBC\nFAFJ/qLIPb1apLOtHb43JnHnDsyZA17/+gq/N+fy119GDFAIIUo4Sf7CqMb3bc/Z7YGEh0P/AQrV\ncCH/296Iy5fT9/8d+zeJKYnGDVIIIUoYGfAnTMqJSzfYurYy48ZpSNMk4znPk71D9uLr4Gvs0IQQ\nwmQU6wF/c+fOxczMjNvZzALj5eWFn58f9evXp1GjRkUcnTCGetWc+PBDDRYWcP3+dfrW7qtP/Fdv\n3MX+nU4s+0HHo0dGDlQIIYoxoyX/6Ohodu3ahaenZ7ZlNBoNYWFh/P333xw5cqQIoxOmwNvWm/md\n5uu3xy5fye0Ya4YNNcfVFYaPjmPL4bNGjFAIIYonoyX/MWPGMGfOnFzLSXe+eOLL14bySdBnNGwI\nd+7AtycW0m3yYqZNM3ZkQghRvBgl+W/atAk3Nzf8/PxyLKfRaGjXrh0BAQEsWbKkiKITpqqybQX+\nO8KVI0fg2DF48UUN5c+8QfPm6fv/s/0/bL2w1bhBCiFEMWCwAX9BQUHExcVlen/69OnMmDGDnTt3\nYm1tjbe3N0ePHsXe3j5T2djYWJydnYmPjycoKIgFCxbQokWLzAchA/5KrQcPoEIFSE57jMtcF07/\n+zTOWmcARiz8kZHtelCvpo2RoxRCiMJV0LxX5KP9T506Rdu2bbGysgLg2rVruLq6cuTIESpXrpzt\n5yZPnkzFihV57733Mu3TaDRMmjRJvx0YGEhgYGChxy5M242HN3Cq6ATA0QvXabi0Lnx2jVZNrRg6\nVNGpRyJOthWMHKUQQjy/sLAwwsLC9NuTJ08uXsn/Wd7e3hw7dgw7O7sM7ycmJqLT6dBqtSQkJNC+\nfXsmTZpE+/btM9UhV/7iWWEnLvPh1/s4vmxY+pMBzn+h6TWM/5Q5wbx5xo5OCCEKplg/6gcZZ3uL\niYmhS5cuAMTFxdGiRQv8/f1p3LgxXbt2zTLxC5GVwHo+/LFoGHFx8O234BUYhjrTkyf/Xwm9FMr8\nw/NzrkQIIUooo1/5Fwa58hd5EXE5jTIWZnh4QN91fWnv057hDYYD8PW2P/Cwq0KnJj5GjlIIIXJX\n7O75G4Ikf/G8Tt44iaeNJzbl0gcDake/xMMtk3ihXEeCg6F99zs0qGNr5CiFECJrxb7bXwhj8HPy\n0yf+x8mpeJdpgu3dtpw6BR9+lEbA0rrUahZBFg+sCCFEsSfJX5R65SwtODnnc+Kul2HzZugx5Arm\nDzxJiK6KkxM8TH5I86XNSU1LNXaoQghRKCT5C/F/LC2ha1f45fuqPJz/Ozt2gEYDWy9sxbqsNRZm\nFvLfx9EAABlUSURBVAD8fiqSjiHzOH0a5G6TEKI4knv+QuQiWZdMfEI8rtauAHSePoft4Zdhy1f4\n+kKLXudp2TqZ4LZ1MZOf00KIIlDQvGdRiLEIUSJZmlvqEz9AzxdbUO5aZ/bZwblzcO7EfJYccOdK\neF0mTEhfjdDByoGyFmWNGLUQQmRPrlOEeE4jOr3EhsUvEBcHv/0GTV5wxvlubzp2TN//5tY32XB2\ng77842QZKyCEMC3S7S9EIXjy9VOkEbgskF8H/EqlcpUAsBrdgLqXfuCVoBfo2hV8ZCoBIUQByXP+\nSPIXput4RCz1FzWEz6+CMgezFMr9uyUjyu7h05nlKVPG2BEKIYojec5fCBPmX9WZmA8iWP6jOX37\ngpXvHzxOTmHn1vTEf+PhDV7d9KqxwxRClDKS/IUwMGfHsgwcCGvXQvyxpqzpvYFPPknft/3Sdu4n\n3deX3XjwFJ0mLiI8HHQ6IwUshCjxpNtfCCOKeRDD/aT7+Dr4AtB68mTC/ngAOz/FwQHqd/+D+g0f\nM7pnG6pUMXKwQgiTId3+QhRjLloXfeIH6N8wiFdqvY63N9y8Cbtufcuc78+wYkX6/t2Xd3Mm/oyR\nohVClBSS/IUwIW90bsqKeb5ERMDZs/BKs0Bau3ajQ4f0/dMOTCPybqS+/PQ1O/gtPI60NOPEK4Qo\nnqTbX4hiZHLYZP7b9L9UsKxAWprCcpw3uh+3UVlTm6AgcG6+kzc6NaOaZwVjhyqEMCDp9heiFJkU\nOIkKlumJ/e7Dx9RUPXEtW4t//oGVax7xaVQfqvumcudOevlfz/+KLk1GDgohMpLkL0QxZWddntNz\n5xEdpeHMGZg05ybet9+keYANtrZw5c4VRmwegZkm/f/mdx8+ZuLyrdy9a+TAhRBGJ93+QpQwSqWv\nRngi7gTbLm5jfIvxAHy2cS/vbRuP2dJwXnwRGrX+B48GZ/h350C0WiMHLYR4LrKwjxAiA40m/X/r\nValHvSr19O8rpah2/3WumsPRo3BUtwUu7+LwmkA2bICI2xHcfnSbhq4NjRS5EKKoSLe/EKXEe73b\ncHHN69y5Azt2QM9OWmokD6Bdu/T9P5z4gY3nNurLfxsaztzVR7l1y0gBCyEMRrr9hRAALP5zMfWq\n1KOpe1MAfN4bxJX/1969h0VZ7Qsc/w43QQEvKYigjHIVZYZRBImjIYWmBqbmzkozU3fS2cftpeR4\nqh0+W00ztinmk0etozvLzC6yU0nTUMoSU1DzSooKCt5vqAQM6/wx+hoJ5k5lnOH3eZ6eh/ddy2H9\n+AW/ed9Z71pZ3WH7KDp0gNYJGTxk9GfkY0aaN7fyYIWo52S2vxDirkjukqwVfgCTdxeiWjyCqyvs\n3g2ZF2Yy6e8n+eEHS/u7P77LgTMHrDRaIcSdsErxT01Nxc/PD5PJhMlkIjMzs8Z+mZmZhIaGEhQU\nxIwZM+p4lELUb59OHMOWzLacPw/ffgt92vWnZ1gMsbGW+QNTNk1Bh07rH/XKRF6ZcYxvv4WrV604\ncCHE77LKbf/Jkyfj4eHB+PHja+1jNpsJCQnh66+/xtfXly5duvDRRx/Rvn37m/rKbX8h6lZlVSUL\nti1gdORodDodZy9e5YEZzWHmKahoiJOzotGovjztsoy0aZ64uVl7xELYF5u97f97g87JySEwMBC9\nXo+zszODBw9m5cqVdTQ6IcStODk4kdwlGZ3uxpX/2NbLeOH5hoSHQ6V7ARdcd7L8Aw9cXeHSL5cI\neyeMKmVZh7iqSnGlrMJawxei3rPao37p6eksWbKEyMhI0tLSaNKkSbX2Y8eO0bp1a+3Yz8+PLVu2\n1PUwhRC3oZmnG7NGJ2rHJWd8WJm9Go/OOnQ6yDmWQzO3ZtqCQ6u37iPpgyfovms3MTFgirpKqPEC\nhnaydaEQdeGeFf+EhARKSkpuOj916lSSk5P529/+BsBrr73GhAkTWLRoUbV+v76iEELYlpYPuPHC\n4wbtOE4fx2dPfqYdr92xA3XcxMaNsHEjEPQNxPyDgZe/ZsUKy1bHxy8dJ7JVpBVGL4T9u2fFf926\ndbfVb+TIkSQmJt503tfXl8LCQu24sLAQPz+/Wl8nNTVV+zouLo64uLjbHqsQ4t5ydHDEq5GXdjzn\nz4OZ1O8JfnwONm+GlUdOcvBYPPoIS/vKfSvZVryNhUkLAfhg/Ta27jnN89170aEDOMnyZKKeycrK\nIisr6669nlUm/BUXF+Pj4wPArFmz2Lp1Kx9++GG1PpWVlYSEhLB+/XpatWpFVFSUTPgTwo5VVVme\nEmjUCJbsWIKHiwf92/cHIPrVFHKy3WHTa7i5gX/vTwgOcuK1J/oTKTcHRD1kkxP+UlJSMBgMGI1G\nNm7cyKxZswA4fvw4ffv2BcDJyYm5c+fSq1cvwsLCePLJJ2ss/EII++DgYCn8AM8an9UKP0D3dlE8\n2i6RgADLG4R9Dp+SkXmJHTss7alZqXy298bHCifPXaaqSi4IhKiNrPAnhLApp0/De+s3cmpfKKOH\neBMQAF0XdmXGIzN4SP8QAN7jEinbPIIHmz1O587QNCyXR6PbEdauMTKdSNiDO617UvyFEDav4FwB\nPh4+uDq5AuA6viO/vP8lnNdbOiQbYOV7bF4RSUwMLNq+iMeCH8Pb3dt6gxbiDkjxR4q/EKK6qipF\nQQFs367jx21VvHc1CT7+lCOHGuDmpmj2ZjP2/2W/Ngkx6KXh9HV5i26dH8BkAn9/haOj3CIQ9y8p\n/kjxF0L8PqUs2x2XVZaRtjmNV7q/AsCRE+fRz24N08+DcgSHCnTj/YndWsCGdQ1wclLkleQR0TJC\nHkEW9w2bnPAnhBB17XrddnVy1Qo/QCNXFyaH/ov/meRIr17QPCQfVebJ8cIGODtDcWkxvT7opfU/\nee4yA6bPZcMGZLtjYbPkaVkhRL3WvHFD/vZs3K/OhPHz0a2UjrAcnSg9Qb+QftpVf8aWXXxe8H98\nPukvAPiEHcSlx0zGBr7L2LFgrjKj0+m01QyFuB/J/51CCPEbgW08iLi24JDJx8SCpAVam1cTTx5u\nNJaYGGjYEIp1P3Lk9EnWr7e0byjYQJ+lfbT+Ow+V8P6aPC5cqMsIhLg1ufIXQoh/Q1LXMJK6hgFg\nNsMPPz3EDzs7Yri2LcG+0/sIbR6q9U/715csycrm+T6Lad0afGO+44HAQ7yUMBRZiFRYixR/IYT4\ngxwdIdbYkljjjQ2J/iv6v6gw39ixsFEDN/TmnpS4QmEhFAashdNmOpohLg7m5szlfNl5Xu3+KgA5\n+4pwwJnOId6yJoG4Z6T4CyHEXebs6Kx9PW/0MzDacpfg559hxZZHKTzoTp+HLe3bi7fT1a+r1n/0\n4jRys1viuSuFjh3BM3IVHdo+QHJiVwIC6joSYa+k+AshRB1wdISQEHglJKba+fTe6dWOnRycaXw5\nigsXLZse4fc+mRsGYnzAUvxfXvsy3f27kxhi2RAta8ch9C1aoG/lUVehCDsgE/6EEMKKGrk0opFL\nI+04Z+qbnM/tQUkJrFsHT3fqx6CusURFWdqzj2bT1K2p1n/g+6NpG5eNlxf06AE9JiwgdU4+Neyo\nLoRGFvkRQggbcvTCUbwaeWlLGfuM68fF5XO4ctzf0uE/w2DFMravMWAyQZ+lfZj+yHQM3gYAPt6Y\nR3RgCP6t3GROgQ2TFf6Q4i+EqN+Uskwm3LMH5ufNoXH+C8xLtyxl3Hh6Y46MPaLdLXCc6EvV/26m\nmaM/YWFQGv0KA1q+xLjRTXF3hypVJWsU2AAp/kjxF0KImiilKC4tppVHKwBKr5bjm9Iblqzj4gUH\ncCyHSZ4w/QJXLjbA1VXRYmYLDo89jLuLOwBjFyxnsHEAHdo74SHTCu4bsryvEEKIGul0Oq3wA7i7\nuXBhznrOn3Pg2DHI+LKSZxq/y6v/3QA3N8tSxs6OzlrhLyg+x+yCEcREO+LpCb7+ZTT/ax/G/FWh\nlOUuQeGFQmuFJ+6AXPkLIYTQlFWWafMJtu4vYkh6Oi4bZ3DgAJQ3+QkGDSLwq73k58OR80eIfS+W\novFFgGWTpD+/u4gR7ScQGgpBQeDmZs1o7Ned1j151E8IIYTmeuEH6BLix/65MwDLOgW7f/bnm50f\n0ObaGgUlpSU82PpBrX/mtj2sLVrO2tQJlhPeO2iQ9BJDqtaxcCGcuXKGXSd3EaePq6twRC2k+Ash\nhPhdjo5gCPHAENJZOxftF83yQcu1Y71XcxJbjEeXBHv3ws+u+/nlogenrlratxzbwuwts7Xi//7a\nHP7+6QqS3N4kKAh8212gZZtSuoT64uhYl9HVP1L8hRBC3BW9IoPpFRmsHV8p68/unx/B41qlcXNy\nIyk4SWv/Zs8OCk6cZvbKayfCV0HISgbpPmb5cthStIU9p/Yw3DT82utV4OrihIODPKN4p6T4CyGE\nuCcaujrTpWMz7bhH2x70aNtDOx6f+BhdfeK4HAsHDkD2pV8oPh9FYCdL+6Yjmzh+6bjWf8Q7C1mR\nvYuOR+YRFAQewbl4+ZbybFw32revs7DsghR/IYQQVhER4ENEtf0KLFf4VVWWo1+/UQA4fP4olafa\nkZcHeXlA/KdgdsGlpBuTJ8M/vv8HSikmPGiZc7B80w7On3EmPjwMf39wdkZcI8VfCCHEfcXh2kPo\nka0iq53//u9vcOqvZg4dhPx8+OKAkYvHWtH12r5IP538iRi/G3snTFm9gF1ZQbAlDEdH8Oz9Jr7u\nrXnz2afo3duy/XLjBo3x8fCpq9DuG1Yp/qmpqSxcuJAWLVoA8MYbb/Doo4/e1E+v1+Pp6YmjoyPO\nzs7k5OTU9VCFEELcR1o0d6RFc4iOhiEMqtb29qNvV1udMKhZMM7eUZxuY1kB8VyjHzi3S8/Fi5b2\nKZumkNAugWERwwDoO20mFUc68x++8QQEgLvvUYzBzfD3cbe7pZCtssiPTqdj/Pjx5ObmkpubW2Ph\nv94vKyuL3Nzcel34s7KyrD2Ee8aeYwOJz9ZJfLbFs4GntkBRVlYWn04cw7aVXTlyBK5ehXUvv8Wy\nqT156CFLfx93H8K9w7V/v/nkGtZ9beb112HIEHh8wYu0jV/PymsTEtM2p5FXkqf13334JJXmqjqL\n726y2gp/t7s4gSzeY3+/oL9mz7GBxGfrJD7b9dvYGjSARzq348l+TWjZ0nJuZs+ZdPLppPWZ0Wsq\nb0/sQkoKPPEENHF3pWFZMO3aWdo//OlDKswVWv+Ytwdw9uLVex3KPWG1z/zT09NZsmQJkZGRpKWl\n0aRJk5v66HQ6HnnkERwdHXnhhRcYNWqUFUYqhBCiPvhz75jfnFnBr68/X+v+GqHNQ7VjN11TvJo2\nwhbdsyv/hIQEwsPDb/ovIyOD5ORkCgoKyMvLw8fHhwkTJtT4Gt999x25ubmsWbOGd955h+zs7Hs1\nXCGEEOImOh3a5/2Phz6OR4MbuxudmPUvK43qzll9bf/Dhw+TmJjIrl27btlv8uTJuLu71/hGQWdv\nMzGEEEKI32Fza/sXFxfj42N5tOLzzz8nPDz8pj5XrlzBbDbj4eHB5cuXWbt2La+//nqNryfzAoQQ\nQojbZ5Xin5KSQl5eHjqdjrZt2zJ//nwAjh8/zqhRo1i1ahUlJSUMGDAAgMrKSp555hl69uxpjeEK\nIYQQdsXqt/2FEEIIUbes9qjf3ZCZmUloaChBQUHMmDHD2sO5K/R6PQaDAZPJRFRUFABnz54lISGB\n4OBgevbsyfnz5608ytv3/PPP4+3tXe2jnVvF88YbbxAUFERoaChr1661xpBvW02xpaam4ufnh8lk\nwmQysWbNGq3NlmIDKCwspEePHnTo0IGOHTsyZ84cwH7yV1t89pLDsrIyoqOjiYiIICwsjEmTJgH2\nkb/aYrOX3F1nNpsxmUwkJiYCdzl3ykZVVlaqgIAAVVBQoMrLy5XRaFR79uyx9rDumF6vV2fOnKl2\n7uWXX1YzZsxQSik1ffp0lZKSYo2h/SGbNm1S27dvVx07dtTO1RbP7t27ldFoVOXl5aqgoEAFBAQo\ns9lslXHfjppiS01NVWlpaTf1tbXYlFKquLhY5ebmKqWUunTpkgoODlZ79uyxm/zVFp895fDy5ctK\nKaUqKipUdHS0ys7Otpv81RSbPeVOKaXS0tLU008/rRITE5VSd/dvp81e+efk5BAYGIher8fZ2ZnB\ngwez8voyTDZO/eaTmIyMDIYNsyw/OWzYML744gtrDOsP6datG02bNq12rrZ4Vq5cyVNPPYWzszN6\nvZ7AwMD7emXHmmKDmieg2lpsAC1btiQiIgIAd3d32rdvz7Fjx+wmf7XFB/aTw4YNGwJQXl6O2Wym\nadOmdpO/mmID+8ldUVERq1evZuTIkVpMdzN3Nlv8jx07RuvWrbVjPz8/7RfXll1f2CgyMpIFCxYA\ncOLECby9vQHw9vbmxIkT1hziHastnuPHj+Pn56f1s9WcpqenYzQaGTFihHZbztZjO3z4MLm5uURH\nR9tl/q7H1/XaDjH2ksOqqioiIiLw9vbWPuKwl/zVFBvYT+7GjRvHzJkzcXC4UabvZu5stvjb67P9\nv7ewkU6ns6vYfy8eW4v1dhewAtuJrbS0lIEDBzJ79mw8PDyqtdlD/kpLS3niiSeYPXs27u7udpVD\nBwcH8vLyKCoqYtOmTXzzzTfV2m05f7+NLSsry25y9+WXX+Ll5YXJZKr1UfY7zZ3NFn9fX18KCwu1\n48LCwmrvfGzV9fUPWrRoQf/+/cnJycHb25uSkhLAskaCl5eXNYd4x2qL57c5LSoqwtfX1ypj/KO8\nvLy0X8qRI0dqt95sNbaKigoGDhzI0KFDefzxxwH7yt/1+IYMGaLFZ285BGjcuDF9+/Zl27ZtdpU/\nuBHbjz/+aDe527x5MxkZGbRt25annnqKDRs2MHTo0LuaO5st/pGRkeTn53P48GHKy8v5+OOPSUpK\nsvaw7siVK1e4dOkSgLawUXh4OElJSSxevBiAxYsXa3+kbFVt8SQlJbFs2TLKy8spKCggPz9fe+LB\nVhQXF2tf/3oBK1uMTSnFiBEjCAsLY+zYsdp5e8lfbfHZSw5Pnz6t3fa+evUq69atw2Qy2UX+aovt\nemEE287dtGnTKCwspKCggGXLlhEfH88///nPu5u7ezNHsW6sXr1aBQcHq4CAADVt2jRrD+eOHTp0\nSBmNRmU0GlWHDh20mM6cOaMefvhhFRQUpBISEtS5c+esPNLbN3jwYOXj46OcnZ2Vn5+feu+9924Z\nz9SpU1VAQIAKCQlRmZmZVhz57/ttbIsWLVJDhw5V4eHhymAwqH79+qmSkhKtvy3FppRS2dnZSqfT\nKaPRqCIiIlRERIRas2aN3eSvpvhWr15tNzncuXOnMplMymg0qvDwcPXmm28qpW7998RW4qstNnvJ\n3a9lZWVps/3vZu5kkR8hhBCinrHZ2/5CCCGE+GOk+AshhBD1jBR/IYQQop6R4i+EEELUM1L8hRBC\niHpGir8QQghRz0jxF6IemjRpEllZWXzxxRdMnz69Tr6nXq/n7NmzdfK9hBC3JsVfiHooJyeHrl27\nsnHjRrp3714n31On09W6TrkQom5J8ReiHpk4cSJGo5GtW7cSExPDokWLSE5OZsqUKRw8eJDevXsT\nGRlJ9+7d2b9/PwDPPfcco0ePpkuXLoSEhLBq1SoAysrKGD58OAaDgU6dOpGVlQWA2WzmpZdeIjw8\nHKPRyDvvvKN9//T0dDp37ozBYNBef+PGjZhMJkwmE506daK0tLRufyhC1Ef3ZkFCIcT9auvWrWrM\nmDGqoqJCxcbGaufj4+NVfn6+UkqpH374QcXHxyullBo2bJjq3bu3Ukqp/Px85efnp8rKytRbb72l\nRowYoZRSat++fapNmzaqrKxMzZs3Tw0aNEiZzWallFJnz55VSiml1+vV3LlzlVJKzZs3T40cOVIp\npVRiYqLavHmzUkqpy5cvq8rKynv9IxCi3nOy9psPIUTd2rZtGwaDgb1799K+fXvAsq3t999/z6BB\ng7R+5eXlgOV2/Z/+9CcAAgMDadeuHfv27eO7775jzJgxAISEhODv78+BAwdYv349ycnJ2j7kTZs2\n1V5zwIABAHTq1InPPvsMgNjYWMaNG8czzzzDgAED7uvd1oSwF1L8hagnduzYwXPPPUdRURHNmzfn\nypUrKKW0W/ZNmjQhNzf3tl7r+l7hqpbP8Gs736BBAwAcHR2prKwEICUlhccee4xVq1YRGxvLV199\nRUhIyL8bnhDi3yCf+QtRTxiNRnJzcwkODmbv3r3Ex8ezdu1atm/fjqenJ23btmXFihWApXjv3LlT\n+/qTTz5BKcXBgwc5dOgQoaGhdOvWjaVLlwJw4MABjh49SmhoKAkJCcyfPx+z2QzAuXPnbjmugwcP\n0qFDByZOnEiXLl20uQBCiHtHir8Q9cipU6do1qwZAPv27SM0NFRrW7p0KYsWLSIiIoKOHTuSkZEB\nWK7y27RpQ1RUFH369GH+/Pm4uLjw4osvUlVVhcFgYPDgwSxevBhnZ2dGjhxJmzZtMBgMRERE8NFH\nH9U4lut3D2bPnq1NDnRxcaF37973+KcghJAtfYUQtzR8+HASExO1z+uFELZPrvyFEEKIekau/IUQ\nQoh6Rq78hRBCiHpGir8QQghRz0jxF0IIIeoZKf5CCCFEPSPFXwghhKhnpPgLIYQQ9cz/A7/EAeay\n4xp2AAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110889250>"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Bonus:_ plot the train set and test set accuracies (scores) over epochs, see overfitting."
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Learning rate schemes and \"pure\" SGD"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Can we decrease the learning rate smartly? Why, yes! Check out below with $\\lambda_t = \\frac{\\lambda_0}{1 + \\lambda_0 \\sqrt t}$, where we also get rid of epochs to do pure SGD. Read [SGD tricks](http://research.microsoft.com/pubs/192769/tricks-2012.pdf) for more good stuff (e.g. you should try $\\lambda_t = \\frac{\\lambda_0}{(1 + \\lambda_0 \\alpha t)^p}$ with powers $p=0.5$ and $p=1$). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from random import randint\n",
      "\n",
      "def train_with_pure_SGD(lr, x_train, y_train, init_learning_rate=0.1):\n",
      "    init(lr, x_train, y_train)\n",
      "    old_loss = 1.E31\n",
      "    new_loss = lr.loss()\n",
      "    print(\"loss before training:\", new_loss)\n",
      "    def get_learning_rate(t):\n",
      "        return init_learning_rate / (1 + init_learning_rate*np.sqrt(t))\n",
      "        #return init_learning_rate / (1 + init_learning_rate*t)\n",
      "    learning_rate = get_learning_rate(0)\n",
      "    losses = [new_loss]\n",
      "    gradients = []\n",
      "    # here we do purely stochastic gradient descent\n",
      "    t = 0\n",
      "    while t < 1000:  # here we see a fixed number of samples\n",
      "        t += 1\n",
      "        old_loss = new_loss\n",
      "        # pick a random sample\n",
      "        i = randint(0, lr.x.shape[0]-1)\n",
      "        xi = lr.x[i]\n",
      "        p_i = softmax(lr.W, xi)\n",
      "        error = np.zeros(p_i.shape)\n",
      "        error[lr.y[i]] = 1\n",
      "        error = error - p_i\n",
      "        grad = np.outer(xi, error)\n",
      "        lr.W += get_learning_rate(t) * grad\n",
      "        new_loss = lr.loss()\n",
      "        losses.append(new_loss)\n",
      "        gradients.append(grad)\n",
      "    plt.plot(np.arange(len(losses)), np.log(losses))\n",
      "    plt.ylabel(\"log of the loss function\")\n",
      "    plt.xlabel(\"#samples seen\")\n",
      "    plt.show()\n",
      "    from sklearn.decomposition import PCA\n",
      "    pca = PCA(n_components=3)\n",
      "    gradients = np.array(gradients)\n",
      "    #gradients = np.log(1 + np.abs(gradients.reshape(gradients.shape[0], gradients.shape[1]*gradients.shape[2])))\n",
      "    gradients = gradients.reshape(gradients.shape[0], gradients.shape[1]*gradients.shape[2])\n",
      "    grad = pca.fit_transform(gradients).transpose()\n",
      "    #grad = np.log(1 + np.abs(grad))\n",
      "    plt.plot(np.arange(grad.shape[1]), grad[0], label=\"#1 direction of the gradient\")\n",
      "    plt.plot(np.arange(grad.shape[1]), grad[1], label=\"#2 direction of the gradient\")\n",
      "    plt.plot(np.arange(grad.shape[1]), grad[2], label=\"#3 direction of the gradient\", linestyle=\":\")\n",
      "    m = np.mean(np.abs(gradients), axis=-1)\n",
      "    plt.plot(np.arange(grad.shape[1]-50), [10*np.sum(m[i:i+50]) for i in xrange(grad.shape[1]-50)],\n",
      "             label=\"smoothed $\\propto$ mean(abs(gradients))\", linestyle=\"--\")\n",
      "    #plt.ylabel(\"gradient log-magnitude\")\n",
      "    plt.ylabel(\"gradient values\")\n",
      "    plt.xlabel(\"#samples seen\")\n",
      "    plt.legend(loc=4)\n",
      "\n",
      "train_with_pure_SGD(lr, x_train, y_train)\n",
      "print(\"training accuracy:\", lr.score(x_train, y_train))\n",
      "print(\"test accuracy:\", lr.score(x_test, y_test))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "loss before training: 2.30258509299\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAgIAAAFHCAYAAADTOCSOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYlOX+P/D3ICDKpqKiLAqyCCqrqKlHAw3NjTSzXFLT\n8pjmmpX6rUzPKVOr8yvztB41K3PXtKOiWaGmAi645K6BAQK5sYvAeP/+uA9jbA7LzDyzvF/XNRfM\nzDPzfOZpmTf3qhJCCBAREZFFslK6ACIiIlIOgwAREZEFYxAgIiKyYAwCREREFoxBgIiIyIIxCBAR\nEVkwRYPAxIkT4erqiqCgoGqPmTFjBvz8/BASEoKkpCQDVkdERGT+FA0CEyZMQGxsbLXP79q1C1eu\nXMHly5fxxRdfYMqUKQasjoiIyPwpGgR69eqFpk2bVvv8jh07MH78eABAt27dkJ2djaysLEOVR0RE\nZPaMeoxAeno6PD09Nfc9PDyQlpamYEVERETmxaiDAABUXAFZpVIpVAkREZH5sVa6gIdxd3dHamqq\n5n5aWhrc3d0rHcdwQERElkZXWwUZdYtATEwMvv76awBAfHw8mjRpAldX1yqPFULwpsfbW2+9pXgN\n5n7jNeZ1Npcbr7H+b7qkaIvAqFGjsH//fty8eROenp5YtGgRSkpKAACTJ0/GwIEDsWvXLvj6+sLe\n3h6rV69WslwiIiKzo2gQWLdundZjVqxYYYBKiIiILJNRdw2Q8YiMjFS6BLPHa2wYvM76x2tsWlRC\n150NClCpVDrvMyEiIjJWuvzeY4sAERGRBWMQICIismAMAkRERBaMQYCIiMiCMQgQERFZMAYBIiIi\nC8YgQEREZMEYBIiIiCyY2QSBoiKlKyAiIjI9ZhME/P2BX39VugoiIiLTYjZBIDUVWLxY6SqIiIhM\ni6K7D+rSoEFAu3ZKV0FERGRazKZF4JlngFu3lK6CiIjItJhNEHBxYRAgIiKqLbMKArdvK10FERGR\naTGbINCsGXDzptJVEBERmRazCQJt2wJ//gnk5ipdCRERkekwmyBgawuEhwPx8UpXQkREZDrMJggA\nQHAwcP680lUQERGZDrMKAu3aAVevKl0FERGR6TCrIODjA1y5onQVREREpsOsgsAjjwCnTgE//aR0\nJURERKbBrIKAqyuwcCHwxRdKV0JERGQazCoIAIC3N9cTICIiqimzCwLOzlxLgIiIqKbMLgg4OTEI\nEBER1RSDABERkQUzyyCQk6N0FURERKbB7IJA48ZAcTFQUqJ0JURERMbP7IKASiVbBfLylK6EiIjI\n+JldEAA4ToCIiKimzDYIcJwAERGRdmYbBNgiQEREpB2DABERkQUzyyDA1QWJiIhqxiyDAFsEiIiI\nakbRIBAbG4uAgAD4+flh6dKllZ6Pi4uDs7MzwsLCEBYWhrfffrtG7+vkBMyaBSxZouuKiYiIzIu1\nUidWq9WYNm0a9u3bB3d3d3Tp0gUxMTEIDAwsd9yjjz6KHTt21Oq9nZyAoiJg/nxg3jxdVk1ERGRe\nFGsRSExMhK+vL7y8vGBjY4ORI0di+/btlY4TQtT6vZs100WFRERE5k+xIJCeng5PT0/NfQ8PD6Sn\np5c7RqVS4fDhwwgJCcHAgQNx7ty5Gr13WJhOSyUiIjJbinUNqFQqrceEh4cjNTUVjRs3xu7duzF0\n6FBcunSpymMXLlyo+b1Hj0gAkQAAtRpo0KD+9RIRESklLi4OcXFxenlvlahL27sOxMfHY+HChYiN\njQUAvPvuu7CyssLcuXOrfY23tzeOHz+OZhXa/lUqVaUuBCEAR0cgI0P+JCIiMhdVfe/VlWJdAxER\nEbh8+TJSUlJQXFyMDRs2ICYmptwxWVlZmg+amJgIIUSlEFAdlQpwcADy83VeOhERkdlQrGvA2toa\nK1asQP/+/aFWq/H8888jMDAQn3/+OQBg8uTJ2Lx5Mz799FNYW1ujcePGWL9+fa3O4eAAFBToo3oi\nIiLzoFjXgC5V10QSEgKsWQOEhipQFBERkZ6YRdeAITRqBBw9qnQVRERExsusWwTKJiaY/ickIiJ6\ngC0CtVRcrHQFRERExsmsg0DZ1gRXrypbBxERkbEy6yDw+uvAsGHAyZNKV0JERGSczDoIAECPHsDh\nw0pXQUREZJzMPggEBwM13KKAiIjI4ph9EGjdGsjKUroKIiIi42T2QaBVKyAzU+kqiIiIjJNZryMA\nAPfvAw0bAtnZgL29gQsjIiLSA64jUAtWVkBpKTBwoNKVEBERGR+zDwIAMHcu4OamdBVERETGxyKC\nQJcuwL17SldBRERkfCwiCDRuzO2IiYiIqmIRQcDenkGAiIioKhYTBAoLla6CiIjI+FhMEGCLABER\nUWUMAkRERBbMIoIABwsSERFVzSKCAFsEiIiIqmYRQaBhQ0CtBkpKlK6EiIjIuFhEEFCpAE9PICVF\n6UqIiIiMi0UEAQAICwOSkpSugoiIyLhYTBAIDgZ++03pKoiIiIyLxQSBNm2A1FSlqyAiIjIuDAJE\nREQWzGKCgKcn8McfSldBRERkXFRCCKF0EfWlUqmg7WPcvQu4uAB37sjphERERKaqJt97NWUxLQKN\nGgF+fsDp00pXQkREZDwsJggAQFAQcO6c0lUQEREZD4sKAvb2souAiIiIJIsKAg0bAvfuKV0FERGR\n8bCuyUHp6elISUmBWq2GEAIqlQq9e/fWd206xyBARERUntYgMHfuXGzYsAEdOnRAgwYNNI8zCBAR\nEZk+rUFg27ZtuHjxIhqawZw7BgEiIqLytI4R8PHxQXFxsSFq0TsGASIiovK0tgg0atQIoaGh6Nu3\nr6ZVQKVSYfny5XovTtcYBIiIiMrTGgRiYmIQExMDlUoFAJrBgroQGxuLWbNmQa1W44UXXsDcuXMr\nHTNjxgzs3r0bjRs3xldffYWwsLA6n49BgIiIqDytQeC5557DvXv3cOnSJQBAQEAAbGxs6n1itVqN\nadOmYd++fXB3d0eXLl0QExODwMBAzTG7du3ClStXcPnyZSQkJGDKlCmIj4+v8zkZBIiIiMrTOkYg\nLi4O/v7+eOmll/DSSy/Bz88P+/fvr/eJExMT4evrCy8vL9jY2GDkyJHYvn17uWN27NiB8ePHAwC6\ndeuG7OxsZGVl1fmcDAJERETlaW0RePnll7F37160b98eAHDp0iWMHDkSJ06cqNeJ09PT4enpqbnv\n4eGBhIQErcekpaXB1dW1TudkECAiIipPa4tAaWmpJgQAgL+/P0pLS+t94pqOM6i4u1J9xicwCBAR\nEZWntUWgc+fOeOGFF/Dss89CCIG1a9ciIiKi3id2d3dHamqq5n5qaio8PDweekxaWhrc3d2rfL+F\nCxdqfo+MjERkZGSlYxgEiIjIFMXFxSEuLk4v760SWjY0Lioqwr///W8cOnQIANCrVy9MnTq13gsM\nlbU0/PTTT3Bzc0PXrl2xbt26SoMFV6xYgV27diE+Ph6zZs2qcrBgTfdl3r8feOMN4ODBepVORESk\nqJp+79WE1hYBOzs7zJkzB3PmzNHJCTUntrbGihUr0L9/f6jVajz//PMIDAzE559/DgCYPHkyBg4c\niF27dsHX1xf29vZYvXp1vc7JFgEiIqLyqm0RGDFiBDZt2oROnTpV6pdXqVQ4ffq0QQqsiZomo1On\ngGeeAS5cMEBRREREeqLLFoFqg8D169fh5uaGa9euVTlgr23btjopQBdqekFKSoC2bYGwMCAqCnjl\nFQMUR0REpGO6DALVzhpwc3MDAHzyySfw8vIqd/vkk090cnJDs7EBNmwAmjcHYmOVroaIiEh5WgcL\nhoWFISkpqdxjQUFBOHPmjF4Lq43aJqPff5ctAteu6bEoIiIiPTHIYMFPP/0Un3zyCa5evYqgoCDN\n43l5eejZs6dOTq6UNm2ArCygqAiws1O6GiIiIuVU2yKQk5ODO3fuYN68eVi6dKkmeTg6OsLFxcWg\nRWpTl2Tk7Q3s2wf4+OipKCIiIj0xyBgBZ2dneHl5YebMmWjatKlmfICNjU2lpYBNkYcHkJamdBVE\nRETK0rrE8JQpU+Dg4KC5b29vjxdffFGvRRkCgwAREVENggAAWFk9OKxBgwZQq9V6K8hQPD3lSoNE\nRESWTGsQ8Pb2xvLly1FSUoLi4mJ89NFHaNeunSFq06tp04C1a4Hbt5WuhIiISDlag8Bnn32GQ4cO\nwd3dHR4eHoiPj8cXX3xhiNr0qk0boHdvQE97OBAREZkEresImIK6jp78v/8DGjUC3nxTD0URERHp\niUE3Hfrzzz/x5ZdfIiUlBaWlpZoCVq1apZMClOTjw50IiYjIsmkNAk888QR69+6N6OhozaDBipsQ\nmap27YB33wV++AEIDAR8fZWuiIiIyLC0dg2Ehobi5MmThqqnTuraRJKXBzzxBHDyJNC9O7Bzpx6K\nIyIi0jGDLChUZvDgwdhppt+Qjo7Azz8Dhw4Bly4pXQ0REZHhaW0RcHBwQGFhIWxtbWFjYyNfpFIh\nNzfXIAXWRH2TUUkJYGsLfPstMGaMDgsjIiLSA122CFj0rIG/+vxz4IMPgIsXATMZAkFERGbKoEHg\nwIEDVT7eu3dvnRSgC7q4IEIAwcHA8uVyi2IiIiJjZdAgMHjwYM0sgaKiIiQmJqJz5874+eefdVKA\nLujqgnz8sRwvsH69DooiIiLSE0W7BlJTUzFz5kxs3bpVJwXogq4uSHY20LQp8M47crEhIiIiY2TQ\nWQMVeXh44Pz58zo5ubFp0gT48EPg9deBy5eVroaIiEj/tLYITJ8+XfP7/fv3cfLkSXh7e+Pbb7/V\ne3E1pctkJARQttliaqrcrpiIiMiYGHSJ4YiIiAcHW1tj9OjR6Nmzp05OboxUKmDTJmDhQrm2AIMA\nERGZs2qDQN++ffHTTz/h7NmzWLZsmSFrUtxTTwG7dwNXrgB9+ihdDRERkf5UGwQyMjJw+PBh7Nix\nAyNHjqz0fHh4uF4LU1qXLsAbbwCursCgQYC11rYTIiIi01PtGIFNmzZh5cqVOHToULnugTK//PKL\n3ourKV32lZQRAtizBxgwAAgIAMx0fCQREZkgg04f/Mc//oEFCxbo5GT6oo8gUObUKWDgQCA9XS9v\nT0REVGtcYrgCfQaBkhLA3h4oLGT3ABERGQdF1xGwNDY2QIsWQEaG0pUQERHpHoNADfj7A/v2KV0F\nERGR7mkNAleuXEFRUREAOUBw+fLlyM7O1nthxmTxYmD+fKB7d/lTrVa6IiIiIt3QGgSGDx8Oa2tr\nXLlyBZMnT0ZqaipGjx5tiNqMRvfuwNGjctDg/v1As2aAEU2aICIiqjOtQcDKygrW1tbYunUrpk+f\njvfeew8ZFthh7ukJvPmm3J1wyRK50NCmTUpXRUREVD9ag4CtrS2+++47fP311xg8eDCEECgpKTFE\nbUZJpQKmTAHWrgU2blS6GiIiovrRGgRWrVqF+Ph4vP766/D29kZKSgrGjh1riNqMWmAgcOGC0lUQ\nERHVT63WEbh9+zbS0tIQHBysz5pqTZ/rCFSnoEBOK/zzT8DBwaCnJiIiC2fQdQQeffRR5Obm4vbt\n2+jcuTNeeOEFzJ49WycnN2X29sAzzwBRUcDq1ZxJQEREpklrEMjJyYGTkxO2bt2KcePGITExEfs4\nqR4A8MUXwIwZwNtvAytXKl0NERFR7WkNAmq1GhkZGdi4cSMGDRoEQDZJ1Mft27cRHR0Nf39/9OvX\nr9p1Cby8vBAcHIywsDB07dq1XufUBxsbYOxY4MMPgXXrlK6GiIio9rQGgQULFqB///7w8fFB165d\ncfXqVfj5+dXrpEuWLEF0dDQuXbqEvn37YsmSJVUep1KpEBcXh6SkJCQmJtbrnPr06KPAsWNyGeIj\nR5SuhoiIqOYU2XQoICAA+/fvh6urKzIzMxEZGYkLVQzB9/b2xrFjx+Di4vLQ91NisGBFM2YAH38s\nf79/X04zJCIi0geDDhZMTU3FsGHD0KJFC7Ro0QLDhw9HWlpavU6alZUFV1dXAICrqyuysrKqPE6l\nUuGxxx5DREQEvvzyy3qdU9+WL5cDBhs1kjMKiIiITIHWjXUnTJiAMWPGYOP/Vs9Zu3YtJkyYgB9/\n/PGhr4uOjkZmZmalx995551y91UqVbVjDg4dOoTWrVvjxo0biI6ORkBAAHr16lXlsQsXLtT8HhkZ\nicjIyIfWpw9WVkDLlsCNG5xSSEREuhMXF4e4uDi9vLfWroGQkBCcOnVK62O1ERAQgLi4OLRq1QoZ\nGRmIioqqsmvgrxYtWgQHBwfMmTOn0nPG0DVQpmtX2UXQrZvSlRARkbkyaNeAi4sLvvnmG6jVapSW\nluLbb79F8+bN63XSmJgYrFmzBgCwZs0aDB06tNIxhYWFyMvLAwAUFBRg7969CAoKqtd5DaF5cyAl\nRekqiIiIakZri0BKSgqmT5+O+Ph4AECPHj3w8ccfo02bNnU+6e3bt/H000/jjz/+gJeXFzZu3Igm\nTZrg+vXrmDRpEnbu3Inff/8dTz75JACgtLQUY8aMwfz586v+EEbUIvDyy3IfgmqGPRAREdWbLr/3\nFJk1oGvGFATu3gWaNgVycwFbW6WrISIic6TL771qBwtOnz79oQUsX75cJwWYm0aNABcXoGFDYOlS\nICxMLkNsrXVYJhERkeFV+/XUuXPnKkfzCyHqvbKgueveHdiyBZg7F2jcWM4meO65B+sMEBERGQt2\nDeiBEPKWng54egI3bwJdugCrVsnWASIiovow6KwBqj2VSrYCeHrK+82bAx99BAwbBvznP8rWRkRE\n9FfsuTaQmBhgxw75s7gYmDpV6YqIiIjYNWBwSUnAk08CyclKV0JERKbKoF0DFy9eRN++fdGxY0cA\nwOnTp/H222/r5OSWKDAQuH5djiEgIiJSmtYgMGnSJCxevBi2/5sUHxQUhHXr1um9MHNlZwfY2wO3\nbytdCRERUQ2CQGFhIbr9ZeF8lUoFGxsbvRZl7lq3BjIylK6CiIioBkGgRYsWuHLliub+5s2b0bp1\na70WZe7c3IDUVKWrICIiqsGsgRUrVuDvf/87Lly4ADc3N3h7e2Pt2rWGqM1s9eoF7NkDDBigdCVE\nRGTpajxroKCgAPfv34ejo6O+a6o1U5o1AAAXLgB9+8pWASuu5EBERLVk0E2HioqKsGXLFqSkpECt\nVmuWGF6wYIFOCtAFUwsCABARIVsG3n8faNBA6WqIiMiUGGTToTJPPPEEmjRpgs6dO8POzk4nJyW5\nVXFAADB5svxJRESkBK1BID09HXv27DFELRalfXs5RuDKFQYBIiJSjtYe6h49euD06dOGqMXi+PoC\nR45wcSEiIlJOtWMEgoKCAABqtRqXL1+Gt7c3GjZsKF+kUhlVODDFMQIA8MsvwIgRwMSJwLJlSldD\nRESmwiCDBVNSUqo9mUqlQtu2bXVSgC6YahAAgD//BEJD5a6EAwcqXQ0REZkCg84aGDt2LL755hut\njynJlIMAILsHnngCOHRIrjjo5CTDARERUVUMOmvgt99+K3e/tLQUx48f18nJSereHZg/H5gwQYYB\nAJg9G+jcGRg1imsNEBGR/lT7FbN48WI4OjrizJkzcHR01NxatmyJmJgYQ9ZoEWbMANzd5UyCb78F\nmjcHnn0WOHdO6cqIiMicae0amDdvHpYsWWKoeurE1LsGqjN4MPD3vwPMXURE9Fe6/N7T2uhs7CHA\nnLVrJ9cZICIi0hf2PhuxkBBgzhw5s4CIiEgfqu0aSE5Ohre3t6HrqRNz7RoA5ABCKyvgo48ABwel\nqyEiImNgkOmDnTt3xvHjx9GnTx/8/PPPOjmZvphzEPjzT7noUKtWcjBhZibw3XdKV0VEREoySBAI\nDQ3FiBEj8Omnn+Lll18ud0KVSoWXX35ZJwXogjkHAQC4e1d2E1y+DDRuDBQUKF0REREpySCDBdev\nX48GDRpArVYjLy8PeXl5yM/P1/xOhtOoEXDxorw1aCCDwP37SldFRETmQOv0wV27dmGgka99a+4t\nAmXu3wdcXYGbN4Hnn5d7FHTqJFciJCIiy2HQJYazs7OxaNEiHDhwAAAQGRmJBQsWwNnZWScF6IKl\nBAFA7lT48cfAzJly5cHz54G2bYEzZ2RrARERmT+DriMwceJEODk5YdOmTdi4cSMcHR0xYcIEnZyc\nak+lAqZPl+sLHDsG5OfLx9atM57ugvPngUmTgKVLga1bla6GiIgeRmuLQEhICE6dOqX1MSVZUotA\nVf7zH+Ddd+WGRf7+wBdfAF26yIBgKP/8p6wjPBw4dQqIjgbs7YH/9//k8x9/DEybZrh6iIjMmUFb\nBBo1aoSDBw9q7v/6669o3LixTk5OuvHCC8DVq3Kq4YsvAr16yXUHDCklRY5ZGDUKmDcP+Pe/gX/9\nC0hLAxYskK0XRERkfLTuPvjZZ59h3LhxyMnJAQA0bdoUa9as0XthVHsODjII+PsDffsCmzcDvXsD\nr78u/zrXp8JCed6nny7/uLs7EBQkxzAQEZHx0RoEQkNDcfr0aU0QMKZBglS1qCggKQm4fVt2E4SG\nylAQEqK/cxYWymmOVWnSBPjfvz5ERGRktAaBMgwApkOlkl/+ANCnD7B+vWwh2LUL6NpVP+csLJSL\nHVXF2RnIztbPeYmIqH5qHATIdI0cKZcmjo6WixK1aqX7c9y9W30QYIsAEZHxUmT3wU2bNqFjx45o\n0KABTpw4Ue1xsbGxCAgIgJ+fH5YuXWrACs3PrFnA5MlARIRcf6C4+MFzeXnAtWtyjYK6YosAEZFp\n0toisGXLFqgqzENzdnZGUFAQWrZsWaeTBgUFYdu2bZg8eXK1x6jVakybNg379u2Du7s7unTpgpiY\nGAQGBtbpnAQsXgw8+aScZRAfL7sJGjaU6xJs2gQEBMgVC6dMeTD1MCVFTge0sQFiY4HgYDk74cIF\n4OuvH7z3w8YIODsDt27Jm4uL3j8mERHVgtYgsGrVKhw5cgRRUVEAgLi4OISHhyM5ORkLFizAuHHj\nan3SgIAArcckJibC19cXXl5eAICRI0di+/btDAL1YG0NPPIIMGwY0K+fXIDIzg4oKQHOngV++w14\n4w0gPR145x35mq++AhYtApo3lyFh716gqAj44Qfgk08ebI38sK6Bhg2Bxx+Xx7/5pkE+KhER1ZDW\nIFBSUoLz58/D1dUVAJCVlYWxY8ciISEBvXv3rlMQqIn09HR4enpq7nt4eCAhIUEv57I0//ynvAkh\nm+xzcgAvL6BdO8DbW44lGD4cCAsD7twBPvgAqLjZZGCg7E7o2FHef1jXAAAMGQKcPq23j0RERHWk\nNQikpqZqQgAAtGzZEqmpqXBxcYGtrW21r4uOjkZmZmalxxcvXowhQ4ZoLaxidwTpnkoFNG0qb2U6\ndZIhYNgwOcgwPR3o2bPya9u0Af74o+ZBwM0N2LNHt/UTEVH9aQ0CUVFRGDRoEJ5++mkIIbBlyxZE\nRkaioKAATZo0qfZ1P/74Y70Kc3d3R2pqquZ+amoqPDw8qj1+4cKFmt8jIyMRGRlZr/NbKpVKrgq4\naJEcWHjtGjB7duXjOnUCnnoKeOwx4P/+T3YNVDdGAJBB4Pp12R2hUhl2+WMiIlMXFxeHuLg4vby3\n1r0G7t+/j61bt+LQoUMAgJ49e2L48OE6+Ys9KioK77//Pjp37lzpudLSUrRv3x4//fQT3Nzc0LVr\nV6xbt67KMQKWvteAvpw8CSxbJvv2K2Y+IYDcXOCVV2STf5MmD/+LPzNT7pLYqJF8nbu7HHPQvbsc\nu2BlBTRrpt9Fj4iIzIVBtyEGgMzMTBw9ehQA0K1btzrPFiizbds2zJgxAzdv3oSzszPCwsKwe/du\nXL9+HZMmTcLOnTsBALt378asWbOgVqvx/PPPY/78+VV/CAYBk5CRAajVQOvWctbBW2/JqYulpfLx\nY8dkV8S9e3JmQrt2D+9uICKyVAYNAhs3bsSrr76KRx99FABw4MABvPfeexgxYoROCtAFBgHz8Pjj\ngJMT8PPPspVg0iQ5qJGIiMozaBAIDg7Gvn37NK0AN27cQN++fXHaiIaAMwiYh6tXge++A3r0AJKT\ngcOHgVWrlK6KiMj46PJ7T+tgQSEEWrRoobnv4uLCL13SCx+fB+sM/PADkJWlbD1798ouiwEDlK2D\niEiftAaBxx9/HP3798fo0aMhhMCGDRswgP9nJD1zdVUuCNy6BXz4IbB8udxa+bHH5MqKRETmSOte\nA8uWLcPkyZNx6tQpnDlzBpMnT8ayZcsMURtZMFdXObiwpAQ4eFAueLR0KfDLL0DF7SmEAPLz5dRE\nXdi9G9i5U+7a6OgoF1k6c0Y3701EZGxqNGvA2HGMgPkpLpbrGFy+LJc0XrhQdheoVHJtA29vuTyy\ns7NcHvn334Hx44Evv5SvFwJYvVruj/Dpp/L95s2Teyd06SJXOgwMBOLi5MyFwkJ5jFoNfPYZMG0a\nMHeufK+33gK2bwe+/14GEiIipRlksKCDg0O1awWoVCrk5ubqpABdYBAwX3fvAleuyBUMrf7XfpWX\nJ/9CLy6WgwpLS+VqiAEBcknksn8VOncGWraUX/ZOTsDgwXK1xPh4uWHSlStATIzcR8HOToaMxo1l\nl8Qrr8igAMgpja+8AuzaBcyZI8cxcEEkIlKSwdcRMHYMAlRGrZaB4a9f1PHxcvOkxYvlokWADA+3\nbskuiJq6eBEYMQLw8JChgIhIKQwCFTAIkKGo1bKV4fBhoH17pashIktl0OmDRPRAgwZAaKjshvD1\nlRs2NWsmd2wMC5PjD2xs5IJI1vyvi4hMAFsEiGqppEQOYLx+XY5JuH5dDmRMSABSU+XzajWwYwfX\nICAi/WDXQAUMAmRs1q4Fnn1W7tDYqNGDGQ7z5gEuLkpXR0SmjkGgAgYBMjZqNTB1quwqsLWVLQg/\n/igHLg4eLHdZbNVKPhcSIndjJCKqKQaBChgEyBSo1XLa45Ytsjvh5k250+K+fcDEiXLdAh8fpask\nIlPAIFABgwCZsjt3gJdflosWtW8vpzQ++yzwyCNyhoKtrdIVEpGxYRCogEGAzEFGhlwg6fx52Wpw\n4gSQkyOYB6mTAAAWw0lEQVQXNurbV85QaNpULo7UsiXQoUPNuhSuXJHLJbu7y8WRxoyRKzMSkeli\nEKiAQYDMVVERcOAAkJQkWw6ys2U4+PNP4PRpuYxykyZyAaXbt+V+CyUlcu+FshUT33pLjkcoKpID\nFg8eBPr0ka/z95fbP9+5Iwcx9uolWyUCApT+5ET0MAwCFTAIkCX66iu5VPK9ezIAuLjIdQ6sreVa\nBkePyr0Rbt+WWyqX7Sa+a5cMErduAZcuya4Ib2/ZEpGQIMcy/Pqrkp+MiLRhEKiAQYBIN27dAtq1\nky0PNd1PISlJ7v/Qu7d+ayOiBxgEKmAQINIdLy9g1Cjg8cflQEUbmwc/27UDGjYsf3yrVnKjpi5d\ngNatZcvEiRMySISEAJ6ecvyDh4fsiggLe7ChExHVDYNABQwCRLpz+TLw0UfAqVNyvEFJidzp8d49\n2c3Qrx/Qo4ccuAgAM2bIboXff5df+Ldvyy99Fxfg+HEZEuztgbQ0ufJifLz8yR0cieqOQaACBgEi\nw0hJAWJj5V/8+fnyFhEBLFhQs9cLIUPChAlyFoMQcnzD00/LmRBlxzAkED0cg0AFDAJEpmPDBjmu\n4MIF2a1w5w6wbZuc4VBYKIPBL78Ajz6qdKVExotBoAIGASLTVlwM3L0r92V46SWgc2fgxReVrorI\neHEbYiIyK7a2D1ZQbNsW+OMPZeshsiRWShdARPRXbdrIPRlOn5YDFIlIvxgEiMiodO4sZxgMGQJM\nnw789pvsOkhMlD+JSLc4RoCIjFJ6upxh4OAgWwZKSuRsgqeeAj7//MH0RSJLpMvvPbYIEJFRcncH\nrl2T6xDk5gKlpTIcNG8uFylq3Fh2I6xfX/Xri4rkmgZE9HBsESAik5OeLndh3LMHmDQJ+PhjeT8x\nEbh+Xa5FcPo0cPEiMHKk3KmxTx8gNFTpyol0g9MHK2AQILJcu3cDy5YBVlbA3/4mWwusrGQYePRR\n+fz588DWrXLMQVCQ3Jjp1i251HGHDrJ1gciUMAhUwCBARNocPQq8/75ctOjePbkN88WLcrnjZcuA\nF15QukKimmMQqIBBgIjqav9+ucHS+PFyC2ZnZ9ly4O4uWw7qQwjgww9lV0bTpnLDpbg4OXZhyhQ5\nQ4KoLhgEKmAQIKL62LZNthh4e8txBjt3Ajdvyv0PHB3l425ussvBykqONejeXT7n5CRvZQsi3b0L\nzJ0rV0l0cgK++Ua2NiQny42Z/PzkssrOzsCKFcp+bjJdDAIVMAgQka7du/dgxkJyMpCZKfdBUKvl\nX/UXL8rnym5CyP0SmjWT2zV36yZ3cly0SI5L+KuEBLmDY2Qk0KKF7K6wspJhYuBA4LnnFPjAZFIY\nBCpgECAipZWWyi/0338HfH3l+gfVuX9ftjrY2sqWBwcHGSQuXgS2bJGtEkQPwyBQAYMAEZmD7Gw5\n6yEnR7YQADJgvPaaDAxlrQZDh8pWh4CA8q8/eRLw8ZFdFmTeGAQqYBAgInPh6yu//K2s5M+iIjnF\nccIE2ZKQmQns3QucPQv07i27MNRq+dyePXLdhDFj5EqMpaXy5uYmj1WplP50pCsMAhUwCBCRubh7\nVy6KpFLJWQsNGshBizY25Y/LypLdC02bylYClUqujbBpk+xmsLZ+8Ppjxx4Mepw9G+jVS5nPRrpj\n8kFg06ZNWLhwIS5cuICjR48iPDy8yuO8vLzg5OSEBg0awMbGBonVdJwxCBARVa+gQO7ouG8f8Pbb\nwJEjciojmS5dfu/Vc5Zs3QQFBWHbtm2YPHnyQ49TqVSIi4tDs2bNDFQZEZH5sbcHHnlE3vbvB27c\nULoiMiaKBIGAiiNcHoJ/6RMR6Y69vZzdQFTGqHcfVKlUeOyxxxAREYEvv/xS6XKIiExe48ayq4Co\njN5aBKKjo5GZmVnp8cWLF2PIkCE1eo9Dhw6hdevWuHHjBqKjoxEQEIBe1YxyWbhwoeb3yMhIREZG\n1qVsIiKz1rgxWwRMUVxcHOLi4vTy3orOGoiKisIHH3xQ7WDBv1q0aBEcHBwwZ86cSs9xsCARUc3M\nmCHXGpg5U+lKqD50+b2neNdAdR+ksLAQeXl5AICCggLs3bsXQRXX6SQiolrhGAGqSJEgsG3bNnh6\neiI+Ph6DBg3CgAEDAADXr1/HoEGDAACZmZno1asXQkND0a1bNwwePBj9+vVTolwiIrPBrgGqiAsK\nERFZkA8+kNsi/+tfSldC9WHy6wgQEZEy6tMicP++fK29ffnlivPy5JLGd+/KRYucnORKhg4Ocnlk\ne3u5wqE2ubkypAQGPnhfe/sH+y6QfjAIEBFZkJoGgWPH5BLGoaHy/p49wNdfyy98T0+5nHHZDou3\nbsktmK2tga5d5Zd+QYHcROn4cflF7ugItGolt14eMgRwdZWvu3pVLomckgL8+CNw6pTcojk/X+7G\nKIR87fDhch+G776Tddjby9dmZgJjx+rralkGdg0QEVmQXbvkjIHhw+UXd/PmwLlzQHw84Ocn9zRw\ndJQbG3XsKI8B5F/pzz8vdz08dw5o0UIGCisroE2b6v/iLywEGjaUf+1fvAgcOABs3Ci/4O3sgKAg\n+buXF9CsGTBiBHDwoHzfsmFhqanAvHlyi+b+/YFLl+RrWrWSdWdnA87OBrl8RsPk9xrQNQYBIqKa\nEUL+RZ2ZKXcuvHVLfulGRMgv5nv3ZJN8y5Zyu2Nj2bFQCLk9c5Mm5R/39wd27Ki8JbO54xgBIiKq\nE5UKGD9e6SpqT6WqHAIAoHVrICPD8oKALjEIEBGRyWrdWm7bnJYGXLsmt2Ru21Z2b1y4ACQny9aO\nNm2UrrS8bduAEyfkmIu0NNm18s9/yq6axo0NWwuDABERmSwPD+DZZ2VrQUAAUFwMXL4sf7q7ywGG\nL70E9OkDqNXA3/4GTJlSs1kM+lJQAIwcCcyfDzRqBPTqBSQmyoGWOTlAt26y7nv3gKIiWaudnbz1\n6/dgkKaucIwAERGZrJISOa3R1rb68QzHj8vWASGA114DwsPl2IKywY7Hj8u/xFUq4PZt+Z4lJfJL\n2c0NWLBAzlJQq+W57O3ll3ZdJScDUVFypkRFBw7IwY8ZGTIk2NnJcxYVyce3bpXBIC6OgwXLYRAg\nIqKayM4GRo8GOnWSMxVu3pRTGnNzZVBo1kzOnLC2lq0Mu3cD33wjv3wbNHgQHM6cka+vi4QEYPp0\n2QpQV5w1UAGDABERGcr48cBvv8nplgMHAiEhQPv28q/3mvjhB+Dzz4H//rfuNTAIVMAgQEREhpKZ\nKQf6ZWUB27fL3yMigKVLZbdBWYuCtbVc/yA/X75OCHlcUZHsqli1qu41MAhUwCBARERKycmRg/8u\nXZLjDkpL5a2kRG75PGPGg4WZnnsOcHEB3n4bePHFup+TQaACBgEiIjIFyclypoONTf3eh0GgAgYB\nIiKyJLr83uOeTkRERBaMQYCIiMiCMQgQERFZMAYBIiIiC8YgQEREZMEYBIiIiCwYgwAREZEFYxAg\nIiKyYAwCREREFoxBgIiIyIIxCBAREVkwBgEiIiILxiBARERkwRgEiIiILBiDABERkQVjECAiIrJg\nDAJEREQWjEGAiIjIgjEIEBERWTAGASIiIgvGIEBERGTBGASIiIgsGIMAERGRBVMkCLz66qsIDAxE\nSEgInnzySeTk5FR5XGxsLAICAuDn54elS5cauEoiIiLzp0gQ6NevH86ePYtTp07B398f7777bqVj\n1Go1pk2bhtjYWJw7dw7r1q3D+fPnFaiWACAuLk7pEswer7Fh8DrrH6+xaVEkCERHR8PKSp66W7du\nSEtLq3RMYmIifH194eXlBRsbG4wcORLbt283dKn0P/wPW/94jQ2D11n/eI1Ni+JjBFatWoWBAwdW\nejw9PR2enp6a+x4eHkhPTzdkaURERGbPWl9vHB0djczMzEqPL168GEOGDAEAvPPOO7C1tcXo0aMr\nHadSqfRVGhEREZURClm9erXo0aOHuHv3bpXPHzlyRPTv319zf/HixWLJkiVVHguAN95444033izq\npit6axF4mNjYWLz33nvYv38/7OzsqjwmIiICly9fRkpKCtzc3LBhwwasW7euymNlFiAiIqLaUmSM\nwPTp05Gfn4/o6GiEhYVh6tSpAIDr169j0KBBAABra2usWLEC/fv3R4cOHfDMM88gMDBQiXKJiIjM\nlkrwz2kiIiKLpfisgfrggkO6kZqaiqioKHTs2BGdOnXC8uXLAQC3b99GdHQ0/P390a9fP2RnZ2te\n8+6778LPzw8BAQHYu3evUqWbHLVajbCwMM2AWV5j3cvOzsZTTz2FwMBAdOjQAQkJCbzOOvbuu++i\nY8eOCAoKwujRo3Hv3j1e43qaOHEiXF1dERQUpHmsLtf0+PHjCAoKgp+fH2bOnFmzk+tstIGBlZaW\nCh8fH5GcnCyKi4tFSEiIOHfunNJlmaSMjAyRlJQkhBAiLy9P+Pv7i3PnzolXX31VLF26VAghxJIl\nS8TcuXOFEEKcPXtWhISEiOLiYpGcnCx8fHyEWq1WrH5T8sEHH4jRo0eLIUOGCCEEr7EejBs3Tqxc\nuVIIIURJSYnIzs7mddah5ORk4e3tLYqKioQQQjz99NPiq6++4jWupwMHDogTJ06ITp06aR6rzTW9\nf/++EEKILl26iISEBCGEEAMGDBC7d+/Wem6TbRHggkO606pVK4SGhgIAHBwcEBgYiPT0dOzYsQPj\nx48HAIwfPx7ff/89AGD79u0YNWoUbGxs4OXlBV9fXyQmJipWv6lIS0vDrl278MILL2gGuPIa61ZO\nTg4OHjyIiRMnApBjjZydnXmddcjJyQk2NjYoLCxEaWkpCgsL4ebmxmtcT7169ULTpk3LPVaba5qQ\nkICMjAzk5eWha9euAIBx48ZpXvMwJhsEuOCQfqSkpCApKQndunVDVlYWXF1dAQCurq7IysoCIAd1\nenh4aF7Da18zs2fPxnvvvadZVRMAr7GOJScno0WLFpgwYQLCw8MxadIkFBQU8DrrULNmzTBnzhy0\nadMGbm5uaNKkCaKjo3mN9aC217Ti4+7u7jW61iYbBLjgkO7l5+dj+PDh+Oijj+Do6FjuOZVK9dBr\nzn8eD/ff//4XLVu2RFhYWLXTXXmN66+0tBQnTpzA1KlTceLECdjb22PJkiXljuF1rp+rV6/iww8/\nREpKCq5fv478/Hx8++235Y7hNdY9bde0Pkw2CLi7uyM1NVVzPzU1tVwSotopKSnB8OHDMXbsWAwd\nOhSATKBlq0NmZGSgZcuWACpf+7S0NLi7uxu+aBNy+PBh7NixA97e3hg1ahR+/vlnjB07ltdYxzw8\nPODh4YEuXboAAJ566imcOHECrVq14nXWkWPHjqFHjx5wcXGBtbU1nnzySRw5coTXWA9q8/8HDw8P\nuLu7l9u7p6bX2mSDwF8XHCouLsaGDRsQExOjdFkmSQiB559/Hh06dMCsWbM0j8fExGDNmjUAgDVr\n1mgCQkxMDNavX4/i4mIkJyfj8uXLmj4pqtrixYuRmpqK5ORkrF+/Hn369ME333zDa6xjrVq1gqen\nJy5dugQA2LdvHzp27IghQ4bwOutIQEAA4uPjcffuXQghsG/fPnTo0IHXWA9q+/+HVq1awcnJCQkJ\nCRBC4JtvvtG85qF0OOjR4Hbt2iX8/f2Fj4+PWLx4sdLlmKyDBw8KlUolQkJCRGhoqAgNDRW7d+8W\nt27dEn379hV+fn4iOjpa3LlzR/Oad955R/j4+Ij27duL2NhYBas3PXFxcZpZA7zGunfy5EkREREh\ngoODxbBhw0R2djavs44tXbpUdOjQQXTq1EmMGzdOFBcX8xrX08iRI0Xr1q2FjY2N8PDwEKtWrarT\nNT127Jjo1KmT8PHxEdOnT6/RubmgEBERkQUz2a4BIiIiqj8GASIiIgvGIEBERGTBGASIiIgsGIMA\nERGRBWMQICIismAMAkQmZv78+YiLi8P3339faflcQ/nqq68wffp0Rc5NRLrFIEBkYhITE/HII49g\n//796N27tyI1cK14IvPBIEBkIl577TWEhITg6NGj6N69O1auXIkpU6bg7bffBgAsX74cHTt2REhI\nCEaNGgVAhoYePXogPDwcPXv21Cy9+9VXX2Ho0KHo168fvL29sWLFCrz//vsIDw9H9+7dcefOHQBA\nZGQkZs2ahbCwMAQFBeHo0aOV6rpx4waeeuopdO3aFV27dsXhw4cBAPv370dYWBjCwsIQHh6O/Pz8\ncq8rKCjAoEGDEBoaiqCgIGzcuBEAcPz4cURGRiIiIgKPP/64Zq31q1evYsCAAYiIiEDv3r1x8eJF\nAMBzzz2HmTNnomfPnvDx8cGWLVt0femJzJuOV0kkIj06evSomDFjhigpKRE9e/Ys95ybm5soLi4W\nQgiRk5MjhBAiNzdXlJaWCiGE+PHHH8Xw4cOFEEKsXr1a+Pr6ivz8fHHjxg3h5OQkPv/8cyGEELNn\nzxYffvihEEKIyMhI8fe//10IIcSBAwdEp06dNK+fNm2aEEKIUaNGiV9//VUIIcS1a9dEYGCgEEKI\nIUOGiMOHDwshhCgoKNDUUWbz5s1i0qRJmvs5OTmiuLhYdO/eXdy8eVMIIcT69evFxIkThRBC9OnT\nR1y+fFkIIUR8fLzo06ePEEKI8ePHi6effloIIcS5c+eEr69v7S8skQWzVjqIEFHNHT9+HMHBwTh/\n/jwCAwPLPRccHIzRo0dj6NChmo1GsrOzMW7cOFy5cgUqlQqlpaWa46OiomBvbw97e3s0adIEQ4YM\nAQAEBQXh9OnTmuPKWhd69eqF3Nxc5OTklDvvvn37cP78ec39vLw8FBQUoGfPnpg9ezbGjBmDJ598\nstIuaMHBwXjllVcwb948DB48GH/729/w22+/4ezZs3jssccAAGq1Gm5ubigoKMDhw4cxYsQIzeuL\ni4sByG6Kss8bGBio2bOdiGqGQYDIBJw6dQrPPfcc0tLS0Lx5cxQWFkIIgfDwcBw+fBh2dnbYuXMn\nDhw4gB9++AHvvPMOzpw5gzfffBN9+/bFtm3bcO3aNURGRmres2HDhprfraysNPetrKzKBYaKrKzK\n9ygKIZCQkABbW9tyj8+dOxeDBw/Gzp070bNnT+zZswft27fXPO/n54ekpCTs3LkTb7zxBvr27Yth\nw4ahY8eOmu6FMrm5uWjatCmSkpKqrOmv5xbcPoWoVjhGgMgEhISEICkpCf7+/jh//jz69OmDvXv3\n4sSJE7Czs4MQAn/88QciIyOxZMkS5OTkID8/H7m5uXBzcwMArF69ukbn+usXqRACGzZsAAD8+uuv\naNKkCRwdHcsd369fPyxfvlxz/+TJkwBkn37Hjh3x2muvoUuXLpo+/TIZGRmws7PDmDFj8MorryAp\nKQnt27fHjRs3EB8fDwAoKSnBuXPn4OTkBG9vb2zevFlT119bLYio7hgEiEzEjRs30KxZMwDAhQsX\nEBAQoHlOrVZj7NixCA4ORnh4OGbOnAlnZ2e89tprmD9/PsLDw6FWqzWj/VUqVbmR/xV//+txdnZ2\nCA8Px9SpU7Fy5cpKxyxfvhzHjh1DSEgIOnbsiC+++AIA8NFHHyEoKAghISGwtbXFgAEDyn2eM2fO\noFu3bggLC8M//vEPvPHGG7CxscHmzZsxd+5chIaGIiwsDEeOHAEArF27FitXrkRoaCg6deqEHTt2\nVFs/EdUctyEmompFRUXhgw8+QHh4uNKlEJGesEWAiIjIgrFFgIiIyIKxRYCIiMiCMQgQERFZMAYB\nIiIiC8YgQEREZMEYBIiIiCwYgwAREZEF+//p0czWJS6oswAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11034aed0>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "training accuracy: 0.956777996071\n",
        "test accuracy: 0.933333333333\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAFHCAYAAABAqi6YAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4VGX68PHvmZbeSIcACSUCFgi9qXEVUFcUGzawt/XV\ntS3q6rrgugo/1HV1XV0LIu4qi21FFClKUem9BQglISG992Ta8/6RZExlAkxmUu7PdeWCmXPmOXdm\nJuc+z3mappRSCCGEEKJb0Hk6ACGEEEK4jyR+IYQQohuRxC+EEEJ0I5L4hRBCiG5EEr8QQgjRjUji\nF0IIIboRjyb+4uJibrjhBgYPHsyQIUPYsmULhYWFTJo0ifj4eCZPnkxxcbEnQxRCCCG6FI8m/kcf\nfZQrr7ySgwcPsnfvXgYNGsS8efOYNGkSycnJXHrppcybN8+TIQohhBBdiuapCXxKSkpISEjg+PHj\njZ4fNGgQ69evJzIykuzsbBITEzl06JAnQhRCCCG6HI/V+FNSUggPD+euu+5i+PDh3HfffVRUVJCT\nk0NkZCQAkZGR5OTkeCpEIYQQosvxWOK3Wq3s3LmThx56iJ07d+Ln59fstr6maWia5qEIhRBCiK7H\n4KkDx8TEEBMTw6hRowC44YYbmDt3LlFRUWRnZxMVFUVWVhYRERHNXisXA0IIIbojV7TOe6zGHxUV\nRe/evUlOTgbghx9+4Nxzz2Xq1KksWrQIgEWLFjFt2rQWX6+Ukp92/pk9e7bHY+jqP/Iey3vcVX7k\nfW7/H1fxWI0f4B//+Ae33XYbZrOZ/v37s3DhQmw2G9OnT2fBggXExsby2WefeTJEIYQQokvxaOIf\nOnQo27Zta/b8Dz/84IFohBBCiK5PZu4TrUpMTPR0CF2evMftT95j95D3ufPw2Dj+s6FpmkvbO4QQ\nQoiOzlW5T2r8QgghRDciiV8IIYToRiTxCyGEEN2IJH4hhBCiG5HEL4QQQnQjkviFEEKIbkQSvxBC\nCNGNSOIXQgghuhFJ/EIIIUQ3IolfCCGE6EYk8QshhBDdiCR+IYQQohuRxC+EEEJ0I5L4hRBCiG5E\nEr8QQgjRjUjiF0IIIboRSfxCCCFENyKJXwghhOhGJPELIYQQ3YgkfiGEEKIbkcQvhBBCdCOS+IUQ\nQohuRBK/EEII0Y1I4hdCCCG6EUn8QgghRDciiV8IIYToRiTxCyGEEN2IJH4hhBCiG5HEL4QQQnQj\nHk/8NpuNhIQEpk6dCkBhYSGTJk0iPj6eyZMnU1xc7OEIhRBCiK7D44n/jTfeYMiQIWiaBsC8efOY\nNGkSycnJXHrppcybN8/DEQohhBBdh0cT/8mTJ1m+fDn33nsvSikAvvnmG+644w4A7rjjDr7++usW\nX1u/vxBCCCHazqOJ//HHH+eVV15Bp/s1jJycHCIjIwGIjIwkJyenxddmms1uiVEIIYToSjyW+L/9\n9lsiIiJISEhotfauaZqjCaCpPeXl7RmeEEII0SUZPHXgjRs38s0337B8+XKqq6spLS1l5syZREZG\nkp2dTVRUFFlZWURERLT4+mdnz2ZrWBgAiYmJJCYmujF6IYQQon2tW7eOdevWubxcTXWAxvL169fz\n6quvsmzZMp566ilCQ0N5+umnmTdvHsXFxc06+GmaRujPP7Nm2DAu8Pf3UNRCCCGE+2ia5pL+bR7v\n1V+v/pb+M888w+rVq4mPj2fNmjU888wzLe4/r18/HkxOdmeIQgghRKfXIWr8p0vTNKqsVtYWF3NF\naKinwxFCCCHanatq/J028XfCsIUQQogz1uVu9QshhBCi/UniFx1HWhpUVHg6CiG6FKXg2289HYXo\nSCTxi47j+edh3z5PRyFEl1JZCXVLoQgBdJE2/t1lZRRZrVwSEuLBqIQQouOpqAB//9qav+jcpI2/\ngaNVVbyanu7pMMTZWrkSTp70dBRCCNGldYnEnxgczC8lJVjsdk+HIs7G999DZqanoxCiS6mvIEqN\nX9TrEok/zGSin48P28vKPB2KOBu5uaDXezoKIboUSfyiKY/N1e9qvwkO5seiIsYFBXk6FHGmPv3U\n0xEI0eVI4hdNdYkaP8BvQkJYWVTk6TBatStrl6dD6PhWr4YTJzwdhRBdSn0LqLSEinpdJvFfFhLC\nA9HRHXZGv+HvDSenPMfTYXRs0sYvhMtJjV801WVu9XvpdMyIivJ0GKdkV3LJfUppabXjjoQQLiM1\nftFUl0n8nYFCLrlP6YsvPB2BEF2OJH7RVJe51S+6gDVrICXF01EI0aXIrX7RlCR+N9LQPB1Cx/b9\n95CR4ekohOhSpMYvmpLELzqO5GQIC/N0FEJ0KVLjF011ucS/vriY6QcOeDqMFkkbvxNLl8KgQZ6O\nQoguRWr8oqkul/h7e3nxeV4ehzrQ8q4ddYhhh7NuHRw75ukohOhS6hO+nIZEvS6X+GO9vRnu78/1\nBw6QbzZ7Ohzg15q+XAA4sWKFLNIjhIvVn3akxi/qdbnEr9M0to0YQaTJxP/y8z0dDvDr+H0Zx+/E\n3r0QE+PpKIToUuRWv2iqS47j12kaFwcHk9VRavx1l9yS+J1YvtzTEQjR5UjnPtFUl0z8AH/q2xe9\n1jGGz0mNv41+/hkiIyE+3tORCNFlSI1fNNXlbvXX6yhJH35N+NKr3wlp4xfC5aTGL5rqsom/I5Ea\nfxtt2QIDB3o6CiG6FKnxi6a67K3+jqS+pi+J34kffvB0BEJ0OTKcTzQlNX4XO1ldTaXN1ug5qfG3\n0YYNcOiQp6MQokuR4XyiqS6f+L/IzSW1qsotx1JKcfHu3fTdvJncBiMKHG38csl9aitXQnq6p6MQ\nokuRW/2iqS6f+JcVFLCisLDdj7OxpITemzZh0unYkJCAsUHnQhnO10br18P553s6CiG6FOncJ5rq\n8m3814SFcf2BA5h0Ou6Ojm634wz19yfDbGZWZCTxvr6Ntv0rKxcMAZL4nVm/3tMRCNHlSI1fNNXl\na/zXhYfzav/+bCwpadfj+On1ZIwbx4txcc227a+sgt43S+J3ZtMm6KALLAnRWUmNXzTV5RM/wHB/\nf5LbqZ2/ukFHvp5eXnjpmr+l90f2gIjLJPE7s2qVjOMXwsWkxi+a8ljiT09P55JLLuHcc8/lvPPO\n48033wSgsLCQSZMmER8fz+TJkykuLj7rYw3x88Pi4svd7JoaCi0W+m/ZQlZNzSn3jfc2gt6LPKvt\nlPt1eytXwsiRno5CiC5FhvOJpjyW+I1GI6+//joHDhxg8+bN/POf/+TgwYPMmzePSZMmkZyczKWX\nXsq8efPO+liRJhMbExJcEHWtb/Lzid60iX6bN3NXVBTRXl6n3F+hwFxIoST+U9u4EUJDPR2FEF2K\nDOcTTXmsc19UVBRRUVEA+Pv7M3jwYDIyMvjmm29YX9fJ64477iAxMdElyV/TNOxKsbyggKTKSips\nNm5roSNeQ/lmM08eO8bO8nLeGTiQEKORc/38+CArC5Omsez887kwONjpse3KDuYi8q3yl3dKW7eC\nj4/07BfCheRWv2iqQ7Txp6amsmvXLsaMGUNOTg6RkZEAREZGkpOT47LjaMDU/fv5Z0YGG0tLeScz\ns9F2i93Oj0VFANiU4unjx9GAKSEhXLh7N+9kZADw78GDOTluXJuSPtQl/pT36Wd02a/SNa1aJeP4\nhXAx6dwnmvL4cL7y8nKuv/563njjDQICAhpt0zQNzYWL7WiaxpHRo+nr7U1yVRUTdu7kz337EmKs\nzchpNTXce/gwKWPHYleKSSEhXBsejpdOx8yoKCLr9gsynN7bppSCskOE6F32q3RNX38Njz7q6SiE\n6FKkxi+a8mjit1gsXH/99cycOZNp06YBtbX87OxsoqKiyMrKIiIiosXXzpkzx/H/xMREEhMT23TM\nAXW39s/18yN/wgQMDXrhH6+qItbbGwCjTsfNdXceoHac/pnqilP2Wu12Sm02ehhdeBtj+3bXlSWE\nAKTG35mtW7eOdevWubxcjyV+pRT33HMPQ4YM4bHHHnM8f/XVV7No0SKefvppFi1a5LggaKph4j9T\nDZP+hpISrt6/n//Xs+dZl9tU08T/f2lphBmN3B4ZibGF4X/utqmkhH4+PkSaTG1+zfayMh49epQt\nI0a4LpDt28FggGHDXFemEN2c1Pg7r6aV2hdeeMEl5Xos62zYsIH//Oc/rF27loSEBBISElixYgXP\nPPMMq1evJj4+njVr1vDMM8+0eyxWu535aWlU2+1MCApyeflNV+ebEBjIvYcPY/rpJ+I2b2ZXWZnL\nj3k6xu/axbX79zseJ1dWOl1XINRopNBqdW0gq1dLG78QLiaJXzTlsRr/xIkTsbfyTfzBzcuzGnQ6\nlp5/Pnal0LmwT0E9xyI9dRcAE4ODeTc+3jHZz5LcXBKa9G9wl/oEP7HugkcpxW/37eP9+HgSQ0Ja\nfV0Po5ECi8W1wfz3v7Bli2vLFKKbk1v9oimPd+7rSNoj6UPLbfz3t9CkUGix0GvTJiovvNClnRpP\n5S8nTgAwv39/oLYD5Kv9+/Pbffu4MyqKf8bHA7UjHlKqqx3DH4MNBkqtVmxKoXdVrHv2uKYcIYSD\n1PhFU55vYO4G2tq5r4fRSJTJxLKCAneEBUB6dTUfDxrU6LlrwsLYN2oUS/PzHc/lWyxcsG0bH2Rm\nsjQ/Hx0QaDBQ7Mrb/bt2wc6dritPCCE1ftGMJH43OJ1leV+Ki+PuQ4fYVlraLrGUW6384ehR3q6b\nk+CDQYOYWTeRUkNx3t6U2Gy8XzfXQZTJRI1SPJ+ayrS6/gBjAgOparBWgeVsqxSrVkFa2tmVIYRo\nRGr8oim51e8Gjjb+Nlxy3xoZia9Ox7X797Nv1CjHHAOu8lZGBq/VLYTzUK9ere6naRprhg6lqK5G\nr2kat0dGMic2lmiTCU3TeGPAAGLqhj8uzMpiQ0kJHzS5e3BaFi2CffvO/PVCiGakxi+akhq/G5zu\nOP5p4eFcEhLCgqyssz52gcXC+rqFjpRSfJCVxdbhw8kZP97pa0cFBjK5Rw/H40WDBxPn44O3vnYm\noobTHV8VGsqC7GzG7NjBsgZNBKclKQn0MsuREK4kNX7RlCR+N2g6nK8t5sbFMaPBBEJnak95OXNS\nUwE4UlWFWSlGBgQQcRpj9tsi3GTi+/PP56rQUO5PTubJo0edrlrYPNg9MomPEC4miV80JYnfDc5k\n5r4Yb2+ivLz4pbgY81n8xRZaLPSom2I4xGDg/XPOOeWIgdJSuOiiMzvW5aGhPB8by+oLLmCYvz+h\np9NMoRT88APUXaQIIVxDbvWLpiTxu0HTcfyn497Dh0murGzz/unV1WwoKXE8LrRaHdPqhptMTGlw\n674lR4/Czz+fdpiNnOfvz8yoKEynMyuh3Q4LFsANN5zxcVOqqhiweTN/kYsHIRykxi+aksTvBmcz\nV/9gPz8OtCHxZ9fUkGs2s6qoiPcarDrYsMbfFm6aPqA5vb62jf8MKaV4MyODMKORD13QN0IIT0iu\nrMTWpGq+rbSUj7Ozz7hMqfGLpiTxu8HpDOdranRAAFtaGNqnlOLF1FTHSeLV9HSm7d/P/LQ0Bvr4\nOPZLrqoi5DRXE/SY/fvPeOa+EquVnWVlvHfOOVTY7ZysrnZxcKI9lbl6+udOyK4U523bhrVBhrYp\nxQ0HDnDHoUMUnuFMmVLjF01J4neDs6nxXxYSwj8yMvixqAi7Usw8eJDemzYxYdcu3s/Kcsyad1NE\nBFV2O4EGA1PDwhyvT/D3Z0xgoGt+kfZkt8PatZCSckYvDzYaWZ+QwAX+/nw+ZAiBneViR2C12wn8\n5RdKunnyL7BYCNDrHVN5A9x16BBpNTVcExrKP+rm3jhd9Qlfavyinpwd3eB0xvE3NSowkE8GD+a5\n48fZPGIEf4mNJcdsZtyuXdwYHt5ov10jRzZ7/SMxMWce+FnKqKkh32JpdUnjDzIzCTMamRYeDjU1\n8MEHLpm291RrDIiOJ6W6mj5eXgR184u1TLOZnl5ejZ7758CBzO/XD4Om8U1BATV2O146HRtLSpi8\nZw/D/P1ZPXQoPqcYBlt/2pEav6gnNX43OJsaP8C0sDA+O/dcAOJ8fBgbFMTGhATeHjjQZTG2h4MV\nFVyxdy+rCgubbTtRXc19ycncn5zMnvJyMnW6dpurv8hi4ePsbHLM5nYpX5ydvRUVnOvn53S/IouF\niTt3krhrF8erqrArxYlO1KSzJDeXEdu381VeHm9nZPDv7GzWFhU5tp+sqaFnk2G2AQYDUV5ehJlM\n3B0d7bgb0MfLi+QxY/DT61mSm3vK48qtftGUJH43OJNx/A2ZdDr61M2QV29cUBBhLh6LD649SVzW\nowd/7NOHKXv3srJJ8rcqxYfnnINVKYZt345/fee+TZvaXH55G28Ne+t0fFdQwIjt23kvM5Mqm+3s\npxcWbbKqsJDHjx7lb+npLMjKapSo88xmojduZMbBg1zboHmqNSFGo2Mxqf5btnDO1q0k7t7d7E7a\n6YyCcZVqm83pHb1xgYGMDgzk3cxMlhcU8G1BAc+lpFBitfJDYSFX79vHdQ3u4p1KjLc3Pb28mBkZ\nybdO1vaQzn2iqe59b81NzrbG7071udRqBVdcVzzQsycLsrKYtn8/R8eMoVfdrcz+Pj709/FheEAA\nuWYzgZoGP/0E/v4wbpzTct/NzOSVtDQOjh6N0cmwQR+9niXnnsvXeXlce+AAf0pJ4bX+/RutUZBW\nXc3oHTuYHRvL704xlbFou98lJ7M0P5/f9ezJN/n5ZJvNjGmw/HSY0cjiwYM518+P8AZftmX5+YwJ\nDOSfGRnoNY3n+vZ19GUZHxTE6qFDSa2uJrmqiikhIY3mpbDY7Vy+dy+jAgIYHRjIrRERRDe5fV5s\nsRBkMLh0BUyfn39m8eDB3Nxg0q306moCDQZHE0Yfb2/eqVvtsqmLg4M5OmYMcQ065rZFQkAAL9at\nsNkaqfGLpiTxu8HZjON3N1cnfpNOx+5Ro/hPdjY/Fxc3OjECv7b/l5bWjuPfts1pmSsKCnjq2DE2\nDR/uNOk3NC08nDcGDKDGbm+2MFFSRQU5Fgv/zc3tlIlfKeW2pZzbQilFL5OJA3XrTTwfG9tsH03T\nWuyPsbm0lKePH+dwZSV2IMJo5MEGn4lRp2Ogry8DG0wZ3XDbLwkJrCwsZHlhIe9kZHB07FjH9oya\nGmI2beLe6GjeHDDglG3jbfXYkSNA7QiaesmVlZyzdSujAwLYNHw4SRUVDPHza3Xpb6NOd9pJHyDe\nx4fRgYHYlXKUnVlTw/UHDnB/dDR3RUdLjV80I7f63eBshvO5W8PE70ozoqKaJf1GAgPJ/n4LeTUW\n7EqR0cp0vwcrKpidmsobAwYwpA3twk39PiaGWX36NHv+8tBQyi+8kEOVlewvLz/tcs+WUooVBQVc\ns28fv9StrdDUwYoK/p6eTnWDFRHrjdyxg4k7d/JzK69tb8sLChpN0axpGn+KjT2jRab+EhfHO/Hx\nrB82jLzx47m/Z8/Ten1PLy/uio7msyFDyLVYKGowDO7hI0cI0uv5d3Y29ycnn3ZsLbknOpore/Tg\n1fR0cuv6kaRVV/PmgAH46fUE//IL52/f3miYnqsYdTr+PXhwowuKEquVPeXlfJKTA0iNXzTnNPEf\nPXqU6rp2ubVr1/Lmm29S7KGTS2fVWW/1u9uoy/cRsWkDA7dsIWbTJlYXFmJvcLJUSvGbPXsY4ufH\n9IgIlx/fT69nVu/e3HnoUKPjtjer3U7Yhg1csW8fl4aEMKzBKIg1RUXcmpTEe5mZzE5NZdbx4y1O\n6PTWwIHcFhnJdQcOsLOszG2x1/v9kSMUuuhLo9c0Lg4OZmJwMGEmU6u1ZGc0TeOWiAiKG8T11bnn\nkj9hAuuGDSOpouK0yttWWsofjx9v1pZ/vr8/X593HquHDiW87kLnsh49eCQmhh+GDiVp1Ch2jhhx\nejNZnoXBfn5kjx/P/ooKrt63jz3G2v41UuMX9Zx+E6+//noMBgNHjx7lgQceID09nVtvvdUdsXUZ\nkvjbduApO7bx6esnuC86mqtDQ3npxAkqGtRu7cC78fEsHDQI33Zaxe+xmBhe6tfPkWyUUpS28mYo\npdhWWsrRykp2lpXxt/R0qlqojTe0u6ys2ZTC523bRl9vbzLHjeP3MTH4NxjWNtzfn1EBAXyQlcWe\n8nJSxoxhRIN28nrjgoL4Xa9e/L5XLybv2cPnDXp6N5wcJ626+oyGldb/vkvz8x2vzzOb2VhSwtL8\nfKxKMbiFW++e9u455zS6ha5pGgadjhEBAawdNuy0yvosL49jVVUtNqkYdTrGBAY226bTNGK8vUlo\n4TNrT4EGAz8lJHBNWBgHvGpHDri7xq+U4ruCgha/byVWK3alWFNUxMSdO90bmBNzUlKI2LCB3R64\ngHYXp238Op0Og8HAV199xSOPPMIjjzxCQkKCO2LrMiTxt0FJCXfzId+eu5GX+7a8i17TuLoNvb/P\nhkGna7SewZ9SUng5LY3X+vfnid69G+37QVYWjx89ikmnI9xo5Fw/P26PjGzUbpxUUcFdhw5h1DRs\nwKHKSh6PiWnUJv+bkBDujIpq1gkNaicmerx3bx5vcuzWPB8by4iAAA7X3RW4NSmJxbm5TA4Jochq\nZVtZGeljxxLTYJTIyepqfPT6VhdVqh87nmM2c+/hw+RNmADAM8eP831hIVlmMysvuOCMa+aeYNTp\nHP1Dcsxmfiwq4paICBSwu7yc8/38GvUf+WtqKsvy8/lrXJyHIj598b6+xPv6YiiC73B/4t9XUcFD\nyckkjxmDV4PvhtVuJ/iXX9BT+zlU2+1U2mzNLuaPVlbS29u70YRG7vBTSQnXhIXR8O1SSvFTSQm7\ny8spsFi4NCSEi4ODm722oG6K9I7U36YlThO/yWTi008/5eOPP2bZsmUAWM5w6sjuqr5T35nWtNzJ\nY4k/NJQJbOSPbj6sM//Lz+fhXr14/eTJRom/wmZjYXY2311wQYsngHpx3t78v169eD8ri/lxcZzv\n50dwkwT7dis9vc/UlaGhXBkaCsCLcXFMCglhd3k5t0REEOPl1ewCY01xMZtLS1uM40hlJVP27iVQ\nryfEaKR/gwuGD845h1KbjT3l5VwYFOTS38GdfHU67j50iAVZWaypa8YsmzixUeKP9fbmcFUVA86g\nA56neaJz3+6yMhJ27ODKHj2aJW6DTse6YcPo5+2NRSmSKytpmCZXFhbyyJEj5NXNZHhszJjT6sR7\nNkqsVg5VVvLRoEGNhlBrmsbNSUlEGI1cExbGtP37SR49utFolOdTUng9PZ1l55/PJR18EjGnif/D\nDz/k3Xff5bnnniMuLo6UlBRmzpzpjti6DKnxt80AjhCXngWc4brALlZutXKwspKdI0ZwSXBwo1q6\nn17PxuHDnZbho9dze1QUtzcZReAu9cMmT2W4vz9PHTvGqIAAVhcV8emQIY5tA3x8+HPfvnjpdBg1\njb5NToZBBgMXneLCpzMIMBhYNXQoG0tKmBkZyd6KikbNLQC3RUaiaVqbJhrqaNzdue9QRQV3Hz7M\nAB8fHm1l5tCGF8v9mnw/Y729WTRoEEP9/Rm5Ywef5ORwZ3Q0UNu8dLSqinEuuNDcW15OSnU119Td\nRTTb7SRs3060yURMC3ffXoqL45LgYOJ8fAgzGh2rnkJtpS5Ir2f3yJEM6IBNXk05Tfznnnsu8+bN\nIy0tDYC4uDiefvrpdg+sK5HE3wYWC+PZSI9CKx0l8Wuaxj8GDMBbr2/zxCqd0Xn+/jzTpw93Hz7M\n9Ca/p6ZpjpNuV3ZRcPApL2A0TeO2U41K6cDqa/p7jYWkpZW3OKrFlezA9PBwHouJwfsM+uKc0yBx\n3h0Vxd9OnuSOqCg0TWNHWRnPpqTwxoABXNjC51VosTB2504G+PgwISiIn4qLWTl0aKN95p44wcSg\nIG4/dIjU6mpUYiJQO/T44OjRrTYt3N3g7+D3TS5oNE3jD+38vrqS0/sn33zzDQkJCVx++eUA7Nq1\ni6uvvrrdAztT7+94nznr5ng6jEZkOF8b5OdzFwvZMfQeNx+4dX56PQ97cK0Dd3o0JobX+vfn7wMG\neDoU4WL1Nf3+NYG8lZHBH48f52hlZbs1PQ7x8+OZvn3PKOk39Yc+fdg7apTjTtulISEM8fXl6v37\nW9x/Q0kJvjodd0VF8dKJE4xtYYEyf72euw8fJtds5romfYbc3Z/AU5z+lnPmzGHLli2E1LVZJCQk\ncPz48XYP7EzNXjebF9a/4OkwGumsE/i4VXQ0l7CODt4npsvSNI0nevdusYNhV7RkCTzxhKejcI/6\nxO9lN7Bz5Ei2lpaSuHs3PTdtOqMVEW1K8X1BAQ8lJ3NLUhK7y8rcNvzVqNPx8eDBKKVaXHtjeEAA\nH5xzDjdGRFBx4YW80EJnzEdiYjgyZgwVF13El+ed546wOxynid9oNBLc5JaKrgNfFXXE3pRyq79t\n+nGMuBPr3H9g0e289hq8/nr7lL0pfRPpJentU/gZaLg6X6jRyI/DhnFy/HimhobyRV7eaZVlV4oZ\nBw9y3+HD9PfxoY+XFwk7dvB9CwtxtRedpjE9IoKRO3aQ3yT59/LyYmRdLb8j5oKOok1t/J988glW\nq5UjR47w5ptvMn78eHfEdkY6Ys95SfxtYDYzls2EFZQDiW4+uOhu2jMnjP9wPJfEXsKaO9a030FO\nQ2ud+94eOBBDk0rcttJSXktPJ9BgYFVhIf56PR8NGuRIplV2O9PCwvho0CC8dDqUUjzft2+zzpDt\n7Z8DB7KzvLx2cS9x2pxW3f/xj39w4MABvLy8uOWWWwgMDOTvf/+7O2LrMs52dT538lTiV1nZ3MlH\nbE14wL0HFt3S/0t7mh+4tN3K70h/660N52ua9KF2LYPz/PwY4uvL9xdcwPXh4dx68KBju59ez00R\nEY62cE3T3J704dcJk1zRj6A7cvqJ+fn58fLLL/Pyyy+7I54uydHG3wHvRjTlqcRv69WHyazmqVNP\nfCeES6wPvoY+2d7tmPo7jtMZzjctPJxpDUZ2zImNZVYbJ48SnYfTxH/JJZc0e07TNNas6Ri3sToD\nudXvXE0dji27AAAgAElEQVQN9CWVPkeOQbc4Hbe/KksVpTWlRPp3zmFo7WlfwHg+ZDyz26n8jtS+\nfDYT+HiqRi/al9NP9JVXXnH8v7q6mi+//BKDfBFOiwznc85cVsNothKan4ckftd48LsH+XjPx6jZ\nHf9Ok7t1oLzc7mR1PtGU0ww+cuTIRo8nTpzIqFGj2i0ggBUrVvDYY49hs9m49957O/2EQVLjd852\n4iR3sIj/xX/Hze49dJeVWpzq6RA6rNnHZlKMDfi0XcrvSM169Qm/A4UkPMxp577CwkLHT35+PitW\nrKC0tLTdArLZbDz88MOsWLGCpKQkFi9ezMEGnUs6IxnH71xldH+u4jsaLOkuzlJnuNBs6pe0Xyit\nab/zS733ev2FMty7Yp6nNBzOJwS0IfEPHz6cESNGMGLECMaNG8drr73GggUL2i2grVu3MmDAAGJj\nYzEajdx8880sXbr0tMs5nH+4HaI7M12mxu9kydmzYTZD734L6Hfyiza/ZtaqWWxI29BuMXV2HanW\n2VYXLryQ+Rvmt7gtr+L0xpyfSpZ3HA/ybqPnlFLszdl7VuXWv+cd6SJfbvWLppze6k9tsnZ4e8vI\nyKB3g16kMTExbNmypdl+23cexNdHQUUVxT7h2GxQcrIKi7V25cC//b+xXH3lIraGpTPOdyyRhkCO\nKwsR3jYMmp7QqChyi8sx5Odh8/WmxtuIXtNDWRFVviZMRh0GvR7NHEBRVTF+JjOmSjPZJguxAQPw\n13wpzU7GHBqGTtOjLysloDIPfewYCqrz0Cxmcg/9jM+5Q3nxx1cB2Lcvj22BZegNOkx+FaiTRdQE\n2Si0gW++GXtkECadNz5mjeLSDGxBIdjsNvwObSSgSEfxhYno9OB98gQ1vXpD3ZAaU1YGhuh+lKsq\nADSLGWU04WUCncWMrbgSa3DtzIs6czXGkhKskdFoGvgVl1FuNFJQY2SAQSMvqwSM/TmUVcCRg8WU\nGIIxm6Hy9c8o+PkoA5e+CDVV6IpzCR9yDln5ldTYqgm1VZKrM1JlreLfe99gVuR9jP36SV6KvZX4\nyaMoqikgqFpHrx4DMRt1KAW++2pPshXnXcDxvaVMSLgXS9oE9h65hOrPP6Nq8Lno+vdFp+nxzs+l\nwt8b5W1CrxnwTU7i9d2vsjP9IC+O+yfemenURPcCnQ5lsaLtO8JB78H0CvcntEcRlqIKrEG1E1EZ\nS4pQOg1b3UIf+tw8vlwfzBVXGzHZa/DOOElF3/78mLGMIBXAQKLw7zmIwECwFFVgqKrC1CeCigqN\ngGoD1aX5eEcOwmiyUZF/ELNdjy0wkABjMNlHy4mOUVR7adjsdoxZJ/GKjeDoCSM9QhV+uSepiYjC\n4G+loLSSoNwiqiOjifKPoar0JAaLmeqAcPIy/AgPzESrqcHao3blPUNlheOxhoahqpxAm5VCn3BS\nUirxzqitOW88kI6upgZdVe33wP7LbkIP7aTo3rtrf//qKuw+Po4V0ky52VgjIkCnQ2e14J1ynKqB\nA0Gnw8fgTc7RbIq9TET0MGEoKkQZDNgDa99LlZuBwScQAkMwGBS6gnzw9kIX7E9lhcIrI52amNo5\nzc22akqObiXsnIn4+CpKCioIroKUvCx+2lqEj38eQb6KDbtNHE7P5dPjY3n75s1Ul0YTFVKDqaiA\n6oieWK1grKnAZK7C2DuMmhrQl5dBjRmfvqGUlYExPw+rjx82H1+MOhtlxYcx9dGzc3Uo1l7RaJqO\nwznbeWrZVbwx/Wei/WLAZkdXUYnVLwC7qv27MpWVoEX2RNPAWlWGsaQEc0Rt58lAq4GKotqJe/JL\ny9i0OQnNbCbT0gNfXwi2l4AGtrr3Sl9aAppGSO9IqmuspCVtJjA6HoN3AAH+CtuxDPaXRNM7To/d\nYsOQmUVleCw6uw+FZXl4Z+VQGdmHgfEaKrf2c7AFBDrOAcWqguTifRzJvQij0Y+jR7LYmBSApimM\nxYUEph0m+9xR2OwKr9xcrAE9KK7xpUeIwjvjpONzwm7HKyuj9pxT99g7u/ZxfnUOATp/AgpKMEf3\nrN1us+GVk4W5V+1015rNjl9ePjU9Y7DYayfc0ZeXYfMPQNNAM9dgLChgv3cRBp2REUEJVGYcorpX\nDMFBUJJSjGaxYAkLp7QUvCpLKPWy0SsyBJtVh76gAGUyYfWrvYNjLMjH7uWFzT+g7vucgyUoCOXl\n3eixwd8bo8lOxq4tBA8Yi0IHaPjmFnDMoCc0wJ/KYhtRKg9zVE/8jP5UF2WBze44dxiKCkEpx9+k\nvrQEzW53nG8NJcUoTcMWGESvsECKTpixBNhQKHRVVZhOnsA8KB6rFezlVRjLi7HUrQmhr6nGUFKM\n/+AoSkpA2V03cV6rif/LL788Zc/U6667zmVBNNTW3rAXPzCauBIbkRU21k0Mgj5e/H5bGZO9TXw6\nDnQVxTy19RqSIuCGAzD2hIE/jBrIorXpfHGON8uG5qNZ/PjntxrL+/uwIjYInd3Gvo9OcuHNfSgw\nGbnkZDlhKof/DrVw2f5o7tlXyszLY7F75TMpvZCb9hu4b3IEaHYuSaticmoZT11Zg76iFzcnVRFX\nbOflMYHYA9KILIeMox+x9tM3eXpy7e/w4dfwSXwoNkw8sruQ6VdHYvfO5eL0au7f4sftV9UOq7l9\nVyXbe8OBtNolIL/+Kp8/XhTEwbDa1aF++iSXa6bbyDFGEVIJ3/wvj4tuicSmYFRWDc9tLeW6a2vL\nGpVVwxPby7jl6h4oFAtW5PLJEF/W9Qniqa25WIJq4Dl44XgkV88uYMydUWg66OFtI3CC4tiSf5GQ\nCc/9pPHX8xWr+vpQQTDff5XNNVNjsRgt9DWfJGP5B4ydAH6L93DdyQHozKG8u2ELCwbr+C4uANB4\n6adi3h3qT9oOA8OyzfzhgA+33LCDeR/FcfdOL/YeMrIz0gCancXfFPC3kUFsi/LFYLey8vNcfG4M\nZ/3BA1x08CL2Lszk4hlRVBl1+FjsrPoil9/dFonZO4eJRwJ4cnslN0yrfQ+e2FZKpUHjX8NqT5Kz\ntpZg0WtMqQikf7GZeT8VM/3qMGw+BVycovHgdo0bpoZit8NVKZVMTqni4d8Eo9crpqTncGWygYev\nsoJdzx/XeVPoo+eDoQHYvQr4cFkFL41QbIwKQbP488OSXG68LJT0HgY0DX75JJubrokg09uAQTPw\ny+J0brk2hJOR2Uzf0YNRWWaevtQHu3c+03f2YHSWmacTg0FTXH+4su5x7Uno+kNVDCso5blLTPzm\nqB+376vix+lw4ScXkHjcwH17KpgxNZSYMisDC62sXfQBKPh8aT4vjA9if3jt9+nw+1lccFcU1QYN\nnV2x4dNcJtwagV2nofTl5L1RQv/7+lFusvHChiIOhxj4dIg/AC/9VMLOKD1fDvJCKY2560vYGeHF\nknhfvFDsW5jN4HtrFzsx2BT7FuYw+N5obFYdQ0osfLvSiwuty1m87zMmH/Tlvr1l3D2j9oL2py8M\nzKi6lsxAA71KrCxaXshlt0QAMPVIFYlp1TyWGIJOB7ckVTC40Mpz44LQ6+HtVYV8PdCHVXE+xJRY\n+U95IU9d6kf+A2VccVMQKI1z8nV8vNqHK7gTpbMwqMDCBysKuPS22pUVR2bV8MyWUq67PgCUxshM\neGpLKTddEw4oxuenc982b5jUl+SyYv6x+iIScsw8mxiMUvD4tlIseo23hgc4vocWncbfJ1SCzYtl\n/7Xz6EX+7A83oZkVP32ey6zrwsj30RNotrN+cS6X3huEXV9Jj7Jwflicyag7o7CsgT9vLSHDX89H\n59d+Dh8tL2DxYD9W9NdxXn4VaaZSPt7ozUWGcFA67t1TTr5Jz9fn+KNpGv/5Lps3RgSwrZcJo8XO\nrkXZjL2n9nMKqKk99tg7oxo9HnNnFJrNiyBLNT8uyWHsHbXbg6vtrP4slwtvr3tcY2H1Z3mMubMH\nms2bviVWPl+az/i67YMKzMxfV8JdNwajdDXEF1fw6iod11wbgd0GM5IrGFBs5cXxQWgaPL61FJNV\nzw3jfNDqziN7Ikx8Nqh2cZ+//lzM3vBfH3+2NJ95YwLZGWVq9Hh7hAml4NiCLM65PxJVl3qSFmTx\n4IwYSk2KuBIr768o5LJbQrB55/HApkCiKmz8dXzt39xd+8oJq7Tzypjac8m9e8oJrbbzf3WP/7C1\nlDKTjneH+XN+aToffa0x7s4INDSGZ1u5b3cFD04OATTGZJl5dEc5t11Vu37A6KwaLvuphBd7eYEG\nBuXCJaFVK+644w515513tvrTXjZt2qSmTJniePzyyy+refPmNdqnYdhW66/Pp+8tVJsX7FdZZVmK\nOSjmoHLLc2v//7xBPfXUr/seKziuiquKmwdgs/36/127VMWiBSq1KLVtsadvUnF/j1N2u73R8//6\n6BG1/7xIVVGSr7b8e556f8f7Kr8iX1VZqpqVsSNzh7rjf3c0eq5JcS2y2evitliUSkpSSillNitV\nU3Pq19ntdke8G9I2qJMlJ9WTK59UZYVV6sTmzGb778rapfZk71ETrzmkXou6WCX9kq+U2azMq390\n7JNalKqYg7p44cXOA29nVpvV+U5nof69s9ltqqSqrNHXx2a3qdLck+powdFGn3VRUe1n07icX/9f\nYa5Q2zK2NTpGS9+VltRYa5TZ+mvhWWVZzb6PzWRlKVVQ0KbylVK137HTUFOjVJWT8B0hthDr9ozt\n6t97/n1ax6zX9H1uTzXW2j+2tvy9NvTSvGr1yX8bB3rokFLFLZyeGv69/vpcC4Xm5ytVWamUUmrn\nmk/V0AdQu7N2N9rFZjv9WF3Cbm/TB2M2K/Wf/7QSY8M/tDNUUqJUWlrb9rXb7Y3+rk5bVpb6r+la\n9c/3Kk/7pevW/XoeP0XKPi1aXWEdhtVq5ZxzzuHHH3+kZ8+ejB49msWLFzN48GDHPpqmnbL9Mr8y\nn/BXwrmo70Wsv3M92gsaPgYfKp6t9MwwnvpYO8MYoqIiuOIKmD8fLmp9edwrroAVK+DAAWiwfDsA\n6SXp9Pl7nw41bakQHdaNN8KkSXD//e1SfI21hoH/GEjKoynodTLTnUdUV2Pfuh3dRRPPqhhnua+t\n2jQg/9tvvyUpKYnq6mrHc3/+85/P+uAtBmQw8NZbbzFlyhRsNhv33HNPo6TfpjJ0hkb/XjHgCqL8\nozyXdztDwq+3fDlceCGMHXvK3cbkfkMwFcAtzbbVn1zkJCNEG1xwAZSUtFvxXgYv0m7bDpVV4O/f\nbscRp+DtjW7UCCguhiaL3nmC08T/wAMPUFVVxZo1a7jvvvv4/PPPGTNmTLsGdcUVV3DFFVec8eub\nJv7lty13SVzdwm231f44UW4KZQjbWrym0Wm6Rv8KIU7h+efbt/wff4SpU+GTT+Daa9v3WKJ1K1fC\n99/Du+8637edOT0zb9y4kY8//pgePXowe/ZsNm/ezOHDHWeoXEv0Wm1Nsz7xC9fL9emLHV2Lib/+\n/ZfEL0QHEBsL69dL0vek6urahN8Bkj60IfH7+NT2JPT19SUjIwODwUB2dna7B3Y26hN+fQISp6Gy\nEry8IDf3lLsV+sYwhxdOWeOX91+INrj1Vnj77fYrv39/aOfZVoUTJhM89ZSno3BwWiW+6qqrKCoq\nYtasWYwYMQKA++67r90DOxvSxnwWDhyAKVMgLOyUu9Un/BZr/Dqp8QvRZgMGgMXSvsfIyoKgIPD1\nbd/jiJbpdDB6NJSVQYDnZ4x0mvjrO/Fdf/31/Pa3v6W6uprgDtA54VSkjfksjBoF33zjdLdTJX5H\njV8uvIRw7i9/ad/yt2yp7ay7Zg20sNqqcJPFiyElBV56ydOROL/Vf8EFF/Dyyy9z7NgxvL29O3zS\nF+5xyhq/tPEL0XH06gW//CJJ39P+9z/40588HQXQhsT/zTffoNfrmT59OiNHjuTVV18lLS3NHbEJ\nT8jLa9OQnzbV+KWNXwjnbr8d/v739is/JgYmTGi/8kXbzJoFRqOnowDakPhjY2N5+umn2bFjB4sX\nL2bv3r3ExcW5IzbhCaGh8OWXTndrS+KXGr8QbdC7d23nr/aUkVHbs1x4zvDhHeYzaNN4t9TUVJYs\nWcJnn32GXq9n/vyWV88SXYBOV9u5z4m2dO6TNn4h2qC923z374fzz4cdO2qTj/CMt9+uHTH1+OOe\njsR54h8zZgxms5np06fz+eef069fP3fEJTo4qfEL0UmEh9d27JOk71n79sErr3g6CqANiX/RokUM\nGjTIHbGITkQSvxAu8tBDEB8Pjz3WPuVHRtb+CM96+OEOMV0vtCHxS9IXLTlV4q/XwdZ/EqJjevDB\n2jH27Sk9vTb5t3dfAtG601xzpj1JlUyckbYkfiFEG1xwAfTt237lp6ZCnz61yV94zl//2qaO0+7g\nNPFXt9ALsaXnRPciiV+ITiI4uHYN7f79PR1J91ZdDXWz33qa08Q/fvz4Nj0nuhdJ/EJ0EsHBbRqp\nI9rZnXfWTqbUAbTaxp+VlUVmZiaVlZXs3LkTpRSaplFaWkplZaU7YxQdkCR+ITqR1NTa+QL0MsTW\nY/r1qx3O1wG0mvhXrVrFRx99REZGBk8++aTj+YCAAF5++WW3BCeEEOIs5eVBXBzk59dO0CU844kn\n4Oab4fLLPR1J64n/jjvu4I477uCLL77ghhtucGdMohNoS41fk9sBQnier2/twluS9D3ro488HYFD\nm5bl/eSTT0hNTcVmszlu+dev2ie6J7nVL0Qn4ecHU6d6OgrRgThN/Ndccw3BwcGMGDECb29vd8Qk\nOgFJ/EJ0IseP197ulz9YQRsSf0ZGBitXrnRHLKITkcQvRCdRWVk7lM9iAUOblmcRXVybhvPt3bvX\nHbGITkQSvxCdhNFYO3GMJH1Rx+k34eeff2bhwoXExcXhVTcUQdM0uRjo5mTKXiE6CaMRrrvO01GI\nDsRp4v/+++/dEYfoZKTGL4QQnZPTW/2xsbGkp6ezdu1aYmNj8fPz6zQ1uc4SZ2ckiV8IITonp4l/\nzpw5zJ8/n7lz5wJgNpuZMWNGuwcmOjZJ/EII0Tk5Tfz/+9//WLp0KX5+fgD06tWLsrKydg/MFYZF\nDfN0CF2WTOAjhBCdk9M2fi8vL3S6X68PKioq2jUgV7E+b0WnyarD7UVq/EII0Tk5zYw33ngjDzzw\nAMXFxbz33ntceuml3Hvvve6I7azodXqpcbYjSfxCCNE5Oa3xz5o1i1WrVhEQEEBycjIvvvgikyZN\nckdsogOTxC+EEJ1Tm2Z0mDx5MpMnT27vWEQnIolfCCE6p1Zv9U+YMAEAf39/AgICGv0EBga6LUDR\nMUniF0KIzqnVxL9hwwYAysvLKSsra/RTWlp6VgedNWsWgwcPZujQoVx33XWUlJQ4ts2dO5eBAwcy\naNAgVq1adVbHEUIIIURjrSb+wsLCU/6cjcmTJ3PgwAH27NlDfHy8Y46ApKQklixZQlJSEitWrOCh\nhx7Cbref1bFE+5Ape4UQonNqtY1/+PDhaJqGUoq0tDRCQkIAKCoqom/fvqSkpJzxQRt2DhwzZgxf\nfvklAEuXLuWWW27BaDQSGxvLgAED2Lp1K2PHjj3jY4n2Ibf6hRCic2q1xp+amkpKSgqTJk3i22+/\npaCggIKCAr777juX9ur/8MMPufLKKwHIzMwkJibGsS0mJoaMjAyXHUu4jkzgI4QQnZPTXv2bNm3i\n/fffdzy+4oormDVrltOCJ02aRHZ2drPnX375ZaZOnQrASy+9hMlk4tZbb221nNaSx5w5cxz/T0xM\nJDEx0WlMwnWkxi+EEO1r3bp1rFu3zuXlOk38PXv25K9//SszZsxAKcWnn35Kr169nBa8evXqU27/\n6KOPWL58OT/++KPjuV69epGenu54fPLkyVaP1TDxC/eTxC+EEO2raaX2hRdecEm5TmfuW7x4Mbm5\nuVx77bVcd9115Obmsnjx4rM66IoVK3jllVdYunQp3t7ejuevvvpq/vvf/2I2m0lJSeHIkSOMHj36\nrI4l2ockfiGE6Jyc1vhDQ0N58803XXrQRx55BLPZ7OgrMG7cON5++22GDBnC9OnTGTJkCAaDgbff\nflvaiTsoSfxCCNE5OU38ubm5zJ8/n6SkJKqqqoDadvc1a9ac8UGPHDnS6rZnn32WZ5999ozLFu4h\niV8IITonp7f6b7vtNgYNGsTx48eZM2cOsbGxjBw50h2xiQ5MEr8QQnROThN/QUEB9957LyaTiYsv\nvpiFCxeeVW1fdA1tSfyyLLIQQnQ8Tm/1m0wmAKKiovj222/p2bMnRUVF7R6Y6NicJf6Nd28kPjTe\nfQEJIYRoE6eJ/09/+hPFxcW89tprPPLII5SWlvL666+7IzbRiY3rPc7TIQghhGjBKRO/zWYjOTmZ\nq666iuDg4HaZSEAIIYQQ7nPKRli9Xn/WY/aFEEII0XE4vdU/ceJEHn74YW666Sb8/PxQSqFpGsOH\nD3dHfEIIIYRwIaeJf9euXWiaxp///OdGz69du7bdghIdn6y4K4QQnZPTxC/t+kIIIUTX4TTxv/ba\na82mzQ0KCmLEiBEMGzas3QITQgghhOs5nWFlx44d/Otf/yIjI4OTJ0/y7rvv8v3333Pffffxf//3\nf+6IUQghhBAu4rTGn56ezs6dO/H39wfgL3/5C1deeSXr169nxIgRPP300+0epBBCCCFcw2mNPy8v\nzzF7H4DRaCQnJwdfX99GS+qK7kXm6BdCiM7JaY3/tttuY8yYMUybNg2lFMuWLePWW2+loqKCIUOG\nuCNGIYQQQriIppTzgVnbtm1jw4YNaJrGhAkTPL46n6ZptCFs0Y6efBL+9jcZ1ieEEO7iqtzntMYP\nMGrUKEaNGnXWBxNCCCGEZ8m6qUIIIUQ3IolfCCGE6EYk8QshhBDdiCR+IYQQohuRxC+EEEJ0I5L4\nhRBCiG5EEr8QQgjRjUjiF0IIIboRSfzijMiMfUII0Tm1aeY+IYT7abISkhDdVntOSy+JX4gOTNak\nEKL7ae+LfrnVL4QQQnQjkvjFGZG70EII0TlJ4hdCCCG6EY8m/tdeew2dTkdhYaHjublz5zJw4EAG\nDRrEqlWrPBidEMLTUlNT0el02O12AK688kr+/e9/u+XYAQEBpKamuuVY9aqqqpg6dSrBwcHcdNNN\nbXpNYmIiCxYsaOfIXGPOnDnMnDkTgLS0NAICAqQfiwd4LPGnp6ezevVq+vbt63guKSmJJUuWkJSU\nxIoVK3jooYccf/BCiI5r9OjRHD16lOPHjzNixIhG29566y1GjhyJt7c3d91111kdZ/ny5Y7E4Uot\nJc+ysjJiY2NdfqxT+eKLL8jNzaWwsJAlS5Y0294wcdbTNK3TjABpGGefPn0oKytzSeyd6eKnI/BY\n4n/iiSeYP39+o+eWLl3KLbfcgtFoJDY2lgEDBrB161YPRSiEaAuLxUJaWhoDBgxgx44dzRJ/r169\neP7557n77rvbNQ6bzXbGr+0oifPEiRPEx8ej03X8Vlir1erpEBw6yufXWXjk27V06VJiYmK44IIL\nGj2fmZlJTEyM43FMTAwZGRnuDk8IcRr279/PkCFDANi+fTsJCQmNtl977bVcc801hIaGOi3Lbrfz\nhz/8gfDwcPr37893333XaHvDmt1HH33EhAkTeOKJJwgLC+OFF17AbDbzhz/8gb59+xIVFcXvfvc7\nqqurHa9funQpw4YNIygoiAEDBrBy5Uqee+45fv75Zx5++GECAgL4/e9/D4BOp+P48eMAlJSUcPvt\ntxMREUFsbCwvvfSS4xb1Rx99xMSJE5k1axY9evSgX79+rFixotXf8eDBgyQmJhISEsJ5553HsmXL\nAJg9ezYvvvgiS5YsISAggIULFzZ63YoVK5g7d65je8P3OTU1lYkTJxIYGMiUKVMoKChwbNu8eTPj\nx48nJCSEYcOGsX79+lZj27lzJwkJCQQGBjJ9+nRuuukmnn/+eQDWrVtHTEwM8+fPJzo6mnvuuYfi\n4mKuuuoqIiIi6NGjB1OnTm10zk5JSeHiiy8mMDCQyZMnk5+f3yjmhs04JSUl3HPPPfTs2ZOYmBie\nf/55x7ZTvcetfX7iFFQ7ueyyy9R5553X7Gfp0qVqzJgxqqSkRCmlVGxsrMrPz1dKKfXwww+r//zn\nP44y7rnnHvXll182K7sdwxZt9MQTSsnH0L46+vd84cKFKjg4WPn6+iofHx8VHBysDAaDCggIUCEh\nISo1NbXR/s8995y68847T1nmO++8owYNGqROnjypCgsLVWJiotLpdMpmsymllEpMTFQLFixwHN9g\nMKi33npL2Ww2VVVVpR577DF1zTXXqKKiIlVWVqamTp2q/vjHPyqllNqyZYsKCgpSP/zwg1JKqYyM\nDHXo0KFm5dbTNE0dO3ZMKaXUzJkz1bRp01R5eblKTU1V8fHxjeIwGo3qgw8+UHa7Xb3zzjuqZ8+e\nLf5+ZrNZ9e/fX82dO1dZLBa1Zs0aFRAQoA4fPqyUUmrOnDlq5syZrb4/LW2/+OKLVf/+/dWRI0dU\nVVWVSkxMVM8884xSSqmTJ0+q0NBQ9f333yullFq9erUKDQ1VeXl5zcquqalRffr0UW+++aayWq3q\nq6++UiaTST3//PNKKaXWrl2rDAaDeuaZZ5TZbFZVVVWqoKBAffXVV6qqqkqVlZWpG2+8UU2bNs1R\n5tixY9WTTz6pzGaz+umnn1RAQIAj/pSUFKVpmuOznTZtmnrwwQdVZWWlys3NVaNHj1bvvvtum97j\nlj6/zqy1v31XnRPcfmbZt2+fioiIULGxsSo2NlYZDAbVt29flZ2drebOnavmzp3r2HfKlClq8+bN\nzcoA1OzZsx0/a9eudeNvIJSSxO8OHT3x17vwwgvV7t271YkTJ9SwYcNa3e9Pf/qT08R/ySWXOE72\nSim1atWqRsmhaeLv06ePY1+73a78/PwcyVoppTZu3Kji4uKUUkrdf//96oknnmjxuImJieqDDz5o\n9Ld6GyIAACAASURBVFx94rdarcpkMqmDBw86tr377rsqMTHREceAAQMc2yoqKpSmaSonJ6fZcX76\n6ScVFRXV6LlbbrlFzZkzRyml1OzZs9WMGTNae3ta3J6YmKheeuklx+O3335bXX755UoppebNm9fs\nQmHKlClq0aJFzcpev3696tWrV6PnJk6c2Cjxm0wmVVNT02p8u3btUiEhIUoppU6cOKEMBoOqrKx0\nbL/11lsd8TdM/NnZ2crLy0tVVVU59v3000/VJZdcopRy/h639Pl1ZvV/+2vXrm2U61x1TnD7zH3n\nnXceOTk5jsdxcXHs2LGDHj16cPXVV3PrrbfyxBNPkJGRwZEjRxg9enSL5cyZM8dNEQvRcbmqafN0\nO1YXFhbSr18/AMrLy0lMTKSmpgaAkJAQ5syZw6OPPtrkGM4PkpWVRe/evR2P+/Tpc8r9G+6bl5dH\nZWVloz4GSinH7eKTJ0/y29/+ttWyWmsnzs/Px2KxNOqI3KdPn0a3tKOiohz/9/X1BWrfl4iIiEZl\nZWZmNooZoG/fvmfdpNnw+D4+PpSXlwO1fQY+//xzR3MC1LbN/+Y3v2lWRmZmJr169Wr0XNNYw8PD\nMZlMjseVlZU8/vjjrFy5kqKiIqD291ZKkZmZSUhICD4+Po79+/btS3p6erNjnzhxAovFQnR0tOM5\nu93e6PN39h53xXb+xMREEhMTHY9feOEFl5Tr8Sl7G35YQ4YMYfr06QwZMgSDwcDbb7/dJT9MIVzF\nUyOhevToQXFxMUuWLGHdunW88847XHfddTz88MMtJhVo24k5OjqatLQ0x+OG/3dWZlhYGD4+PiQl\nJTVKIPV69+7N0aNHTzu2sLAwjEYjqampDB482BFXw/5IbdWzZ0/S09NRSjmOeeLECQYNGtSm159u\np78+ffowc+ZM3nvvPaf7RkdHN7sAqe+0Wa/p+/Taa6+RnJzM1q1biYiIYPfu3QwfPhylFNHR0RQV\nFVFZWelI1CdOnECv1zc7du/evfHy8qKgoOCMOjZKnjg9Hu86evz4cXr06OF4/Oyzz3L06FEOHTrE\nlClTPBiZEMKZ7du3M3z4cAB27drVrEc/1Pa2r66uxmq1YrPZqKmpabUH/vTp03nzzTfJyMigqKiI\nefPmtTkWnU7Hfffdx2OPPUZeXh4AGRkZjvlA7rnnHhYuXMiaNWuw2+1kZGRw+PBhACIjIzl27FiL\n5er1eqZPn85zzz1HeXk5J06c4PXXX2fGjBltjq3e2LFj8fX1Zf78+VgsFtatW8e3337LzTff3KbX\nR0ZGkpqa2uzuSWt3U2bMmMGyZctYtWqV43NYt25di3cYxo8fj16v56233sJq/f/t3XlUVHeWB/Bv\nyaLgKaDY912CgLIMCm6xNK1oBOMSiBuahDGT2EQRTaKjjmiCoBOzoB07Hgkao5EOQ7eicUsUice4\ngukmGFSEUkAUWZRNqgru/OHhNQVVgKYQCPdzjkfe9vvduu9V3XpLvafEwYMHcenSpQ7jqa2thYGB\nAYyNjVFZWamyR+rk5ITAwECsX78eCoUCZ8+exeHDh9W2Y2Njg8mTJyM2NhY1NTVobm5GQUEBsrKy\nOksJgI7XH2uvxws/Y6zvys7ORkBAACoqKqCrqwtjY+N283z44YcwNDTE5s2b8c0338DAwADx8fFq\n21u8eDFCQkLg6+uLwMBAzJ49W+PenLrfr2/evBnu7u4IDg6GsbExJk2ahOvXrwMARowYgZSUFCxf\nvhwmJiaQSqXCEYVly5YhLS0NpqamiImJadfXtm3bMHjwYLi6umLcuHGYP3++cE8CdXFoillPTw8Z\nGRk4evQoLCwsEB0djb1798LDw0NjW62Fh4cDAMzMzBAYGKi2v9Zt2Nvb4+DBg9i0aRMsLS3h6OiI\nrVu3qr0/ip6eHtLT05GcnAyJRIJ9+/YhNDRU5dB+29hiYmLQ0NAAc3NzjB49GlOnTlWZZ//+/bhw\n4QJMTU2xceNGLFq0SGOevv76a8jlcnh5ecHU1BTh4eEoKyvTmJfWw52tP6ZKRF058dbLiEQivttT\nD4uNBT79tOcONfcHvJ2znhYUFIQlS5a0K9ise2l672vrM4H3+BljjAEAsrKyUFZWBqVSiT179iA3\nNxdTpkzp6bCYlvX4xX2MMcZ6h/z8fERERKCurg5ubm5IS0uDlZVVT4fFtIwP9bNnsmIF8MknfKi/\nO/F2zlj/xIf6GWOMMaY1XPgZY4yxfoQLP2OMMdaPcOFnjDHG+hEu/Iwxxlg/woWfMdZrtX1m+8sv\nv4y9e/c+l77FYjGKioqeS18tGhoaEBYWBhMTE7z22mtdWkYqlSI5ObmbI9OOuLg4REZGAnjyHACx\nWMy/XOkBXPgZY7/byJEjcfPmTdy6dUvlfv1yuRxRUVFwdnaGkZER/P39cezYsWfu5/vvvxcKhzap\nK541NTVwdnbWel8dSUtLw/3791FZWYnU1NR201sXzhad3ea3N2kdp6OjI2pqarQSe1/68tMbcOFn\njP0uCoVCeIrblStXVAq/UqmEo6MjsrKy8OjRI3z00UeIiIiATCbTehyaHvzTFb2lcMpkMnh4eDzT\nE+qeN6VS2dMhCHrL+usrev/WxRjr1XJzc+Hl5QXgydP6/P39hWmGhoZYv3698Fz1adOmwcXFBdnZ\n2Wrbam5uxsqVK2FhYQE3NzccOXJEZXrrPbvdu3djzJgxiI2Nhbm5OTZs2AC5XI6VK1fCyckJ1tbW\neOedd/D48WNh+YMHD8LPzw/GxsZwd3fH8ePHsWbNGvz000+Ijo6GWCzG0qVLATx52t+tW7cAAA8f\nPsTChQthaWkJZ2dnxMfHC4eod+/ejbFjx+K9996DqakpXF1dOzyqce3aNUilUkgkEvj4+CAjIwMA\nsH79enz44YdITU2FWCxGSkqKynLHjh1DQkKCML11nouKijB27FgYGRkhJCQEFRUVwrTz589j9OjR\nkEgk8PPzw5kzZzTGlp2dDX9/fxgZGSEiIgKvvfYa1q1bBwDIzMyEvb09tmzZAhsbG0RFRaG6uhqh\noaGwtLSEqakpwsLCVJ78V1hYiPHjx8PIyAiTJ0/GgwcPVGJufRrn4cOHiIqKgq2tLezt7bFu3Tph\nWkc51rT+WAeoD+qjYf+hxMYS8WroXr19O09JSSETExMyNDQkAwMDMjExIV1dXRKLxSSRSKioqKjd\nMmVlZTRo0CDKz89X2+aOHTvI09OTiouLqbKykqRSKQ0YMICampqIiEgqlVJycrLQv66uLm3fvp2a\nmpqooaGBYmJi6JVXXqGqqiqqqamhsLAwWr16NRERXbhwgYyNjemHH34gIqKSkhL67bff2rXbQiQS\nUUFBARERRUZG0owZM6i2tpaKiorIw8NDJQ49PT3atWsXNTc3044dO8jW1lbt65PL5eTm5kYJCQmk\nUCjo1KlTJBaLhXzExcVRZGSkxpyrmz5+/Hhyc3OjGzduUENDA0mlUlq1ahURERUXF5OZmRkdPXqU\niIhOnjxJZmZmVF5e3q7txsZGcnR0pKSkJFIqlZSenk76+vq0bt06IiI6ffo06erq0qpVq0gul1ND\nQwNVVFRQeno6NTQ0UE1NDYWHh9OMGTOENoODg2nFihUkl8spKyuLxGKxEH9hYSGJRCJh3c6YMYPe\nfvttqq+vp/v379PIkSPpyy+/7FKO1a2/vkzTe19bnwm9+5NFg97+gdgfcOHvfn1lOx83bhxdvXqV\nZDIZ+fn5aZxPLpfTSy+9RG+//bbGeSZMmCB82BMRnThxQqU4tC38jo6OwrzNzc00ePBgoVgTEZ07\nd45cXFyIiOitt96i2NhYtf1KpVLatWuXyriWwq9UKklfX5+uXbsmTPvyyy9JKpUKcbi7uwvT6urq\nSCQS0b1799r1k5WVRdbW1irj5s6dS3FxcUREtH79elqwYIGm9KidLpVKKT4+Xhj+4osvaMqUKURE\nlJiY2O6LQkhICO3Zs6dd22fOnCE7OzuVcWPHjlUp/Pr6+tTY2KgxvpycHJJIJEREJJPJSFdXl+rr\n64Xp8+bNE+JvXfjLyspo4MCB1NDQIMy7f/9+mjBhAhF1nmN1668v6+7Czw/pYawPE23QzrlNWv90\nV1ZXVlbC1dUVAFBbWwupVIrGxkYAgEQiQVxcHJYtWybM39zcjMjISAwaNAjbt2/X2O7du3fh4OAg\nDLecItCk9bzl5eWor69XucaAiITDxcXFxZg2bZrGtjSdJ37w4AEUCgWcnJxU4mp9SNva2lr429DQ\nEMCTvFhaWqq0VVpaqhIzADg5Oam09Sxa929gYIDa2loAT64Z+O6774TTCcCTc/MTJ05s10ZpaSns\n7OxUxrWN1cLCAvr6+sJwfX09li9fjuPHj6OqqgrAk9dNRCgtLYVEIoGBgYEwv5OTE+7cudOub5lM\nBoVCARsbG2Fcc3OzyvrvLMd8nr/ruPAz1oc9bcHWFlNTU1RXVyM1NRWZmZnYsWMHZs2ahejo6HZF\nhYgQFRWF8vJyfP/999DR0dHYro2NDW7fvi0Mt/5bndYf9ubm5jAwMEBeXp5KAWnh4OCAmzdvdtpO\nW+bm5tDT00NRURGGDh0qxGVvb99hbOrY2trizp07ICKhT5lMBk9Pzy4t/7QX/Tk6OiIyMhI7d+7s\ndF4bG5t2X0BaLtps0TZPW7duxfXr13Hx4kVYWlri6tWrCAgIABHBxsYGVVVVqK+vFwq1TCZTu/4d\nHBwwcOBAVFRUPNOFjVz0nw5f3MeeCf/0lgFPLuYLCAgAAOTk5Kjsbbd455138Ntvv+HQoUMYOHBg\nh+1FREQgKSkJJSUlqKqqQmJiYpdjGTBgABYvXoyYmBiUl5cDAEpKSnDixAkAQFRUFFJSUnDq1Ck0\nNzejpKQE+fn5AAArKysUFBSobVdHRwcRERFYs2YNamtrIZPJ8Omnn2LBggVdjq1FcHAwDA0NsWXL\nFigUCmRmZuLw4cOYM2dOl5a3srJCUVFRu9++tx1usWDBAmRkZODEiRNoamrC48ePkZmZqfYIw+jR\no6Gjo4Pt27dDqVTi4MGDuHTpUofx1NbWwsDAAMbGxqisrMSGDRuEaU5OTggMDMT69euhUChw9uxZ\nHD58WG07NjY2mDx5MmJjY1FTU4Pm5mYUFBQgKyurs5QA6Hj9sfa48DPGnll2djYCAgJQUVEBXV1d\nGBsbq0yXyWTYuXMnfvnlF1hbW0MsFkMsFuPbb79V297ixYsREhICX19fBAYGYvbs2Rr35tT9fn3z\n5s1wd3dHcHAwjI2NMWnSJFy/fh0AMGLECKSkpGD58uUwMTGBVCoVjigsW7YMaWlpMDU1RUxMTLu+\ntm3bhsGDB8PV1RXjxo3D/Pnz8cYbb2iMQ1PMenp6yMjIwNGjR2FhYYHo6Gjs3bsXHh4eGttqLTw8\nHABgZmaGwMBAtf21bsPe3h4HDx7Epk2bYGlpCUdHR2zdulU4/dE2tvT0dCQnJ0MikWDfvn0IDQ1V\nObTfNraYmBg0NDTA3Nwco0ePxtSpU1Xm2b9/Py5cuABTU1Ns3LgRixYt0pinr7/+GnK5HF5eXjA1\nNUV4eDjKyso05qX1cGfrj6kSkaavir0YP6e8561YAXzyCe/5dyfezllPCwoKwpIlS9oVbNa9NL33\ntfWZwHv8jDHGAABZWVkoKyuDUqnEnj17kJubiylTpvR0WEzL+OI+xhhjAID8/HxERESgrq4Obm5u\nSEtLg5WVVU+HxbSMD/WzZ8KH+rsfb+eM9U98qJ8xxhhjWsOFnzHGGOtHuPAzxhhj/QgXfsYYY6wf\n4cLPGGOM9SNc+BljvVbbZ7a//PLL2Lt373PpWywWo6io6Ln01aKhoQFhYWEwMTHBa6+91qVlpFIp\nkpOTuzky7YiLi0NkZCSAJ88BEIvF/MuVHsCFnzH2u40cORI3b97ErVu32t2vf8GCBbCxsYGRkRFc\nXV0RHx//zP18//33QuHQJnXFs6amBs7OzlrvqyNpaWm4f/8+KisrkZqa2m5668LZorPb/PYmreN0\ndHRETU2NVmLvS19+eoMeK/zbtm3D0KFD4ePjgw8++EAYn5CQgCFDhsDT01N4uAZjrPdSKBTCU9yu\nXLnSrvCvXr0ahYWFePToEY4ePYpt27bh2LFjWo+jqanpmZftLYVTJpPBw8PjmZ5Q97wplcqeDkHQ\nW9ZfX9EjW9fp06dx6NAh/POf/0Rubi5WrlwJAMjLy0Nqairy8vJw7NgxLFmyRO3DJBhjvUdubi68\nvLwAPHlan7+/v8p0b29vDBo0SBjW1dVt95z6Fs3NzVi5ciUsLCzg5uaGI0eOqExvvWe3e/dujBkz\nBrGxsTA3N8eGDRsgl8uxcuVKODk5wdraGu+88w4eP34sLH/w4EH4+fnB2NgY7u7uOH78ONasWYOf\nfvoJ0dHREIvFWLp0KYAnT/u7desWAODhw4dYuHAhLC0t4ezsjPj4eOEQ9e7duzF27Fi89957MDU1\nhaura4dfbK5duwapVAqJRAIfHx9kZGQAANavX48PP/wQqampEIvFSElJUVnu2LFjSEhIEKa3znNR\nURHGjh0LIyMjhISEoKKiQph2/vx5jB49GhKJBH5+fjhz5ozG2LKzs+Hv7w8jIyNERETgtddew7p1\n6wAAmZmZsLe3x5YtW2BjY4OoqChUV1cjNDQUlpaWMDU1RVhYmMqT/woLCzF+/HgYGRlh8uTJePDg\ngUrMrU/jPHz4EFFRUbC1tYW9vT3WrVsnTOsox5rWH+sA9YDw8HD68ccf243ftGkTJSYmCsMhISH0\n888/t5uvh8JmrcTGEvFq6F69fTtPSUkhExMTMjQ0JAMDAzIxMSFdXV0Si8UkkUioqKhImPedd94h\nQ0ND0tHRoR07dmhsc8eOHeTp6UnFxcVUWVlJUqmUBgwYQE1NTUREJJVKKTk5WehfV1eXtm/fTk1N\nTdTQ0EAxMTH0yiuvUFVVFdXU1FBYWBitXr2aiIguXLhAxsbG9MMPPxARUUlJCf3222/t2m0hEomo\noKCAiIgiIyNpxowZVFtbS0VFReTh4aESh56eHu3atYuam5tpx44dZGtrq/b1yeVycnNzo4SEBFIo\nFHTq1CkSi8WUn59PRERxcXEUGRmpMT/qpo8fP57c3Nzoxo0b1NDQQFKplFatWkVERMXFxWRmZkZH\njx4lIqKTJ0+SmZkZlZeXt2u7sbGRHB0dKSkpiZRKJaWnp5O+vj6tW7eOiIhOnz5Nurq6tGrVKpLL\n5dTQ0EAVFRWUnp5ODQ0NVFNTQ+Hh4TRjxgyhzeDgYFqxYgXJ5XLKysoisVgsxF9YWEgikUhYtzNm\nzKC3336b6uvr6f79+zRy5Ej68ssvu5RjdeuvL9P03tfWZ0KPfLL4+fnR+vXrKSgoiMaPH0+XLl0i\nIqLo6Gj65ptvhPmioqIoLS2t3fK9/QOxP+DC3/36ynY+btw4unr1KslkMvLz89M4X3NzM50+fZrM\nzMzowoULaueZMGGC8GFPRHTixAmV4tC28Ds6Oqq0P3jwYKFYExGdO3eOXFxciIjorbfeotjYWLX9\nSqVS2rVrl8q4lsKvVCpJX1+frl27Jkz78ssvSSqVCnG4u7sL0+rq6kgkEtG9e/fa9ZOVlUXW1tYq\n4+bOnUtxcXFERLR+/XpasGCB2hg1TZdKpRQfHy8Mf/HFFzRlyhQiIkpMTGz3RSEkJIT27NnTru0z\nZ86QnZ2dyrixY8eqFH59fX1qbGzUGF9OTg5JJBIiIpLJZKSrq0v19fXC9Hnz5gnxty78ZWVlNHDg\nQGpoaBDm3b9/P02YMIGIOs+xuvXXl3V34e+2h/RMmjRJeJZya/Hx8VAqlaiqqsL58+dx6dIlRERE\nCIfU2uJzN4x1IC5Ou/93UWVlJVxdXQEAtbW1kEqlaGxsBABIJBLExcVh2bJlKsuIRCJIpVKEh4fj\n22+/xciRI9u1e/fuXTg4OAjDjo6OHcbRet7y8nLU19erXGNARMLh4uLiYkybNk1jW5o+ax48eACF\nQgEnJyeVuFof0ra2thb+NjQ0BPAkL21PaZSWlqrEDABOTk4qbT2L1v0bGBigtrYWwJNrBr777jvh\ndALw5Nz8xIkT27VRWloKOzs7lXFtY7WwsIC+vr4wXF9fj+XLl+P48eOoqqoC8OR1ExFKS0shkUhg\nYGAgzO/k5IQ7d+6061smk0GhUMDGxkYY19zcrLL+O8sx14qu67bCf/LkSY3TduzYgVmzZgEARowY\ngQEDBuDBgwews7NT2SiKi4vbbYgt4lp9UEmlUkilUq3EzVif0rZg/97hLjI1NUV1dTVSU1ORmZkp\nvKejo6PVFpXWFAoFzMzM1E6zsbHB7du3heHWf6vT+sPe3NwcBgYGyMvLUykgLRwcHHDz5s1O22nL\n3Nwcenp6KCoqwtChQ4W47O3tO4xNHVtbW9y5cwdEJPQpk8ng6enZpeWf9qI/R0dHREZGYufOnZ3O\na2Nj0+4LSMtFmy3a5mnr1q24fv06Ll68CEtLS1y9ehUBAQEgItjY2KCqqgr19fVCoZbJZNDR0WnX\nt4ODAwYOHIiKiopnurDxj1r0MzMzkZmZqfV2e+TivhkzZuDUqVMAgOvXr0Mul8Pc3BzTp0/HgQMH\nIJfLUVhYiBs3bqjdKwCeFP6Wf1z0GesZly9fRkBAAAAgJyen3RX95eXlOHDgAOrq6tDU1ITjx4/j\nu+++wyuvvKK2vYiICCQlJaGkpARVVVVITEzsciwDBgzA4sWLERMTg/LycgBASUmJ8OugqKgopKSk\n4NSpU2hubkZJSQny8/MBAFZWVigoKFDbro6ODiIiIrBmzRrU1tZCJpPh008/xYIFC7ocW4vg4GAY\nGhpiy5YtUCgUyMzMxOHDhzFnzpwuLW9lZYWioqJ2v31vO9xiwYIFyMjIwIkTJ9DU1ITHjx8jMzNT\n7RGG0aNHQ0dHB9u3b4dSqcTBgwdx6dKlDuOpra2FgYEBjI2NUVlZiQ0bNgjTnJycEBgYiPXr10Oh\nUODs2bM4fPiw2nZsbGwwefJkxMbGoqamBs3NzSgoKEBWVlZnKQHQ8frry6RSqUqt05YeKfxvvvkm\nbt26hWHDhmHu3Ln4+uuvAQBeXl6IiIiAl5cXpk6dii+++OIP+02ur1u4EIiO7ukoWE/Lzs5GQEAA\nKioqoKurC2NjY5XpIpEIf/3rX2Fvbw8zMzOsW7cOe/fuxYgRI9S2t3jxYoSEhMDX1xeBgYGYPXu2\nxs8Adb9f37x5M9zd3REcHAxjY2NMmjQJ169fB/Dk6GJKSgqWL18OExMTSKVS4YjCsmXLkJaWBlNT\nU8TExLTra9u2bRg8eDBcXV0xbtw4zJ8/H2+88YbGODTFrKenh4yMDBw9ehQWFhaIjo7G3r174eHh\nobGt1sLDwwEAZmZmCAwMVNtf6zbs7e1x8OBBbNq0CZaWlnB0dMTWrVvV/lpKT08P6enpSE5OhkQi\nwb59+xAaGqpyaL9tbDExMWhoaIC5uTlGjx6NqVOnqsyzf/9+XLhwAaampti4cSMWLVqkMU9ff/01\n5HI5vLy8YGpqivDwcOF0cWc57mz9MVUi0vRVsRfj55Sz/oC3c9bTgoKCsGTJknYFm3UvTe99bX0m\n9P67RDDGGHsusrKyUFZWBqVSiT179iA3NxdTpkzp6bCYlnXbxX2MMcb6lvz8fERERKCurg5ubm5I\nS0uDlZVVT4fFtIwP9TPWS/F2zlj/xIf6GWOMMaY1XPgZY4yxfoQLP2OMMdaPcOFnjDHG+hEu/Iwx\nxlg/woWfMcYY60e48DPGGGP9CBd+xtgfirOzM3788UettPX6669j3bp1WmmrN1u9ejU+//zzLs37\ne/L7NP1oQ+v15+Pj0+WH/mhLUFAQ8vLyOh33vHHhZ4z1Wc7OzsKTPlt09qCbp6HNtnqr8vJy7N27\nF2+//XaX5n/WnDxtP9rQOtbc3Fy8+OKLv7tNdducJitXrsT//M//dDrueePCzxjrs57H3Q3/6HdP\n3L17N6ZNm4aBAwf22n6USuUz96vt9fc021xYWBhOnz6Ne/fudTjueePCzxh7Zps3b4a9vT2MjIzg\n6emJ06dPA3iyV/Txxx9j+PDhEIvFiIqKwr179zB16lThcbnV1dUAgGvXrkEqlUIikcDHxwcZGRkq\nfWiaHhkZidu3byMsLAxisRgff/yxsExOTg58fX1hYmKCOXPmoLGxEQBQWlqK2bNnw9LSEq6urti2\nbZtKXzk5OQgICICRkRHmzJmDx48fd1vuniZHncWemJgId3d3GBkZwdvbG//4xz/a9bV161a1OTl2\n7BjGjx+vMn9n7V28eBHe3t4wNTXFm2++KbQFaN4mjh492q6f7Oxs+Pv7w8jICBEREXjttdeEQ/PO\nzs7YsmWLkJ+mpqZO41K3/lr2+Nueougon5rypWmba/uaW44IDBo0CP/xH/+B48ePC22rG/fcUR/U\nR8Nm7Kn09u38t99+IwcHB7p79y4REclkMiooKCAiIicnJxo1ahTdv3+fSkpKyNLSkvz9/enq1av0\n+PFjmjhxIm3YsIEUCgW5ublRQkICKRQKOnXqFInFYsrPzyciIrlcrnb69evXiYjI2dmZfvzxR5W4\nnJycKCgoiO7evUuVlZU0dOhQ+utf/0rNzc0UEBBAH374ISkUCrp16xa5urrS8ePHiYiosbGRHB0d\n6bPPPiOlUklpaWmkp6dH69at6zAPZ8+epdjYWFq6dCmVl5cTEdHNmzfpk08+ofPnz2tcztnZuUs5\nIiJqamrqMPbvvvtOWA+pqak0ePBgYbilL3U5ISKysLCgy5cvq8Smrr2ysjIhv8OGDaPi4mKqrKyk\nMWPG0Nq1azvdJtr205LvpKQkUiqVlJ6eTvr6+kK+nZycyN/fn4qLi+nx48edvs7O1l/rbaWzfGra\nhtq209lrJiJaunQpxcbGquRX3bjWNL33tfWZwHv8jLFnoqOjg8bGRvz6669QKBRwdHSEq6srsLJ0\nYQAAElRJREFUgCeHQ999911YWFjA1tYW48aNw6hRo+Dr64uBAwdi5syZyMnJwfnz51FXV4dVq1ZB\nV1cXEyZMQGhoKL799lsA0Dh9//79GuMSiURYunQprK2tIZFIEBYWhqtXr+LixYt48OAB1q5dC11d\nXbi4uOA///M/ceDAAaEvpVKJZcuWQUdHB7Nnz8aIESM6zEFtbS0ePHiArVu3Ijw8HFOnTkVRURF2\n7dqF5cuXIygoqMPlu5IjALh06VKHsb/66quwtrYGAERERGDIkCG4ePGiSl9tc/LLL78AAKqrqyEW\ni1Xm7ag9kUiE6Oho2NnZQSKRYM2aNcL66mibaNvP+fPn0dTUhHfffRc6OjqYOXMmRo4c2W492tnZ\nCacHOorradZfZ/nUtA2p09FrBgCxWKxy5EbTuOeJH8vLWB8WV1iIDTJZu/HrnZwQ5+LSpfk1zdsZ\nd3d3fPbZZ4iLi8Ovv/6KkJAQfPLJJ7CxsQEAlce5GhgYqAwPGjQItbW1KC0thYODg0q7Tk5OKCkp\nAQCN00tLSzuMraU4AIChoSFKS0tx+/ZtlJaWQiKRCNOampqEC75KS0thZ2fXri/q4HxudnY2pk2b\nBgAYO3YsZs2ahalTp+LKlSsdxteiKzkCAJlM1mHsX3/9NT799FMUFRUBePKFpKKiQqWv1jkxMDDA\n3bt3AQASiQQ1NTUq86pr78GDB8L01uvE0dFRWB8dbRNt+1GX77bruu1wR69T0/pTp7N8Aprz1VZn\n74NHjx6p9KNp3PPEe/yM9WFxLi4gqbTdP02FXN38z1L0W8ydOxc//fQTZDIZRCIRPvjgA43zqiug\ntra2uHPnjso0mUwGe3v7Dqe3fMA/zdXlDg4OcHFxQVVVlfDv0aNHOHz4MADAxsZG+MLRuq+O+pDL\n5dDV/ff+06JFiyCTyVBQUNDluFrT9CWjo9hlMhneeust/OUvf0FlZSWqqqrg4+PT5QvQhg8fjvz8\nfGG4K+3dvn1b5W9bW1thWNM20bYfdflu3S6gun47i0vT+lOns22hI+q2h47eB9euXYOvr6/K/OrG\nPU9c+Bljz+T69es4deoUGhsbMXDgQAwaNAg6OjpP1UZQUBAMDQ2xZcsWKBQKZGZm4vDhw5gzZw4A\nIDg4uMPpVlZWnRbZlsIwcuRIiMVibNmyBQ0NDWhqakJubi4uX74MABg1ahR0dXWRlJQEhUKB9PR0\nXLp0qcO2vb29kZ2dLQx/8cUXOHToEP785z+joaHhqXLRkY5ir6urg0gkgrm5OZqbm5GSkoLc3NxO\n22zJy8svv4wzZ84I4ztrj4jwl7/8BSUlJaisrER8fLywPjraJtr2M3r0aOjo6GD79u1QKpU4ePBg\nh/nuLK6nWX9BQUEdbgsd5avtNtfRa378+DGys7MxadIkYX514543LvyMsWfS2NiI1atXw8LCAjY2\nNnjw4AESEhI0zt96T6nl99V6enrIyMjA0aNHYWFhgejoaOzduxceHh4A0On01atX46OPPoJEIsEn\nn3yisV+RSIQBAwbg8OHDuHr1KlxdXWFhYYG33noLjx49AgDo6+sjPT0du3fvhpmZGf72t79h9uzZ\nHebAxsYG9+/fx1dffYW1a9di4cKF+NOf/oT//u//xpw5c/D3v//9qXKqLkfAk/PImmL38vLCihUr\nMGrUKFhbWyM3Nxdjx47ttJ+WtiMjI/H9998Lv2DorD2RSIT58+dj8uTJcHNzw5AhQ7B27VoAHW8T\nCxcuVOlHT08P6enpSE5OhkQiwb59+xAaGqrx536dxfU066+zbaGjfLXd5jp6zRkZGZgwYYLKaQN1\n4543EXX1eFAv8jx+u8tYT+PtnD0va9asgaWlJZYtW9aj/QQFBWHJkiVYtGhRt8bxvAQHB+Orr76C\nl5dXh+Pa0vTe19ZnAhd+xnop3s7ZH11WVhY8PDxgbm6Offv2YcmSJbh165bKRY79UXcXfr6qnzHG\nWI/Iz89HREQE6urq4ObmhrS0tH5f9J8H3uNnrJfi7Zyx/qm79/j54j7GGGOsH+HCzxhjjPUjXPgZ\nY4yxfoQLP2OMMdaPcOFnjDHG+hEu/Iwxxlg/wr/jZ6wXe5qH0DDGWFf0SOG/ePEioqOjoVAooKur\niy+++EJ4bnJCQgK++uor6OjoICkpCZMnT+6JEBnrcfwbfsZYd+iRQ/3vv/8+PvzwQ+Tk5GDjxo14\n//33AQB5eXlITU1FXl4ejh07hiVLlqC5ubknQmQAMjMzezqEPzzOcffjHD8fnOe+o0cKv42NDR4+\nfAgAqK6uFp6tffDgQcydOxd6enpwdnaGu7s7Ll682BMhMvAb+XngHHc/zvHzwXnuO3rkUH9iYiLG\njh2LlStXorm5GT///DMAoLS0FMHBwcJ89vb2KCkp6YkQGWOMsT+kbiv8kyZNQllZWbvx8fHxSEpK\nQlJSEmbOnInvvvsOb775Jk6ePKm2Hb64iTHGGNOeHnlIj5GRER49egTgyQVMJiYmePjwIRITEwEA\nq1atAgBMmTIFGzZsQFBQkMry/GWAMcZYf9RnH8vr7u6OM2fOYPz48Th16hQ8PDwAANOnT8e8efMQ\nGxuLkpIS3LhxAyNHjmy3PF/tzBhjjD2bHin8O3fuxJ///Gc0NjbCwMAAO3fuBAB4eXkhIiICXl5e\nws/8eO+eMcYY054eOdTPGGOMsZ7R527Ze+zYMXh6emLIkCHYvHlzT4fTZ925cwcTJkyAt7c3fHx8\nkJSUBACorKzEpEmT4OHhgcmTJ6O6ulpYJiEhAUOGDIGnpydOnDjRU6H3OU1NTfD390dYWBgAzrG2\nVVdX49VXX8XQoUPh5eWFCxcucI67QUJCAry9vTFs2DDMmzcPjY2NnOff6c0334SVlRWGDRsmjHuW\nnF65cgXDhg3DkCFDsGzZss47pj5EqVSSm5sbFRYWklwuJ19fX8rLy+vpsPqku3fvUk5ODhER1dTU\nkIeHB+Xl5dF7771HmzdvJiKixMRE+uCDD4iI6NdffyVfX1+Sy+VUWFhIbm5u1NTU1GPx9yVbt26l\nefPmUVhYGBER51jLFi5cSMnJyUREpFAoqLq6mnOsZYWFheTi4kKPHz8mIqKIiAjavXs35/l3ysrK\nouzsbPLx8RHGPU1Om5ubiYhoxIgRdOHCBSIimjp1Kh09erTDfvtU4T937hyFhIQIwwkJCZSQkNCD\nEf1xvPLKK3Ty5El64YUXqKysjIiefDl44YUXiIho06ZNlJiYKMwfEhJCP//8c4/E2pfcuXOHXnrp\nJTp16hSFhoYSEXGOtai6uppcXFzajecca1dFRQV5eHhQZWUlKRQKCg0NpRMnTnCetaCwsFCl8D9t\nTktLS8nT01MY/+2339J//dd/ddhnnzrUX1JSAgcHB2GYb/CjHUVFRcjJyUFQUBDu3bsHKysrAICV\nlRXu3bsH4MnNlezt7YVlOPdds3z5cvzv//4vBgz491uNc6w9hYWFsLCwwBtvvIGAgAAsXrwYdXV1\nnGMtMzU1xYoVK+Do6AhbW1uYmJhg0qRJnOdu8LQ5bTvezs6u01z3qcLPV/hrX21tLWbPno3PP/8c\nYrFYZZpIJOow57w+Onb48GFYWlrC399f409QOce/j1KpRHZ2NpYsWYLs7GwMHjxYuB9IC87x71dQ\nUIDPPvsMRUVFKC0tRW1tLb755huVeTjP2tdZTp9Vnyr8dnZ2uHPnjjB8584dlW867OkoFArMnj0b\nkZGRmDFjBoAn3zBb7rh49+5dWFpaAmif++LiYuEZC0y9c+fO4dChQ3BxccHcuXNx6tQpREZGco61\nyN7eHvb29sLTPV999VVkZ2fD2tqac6xFly9fxujRo2FmZgZdXV3MmjULP//8M+e5GzzN54O9vT3s\n7OxQXFysMr6zXPepwh8YGIgbN26gqKgIcrkcqampmD59ek+H1ScREaKiouDl5YWYmBhh/PTp07Fn\nzx4AwJ49e4QvBNOnT8eBAwcgl8tRWFio8eZK7N82bdqEO3fuoLCwEAcOHMDEiROxd+9ezrEWWVtb\nw8HBAdevXwcA/PDDD/D29kZYWBjnWIs8PT1x/vx5NDQ0gIjwww8/wMvLi/PcDZ7288Ha2hpGRka4\ncOECiAh79+4VltFIWxcoPC/ff/89eXh4kJubG23atKmnw+mzfvrpJxKJROTr60t+fn7k5+dHR48e\npYqKCnrppZdoyJAhNGnSJKqqqhKWiY+PJzc3N3rhhRfo2LFjPRh935OZmSlc1c851q6rV69SYGAg\nDR8+nGbOnEnV1dWc426wefNm8vLyIh8fH1q4cCHJ5XLO8+80Z84csrGxIT09PbK3t6evvvrqmXJ6\n+fJl8vHxITc3N3r33Xc77Zdv4MMYY4z1I33qUD9jjDHGfh8u/Iwxxlg/woWfMcYY60e48DPGGGP9\nCBd+xhhjrB/hws8YY4z1I1z4GesDVq9ejczMTPzjH/9od0va52X37t149913e6Rvxpj2cOFnrA+4\nePEigoODcebMGbz44os9EgPfa52xPwYu/Iz1Yu+//z58fX1x6dIljBo1CsnJyXjnnXfw0UcfAQCS\nkpLg7e0NX19fzJ07F8CTLwmjR49GQEAAxowZI9zOdvfu3ZgxYwYmT54MFxcXbN++HR9//DECAgIw\natQoVFVVAQCkUiliYmLg7++PYcOG4dKlS+3iKi8vx6uvvoqRI0di5MiROHfuHADgzJkz8Pf3h7+/\nPwICAlBbW6uyXF1dHaZNmwY/Pz8MGzYMf/vb3wAAV65cgVQqRWBgIKZMmSLcq7ygoABTp05FYGAg\nXnzxReTn5wMAXn/9dSxbtgxjxoyBm5sb/u///k/bqWfsj6sb7kLIGNOiS5cu0dKlS0mhUNCYMWNU\nptna2pJcLicioocPHxIR0aNHj0ipVBIR0cmTJ2n27NlERJSSkkLu7u5UW1tL5eXlZGRkRF9++SUR\nES1fvpw+++wzIiKSSqX01ltvERFRVlaW8KzwlJQUio6OJiKiuXPn0tmzZ4mISCaT0dChQ4mIKCws\njM6dO0dERHV1dUIcLdLS0mjx4sXC8MOHD0kul9OoUaPowYMHRER04MABevPNN4mIaOLEiXTjxg0i\nIjp//jxNnDiRiIgWLVpEERERRESUl5dH7u7uT59Yxvop3Z7+4sEY69iVK1cwfPhwXLt2DUOHDlWZ\nNnz4cMybNw8zZswQHsxRXV2NhQsX4ubNmxCJRFAqlcL8EyZMwODBgzF48GCYmJggLCwMADBs2DD8\n85//FOZrOXowbtw4PHr0CA8fPlTp94cffsC1a9eE4ZqaGtTV1WHMmDFYvnw55s+fj1mzZrV7Stjw\n4cOxcuVKrFq1CqGhoRg7dixyc3Px66+/4k9/+hMAoKmpCba2tqirq8O5c+cQHh4uLC+XywE8Oe3Q\n8nqHDh0qPLOcMdY5LvyM9VK//PILXn/9dRQXF8Pc3Bz19fUgIgQEBODcuXMYNGgQjhw5gqysLGRk\nZCA+Ph7/+te/sG7dOrz00kv4+9//DplMBqlUKrQ5cOBA4e8BAwYIwwMGDFD5gtDWgAGqZwWJCBcu\nXIC+vr7K+A8++AChoaE4cuQIxowZg+PHj+OFF14Qpg8ZMgQ5OTk4cuQI1q5di5deegkzZ86Et7e3\ncLqgxaNHjyCRSJCTk6M2ptZ9Ez9yhLEu43P8jPVSvr6+yMnJgYeHB65du4aJEyfixIkTyM7OxqBB\ng0BEuH37NqRSKRITE/Hw4UPU1tbi0aNHsLW1BQCkpKR0qa/WhZOIkJqaCgA4e/YsTExMIBaLVeaf\nPHkykpKShOGrV68CeHJO3tvbG++//z5GjBghnJNvcffuXQwaNAjz58/HypUrkZOTgxdeeAHl5eU4\nf/48AEChUCAvLw9GRkZwcXFBWlqaEFfroxKMsWfDhZ+xXqy8vBympqYAgN9++w2enp7CtKamJkRG\nRmL48OEICAjAsmXLYGxsjPfffx+rV69GQEAAmpqahKvxRSKRypX5bf9uPd+gQYMQEBCAJUuWIDk5\nud08SUlJuHz5Mnx9feHt7Y2dO3cCAD7//HMMGzYMvr6+0NfXx9SpU1Vez7/+9S8EBQXB398fGzdu\nxNq1a6Gnp4e0tDR88MEH8PPzg7+/P37++WcAwL59+5CcnAw/Pz/4+Pjg0KFDGuNnjHUNP5aXMaZi\nwoQJ2Lp1KwICAno6FMZYN+A9fsYYY6wf4T1+xhhjrB/hPX7GGGOsH+HCzxhjjPUjXPgZY4yxfoQL\nP2OMMdaPcOFnjDHG+hEu/Iwxxlg/8v84WswBa9qJnAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x1104d6490>"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Can you think about an even more \"stochastic\" gradient descent? (yes, 2 answers at least: 1) just care about the sign, not the magnitude, 2) just update one of the weights per datapoint)"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "_Bonus:_ Adding momentum"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can accumulate gradient in directions that are consistent thorough samples by using a momentum update. Instead of doing: $$W_{t+1} = W_t + \\lambda \\nabla f(W_t)$$ add a momentum $v$ (with a parameter $\\mu$ and $v_0=0$, $\\mu$ being a \"friction\", it makes sense to take it $\\in [0\\dots1]$, I would advise trying $0.9$) s.t.: $$v_{t+1} = \\mu v_t + \\lambda \\nabla f(W_t)$$ $$W_{t+1} = W_t + v_{t+1}$$"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "3. Adding hidden layers"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Some common non-linearities"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a = np.arange(-3, 3, 0.01)\n",
      "plt.plot(a, a, label=\"linear\", linewidth=2, linestyle='--')\n",
      "plt.plot(a, 1./(1+np.exp(-4*a)), label=\"sigmoid\")  # I multiply by 4 so that's it's less flat\n",
      "plt.plot(a, np.tanh(4*a), label=\"tanh\")            # I multiply by 4 so that's it's less flat\n",
      "plt.plot(a, (a+abs(a))/2, label=\"ReLU\", linewidth=1.5)\n",
      "plt.legend(loc=4)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "<matplotlib.legend.Legend at 0x11036a610>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAE4CAYAAAAaQfliAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xlc1VX+x/EXq4oKgsWiIAqK4lLYzxlTzLDCytIabXFJ\nWyxLnSZbLJfJyExbnbJFbbJsz7RMc0bT0dBMW0wxzQ0FUnBJDBVc4ML9/v74KkKCbHeF93PmO3f7\nLh/v+Lgfzzmf7zkehmEYiIiIiMN4OjsAERGRukbJV0RExMGUfEVERBxMyVdERMTBlHxFREQcTMlX\nRETEwWqUfE+fPk3Xrl2Ji4ujffv2jB8/3lZxiYiI1FoeNb3P9+TJk/j5+VFYWEiPHj146aWX6NGj\nh63iExERqXVq3O3s5+cHQEFBAUVFRQQFBdU4KBERkdqsxsnXarUSFxdHSEgIvXr1on379raIS0RE\npNaqcfL19PQkJSWFzMxM1qxZQ3Jysg3CEhERqb28bXWigIAAbrjhBjZs2EBCQkLx+x4eHra6hIiI\niFuoqJyqRi3f7Oxsjh49CsCpU6dYsWIFnTt3LjMIbRfennrqKafH4C6bvit9T/qu9D25wma1Grz5\n2Wl6rd4C33xD+x9+YO2ZnFiRGrV8Dxw4wJ133onVasVqtTJ06FCuvvrqmpxSRETE5e1OM+g3J4vt\n3dPxOG3wTOtWPN4yAl/PyrVpa5R8O3XqxMaNG2tyChEREbdhscBjs3J5zXcXRmIu3imBJDWJYfw1\nDahk3gVsOOYrNVNynFwuTN9V5eh7qjx9V5VT17+nvMJCOryUwd6/ZsIxH+JXxbJgVDChoVWvbarx\nJBsVXsDDAztfQkRExK6WZGczOjWVvfn5+K8O493uUfRP9Clz38rkPSVfERGRcmTl5/NQaiqfZ2fT\n3s+P2TFt6VI/gPr1yz+mMnlP3c4iIiJ/sifdYIlXFk/+lo7FMJjaqhWPRlS+oKoiSr4iIiJnWCzw\n6MxcXq+3C6NtLomBgcyMiSG6QQObXkfdziIiIsDKdYXcvjSDIwlmQdVVO1rzv/HBVZ4sSmO+IiIi\nFTh9Gvq/kM3SmFQIrbigqiJKviIiIhdQsqCKDD+GHW7LrL8HUJNeZiVfERGRMhQZBjOzspiQbhZU\njWoUyVDfCOI61rygStXOIiIif5KSm8uIXbv4Kdd+BVUVUfIVEZE6YeW6QoatyuBQj0ya+vjwcWws\nA4OrXlBlC+p2FhGRWi0nBwa/ls2yMwVVHdLC+HZoFIE+1SuoqojGfEVEpM4yDJg5P59H0lPJ72q7\ngqqKKPmKiEidVGQYTN6cxeSsdPA2iFwdyZeDbVNQVRElXxERqXNKFlS1PhrI3UdjGDesakv+1YSq\nnUVEpM7IKywkKSODVzKdX1BVESVfERFxazk5kPR1Nl9Gmkv+jQgL47ko+xVU2YKSr4iIuKU/F1RF\nnvZj7WWdiQ8IcHZoFdKYr4iIuJ3UPQb95mSxI75EQdWQCOI6OGhg9wJUcCUiIrXO7JW5jNxtLvnn\nnRJIUpMYxjuwoKoiKrgSEZFao7igyisTj1Afuq2K5fNRwYSGul5BVUXU8hUREZe3JDub0annCqqe\nCIwiKtg1C6rU8hUREbdlGLDlQD6T/zCX/Gvv58fazu5RUFURJV8REXE5u9PMgqqdPdLx9TOY2qoV\nj0ZE4OsqA7s1pOQrIiIuw2KBx2bl8prvLoxEs6Dq0x4x3BTp2CX/7E1jviIi4hKSfyjk1iUZZCdk\nwjEf4lNas8ANC6p0q5GIiLiFJdnZ3L05lWyvfPxXh/Fu9yj6J7pmQVVFlHxFRMSlZeXn81DquYKq\nm3a35cm/2XfJP3tT8hUREZdUZBjMzMpiQno6FsNgUmRkrSmo0q1GIiLiUs4WVH0ZvYu9frkkBgYy\nMyaGaHdu6laDkq+IiDjEynWF3L40gyMJmXDch1kxsYyIds0l/+xNyVdEROzq6FEYPCObpTGpcHWJ\ngqrW7llQZQsa8xUREbvJys/n8g9TyYzOhgw/hv7eltkPundBVUVUcCUiIk5RsqCqoMggfFUkC26L\nIK6j+xdUVUTJV0REHC4lN5cRu3bxU27dLKhStbOIiDjMynWFfNYggznHMmnq48PHsbEMDK6bBVUV\nUfIVEZEaKVVQFWou+fdcVBSBPnW3oKoi6nYWEZFqMQyYtSCfh9NSye9qFlQNO9yWdx4JwMvL2dE5\nT2XyXo1Gvvft20evXr3o0KEDHTt2ZMaMGTU5nYiIuIlCq8GlT2Uyyu9H8uP+IHJ5KzZ16cJ7Y+t2\n4q2sGrV8Dx48yMGDB4mLiyMvL4//+7//48svvyQ2NvbcBdTyFRGpVUoWVHmnBPJ0kxjGDWtALZgZ\n0ibsXnAVGhpKaGgoAI0aNSI2Npb9+/eXSr4iIlI75BUWkpSRwSuZZkHV3NaxXNvO/Zb8cwU2K7jK\nyMhg06ZNdO3a1VanFBERF3DyJKw6mc3o1FT25qugyhZsknzz8vK45ZZbePXVV2nUqNF5nyclJRU/\nT0hIICEhwRaXFRERO/pzQVV7Pz/Wdu5MfECAs0NzKcnJySQnJ1fpmBpXO1ssFm688Uauv/56xowZ\nc/4FNOYrIuJ2UvcY9JuTxY74dPA2uHRzJD8+VjuW/LM3u89wZRgGd955J02bNuVf//pXtYMQERHX\nYLHAozNzeb3eLoy2KqiqDrsn37Vr19KzZ08uueSS4hlMpk2bxnXXXVelIERExPnyCguZmJrBjP2Z\ncMyH+JTWLBilgqqq0tzOIiJSKUuyzxVU3WCEcbcligG9VVBVHZrbWURELigrP5+HUlP5PFsFVY6k\n5CsiUgel7jG484sstnZNx4LB1FateDRCBVWOom5nEZE65M8FVVF/BLL8+rq15J+9qdtZRESK/e+7\nQgYuy+BIwpmCqlWxZkFVAxVUOZpaviIidcBrP2Xzjz3mkn/+q8N4t3sU/RNVUGUPqnYWEanjShZU\n+f/hx8172jLr7wGol9l+lHxFROqoIsNgZlYWE9LTsRgGkyIjeSQ8gnpeKqiyN435iojUMRYLfPRD\nLm82MJf8SwwMZGaMCqpcjZKviEgtUbKgqqnVh49jYxkYHFw8A6G4DiVfERE3l5MDg1/LZllMKlxt\nFlR9cm0UiSEqqHJVGvMVEXFj85PzGfqDueQfGX4MO6yCKmfTmK+ISC11tqBqvEc6+XEGkctb8eXg\nCOI6qqDKHajlKyLiZlJycxmx61xB1T+KYugTpyX/XIVaviIitcjR/EKm7MvglcxMmvqooMqdKfmK\niLi4swVVq+NSOeWfz4iwMJ6LiiLQRwVV7krJV0TERRkGzJyfzyPpqeT3NAuq3o/ozNC2WvLP3WnM\nV0TEBe1OM+g3J4vt3dPB2yBydaQKqtyEppcUEXFDKbm53Ji8i6zGuXinBJLUJIbxw1RQ5S6UfEVE\n3EheYSFJGecKqrr80Jo5g4MJDVVBlTtR8hURcRNLsrMZnZrK3nwVVLk73WokIuLCDANmLcjno6BU\nvvPKpr2fH2s7dyY+QAVVtZ2Sr4iIE5QqqCoyeKpFKyZER+Crgd06QclXRMSBLBZ4bFYur/nuwkgs\nUVB1jQqq6hKN+YqIOEheYSFxr2awJy4TjvnQPaU1n49SQVVto4IrEREXUbKgyn91GO90i2JAbxVU\n1UZKviIiTpaVn89Dqal8nm0WVL3Vti1/9QtAhcy1l6qdRUScZHeawWKPLJIy07EYBlNbteLRCBVU\niUnJV0TEhkoVVLU1l/ybGRNDtFa3lxLU7SwiYiMr1xVy+9IMjiSYBVVXbGlN8pPBeHqqoKou0Ziv\niIgDFBZCv6nZLI1JhVCzoOrd7lH0T9TAbl2k5CsiYmclC6rI8GPY4bbM+nsA6mWuu5R8RUTspMgw\nmJmVxYR0s6BqTFAkt6Ml/0TVziIiNmcYsDkvlxG7dvFTrgqqpHqUfEVEKmnlukKGLM8gO8Fc8u/j\n2FgGBgfj4aGCKqkadTuLiFTg6FEYPCObpW1TISSf1jvC+HG4lvyTsmnMV0SkBs4u+fdwWir5XVVQ\nJZWj5CsiUk1FhsHzO7KYmJ4OXgaRqyP5crAKqqRilcl7Nf5bdM899xASEkKnTp1qeioREZeQkptL\nt40bmXhoN7GGP88e+AtpUyKVeMVmatzy/fbbb2nUqBHDhg1jy5Yt519ALV8RcRN5hYUkZWTwSqZZ\nUPVK69YqqJIqc8itRldccQUZGRk1PY2IiNMcPQpjv8hmeay55N+IsDCei1JBldiPbjUSkTrrzwVV\nzfP8WPvXzsQHBDg7NKnllHxFxHUYBpw4AceOmdvRo+bjqVNw+jTk55vbn58XFkJREVit5x5LPi/j\n8ViewVNNO/DWbZdjjfPk3o9+YmpmCk0bFmKxFmK1WrEaVooMK4a1iCLDfG21FpmPhhWrUVS8n4Fh\ndjUaVgzDKH599jkYGNYS73N2X84de2ZPSnRZGuX8b+ndLvT5ee8WP5R9VJX/T6v2kQ67kuMPrJBD\nkm9SUlLx84SEBBISEhxxWRFxNXl5sHMn7NgBqamQlQX795/b/vgDfH0hIMDcmjQBf3/w84P69aFe\nPXM7+7x+fWjYEHx8wNOTAorIKzzJicLTnLCe4kShhbyiU+QVniKv6CQnCk9x0shnq3dT5l92C380\nb0lI+hZ6rJtF4am9/CPgNFbDiq+vL96e3nh7euPj5WM+9/LB29MHn+Ln3nh7eePt6YO3lzdeHl54\nenjiiScenp7mcw9PPD29zj0v47W5rxdeHp544IGHx5mNM+PMHuYzjzP/Mf/rUWoc+txnZ/Y989nZ\nc5Tcv6JzV4dHNQ905Fh6tWOsxHE/78pg467fzr2x6feKz2uLW40yMjLo27evCq5E5ByrFTZtgrVr\nYf16+OEHOHQI2rSBdu0gJgbCw6FZM2jeHMLC4KKLzERahtz8XNJy0tiTs4ffjv7G/tz9HMg7UOox\nvzCf4IbBBDUIKnNrUC+IrwtDWXq6IeRBx031mDEggJbNGtHItxENfRtSz6ueCqykRhxyn++gQYNY\nvXo1R44cITg4mMmTJ3P33XdXKQgRqSUKC2H5cli4EJYsMVuvCQnQrRtcfjm0bg1eXuUebjWs/Hb0\nN7b8voUth7awPXs7e3L2sOePPeQV5BEVGEV0UDQtA1rSrHEzmjVuRljjMPOxURhN6jcpN3Euyc5m\ndOq5gqqJwVG0CFRBldieJtkQEcc4eBBefx3efRdatIDbboO+fc1kWw7DMEjLSWN95nq+z/yenw/8\nzK+//4p/PX86hXSiU3An2l/cntZBrYkOjCa0UWiVW6SGARt+y+f5PHPJv/Z+frzVtq0KqsSutKqR\niNjXoUMwZQp89BEMHmy2ejt0KHNXq2El5WAK/0v7H2v3ruX7zO/x9fKlW0Q3uoV347YOt9EpuBOB\nDQJtElrqHoN+c7LY2SOdeg0NprZqxaMREfh6aqIMcT61fEWk6goL4c034ZlnYOhQePxxCA09b7fD\nJw6zZNcSlqctZ2XaSgIbBJIYlciVkVfSLaIb4f7hNg/NYoFHZ+byer1dGG1z8U4J5MP/i+H2KzQZ\nsziGup1FxPbS02HQILMC+Y03IDa21MdZx7P4YvsXfL79czYd3ETv6N5cG30tiVGJRDaJtGtoq38s\nZMBXGRxJyIRjPsSntGbBqGBCQ1VAJY6j5CsitrVwIdx/P4wbBw8/DGfGYI/nH2f+r/OZu3ku2w5v\no29MX/rH9icxKpEGPo5pcS7JzuberakcIh//1WG82z2K/okqqBLHU/IVEdv5179g+nT44gv4y18w\nDIPkjGTeTXmXxTsX06tVL+6Ou5vrW1+Pj5fjkl5Wfj4PpZ4rqBp8oC2P9NaSf+I8Sr4iUnOGYbZ0\nv/oKli3jZNhFfPjLh7z6w6t44MG9l93LkE5DuLjhxQ4Nq8gwmJmVxYT0dCyGwaTISBVUiUtQ8hWR\nmps4Eb7+mv0L5vLa7o94e9PbdAvvxpjLx9CrZS+HT0hxtqBqQeQuDgTkkhgYyMyYGKLV1BUXoVuN\nRKRmpk/HMn8eE5Ou4O15PbnjkjtYP3w9rYPKv3/Xnv73XSEDl50pqDruw6sXxfJgBy35J+5HyVdE\nynTovTfxmTaJhHu96RMcwo6bdhDcMNgpseTkwODXslkWkwpX59M42SyoGtBRBVXintTtLCKl/H7i\nd2Z/9AgjH/mY+S/ezW13vkBTv6ZOiycrP5+en6WSFpENGX4M/b0tsx9UQZW4Lo35ikilnSg4wb++\n/xf/Xv0vvn8bGj3+TxqPfthp8ZQqqLIaRK6O5NP+EcR1VEGVuDYlXxGpUJG1iHdT3uWp5Ke4osUV\nvLXYwN+zgTlPs5PGUlNycxmxaxc/5aqgStyPCq5E5IJ+yvqJB/7zAH4+fiy8fSF//fUorL4Ptmxx\nSuJdua6QD70y+OBUJk19fPg4NpaBwSqoktpHyVekDso5lcPEVRP5YvsXvJD4AkMvGYpHbi7c1wn+\n/W9zAXsHOnoUBs/IZmnbVAjJ577QMJ6PjiKwnLV9RdydBk9E6hDDMHh/8/u0f7M9hmGwffR2hl06\nzGxZjhsHvXubm8PigZnz8wmdvZWlPbfCKS+G/tiZ16PaKvFKraaWr0gdse3wNkb9ZxS5BbksGriI\nvzb/67kPN2+Gzz+HHTscFk+RYfDXKVlsvCwd4gwil7fiy8EqqJK6QQVXIrXciYITPLPmGeZsmsNT\nVz7FyC4j8fL0OreDYcBVV8Ftt8HIkQ6JqWRBlXdKIElNYhg/rAGaGVJqAxVcidRxi3Ys4qFlDxHf\nIp4tI7cQ2uj8NXf54gvIzob77rN7PHmFhSRlZPBKpllQ9WHbWK6JDSYkRAVVUrco+YrUQhlHM/jH\n0n+w68gu3rnpHa5qdVXZOxYUwOOPw+zZ4G2/n4Pjx2F1QTZ/T01lb34+I8LCeC5KBVVSdyn5itQi\nBUUFvLTuJaavn84j3R5h/q3zqeddr/wD3n0XoqPhmmvsEo9hwKwF+Tyclkp+V3PJv7WdOxMfEGCX\n64m4C435itQSq9JXMfq/o4kOjOa161+jVWCrCx+Qnw9t2sBnn8Hll9s8nt1pBv3mZLG9ezp4G8T+\nFEnKBC35J7WfxnxF6oADuQcYu2Is3+79lleve5Wb2t5UuUkp3n4bOnWyeeK1WOCxWbm85rsLI7FE\nQdUEFVSJnKXkK+KmLEUWXv/xdZ799lnuvexeto3aRkPfhpU7+PRpmDoVFi2yaUx5hYVMyshgRvtM\nOOZD/KpYFowKJjRUBVUiJSn5irihNb+tYfR/RxPWKIzv7vmOthe1rdoJ3nsPOneGLl1sFtOS7GxG\nnymo+lv9MIYURTFgsgqqRMqi5CviRs52Ma/5bQ3Tr53OgNgBVZ/32GqFl182p5G0gaz8fB5KTeXz\nbBVUiVSWkq+IGzhRcILp66fzyg+vMOKyEWwfvb3yXcx/9tVX0KQJ9OxZo5hS9xgM+jSLnVekU4jB\n1FateDRCBVUilaHkK+LCiqxFvLf5PSZ9M4keLXrw030/ERUYVbOTvvgiPPZYtVctsljg0Zm5vF5v\nF0Z8LhEHAvmmn5b8E6kKJV8RF2QYBsv3LGfsirH41/NnwW0LuDzcBlXJ69dDVhb071+tw1euK+T2\npRkcSfhTQVUDFVSJVIXu8xVxIYZhsCp9FUmrk/j9xO9Mu3oaf2v3N9utZztgACQkwIMPVvnQt3/N\n5r6t5pJ//qvDeLd7FP0TVVAl8meVyXtKviIu4M9J98meTzKw40C8PW3YOZWWBn/9K2RkQKNGlT6s\nZEFV4DE/+u5qy6y/B6BeZpGyaZINERdXaC3ki+1fMH39dP449QeTrpxk+6R71qxZcPfdlU68RYbB\nzKwsJqSnYzHMgqpHwiOo56WCKpGaUstXxAmOnT7GnE1zePWHV4kMiOSRbo/QN6Zv6aX+bOnUKWjR\nAr7/3pzL+QIsFpiTnMs7QeaSf4mBgcyMUUGVSGWp5SviQgzDYH3meuZsnMMXO76gT5s+fH7b53Rp\nZruJLso1f745oUYFibdkQVVgrg8fx8YyMDjYdmPOIgIo+YrY3YHcA3y85WPmbJpDkVHE8M7D2T56\ne9lr69rLm2/CxInlfnz0KAyekc3SmFS4Op/GyWF8dHUU14eooErEHtTtLGIHWcez+Hz758zfNp+t\nv2/lprY3MbzzcHq06OH4VuTPP5u3FqWlgdf53dqL1+Vz27fmkn9k+DH097bMflAFVSLVpW5nEQcp\nshax8cBGlu1exrI9y9h+eDt92/blifgnSIxKvPCauvY2cyY88MB5ifdsQdX4wnTy4wwil7fiy8ER\nxHVUQZWIvanlK1INVsPK9sPb+W7fd3yT8Q0r9qwgpFEI10Zfy7XR15LQMsG5CfesnByIioKdOyE4\nuPjtlNxcRuw6V1D1hG8MvdpqyT8RW1DLV8QGDMPgQN4BUg6msOnAJtZnrmfdvnUENQgivkU8V7e6\nmheueYGIgAhnh3q+996DPn2KE++Rk4VM25/BK5mZNPVRQZWIs9S45bts2TLGjBlDUVER9957L088\n8UTpC6jlK27ibJLddWQXqUdS2XVkF5sPbSblYAoGBp1DOxMXGkfX5l2JbxHv2IKp6jAMaNcO5swh\np0MPBr+WzapOqRQE5jMiLIznoqII9FFBlYit2X2Gq6KiItq2bcv//vc/mjdvzl/+8hc++eQTYmNj\nqxSEiCPkF+azP3c/mcczycrNMh+PZ7Hv+D52/7Gb3X/spqFvQ9oEtaFN0zbEBMVwScglxIXG0axx\nM/drHa5ahfHQQ8z85088krG7uKBqVpu23B+vJf9E7MXu3c4//vgjrVu3pmXLlgAMHDiQRYsWlUq+\nIrZiNaycKDhBbkEueQV55OafeTzz+nj+cY6cPMKRU+aWfTKbIyfPPJ46Qm5+LmGNwwj3D6d54+aE\n+4cT7h/O5eGXEx0UTZugNgTUrz1J6dj0t7gvfjzzG/0EKqgScSk1Sr5ZWVlERJwb5woPD+eHH344\nb78xW9eVebxB6X8ZVPQvhT/vf97nDj7+vPOd97KKn1dwvYqOv9D5qvPdGIaB1bBiNawYhoFBideU\n/qzke8XHYcWwGlgpvV+htZBCayFFRtG559ZzzwuNQgqLzjye+aygqICCogJ8vXyp51UPX2/zsb53\nffM973rU86pHQ59A/Oo3p6F/Qy72aUikjx8NfRvi5+NHQ5+GZbZeM4HMIlh9+Dhw/ILfk7swcnOZ\n1/dqNrRthneKP083iWHcFBVUibiKGiXfynbDvfrSW+dexMWZm1v685/XzbohncHGP/YFZ7YLsgKn\nz2xYgdwzW90S0iKc/ttjeWNgMKGh+rsqYi/JyckkJydX6ZgaJd/mzZuzb9++4tf79u0jPDz8vP2O\nv/12TS4jIlVhtUJcHA3mzsX7+hBnRyNS6yUkJJCQkFD8+umnn67wmBol3y5dupCamkpGRgbNmjVj\n3rx5fPLJJ+ft19hbdzSJ2INhwLx5Zr4dPPjMm19/DfXrw1/+4tTYRKR8NcqK3t7evP7661x77bUU\nFRUxfPhwFVuJOEhaGowaZebaJk0gMREuvhiYPduc0crdqrNF6hDNcCXiZiwWePllePppOH3aTLwv\nvADDh4Pnwf3QsSP89hs0buzsUEXqJM1wJVILDR8OH3xgPh88GKZPh5CzQ7tz5sDttyvxirg4tXxF\n3MzGjTBoELz2GvTuXeKDoiJo1QoWL3bjOwpE3J9aviK10GWXwbZtZawOuHQpNGumxCviBnTLvYiL\nSkuD7OyyPytjWV6YNQvuv9+uMYmIbSj5irgYiwWeew46dICxYyt5UEYGrF9vjveKiMtTt7OIC1m/\nHkaMgK1bzdcWizmUW2ZLt6Q33oC77wY/P7vHKCI1p4IrERdgGPDgg/Dmm+bz6GiYOdO8d7dCJ05A\nZCRs2ABnFjkREeepTN5Tt7OICzg7H4aXF0yYAFu2VDLxArz/PlxxhRKviBtRy1fERRw7BpmZ5lhv\npVmt5gEzZ0KJuWVFxHl0q5GIC7JaKXNpv4AAc6uSFSvA1xeuvNImsYmIY6jbWcSB1q+Hzp1hXdlL\nXFfdjBnw0EOax1nEzSj5ijjA0aMwciR07w6//GLeSlRj27aZRVaDBtngZCLiSEq+InZ0dsm/du3M\nOTC8vc2CqnnzbHDy554zW70NGtjgZCLiSCq4ErGj48ehdWs4fBji483V/qpUUFWe9HTo0gX27DGX\nNRIRl1GZvKfkK2JnCxZATs6ZJf9s1dc0ejT4+8O0aTY6oYjYipKvSG108CC0bw/bt5dYS1BEXIUm\n2RBxkKNHYcoUKCx0wMWmToVhw5R4RdyY7vMVqQHDgM8+M+ueDh0yp1Z+5BE7XjAjAz76yGz1iojb\nUvIVqaa0NBg1Cr7+2nwdHw/XXmvniz71lDneGxxs5wuJiD0p+YpUw5Yt0LUrnDplFhu/+CLcc48N\nC6rK8uuvsHQppKba8SIi4ggquBKpBsOAXr0gPBxeftkBw6+GAX36mKst2LVfW0RqStXOInZ0+jTU\nr++giy1cCP/8J6SkgI+Pgy4qItWhameRGjIMcx6Lsjgs8Z44AWPGwOuvK/GK1BJKviLlSEuD6683\nF0LYv9+JgTz7rFnN1auXE4MQEVtSwZXIn1gs5jju00+bXctNmphrGDRr5oRgfvgB5swxu5tFpNZQ\n8hUpYeNGuPNO2LrVfD1kiIMKqspy4gQMHQpvvAFhYU4IQETsRclXpARvb9ixA6KjYeZMs7jYaR57\nDC6/HG65xYlBiIg9KPmKlHDJJbBkCfTs6eSV+t5/H5YvN5viIlLr6FYjEVfz/ffQrx98842N1h8U\nEUfSrUYiZbBY4Pnn4b77nB1JGX77zexmnjNHiVekFlPLV+qU9ethxIhzBVW//AKdOjk3pmL79kFC\ngnlP74MPOjsaEakmtXxFzsjJgQcegO7dzcQbHW0OqbpM4s3KgquuMldqUOIVqfWUfKVOeOklmD3b\nrGaeONFcGMGplcwlbdoE3brB/ffDo486OxoRcQB1O0udkJsL994Lkya52FDq55+bTfKZM3VLkUgt\noYUVRFxjTHsLAAAYW0lEQVTV8ePm2O7q1TBvHnTp4uyIRMRGNOYrdc769eadOi7LMOCTT8zBZh8f\n2LxZiVekDtIkG1Ir5OTA+PHmuG7btmZOq1fP2VGVYLXCf/8LU6aY9zq9/z5ceaWzoxIRJ6l2y3f+\n/Pl06NABLy8vNmoWHnESw4BPP4XY2HMFVbfcYr7vEn7/HV591RxonjTJ7Gr+6SclXpE6rtot306d\nOrFw4ULuv/9+W8YjUiV33w3vvWc+j483E7BTC6oMA7Zvh2XLzO3HH6FvX3jzTfMeXg8PJwYnIq6i\n2sm3Xbt2toxDpFr69IFFi+CFF2D4cPB0ZBXD6dOwZw+kppr93D/9ZG4NGsB118HIkfDFF9CokQOD\nEhF3oDFfcWu33grXXANBQTU4SVERFBTAqVPmPUnHj597PLsdPgwHDsDBg+bjvn3m88hIiImBjh3h\nnnvMW4bCw9XCFZELumDyTUxM5ODBg+e9P3XqVPr27Vv5q9x4Y+nX5Q3I6X3XisWF3i8sMlu1npR+\n3wMIutA5iorMAqeCgvIfAXx9oX598PcvvTVubG7BwRAVZfZth4aaCTYy0hxkFhGpogv+cqxYscIm\nF0ny9y9+ntCpEwmdOpXfMtD75b/vSrE46H3DMKeBnD4dBg82F7qv0nm8vMzE6uNT/qOXV9nnExGp\nhOTkZJKTk6t0TI0n2ejVqxcvvfQS//d//1f2BTTJhlRTero51fGyZebrxET4+mv16IqIa7PrJBsL\nFy4kIiKC77//nhtuuIHrr7++uqcSKaWoyFzyr0MHM/E2aQJvvWU+V+IVkdpA00uKyzEMs5W7cqXZ\n1Tx9OoSEODsqEZHK0dzO4rZ274a0NOjd29mRiIhUjZKviIiIg2lhBXFp6ekwYADs3evsSEREHEs3\nKYrDWSzmOO7TT5vzWjRoAB9+6OyoRGofD1Uo2l11e3aVfMWh1q+HESNg61bz9eDB8PLLzo1JpDbT\nsJ/91OQfNxrzFYf5/Xdo0QLy883JombOVEGViD3p99e+yvt+VXAlLmfyZDP5/vOfZneziNiPfn/t\nS8lXRETOo99f+6pJ8lW1s9icxQILFzo7ChFxVS1btmTlypVMmzaN++67z9nhOIUKrsSmShZULVkC\nN9zg7IhExNV4eHjg4eHB+PHjnR2K06jlKzZx9Ki5dnx8vJl4o6KgYUNnRyUicj6r1ersEJR8peY2\nbYJ27WDWLHN1vgkTzASckODsyETEVRmGQVJSEkOHDgUgIyMDT09P3n//fSIjI7n44ouZOnVqqf2f\ne+45WrduzUUXXcTtt99OTk5O8ee33norYWFhNGnShCuvvJJt27YVf3bXXXcxcuRI+vTpQ6NGjaq8\n/J89KPlKjcXEmOvQx8dDSgo8+6wqmUWkYmXdJ/vdd9+xa9cuVq5cyeTJk9m5cycAM2bMYPHixaxZ\ns4YDBw4QGBjI6NGji4+74YYb2L17N4cPH+ayyy5jyJAhpc77ySef8OSTT5KXl0d8fLx9/2CVoGpn\nsYm9eyE8HDz1zzkRl1HR7295c0SUd0hZ+1fn571Vq1a8/fbbrF27lt27d/PBBx+QkZFBVFQUmZmZ\nNGvWDICuXbvy6KOPcttttxEbG8sbb7zBVVddBcCBAweIjIzk9OnTeP7ph+fo0aMEBQVx7NgxGjdu\nzF133QXA3Llzqx7sBdSk2lkFV1IlJ0+Cn9/577do4fhYRKT2CQ0NLX7u5+dHXl4eAL/99ht/+9vf\nSiVab29vDh06RHBwMBMnTmTBggUcPny4eJ/s7GwaN26Mh4cH4eHhjv2DVEDtFKmUswVVnTub8zGL\niPszjLK3quzvKC1atGDZsmXk5OQUbydPniQsLIyPP/6YxYsXs3LlSo4dO0Z6evqZeF2311XJVy7I\nMGDevHMFVWlpsHats6MSkdqgKsnxgQceYMKECew9swza4cOHWbx4MQB5eXnUq1ePoKAgTpw4wYQJ\nE6p9HUdR8pVypaVBnz4wcCAcOnSuoCox0dmRiYi7O3uvb8miqwstVPDQQw/Rr18/evfujb+/P926\ndePHH38EYNiwYURGRtK8eXM6duxIt27dzjuvq63wpIIrKdeXX8Lf/gZNmsCLL8I996igSsSd6PfX\nvjS3s9jNK6/AoEEQEuLsSESkqvT7a19KviIich79/tqXFlaQajtbUPXaa86ORESk7lDLtw5LS4NR\no+Drr6FePdi+HVq1cnZUImIr+v21L7V8pUosFnjuOejQwUy8TZrA669DZKSzIxMRqRs0w1Ud9PDD\n8MYb5vMhQ+Dll1VQJSLiSOp2roPS0+Gmm8ykq3t2RWov/f7al6qdpcoMo/xJ1UWkdtDvr31pzFfK\nlJZmrjZUFiVeEXE106ZN47777nO567Zs2ZKVK1fa9Jpq+dZCFovZpfz00+aC9v/9r5KtSF2k31/b\naNWqFXPmzClezvAsLSkoxdavhxEjYOtW83XTppCfby52LyIirkHdzrXIww9D9+5m4o2OhuXL4cMP\nlXhFxPU8//zzhIeH4+/vT7t27Vi1ahVJSUkMHTq0eJ/333+fyMhILrroIqZMmULLli1ZtWoVAElJ\nSdx6660MHToUf39/LrnkElJTU5k2bRohISFERkayYsWK4nPt37+ffv360bRpU9q0acPbb79d/Nmf\nr/vBBx8UX3fq1Kl2+fMr+dYiQUHg4wMTJ8KWLapkFhHXtHPnTt544w02bNjA8ePHWb58OS1btiy1\n8tC2bdsYPXo0n3zyCQcOHODYsWPs37+/1HmWLFnCsGHDyMnJoXPnziSe+dHbv38/Tz75JPfff3/x\nvgMHDqRFixYcOHCABQsWMGHCBL755huA8647atQoPvroI/bv38+RI0fIzMy0+Xeg5FuLPP44bN4M\nU6ZAgwbOjkZEpGxeXl7k5+fz66+/YrFYaNGiBVFRUaXGSRcsWEC/fv3o3r07Pj4+TJ48+bxlAXv2\n7EliYiJeXl7ccsstHDlyhHHjxuHl5cXtt99ORkYGx48fZ9++faxbt47nn38eX19fLr30Uu69917e\nf/99gPOu27dvX3r06IGvry/PPPMMnnZYzk1jvm6osBC8y/h/rl49iI11fDwi4p48nrZNJabxVNWK\nulq3bs0rr7xCUlISv/76K9deey3Tp08vtc/+/fsJDw8vft2gQQOaNm1aap/g4OBSn1900UXFCbrB\nmRZIXl4e+/fvJygoiIYNGxbv36JFCzZs2HBebH++rp+f33nXtQUlXzezbh3cfz888wzcfLOzoxER\nd1bVpGlLgwYNYtCgQeTm5nL//ffzxBNPEB0dXfx5s2bN2LlzZ/HrU6dOceTIkWpdq1mzZvzxxx/k\n5eXRqFEjAPbu3VsqyZbcd/v27cWvT548We3rXoi6nd1ETg488ADEx5sFVa+84uyIRESqZ9euXaxa\ntYr8/Hzq1atH/fr18fLyKrXPgAED+Oqrr1i/fj0FBQUkJSVV+7apiIgIunfvzvjx48nPz+eXX37h\nnXfe4Y477jhv3wEDBrBkyRK+++47CgoKmDRpElartVrXvRAlXxdnGPDpp2Z38uzZZnfzhAmwdKmz\nIxMRqZ78/HzGjx/PxRdfTFhYGNnZ2UybNg04V/zUoUMHXnvtNQYOHEizZs1o3LgxwcHB1KtXr3i/\nP48BX+j1J598QkZGBs2aNaN///5Mnjy5+L7dkufq0KEDb7zxBoMHD6ZZs2YEBQURERFh8+9Ak2y4\nuPx8c/WhPXvMVu/s2eZrEZGK1Kbf37y8PAIDA9m9ezeRLrIEm1Omlxw7diyxsbFceuml9O/fn2PH\njlX3VHIB9erBW2+Z25o1SrwiUnd89dVXnDx5khMnTvDYY49xySWXuEziralqJ9/evXvz66+/snnz\nZmJiYoq7DMT2rroK7rsP7FDtLiLishYvXkzz5s1p3rw5e/bs4dNPP3V2SDZjk27nhQsX8vnnn/Ph\nhx+ef4Fa1O1hT0ePwgsvwD//CX5+zo5GRGoD/f7al9Pndn7nnXcYNGiQLU5V5xgGfPYZPPQQHDoE\nVis895yzoxIREXu6YPJNTEzk4MGD570/depU+vbtC8Czzz6Lr68vgwcPLvc8SUlJxc8TEhJISEio\nXrS1THo6jBoFy5aZr+PjocT0oiIi4gaSk5NJTk6u0jE16naeO3cu//73v1m5ciX1y5m9X90eZcvI\ngPbt4dQpaNLE7HIePlzjuiJiO/r9tS+ndDsvW7aMF198kdWrV5ebeKV8LVvCDTeY1cwvvwwhIc6O\nSEREHKXaLd82bdpQUFBAUFAQAN26dePNN988/wL6l1e5LBZzFSIREXvQ76991aTlq0k27MwwYNs2\n3Z8rIo5XV35/PT092b17N1FRUQ69rlMm2ZCKpadDnz7QuTOUmKdbRKTOa9myJatWrXJ2GE6j5GsH\nFgs8/7zZ2l22DBo2NKeHFBERU11plZdHydfGtmyByy6DcePMSuYhQ2DHDrjxRmdHJiLiGoYOHcre\nvXvp27cvjRs35sUXX+TWW28lLCyMJk2acOWVV7Jt27bi/e+66y5Gjx7NjTfeiL+/P5dffjlpaWml\nzrlixQpiYmIIDAzk73//u6P/SFWm5GtjAQFmd3N0NCxfDh9+qEpmEZGSPvjgA1q0aMGSJUvIzc1l\n7Nix3HDDDezevZvDhw9z2WWXMWTIkFLHzJs3j6SkJHJycmjdujUTJ04s9fl//vMfNmzYwC+//MJn\nn33G119/7cg/UpUp+dpYixZmV/OWLZCY6OxoRETcw1133UXDhg3x8fHhqaeeYvPmzeTm5gJmF3X/\n/v3p0qULXl5eDBkyhJSUlFLHjxs3Dn9/fyIiIujVq9d5n7saJd8aKG+4okcPaNDAsbGIiFSZh4dt\nthoqKipi3LhxtG7dmoCAAFq1agVAdnZ28T4hJboQGzRoQF5eXqlzhIaGFj/38/M773NXo+RbDRaL\nOf/yzTeXn4BFRFyeYdhmq4aSC91//PHHLF68mJUrV3Ls2DHS09PPhFd7f2CVfKto/XqzoGr8eFi8\nGNaudXZEIiLuJyQkhD1nbgPJzc2lXr16BAUFceLECSZMmFBq36omYXdI2kq+lXT0KIwcCd27w9at\n5wqqrrjC2ZGJiLif8ePHM2XKFAIDA8nJySEyMpLmzZvTsWNHunXrVqpl7OHhUer12ffKel7e/q5G\nM1xV0vTp8Oij4O0Njz9urrurcV0RcWW15ffXVWl6SQewWOCBB+CRRzRVpIi4h9ry++uqlHxFROQ8\n+v21L83tbEPr14OL35stIiJuTsn3jJIFVXfdBceOOTsiERGprbydHYCzGQZ89hk89BAcOmQWVN1z\nD/j6OjsyERGprep88n3wQXjjDfN5fDzMnq2CKhERsa863+18yy0QGAj//jesWaPEKyIi9qdqZyA3\nFxo3dnYUIiK25Q6/v+5M1c6VcPQolDfPthKviIg4Uq1PvoYBn34K7drBpEnOjkZERFq2bImfnx+N\nGzcmNDSUoUOHcvz48QqPS0hIYM6cOWWeb+XKlaXemzt3Lle48Py/tTr5pqXB9dfDoEFmJfPGjVBY\n6OyoRETqNg8PD5YsWUJubi6bN29my5YtTJkypVLHlTVnszvM5fxntTL5GgY8/zx07GhOmNGkiVlQ\ntWqVeSuRiIi4hpCQEHr37s2vv/4KwPfff0/37t0JDAwkLi6O1atXOzlC+6iVydfDA375BU6dgiFD\nYMcOuPde8KyVf1oREfdztiApMzOTZcuW0bVrV7KysrjxxhuZNGkSOTk5vPTSSwwYMIAjR444OVrb\nq7XtwOnTzZmqEhOdHYmIiOsZk5pKSnlVqFUQ16gRr7RpU6VjDMPg5ptvxsPDg7y8PG666SYmTpzI\nSy+9RJ8+fbjuuusAuOaaa+jSpQv/+c9/GDZsWI1jdSW1ti0YEqLEKyLiijw8PFi0aBHHjx8nOTmZ\nVatW8fPPP/Pbb78xf/58AgMDi7fvvvuOgwcPXvB83t7eWCyWUu9ZLBZ8fHzs+ceoEbdu+aalmdNC\nTp0KnTo5OxoREfdR1daqvfTs2ZMHH3yQJ554gmuvvZahQ4fy1ltvVekcLVq0ID09vdR76enptGzZ\n0oaR2pZbtnwtFnjuOXM2qiVLzMXtRUTEPY0ZM4Yff/yRHj168NVXX7F8+XKKioo4ffo0ycnJZGVl\nFe9rsVg4ffp08WaxWLj99tt55ZVX2LlzJ4ZhsGHDBt59910GDhzoxD/Vhbldy3fdOrj/fti61Xw9\nZAi8/LJzYxIRkeq76KKLuPPOO5k+fTqLFy9m7NixDBo0CC8vL7p27crMmTOL9x05ciQjR44sfn3H\nHXfw3nvvkZOTQ9++fTl06BDh4eFMnTqV3r17O+OPUyluNb1kbi5ERJjL/UVHw8yZGtcVESmPppe0\nr5pML+lWyRdg1izYtw/++U9o0MBmpxURqXWUfO2rTiVfERGpHP3+2letW1jBYoH33zdnqhIREalt\nXC75rlsHl10Gd94Jc+c6OxoRERHbc5nke/QojBwJ8fFmJXNUFLRo4eyoREREbM8lbjXasQMSEsyV\nh7y9zft2VVAlIiK1lUsUXBUWQpcu0KgRzJ5tTp4hIiI1o4Ir+6oV1c6HDsHFF2vlIRERW3G3NW7d\nkcOT75NPPsnixYvx8PCgadOmzJ07l4iIiAqDOHYMAgKqc0URERHXZ9dbjR5//HE2b95MSkoKN998\nM08//fQF9z9bUNWuHfzxR3WvWnslJyc7OwS3oe+qcvQ9VZ6+q8rR92Q71U6+jRs3Ln6el5fHRRdd\nVO6+8+aZSXfWLMjOhjVrqnvV2kt/qStP31Xl6HuqPH1XlaPvyXZqVO08ceJEPvjgA/z8/Pj+++/L\n3e/swhLx8SqoEhERuWDLNzExkU6dOp23ffXVVwA8++yz7N27l7vuuouHH3643PM0aQL//rfZ4lXi\nFRGRus4m1c579+6lT58+bD27zl/JC6jaTkRE6piKUmu1u51TU1Np06YNAIsWLaJz587VCkBERKSu\nqXbL95ZbbmHnzp14eXkRHR3NzJkzCQ4OtnV8IiIitY7dJ9kQERGR0hwyn9STTz7JpZdeSlxcHFdf\nfTX79u1zxGXdztixY4mNjeXSSy+lf//+HDt2zNkhuaT58+fToUMHvLy82Lhxo7PDcUnLli2jXbt2\ntGnThueff97Z4bike+65h5CQEDp16uTsUFzevn376NWrFx06dKBjx47MmDHD2SG5pNOnT9O1a1fi\n4uJo374948ePL39nwwGOHz9e/HzGjBnG8OHDHXFZt7N8+XKjqKjIMAzDeOKJJ4wnnnjCyRG5pu3b\ntxs7d+40EhISjJ9//tnZ4bicwsJCIzo62khPTzcKCgqMSy+91Ni2bZuzw3I5a9asMTZu3Gh07NjR\n2aG4vAMHDhibNm0yDMMwcnNzjZiYGP2dKseJEycMwzAMi8VidO3a1fj222/L3M8hLd+qTMhRlyUm\nJuJ5ZnLrrl27kpmZ6eSIXFO7du2IiYlxdhgu68cff6R169a0bNkSHx8fBg4cyKJFi5wdlsu54oor\nCAwMdHYYbiE0NJS4uDgAGjVqRGxsLPv373dyVK7Jz88PgIKCAoqKiggKCipzP4ctYzBx4kRatGjB\ne++9x7hx4xx1Wbf1zjvv0KdPH2eHIW4oKyur1Dzr4eHhZGVlOTEiqU0yMjLYtGkTXbt2dXYoLslq\ntRIXF0dISAi9evWiffv2Ze5ns+Rrqwk5aruKvicwvytfX18GDx7sxEidqzLfk5RN99aLveTl5XHL\nLbfw6quv0qhRI2eH45I8PT1JSUkhMzOTNWvWlDslZ42mlyxpxYoVldpv8ODBdbpFV9H3NHfuXP77\n3/+ycuVKB0Xkmir790nO17x581JFjfv27SM8PNyJEUltYLFYGDBgAHfccQc333yzs8NxeQEBAdxw\nww1s2LCBhISE8z53SLdzampq8fMLTchR1y1btowXX3yRRYsWUb9+fWeH4xYM3Sl3ni5dupCamkpG\nRgYFBQXMmzePfv36OTsscWOGYTB8+HDat2/PmDFjnB2Oy8rOzubo0aMAnDp1ihUrVpSb7xxyn68m\n5KicNm3aUFBQUDxA361bN958800nR+V6Fi5cyD/+8Q+ys7MJCAigc+fOLF261NlhuZSlS5cyZswY\nioqKGD58+IVveaijBg0axOrVqzly5AjBwcFMnjyZu+++29lhuaS1a9fSs2dPLrnkkuJhjWnTpnHd\nddc5OTLXsmXLFu68806sVitWq5WhQ4cyduzYMvfVJBsiIiIO5rBqZxERETEp+YqIiDiYkq+IiIiD\nKfmKiIg4mJKviIiIgyn5ioiIOJiSr4iIiIMp+YqIiDjY/wMIMr2/EHRuRAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110f2f910>"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a = np.random.standard_normal((20,20))\n",
      "def non_linearities_imshows(a):\n",
      "    plt.imshow(a, label=\"linear\", cmap=plt.cm.gray_r, interpolation=\"none\")\n",
      "    print(\"linear\")\n",
      "    plt.colorbar()\n",
      "    plt.show()\n",
      "    plt.imshow(1./(1+np.exp(-4*a)), label=\"sigmoid\", cmap=plt.cm.gray_r, interpolation=\"none\") # /!\\ mult by 4\n",
      "    print(\"sigmoid\")\n",
      "    plt.colorbar()\n",
      "    plt.show()\n",
      "    plt.imshow(np.tanh(4*a), label=\"tanh\", cmap=plt.cm.gray_r, interpolation=\"none\") # /!\\ mult by 4\n",
      "    print(\"tanh\")\n",
      "    plt.colorbar()\n",
      "    plt.show()\n",
      "    plt.imshow((a+abs(a))/2, label=\"ReLU\", cmap=plt.cm.gray_r, interpolation=\"none\")\n",
      "    print(\"Rectified Linear Unit\")\n",
      "    plt.colorbar()\n",
      "    plt.show()\n",
      "\n",
      "non_linearities_imshows(a)\n",
      "#non_linearities_imshows(digits.images[0])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "linear\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEzCAYAAADD4uP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X1UVHX+B/D3CPw0VEQFh4fBxeRpQB6mMNNNXdNR1CQo\ntnStPEhGlttmVutuZ0/ZORpW55hp2zHPrqEV6W6alDib1oF8iCgBS0FgW5BHSTJMIkVGfn9044Q8\nzXyYYZjL+3UO5zgz932/35DeXO/c+x1NW1tbG4iIyKkNcfQEiIio71jmREQqwDInIlIBljkRkQqw\nzImIVIBlTkSkAq722rFGo7HXrolokLPHFdV96ayBcIW33cpcavXq1aKct7e3KHfbbbeJcjt37hTl\nwsPDu33NZDIhLi6uy9e2b98uGi8sLEyUc3NzE+ViYmJEudOnT3f72ldffYWoqKguX/P397d6rOHD\nh1udAYC///3volx6eroo99FHH3X72vHjxzFt2rQuX5N8T4Ce/w56MmXKFFFOWp4PPfSQKGcJyZwG\nQpEDfTjNYjKZEBYWhuDgYGzcuNGWcyIicgiNRmP110AhKnOz2YxVq1bBZDKhqKgIGRkZKC4utvXc\niIjIQqIyz8vLQ1BQEAIDA+Hm5obFixdj//79tp7boBMUFOToKQxIWq3W0VMYcAICAhw9BVUadEfm\nNTU1HX6YdDodampqbDapwYpl3jWWeWcsc/tw5jIXvQE6kP4DiIhsxZm7TVTm/v7+qKqqan9cVVUF\nnU5ns0kREf2ipKQEJSUl/TLWoCvz2NhYlJWVoaKiAn5+fti9ezcyMjJsPTciIoSGhiI0NLT98Ycf\nfmi3sQZdmbu6umLr1q2YN28ezGYzUlJSoNfrbT03IqJ+NejKHADmz5+P+fPn23IuREQONSjLnIhI\nbVjmREQqwDInIlIBlnk3duzYYXVm69atorFWrVolyo0dO1aUi46OFuVycnJEOeniSY2NjaLcnDlz\nRLmsrCxRTvoG+gMPPGB1Rvp3MH36dFHu7bffFuVaW1tFOemCWdIFyLKzs/t1PHty5jLneuZERCrA\n0yxERApnPjJnmRMRKVjmREQqwDInIlIBljkRkQqwzImIVIBlTkSkAixzIiIVcOYy501DREQKW3xs\nXFVVFWbNmoWIiAhMmjQJr776arfjffHFF3B1dcXevXv7PHcemRMRKWxxZO7m5oZNmzYhJiYGTU1N\nuPnmm2E0GjstWWE2m/HnP/8ZcXFxaGtr6/O4PDInIlLY4sjcx8cHMTExAIARI0ZAr9ejtra203Zb\ntmxBUlISvL29bTJ3ljkRkZ1UVFSgoKAAU6ZM6fB8TU0N9u/fj5UrVwKwzb8I7Hqapbm52erMxIkT\nRWOZTCZR7o033hDlkpOTRbkTJ06Icq+//rooV19fL8pduXJFlLv33ntFuaSkJFFOsjLnqVOnRGPt\n2bNHlJOs7AgA4eHhoty3334rynl4eIhyXl5eolx5ebkoZ0+WlOqVK1fQ0tLS63ZNTU1ISkrC5s2b\nMWLEiA6vPf7440hLS4NGo0FbW5tNTrPwnDkRkcKSMh82bBiGDRvW/ripqanTNlevXsXdd9+N++67\nDwkJCZ1eP3HiBBYvXgwAaGhowMGDB+Hm5ob4+Hjx3FnmREQKW5zuaGtrQ0pKCsLDw/H44493uc3/\n/ve/9j8nJydj0aJFfSpygGVORNTOFmV+7NgxvPXWW4iKioLBYAAAbNiwAZWVlQCA1NTUPo/RFZY5\nEZHCFmV+22234dq1axZvL3nfpysscyIihTPfAcoyJyJSsMyJiFSAZU5EpAIscyIiFWCZExGpgDOX\nOddmISJSAR6ZExEpnPnInGVORKRw5jLXtNliua6udqzR4Pnnn7c6d+edd4rGy8zMFOVuuukmUU7q\n008/FeW6Wg/ZEpas7tYV6Yp9n332mSg3dOhQUW7GjBlWZ6QrV65fv16Uk/63vffee6Kci4uLKFdT\nUyPKSUn/37v77rttssrg9TQaDUJCQqzOlZaW2mU+1uKRORGRwpmPzFnmREQKljkRkQqwzImIVIBl\nTkSkAixzIiIVYJkTEakAy5yISAWcucy5NgsRkQrwyJyISOHMR+YscyIiBcuciEgFWOZERCrAMu9G\nfX291Zl///vforG++OILUc7Dw6Nfcz4+PqLcmTNnRLmwsDBRrri4WJSbMGGCKOfr6yvKubu7W52R\nrkD56quvinLSlSuHDJFdn3DrrbeKcufPnxflcnJyRLlFixaJcvbEMiciUgGWORGRCrDMiYhUwJnL\nnDcNEREpNBqN1V/XW758ObRaLSIjI7sdJzs7GwaDAZMmTcLvfvc7m8ydZU5EpLBFmScnJ8NkMnU7\nRmNjIx599FF88MEHOHXqlPiij+uxzImIFLYo8+nTp2P06NHdjvHOO+/g7rvvhk6nAwB4eXnZZO4s\ncyKiflRWVoYLFy5g1qxZiI2Nxa5du2yyX74BSkSksOQN0IsXL+LixYviMa5evYr8/Hx8/PHHaG5u\nxtSpU3HrrbciODhYvE+AZU5E1M6SMvf09ISnp2f74+rqaqvGCAgIgJeXF2644QbccMMNmDFjBk6e\nPNnnMudpFiIihS3OmffmzjvvxNGjR2E2m9Hc3IzPP/8c4eHhfZ47j8yJiBS2uM58yZIlyMnJQUND\nAwICArBu3TpcvXoVAJCamoqwsDDExcUhKioKQ4YMwYoVK1jmRES2ZIsyz8jI6HWbJ598Ek8++WSf\nx/o1ljkRkcKZ7wC1a5nPnDnT6syePXtEY4WEhIhyX375pSh34403inJjx44V5aS/xXu63rUna9as\nEeWkK/b95z//EeUiIiKszuj1etFYkhUaAfmKnt7e3qLcu+++26/j3XPPPaJceXm5KGdPLHMiIhUY\nlGUeGBgIDw8PuLi4wM3NDXl5ebacFxFRvxuUZa7RaJCdnY0xY8bYcj5ERA4zKMscANra2mw1DyIi\nh3PmMhffNKTRaDBnzhzExsZi+/bttpwTEZFD9MdNQ/YiPjI/duwYfH19cf78eRiNRoSFhWH69Om2\nnBsREVlIXOa/fACvt7c3EhMTkZeX16nMf32ZYUREhOgyMiIa3IqKilBUVNQvYw2kI21ricq8ubkZ\nZrMZI0eOxI8//oiPPvoIzz77bKftpNefEhH9Ijw8vMPt7u+9957dxhp0ZV5fX4/ExEQAQGtrK5Yu\nXYq5c+fadGJERP1t0JX5hAkTUFhYaOu5EBE51KArcyIiNWKZExGpAMu8G1VVVVZnpkyZIhrr6NGj\notxf/vIXUU66SFBjY6Mot23bNlEuNDRUlHv44YdFuTNnzohyCQkJolxAQIDVmcrKStFY58+fF+Wk\ni8BJ/+6mTp0qyp04cUKUKygoEOVmzJghytkTy5yISAVY5kREKsAyJyJSAZY5EZEKOHOZixfaIiKi\ngYNH5kRECmc+MmeZExEpWOZERCrAMiciUgGWORGRCrDMiYhUwJnLnJcmEhEpbPUZoCaTCWFhYQgO\nDsbGjRs7vd7Q0IC4uDjExMRg0qRJePPNN/s8d5Y5EZHCFmVuNpuxatUqmEwmFBUVISMjA8XFxR22\n2bp1KwwGAwoLC5GdnY01a9agtbW1T3O362mW+vp6qzPSf+YMGzZMlMvPzxflRo4cKcqVlpaKcmaz\nWZS7evWqKPfuu++KcsHBwaLczTffLMpJVsuUrg540003iXL333+/KFdSUiLKvfzyy6Lc2LFjRblp\n06aJctcX3EBgi9MseXl5CAoKQmBgIABg8eLF2L9/P/R6ffs2vr6++OqrrwAAP/zwA8aOHQtX177V\nMc+ZExEpbFHmNTU1HZZm1ul0+Pzzzztss2LFCtx+++3w8/PDpUuXsGfPnj6Py9MsREQKW5xmseQX\nwoYNGxATE4Pa2loUFhbi0UcfxaVLl/o0dx6ZExFZob6+vsdTyP7+/h0+mKeqqgo6na7DNsePH8cz\nzzwDAJg4cSImTJiAkpISxMbGiufFI3MiIoUlR+I+Pj6Ijo5u/7pebGwsysrKUFFRgZaWFuzevRvx\n8fEdtgkLC8Phw4cB/PzLoaSkBDfeeGOf5s4jcyIihS3Ombu6umLr1q2YN28ezGYzUlJSoNfr2z/6\nMTU1FX/961+RnJyM6OhoXLt2DS+++CLGjBnTt3H7PHMiIpWw1U1D8+fPx/z58zs8l5qa2v5nLy8v\nfPDBBzYZ6xcscyIihTPfAcoyJyJSsMyJiFSAZU5EpAIscyIiFWCZExGpAMuciEgFWObdGDp0qNUZ\n6Sp/t9xyiygnHa+iokKU279/vyiXkpIiyklXMfT39xfl3N3dRbl9+/aJcpL/PunqdJMnTxblzp07\n16/j7d27V5RramoS5S5fvizKlZWViXL25Mxlztv5iYhUgKdZiIgUznxkzjInIlKwzImIVIBlTkSk\nAixzIiIVYJkTEakAy5yISAVY5kREKsAyJyJSAZY5EZEKsMyJiFTAmcuca7MQEamAXY/MJSsL3nrr\nraKxjEajKLdlyxZR7ttvvxXloqKi+nW8YcOGiXLSTw5/5JFHRDnp319JSYnVGRcXF9FYX375pSh3\n5coVUe7UqVOi3KRJk0S5oqIiUc7Ly0uUu+OOO0S53bt3i3KWcOYjc55mISJSsMyJiFSAZU5EpAIs\ncyIiFWCZExGpgDOXOS9NJCJSaDQaq7+6YjKZEBYWhuDgYGzcuLHLbR577DEEBwcjOjoaBQUFfZ47\ny5yISGGLMjebzVi1ahVMJhOKioqQkZGB4uLiDttkZWXhv//9L8rKyvDGG29g5cqVfZ47y5yISGGL\nMs/Ly0NQUBACAwPh5uaGxYsXY//+/R22yczMxLJlywAAU6ZMQWNjI+rr6/s0d5Y5EZHCFmVeU1OD\ngICA9sc6nQ41NTW9blNdXd2nufMNUCIihSVvgFZUVODs2bN92gcAtLW1iXLdYZkTEVkhMDAQgYGB\n7Y+PHDnS4XV/f39UVVW1P66qqoJOp+txm+rqavj7+/dpXjzNQkSksMVpltjYWJSVlaGiogItLS3Y\nvXs34uPjO2wTHx+PnTt3AgByc3Ph6ekJrVbbp7nzyJyISGGL68xdXV2xdetWzJs3D2azGSkpKdDr\n9di2bRsAIDU1FQsWLEBWVhaCgoIwfPhw7Nixo8/jatquP3FjIxqNBiaTyercmTNnROP9+OOPotwr\nr7wiyj388MOi3OXLl0W5W265RZTz9PQU5XJzc0U56SqN0lUhKysrrc4sWLBANNbIkSNFuev/GW6p\nuro6UW7mzJmi3KhRo0S5devWiXIJCQmi3MaNGzudb7YFjUaDZ5991urcunXr7DIfa/HInIhI4cx3\ngLLMiYgUzlzmPb4Bunz5cmi1WkRGRrY/d+HCBRiNRoSEhGDu3LlobGy0+ySJiPqDrW7nd4Qeyzw5\nObnTee+0tDQYjUaUlpZi9uzZSEtLs+sEiYj6i2rLfPr06Rg9enSH5359G+qyZcvw/vvv2292RET9\nyJnL3Opz5vX19e3XQ2q12j6vJ0BENFAMpHK2Vp/eAO3tN9OuXbva/xwVFYXo6Oi+DEdEg1BlZaXo\nElSJQVXmWq0W586dg4+PD+rq6jBu3Lhut73//vv7NDkiovHjx2P8+PHtj48dO2a3sZy5zK2+nT8+\nPh7p6ekAgPT0dPGF/0REZDs9lvmSJUswbdo0lJSUICAgADt27MDatWtx6NAhhISE4JNPPsHatWv7\na65ERHal2jdAMzIyunz+8OHDdpkMEZEjDaRythbvACUiUrDMiYhUgGXejeTkZKsz9913n2isxMRE\nUU66El5FRYUoFxMTI8pdvHhRlDt69Kgo19NVSj355JNPRLkhQ2RL67u7u1udKSoqEo0VFBQkyklX\nAvXw8BDlXFxcRLnNmzeLctIVL6X/L9gTy5yISAVY5kREKsAyJyJSAZY5EZEKsMyJiFSAZU5EpALO\nXOay68GIiGhA4ZE5EZHCmY/MWeZERAqWORGRCrDMiYhUgGVORKQCLHMiIhVgmXdj06ZNVmc+/vhj\n0VjZ2dmi3MSJE0W5goICUa64uFiUCw0NFeVGjBghyk2ePFmU+81vfiPKST8vNjU11epMfn6+aKyh\nQ4eKciEhIaLc2bNnRbl3331XlFu9erUo9/3334tyjY2Nopw92bvML1y4gHvvvRdnz55FYGAg9uzZ\nA09Pz07bvfDCC3jrrbcwZMgQREZGYseOHb3+/PE6cyIihb0/Ni4tLQ1GoxGlpaWYPXs20tLSOm1T\nUVGB7du3Iz8/H19//TXMZrNFv6BZ5kRECnuXeWZmJpYtWwYAWLZsGd5///1O23h4eMDNzQ3Nzc1o\nbW1Fc3Mz/P39e903y5yISGHvMq+vr4dWqwUAaLVa1NfXd9pmzJgxWLNmDcaPHw8/Pz94enpizpw5\nve6bb4ASESksKeczZ86gpKSk29eNRiPOnTvX6fn169d3Gqur8b755hu88sorqKiowKhRo/D73/8e\nb7/9NpYuXdrjvFjmREQKS8pcr9dDr9e3P87MzOzw+qFDh7rNarVanDt3Dj4+Pqirq+vy4xm//PJL\nTJs2DWPHjgUA3HXXXTh+/HivZc7TLERE/SQ+Ph7p6ekAgPT0dCQkJHTaJiwsDLm5ufjpp5/Q1taG\nw4cPIzw8vNd9s8yJiBT2Pme+du1aHDp0CCEhIfjkk0+wdu1aAEBtbS0WLlwIAIiOjsYDDzyA2NhY\nREVFAQAeeuihXvfN0yxERAp7X2c+ZswYHD58uNPzfn5+OHDgQPvjp59+Gk8//bRV+2aZExEpeAco\nEZEKsMyJiFSAZU5EpAIscyIiFWCZd0OyypyXl5doLB8fH1Hu2LFjotzMmTNFOenqjtKV9yIjI0W5\n62+EsNSsWbNEucDAQFFOsrLgokWLRGNJV4T88MMPRTlpsdTW1opye/fuFeWkP2NffPGFKGdPLHMi\nIhVgmRMRqQDLnIhIBZy5zHk7PxGRCvDInIhI4cxH5ixzIiIFy5yISAVY5kREKsAyJyJSAZY5EZEK\nsMyJiFSAZU5EpAIscyIiFWCZd2P79u1WZ2bPni0a69ChQ6LclStXRDlLPi27Kx4eHqLcp59+Kspd\nu3ZNlEtKShLlTpw4IcrFx8eLco2NjVZnduzYIRrrlw/ftVZcXJwoJ11V8JtvvhHlfvlAYWv99NNP\nolxLS4soZ08scyIiFXDmMufaLEREKsAjcyIihTMfmbPMiYgULHMiIhVgmRMRqYAzlznfACUiUmg0\nGqu/rPGvf/0LERERcHFxQX5+frfbNTY2IikpCXq9HuHh4cjNze113yxzIiKFvcs8MjIS+/btw4wZ\nM3rc7k9/+hMWLFiA4uJifPXVV9Dr9b3um6dZiIgU9j7NEhYW1us2Fy9exJEjR5Ceng4AcHV1xahR\no3rN8ciciEhh7yNzS5SXl8Pb2xvJycm46aabsGLFCjQ3N/eaY5kTESlsUeZGoxGRkZGdvj744AOL\n5tDa2or8/Hw88sgjyM/Px/Dhw5GWltZrjqdZiIgUlhxpFxYWorCwsNvXpetE/UKn00Gn02Hy5MkA\nfl4niWVORGQFS8rcYDDAYDC0P965c6dorLa2ti6f9/HxQUBAAEpLSxESEoLDhw8jIiKi1/3ZtcwT\nEhKszpw/f140lnSVv71794pyPf1m7ol0pbiHHnpIlCsvLxflcnJyRDkXFxdRbtGiRaKcq6v1P8JP\nPfWUaKzq6mpR7uTJk6KcJW96dWXTpk2i3KVLl0S52tpaUS4qKkqUM5lMotxAsG/fPjz22GNoaGjA\nwoULYTAYcPDgQdTW1mLFihU4cOAAAGDLli1YunQpWlpaMHHiRItW+uSRORGRwt5XsyQmJiIxMbHT\n835+fu1FDgDR0dFWL4Hc4xugy5cvh1arRWRkZPtzzz33HHQ6Xfs/NZz5tyQR0a8NhKtZpHos8+Tk\n5E5lrdFo8MQTT6CgoAAFBQXihfeJiAYa1Zb59OnTMXr06E7Pd3finojImam2zLuzZcsWREdHIyUl\nRfSxXUREA9GgKvOVK1eivLwchYWF8PX1xZo1a+wxLyKifufMZW711Szjxo1r//ODDz7Y4yVlmZmZ\n7X8ODQ1FaGiotcMR0SBXWVmJysrKfhlrIJWztawu87q6Ovj6+gL4+ZrJX1/pcj3pJ64TEf1i/Pjx\nGD9+fPvj48eP220s1Zb5kiVLkJOTg4aGBgQEBGDdunXIzs5GYWEhNBoNJkyYgG3btvXXXImI7Eq1\nZZ6RkdHpueXLl9ttMkREjqTaMiciGkycucy5BC4RkQrwyJyISOHMR+Z2LfOGhgarM5999plorGvX\nrolyKSkpopx05T1L1iXuSlZWlijn4eEhymm1WlHuyJEjotzUqVNFuYMHD1qdkf6dS2+Qk6weCgAj\nR44U5b7++mtRzpJlVrvy/fffi3L33XefKPfiiy+KcpZgmRMRqQDLnIhIBVjmREQqwDInIlIBljkR\nkQqwzImIVIBlTkSkAixzIiIVYJkTEamAM5c512YhIlIBHpkTESmc+cicZU5EpGCZExGpgDOXuaat\nra3NLjvWaBAXF2d1bubMmaLx3N3dRbmioiJRbvLkyaLcO++8I8o1NTWJctLvS2trqyg3atQoUS4m\nJkaUk/zP5+LiIhqrrq5OlCsvLxflbr/9dlHu7Nmzopyrq+zYLjw8XJQ7duyYKPf222/DHrWl0WhQ\nUlJidS40NNQu87EW3wAlIlJoNBqrv6zx1FNPQa/XIzo6GnfddRcuXrzY7bZmsxkGgwGLFi2yaN8s\ncyIihb3LfO7cuTh9+jROnjyJkJAQvPDCC91uu3nzZoSHh1s8BsuciEhh7zI3Go0YMuTn2p0yZQqq\nq6u73K66uhpZWVl48MEHLT6FwzInIlLYu8x/7Z///CcWLFjQ5WurV6/GSy+91F78luDVLERECltc\nzWI0GnHu3LlOz2/YsKH9/Pf69evxf//3f/jDH/7QabsPP/wQ48aNg8FgQHZ2tsXjssyJiBSWlHlu\nbi5yc3O7ff3QoUM95t98801kZWXh448/7vL148ePIzMzE1lZWbh8+TJ++OEHPPDAA9i5c2eP+2WZ\nExEpLCnzqVOndvgA8s2bN1u8f5PJhJdeegk5OTkYNmxYl9ts2LABGzZsAADk5OTg5Zdf7rXIAZ4z\nJyLqN3/84x/R1NQEo9EIg8GARx55BABQW1uLhQsXdpmx9NQPj8yJiBT2vgO0rKysy+f9/Pxw4MCB\nTs/PnDnT4hspWeZERApnvp2fZU5EpGCZExGpAMuciEgFWObdGDFihNWZI0eOiMaaP3++KHfbbbeJ\nct9//70o99vf/laU8/b2FuWGDh0qyr3//vuiXGJioih38803i3J5eXlWZ6QrQl6+fFmUa2xsFOXG\njh0rykn/7qSrNH733Xei3MiRI0U5e2KZExGpAMuciEgFWOZERCrAMiciUgGWORGRCjhzmXNtFiIi\nFeCRORGRwpmPzFnmREQKljkRkQqwzImIVIBlTkSkAixzIiIVYJkTEamAM5e5pq2trc0uO9Zo8Nln\nn1md279/v2i8wMBAUe7MmTOiXHNzsyhXV1cnykVERIhyJSUlolxqaqoo98Ybb4hyixcvFuWqq6ut\nzkhXFZTOsaqqSpST/kxLVyPct2+fKOfqKjsmXLZsmSgXFxcHe9SWRqPBpUuXrM6NHDnSLvOxFo/M\niYgUznxkzjInIlKwzImIVIBlTkSkAs5c5lxoi4hIBXhkTkSkcOYjc5Y5EZGCZU5EpALOXOY8Z05E\npNBoNFZ/WeNvf/sboqOjERMTg9mzZ3d5Q1lVVRVmzZqFiIgITJo0Ca+++qpF+3ZImefn5zti2AGv\noaHB0VMYkE6fPu3oKQw40jt7qWf2LvOnn34aJ0+eRGFhIRISErBu3bpO27i5uWHTpk04ffo0cnNz\n8dprr6G4uLjXfbPMB5DvvvvO0VMYkFjmnZWWljp6Cqpk7zL/9VILTU1N8PLy6rSNj48PYmJiAAAj\nRoyAXq9HbW1tr/vmOXMiIkV/nDN/5plnsGvXLri7uyM3N7fHbSsqKlBQUIApU6b0ul+7lvnw4cO7\nfN7Nza3b1/z8/ERjdfUbzhI6nU6Uu3z5sijn7u7e7WvffvstJk6c2OVr0u/L1atXRTkPDw9R7sYb\nbxTlPD09u31t2LBh3b7e0tJi9VghISFWZwD5z5jZbLb5eO7u7t2+3tPPWE8mTJggyrm4uIhy0p8x\ne7KkzLOzs5Gdnd3t60ajEefOnev0/IYNG7Bo0SKsX78e69evR1paGlavXo0dO3Z0uZ+mpiYkJSVh\n8+bNGDFiRO9zt+eqiURE9mCvVROlJPOprKzEggULcOrUqU6vXb16FXfccQfmz5+Pxx9/3KL92e3I\nfCAsCUlEZKn+6KyysjIEBwcD+Hm5b4PB0OU8UlJSEB4ebnGRA3Y8Micioo6SkpJQUlICFxcXTJw4\nEa+//jrGjRuH2tparFixAgcOHMDRo0cxY8YMREVFtf9r4YUXXkBcXFyP+2aZExGpQL9fmmgymRAW\nFobg4GBs3Lixv4cfkAIDAxEVFQWDwYBbbrnF0dNxmOXLl0Or1SIyMrL9uQsXLsBoNCIkJARz585F\nY2OjA2foGF19X5577jnodDoYDAYYDAaYTCYHzpAGgn4tc7PZjFWrVsFkMqGoqAgZGRkWXQyvdhqN\nBtnZ2SgoKEBeXp6jp+MwycnJnUopLS0NRqMRpaWlmD17NtLS0hw0O8fp6vui0WjwxBNPoKCgAAUF\nBb3+E5zUr1/LPC8vD0FBQQgMDISbmxsWL14s/sxPteHZLmD69OkYPXp0h+cyMzPbPyty2bJl4s/v\ndGZdfV8A/sxQR/1a5jU1NQgICGh/rNPpUFNT059TGJA0Gg3mzJmD2NhYbN++3dHTGVDq6+uh1WoB\nAFqtFvX19Q6e0cCxZcsWREdHIyUlZVCefqKO+rXMee15144dO4aCggIcPHgQr732Go4cOeLoKQ1I\nktun1WrlypUoLy9HYWEhfH19sWbNGkdPiRysX8vc39+/wyphVVVV4jsw1cTX1xcA4O3tjcTExEF9\n3vx6Wq1RUcXBAAABHklEQVS2/W66uro6jBs3zsEzGhjGjRvX/svtwQcf5M8M9W+Zx8bGoqysDBUV\nFWhpacHu3bsRHx/fn1MYcJqbm3Hp0iUAwI8//oiPPvqow1ULg118fDzS09MBAOnp6UhISHDwjAaG\nurq69j/v27ePPzPUvwttubq6YuvWrZg3bx7MZjNSUlKg1+v7cwoDTn19PRITEwEAra2tWLp0KebO\nnevgWTnGkiVLkJOTg4aGBgQEBOD555/H2rVrcc899+Af//gHAgMDsWfPHkdPs99d/31Zt24dsrOz\nUVhYCI1GgwkTJmDbtm2OniY5GG8aIiJSAX7SEBGRCrDMiYhUgGVORKQCLHMiIhVgmRMRqQDLnIhI\nBVjmREQqwDInIlKB/wdOdonD6tcfvAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11056fc90>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "sigmoid\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAE4CAYAAACdXNIhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X90U/X9BvAnkBx+/xJKaZNIwBYTFUowHSBWUIESJgF0\n07K5daVqxrHHeZQJurkhOqDb1E2ys1NREFRqPVMoOIxatQgiZtNWmMVS1GAI0GOFQgGlEPr9w0u+\nhv5I8iZpc9vndU7P6W3ucz8fa3m43N77iaapqakJRESU0Lp19ASIiCg8ljURkQqwrImIVIBlTUSk\nAixrIiIVYFkTEakAy5qIKIbmz5+P5ORkjB49utV97rnnHqSnpyMjIwMVFRURHZdlTUQUQ3l5eXC7\n3a2+vmXLFuzbtw81NTV4+umnsWDBgoiOy7ImIoqhrKwsDBo0qNXXN23ahNzcXADA+PHjUV9fj9ra\n2rDHZVkTEbUjv98Po9EY3DYYDDhw4EDYHMuaiKidXbjKh0ajCZvRxmsykQxORCQRjyWNLqazopmP\nXq+Hz+cLbh84cAB6vT5sLm5lLXXDDTeIcr/97W9FuezsbFFu2LBhotz27dtbfe2pp57CPffc0+Jr\no0aNEo2n1cr+F3fv3l2U27Rpkyg3e/bsVl87c+YMdDpdi6/NnTs36rGmT58edQb4/rf8EoFAQJS7\n6aabWn1t7969rf5MPP7446LxrrvuOlFu3759oly3brJ/2Pfr10+Ui4SksKP9i8PhcMDlciEnJwc7\nd+7EwIEDkZycHDYnLmu32417770XgUAAd9xxBxYtWiQ9FBFRQohFWc+bNw9bt25FXV0djEYjHnnk\nEZw5cwYA4HQ6MXPmTGzZsgVpaWno06cP1qxZE9E4orIOBAIoKChAWVkZ9Ho9MjMz4XA4YLFYJIcj\nIuo0iouLw+7jcrmiPq7o3yEejwdpaWkwmUzQ6XTIyclBaWmp5FD0A+PHj+/oKSQk6T+XO7PBgwd3\n9BQ6JY1GE/VHexH9KWjp1hO/3x+zSXVVLOuWSa+fd2Ys6/hI5LIWXQbhnR5E1BklcreJyvrCW098\nPh8MBkPMJkVEdN62bduwbdu2dhmr05W1zWZDTU0NvF4vUlNTUVJSEtFFdSKiaGVlZSErKyu4vXz5\n8riN1enKWqvVwuVyITs7G4FAAPn5+bwThIhUr9OVNQDY7XbY7fZYzoWIqEN1yrImIupsWNZERCrA\nsiYiUoFELmtNUzyWr8L3/9GnTp2KOtenTx/ReJKxAKBnz56i3KFDh0S54cOHi3Ln1xaIlvSHb/Pm\nzaLcrFmzRDnpwliSh7G++eYb0VgpKSmiXK9evUS5xsbGds1JHz5qbZGtcPr37y/K1dbWxm3VPcki\nUQ0NDXGZz4X4HC8RkQrwMggRkSKRL4OwrImIFCxrIiIVYFkTEakAy5qISAVY1kREKsCyJiJSgUQu\na95nTUSkiNU7xbjdbpjNZqSnp6OwsLDZ60ePHsXcuXORkZGB8ePH49NPPw07N5Y1EZEiFmV9/g3F\n3W43qqqqUFxcjD179oTss2zZMowbNw6ffPIJ1q1bh9/85jdh58ayJiJSxKKsI3lD8T179uD6668H\nAFx++eXwer34+uuv25wby5qISBGLso7kDcUzMjLw6quvAvi+3Pfv348DBw60OTeWNRFRDEXyS8rF\nixejvr4eVqsVLpcLVqs17EJacb0b5MSJE1FnpCt4DRw4UJSTrmZ37NgxUe7s2bOi3Llz50S59l65\nLdzZQWukK9q98cYbUWduueUW0VinT58W5SQruQHAunXrRLlly5aJctIVE3Nzc0W5n/3sZ6JcbW2t\nKBeJSIr29OnTbf75iOQNxfv164fVq1cHt0eMGIGRI0e2OS7PrImIFJFc9ujZsyf69+8f/LjQD99Q\nvLGxESUlJXA4HCH7HDt2LFj4q1atwuTJk9G3b98258b7rImIFLG4z7q1NxQvKioCADidTlRVVeFX\nv/oVNBoNrrrqKjz77LPhj3vRMyMi6iRi9VBMS28o7nQ6g59PnDgR1dXVUR2TZU1EpEjkJxhZ1kRE\nCpY1EZEKsKyJiFSAZU1EpAIsayIiFWBZExGpQCKXNZ9gJCJSAZ5ZExEpEvnMmmVNRKTosmX95ptv\nRp357rvvRGO9/PLLotytt94qyklJVyhracGYSEhX+fvnP/8pys2fP1+U69ZNdkVu9+7dUWekKxhK\n5yhZfRKQry534403inJ/+ctfRLkfrh4XjQ8++ECU02rjV1tdtqyJiNSEZU1EpAIsayIiFWBZExGp\nAMuaiEgFWNZERCrAsiYiUgGWNRGRCiRyWXNtECKiGHO73TCbzUhPT0dhYWGz1+vq6jBjxgyMHTsW\nV111FZ577rmwx2RZExEpNBpN1B8XCgQCKCgogNvtRlVVFYqLi7Fnz56QfVwuF6xWKyorK1FeXo77\n778/7NPGLGsiIkUsytrj8SAtLQ0mkwk6nQ45OTkoLS0N2SclJQXHjx8HABw/fhyDBw8O+xg9r1kT\nESlicc3a7/fDaDQGtw0GAz788MOQfe68807ccMMNSE1NRUNDQ0RrG/HMmohIEYsz60gKf9myZRg7\ndiwOHjyIyspK3H333WhoaGgzE9cz6z/96U9RZ/bv3y8a6+GHHxblJk2aJMolJSWJcr/73e9EuXXr\n1olyt99+uyi3YMECUW7EiBGi3COPPCLKDRs2LOqMdGVHu90uynm9XlGuV69eopzH4xHlLBaLKJeX\nlyfKLVu2TJSLp0iKtqGhoc1i1ev18Pl8wW2fzweDwRCyz44dO4JdcNlll2HEiBGorq6GzWZr9bg8\nsyYiUkRyJt2/f3/o9frgx4VsNhtqamrg9XrR2NiIkpISOByOkH3MZjPKysoAfL8UbnV1NUaOHNnm\n3HjNmohIEYtr1lqtFi6XC9nZ2QgEAsjPz4fFYkFRUREAwOl04qGHHkJeXh4yMjJw7tw5/PnPf8Yl\nl1zS9nEvemZERJ1ErB6KsdvtzS6bOZ3O4OdDhgzB5s2bozomy5qISJHITzCyrImIFCxrIiIVSOSy\n5t0gREQqwDNrIiJFIp9Zs6yJiBQsayIiFWBZExGpAMuaiEgFWNZERCrQZct6165dUWf69OkjGisj\nI0OUk640Jl0x7Oabbxblwr2LRGuamppEueeff16UO784TbQuu+wyUW727NlRZ9LT00VjXX311aKc\n2+0W5bp1k91Z26NHD1Fu0KBBotw333wjygUCAVEunrpsWRMRqUmnLGuTyYT+/fuje/fu0Ol04jV0\niYgSRacsa41Gg/Ly8rDL+hERqUWnLGtAfj2UiCgRJXJZi9cG0Wg0mDp1Kmw2G1atWhXLORERdYhY\nvAdjvIjPrN9//32kpKTg66+/xrRp02A2m5GVlRXLuRERkUJc1ikpKQC+f+PYuXPnwuPxNCvrpUuX\nBj+fPHkyJk+eLB2OiLqo8vJybN26tV3GSuTLIKKyPnXqFAKBAPr164eTJ0/izTffxB//+Mdm+/3h\nD3+46AkSUdc2ZcoUTJkyJbj96KOPxm2sRC5r0TXr2tpaZGVlYezYsRg/fjxuuukmTJ8+PdZzIyJq\nV7G6Zu12u2E2m5Geno7CwsJmr//1r3+F1WqF1WrF6NGjodVqUV9f3+bcRGfWI0aMQGVlpSRKRJSw\nYnFmHQgEUFBQgLKyMuj1emRmZsLhcIQ8Lb1w4UIsXLgQAPDaa6/hb3/7GwYOHNjmcflOMUREilic\nWXs8HqSlpcFkMkGn0yEnJwelpaWtjrl+/XrMmzcv7NxY1kREiliUtd/vh9FoDG4bDAb4/f4Wxzt1\n6hTeeOMN3HLLLWHnFte1QWpqaqLOfPDBB6KxMjMzRTnpAknnzp0T5Y4fPy7K9e3bV5Rbv369KCf9\nvrz77rui3NGjR0W5/v37R50Jd22wNW+//bYoN3ToUFFuw4YNotzEiRNFuX379olyY8eOFeU+//xz\nUS6eYnEZJJpjbN68Gddee23YSyAAF3IiIgqKpGjr6uraXGlQr9fD5/MFt30+HwwGQ4v7vvTSSxFd\nAgFY1kREQZGUdVJSEpKSkoLbe/fuDXndZrOhpqYGXq8XqampKCkpQXFxcbPjHDt2DO+9917E//pl\nWRMRKWJxGUSr1cLlciE7OxuBQAD5+fmwWCwoKioCADidTgDAxo0bkZ2djV69ekV23IueGRFRJxGr\nh2LsdjvsdnvI186X9Hm5ubnIzc2N+Ji8G4SISAV4Zk1EpEjkx81Z1kRECpY1EZEKsKyJiFSAZU1E\npAIsayIiFWBZExGpAMuaiEgFumxZL1++POqMTqcTjSXNSVdg69Onjyg3Z84cUa6pqUmUu3Ddgkj1\n7t1blPv9738vyt1www2i3OOPPx51ZtGiRaKxfvieotE4fPiwKCf92dRqZX+sI1n5rSX79+8X5ST9\nEG9dtqyJiNSEZU1EpAKJXNZcG4SISAV4Zk1EpEjkM2uWNRGRgmVNRKQCLGsiIhVgWRMRqQDLmohI\nBRK5rHnrHhGRQqPRRP3RErfbDbPZjPT0dBQWFra4T3l5OaxWK6666ipMmTIl7Nx4Zk1EpIjFmXUg\nEEBBQQHKysqg1+uRmZkJh8MBi8US3Ke+vh5333033njjDRgMBtTV1YU9Ls+siYgUsTiz9ng8SEtL\ng8lkgk6nQ05ODkpLS0P2Wb9+PW655RYYDAYAwJAhQ8LOjWVNRKSIRVn7/X4YjcbgtsFggN/vD9mn\npqYGR44cwfXXXw+bzYbnn38+7NziehnkyiuvjDrzxRdfiMaqrKwU5RoaGkS57du3i3Lbtm0T5V57\n7TVRbsaMGaLcXXfdJcr16tVLlEtLSxPltmzZEnVm8eLForEeeughUe7cuXOinHQVPOl4J06caNfc\nY489JsrFUySXQQ4ePIiDBw9e1DHOnDmDjz/+GG+//TZOnTqFiRMnYsKECUhPT281w2vWRERRSE1N\nRWpqanD7448/Dnldr9fD5/MFt30+X/Byx3lGoxFDhgxBr1690KtXL1x33XX45JNP2ixrXgYhIlLE\n4jKIzWZDTU0NvF4vGhsbUVJSAofDEbLP7NmzsX37dgQCAZw6dQoffvghrrjiijbnxjNrIiJFLO4G\n0Wq1cLlcyM7ORiAQQH5+PiwWC4qKigAATqcTZrMZM2bMwJgxY9CtWzfceeedLGsiokjF6qEYu90O\nu90e8jWn0xmyvXDhQixcuDDiY7KsiYgUifwEI8uaiEjBsiYiUgGWNRGRCrCsiYhUgGVNRKQCLGsi\nIhVgWRMRqUAilzUfNyciUoG4nllLVtWSrroXyXqwLbFaraLcZ599JspJ/+Z+9NFHRTmTySTKjR49\nWpQ7c+aMKLd3715RbuvWrVFnunWTnaM8+eSTolxVVZUol5KSIsrddNNNoty///1vUS4pKUmUk654\n2aNHD1EuEol8Zs3LIERECpY1EZEKsKyJiFSAZU1EpAIsayIiFWBZExGpAMuaiEgFWNZERCrAsiYi\nUoFELms+bk5EpIjFu5sDgNvthtlsRnp6OgoLC5u9Xl5ejgEDBsBqtcJqtUb0tDfPrImIYigQCKCg\noABlZWXQ6/XIzMyEw+GAxWIJ2W/y5MnYtGlTxMflmTURkSIWZ9YejwdpaWkwmUzQ6XTIyclBaWlp\ns/2ampqimhvLmohIEYuy9vv9MBqNwW2DwQC/399snB07diAjIwMzZ86MaLGvuF4GOX78eNSZ/fv3\ni8b68MMPRbldu3aJcpIVBQFg9+7dotwLL7wgynXv3l2Ue+WVV0Q56fdz+fLlolw0/4w8T7ram06n\nE+WmTp0qyr344oui3MGDB0W5vn37inLS70teXp4oF0+R/ILR6/XC6/Ve1DHGjRsHn8+H3r174/XX\nX8ecOXPCrjzJM2siIkUkZ9IjRozA9ddfH/y4kF6vh8/nC277fD4YDIaQffr164fevXsDAOx2O86c\nOYMjR460OTeWNRGRIhaXQWw2G2pqauD1etHY2IiSkhI4HI6QfWpra4PXrD0eD5qamnDJJZe0Obc2\ny3r+/PlITk4OWYj+yJEjmDZtGkaNGoXp06ejvr4+4m8EEVEii0VZa7VauFwuZGdn44orrsBtt90G\ni8WCoqIiFBUVAQD+9a9/YfTo0Rg7dizuvfdevPTSS2Hn1mZZ5+Xlwe12h3xtxYoVmDZtGvbu3Ysb\nb7wRK1asiOZ7QUSUsGJ1n7Xdbkd1dTX27duHBx98EADgdDrhdDoBAHfffTf+97//obKyEjt27MCE\nCRPCzq3Nss7KysKgQYNCvrZp0ybk5uYCAHJzc7Fx48bw3wEiIhWIVVnHQ9R3g9TW1iI5ORkAkJyc\njNra2phPioioIyTy4+YXdeteuL9ZlixZEvx8ypQpmDJlysUMR0Rd0KFDh3Do0KF2GatTlXVycjIO\nHz6MYcOG4dChQxg6dGir+/6wrImIJFJSUkLe6b2ysjJuYyVyWUd9657D4cDatWsBAGvXrsWcOXNi\nPikiIgrVZlnPmzcP11xzDaqrq2E0GrFmzRosXrwYb731FkaNGoV33nkHixcvbq+5EhHFlWp/wVhc\nXNzi18vKyuIyGSKijpTIl0G4RCoRkYJlTUSkAl22rCUrvt12222isVq7ZBOO9JYg6YphGzZsEOW+\n++47UW7atGmi3PmnrqI1btw4UU5Kq43+R/jXv/61aKyVK1eKcu+++64o16NHD1FOutKidDyp849e\nR2v16tUxnsn/67JlTUSkJixrIiIVYFkTEakAy5qISAVY1kREKsCyJiJSgUQua76tFxGRCvDMmohI\nkchn1ixrIiJFIpc1L4MQESliteqe2+2G2WxGeno6CgsLWx3vP//5D7RaLV599dWwc2NZExEpYlHW\ngUAABQUFcLvdqKqqQnFxMfbs2dPifosWLcKMGTPQ1NQUdm4sayIiRSzK2uPxIC0tDSaTCTqdDjk5\nOSgtLW2238qVK/GTn/wESUlJEc2NZU1EpIhFWfv9fhiNxuC2wWCA3+9vtk9paSkWLFgQHDecuP6C\n8fTp01Fnhg8fLhpL+oYI0jfx3blzpyj38MMPi3JLly4V5a677jpRbtasWaKcdHVA6YpvqampUWfO\nvy1dtK644gpRbsKECaKcx+MR5QYNGiTKNTY2inInTpwQ5RoaGkS5eIqkNKurq7F3796LOsa9996L\nFStWQKPRoKmpKaLLILwbhIhIEUnRms1mmM3m4PZrr70W8rper4fP5wtu+3w+GAyGkH0++ugj5OTk\nAADq6urw+uuvQ6fTweFwtDouy5qISBGLW/dsNhtqamrg9XqRmpqKkpKSZuvtf/HFF8HP8/LyMGvW\nrDaLGmBZExEFxaKstVotXC4XsrOzEQgEkJ+fD4vFEnyzBafTKTvuRc+MiKiTiNVDMXa7HXa7PeRr\nrZX0mjVrIjomy5qISMEnGImI6KLwzJqISJHIZ9YsayIiBcuaiEgFWNZERCrAsiYiUgGWNRGRCrCs\niYhUoMuWdffu3aPO3HrrraKxJk2aJMpNnDhRlDt48KAol5KSIso9+uijotxjjz0myl155ZWi3O7d\nu0U5qa+++qpdMoBshT8AeOKJJ0Q5qXPnzolyffv2FeXWr18vyv30pz8V5eKpy5Y1EZGasKyJiFSA\nZU1EpAKJXNZcG4SISAV4Zk1EpEjkM2uWNRGRgmVNRKQCLGsiIhVI5LLmLxiJiBQajSbqj5a43W6Y\nzWakp6ejsLCw2eulpaXIyMiA1WrF1VdfjXfeeSfs3HhmTUSkiMWZdSAQQEFBAcrKyqDX65GZmQmH\nwwGLxRLcZ+rUqZg9ezaA75/6nTt3Lvbt29fmcXlmTUSkiMWZtcfjQVpaGkwmE3Q6HXJyclBaWhqy\nT58+fYKfnzhxAkOGDAk7N5Y1EZEiFmXt9/thNBqD2waDAX6/v9l+GzduhMVigd1ux1NPPRV2bixr\nIiJFLMo60kspc+bMwZ49e7B582b84he/CLt/XK9Z63S6qDOffvqpaKzBgweLcmfPnhXlTp8+LcqN\nHDlSlJOu+BYIBES5I0eOiHLPPPOMKDdhwgRRTvLzMnz4cNFYjY2NotyBAwdEOafTKcqtXr1alDt2\n7Jgo9+2334py0tUB4ymSot29e3ebq0vq9Xr4fL7gts/ng8FgaHX/rKwsnD17Ft98802bPcZfMBIR\nKSIp6zFjxmDMmDHB7eLi4pDXbTYbampq4PV6kZqaipKSkmb7fP755xg5ciQ0Gg0+/vhjAOFPOFnW\nREQxpNVq4XK5kJ2djUAggPz8fFgsFhQVFQH4/l9Mr7zyCtatWwedToe+ffvipZdeCn/ceE+ciEgt\nYvVQjN1uh91uD/naDy9rPfDAA3jggQeiOibLmohIkchPMLKsiYgULGsiIhVgWRMRqQDLmohIBVjW\nREQqwLImIlIBljURkQqwrImIVIBlTUSkAl22rI8ePRp1ZseOHaKxGhoaRLm+ffuKcmPHjhXlpPOs\nq6sT5c6cOSPK3XzzzaJcJIuot0T6/72pqSnqjFYr+7Gvrq4W5aQrCqalpYly0pUWpd5//31R7pe/\n/KUot27dOlFO7XhmTUSkSOQz6zbffGD+/PlITk7G6NGjg19bsmQJDAYDrFYrrFYr3G533CdJRNQe\nYvWGufHQZlnn5eU1K2ONRoP77rsPFRUVqKiowIwZM+I6QSKi9qLass7KysKgQYOafV1ynZCIKNGp\ntqxbs3LlSmRkZCA/Px/19fWxnhMRUYfoVGW9YMECfPnll6isrERKSgruv//+eMyLiKjdJXJZR303\nyNChQ4Of33HHHZg1a1ar+y5fvjz4+bXXXousrKxohyOiLu7w4cM4fPhwu4yVyHeDRF3Whw4dQkpK\nCgBgw4YNIXeKXOjBBx+Uz4yICMCwYcMwbNiw4PauXbviNlYil3Wbl0HmzZuHa665BtXV1TAajVi9\nejUWLVqEMWPGICMjA1u3bsWTTz7ZXnMlIoqrWF0GcbvdMJvNSE9PR2FhYbPXX3zxRWRkZGDMmDGY\nNGlSRH8BtXlmfeHbpwPf33tNRNQZxeLMOhAIoKCgAGVlZdDr9cjMzITD4YDFYgnuM3LkSLz33nsY\nMGAA3G437rrrLuzcubPN44ruBiEi6oxicWbt8XiQlpYGk8kEnU6HnJwclJaWhuwzceJEDBgwAAAw\nfvx4HDhwIOzcWNZERDHk9/thNBqD2waDAX6/v9X9n332WcycOTPscbk2CBGRIhaXQaI5xrvvvovV\nq1dHtBhWXMtaspLaj3/8Y9FYGzduFOVOnjwpyklXbpOuiNbSLykiYbfbRbklS5aIcldffbUo9/TT\nT4tyJpMp6syRI0dEY0nvQpA+OCYtjv3794tyw4cPF+XWrl0ryj333HOiXDxX3Yvke/7f//4XH330\nUauv6/V6+Hy+4LbP54PBYGi2365du3DnnXfC7Xa3+KT4hXhmTUSkiKSsMzMzkZmZGdxetWpVyOs2\nmw01NTXwer1ITU1FSUlJs5s1vvrqK9x888144YUXIl4Kl2VNRKSIxWUQrVYLl8uF7OxsBAIB5Ofn\nw2KxoKioCADgdDqxdOlSHD16FAsWLAAA6HQ6eDyeto970TMjIuokYvVQjN1ub3YJ0ul0Bj9/5pln\n8Mwzz0R1TJY1EZEikZ9gZFkTESlY1kREKsCyJiJSAZY1EZEKsKyJiFQgkcuaa4MQEakAz6yJiBSJ\nfGbNsiYiUrCsiYhUIJHLWtPU1NQUlwML3/n3rbfeEo03atQoUW7SpEmi3J49e0S5fv36iXKd3aJF\ni0S5Hj16RJ0ZMmSIaCzpyocHDx4U5d5++21RTvpuTv379xflysrKRLlLL71UlPv2228Rj9rSaDSo\nrq6OOnf55ZfHZT4X4pk1EZEikc+sWdZERAqWNRGRCrCsiYhUgGVNRKQCLGsiIhVgWRMRqUAilzXX\nBiEiUgGWNRGR4vzDfNF8tMTtdsNsNiM9Pb3Fh6k+++wzTJw4ET179sTjjz8e0dx4GYSISBGLyyCB\nQAAFBQUoKyuDXq9HZmYmHA4HLBZLcJ/Bgwdj5cqV2LhxY8TH5Zk1EZEiFmfWHo8HaWlpMJlM0Ol0\nyMnJQWlpacg+SUlJsNls0Ol0Ec+NZU1EpIhFWfv9fhiNxuC2wWCA3++/6LnxMggRkSIWl0HidUdJ\nXMtaq43+8NOnTxeNdeDAAVFu3759otzXX38tyi1fvlyUu/3220U56Sp/AwcOFOUaGhpEud69e7fb\neKdPnxaN5fF4RLmSkhJRbty4caKc1+sV5RYuXCjKbd26VZQbOnSoKLd//35RLhKRFO2OHTuwY8eO\nVl/X6/Xw+XzBbZ/PB4PBcNFz45k1EZEikrKeNGlSyNLKTzzxRMjrNpsNNTU18Hq9SE1NRUlJCYqL\ni1s8VjRLq7KsiYgUsbiEodVq4XK5kJ2djUAggPz8fFgsFhQVFQEAnE4nDh8+jMzMTBw/fhzdunXD\n3//+d1RVVaFv376tH/eiZ0ZE1EnE6nqz3W6H3W4P+ZrT6Qx+PmzYsJBLJZFgWRMRKRL5cXOWNRGR\nIpHLmvdZExGpAM+siYgUiXxmzbImIlKwrImIVIBlTUSkAixrIiIVYFkTEakAy5qISAW6bFl/9913\nUWcKCgpEY/Xs2VOUmzZtmij3xRdfiHIHDx5s1/FWr14typ09e1aU69Wrlyj37bffinK1tbVRZ8xm\nc7uNBQADBgwQ5aSkqwr26dNHlJOsrgkAJ0+eFOW6dYvf4yFdtqyJiNSEZU1EpAIsayIiFWBZExGp\nQCKXNRdyIiJSAZ5ZExEpEvnMmmVNRKRgWRMRqUAilzWvWRMRKTQaTdQfLXG73TCbzUhPT0dhYWGL\n+9xzzz1IT09HRkYGKioqws6tQ8q6vLy8I4ZNeNG8LX1Xwp+X5rZt29bRU+iUYlHWgUAABQUFcLvd\nqKqqQnFxMfbs2ROyz5YtW7Bv3z7U1NTg6aefxoIFC8LOrUPKeuvWrR0xbMJjWbeMPy/Nbd++vaOn\n0CnFoqw9Hg/S0tJgMpmg0+mQk5OD0tLSkH02bdqE3NxcAMD48eNRX18fdjkDXgYhIlLEoqz9fj+M\nRmNw22DsjwdmAAADiklEQVQwwO/3h93nwIEDbc4t4X7B+MP/gGh0795dlBs1apQoJ130ZtiwYa2+\n5vf7odfrW3zt0ksvFY1ntVpFOan2Hq9Hjx5RZ8aNGycaS/ozJv2ZbmvBIo1GE/MFjaT/76Tfl0QU\ni18wRnqMC/8lHTbXFCcA+MEPfvAjLh+J1lk/9MEHHzRlZ2cHt5ctW9a0YsWKkH2cTmdTcXFxcPvy\nyy9vOnz4cJvzi9uZdROvvxKRisSqs2w2G2pqauD1epGamoqSkhIUFxeH7ONwOOByuZCTk4OdO3di\n4MCBSE5ObvO4CXcZhIhIzbRaLVwuF7KzsxEIBJCfnw+LxYKioiIAgNPpxMyZM7FlyxakpaWhT58+\nWLNmTdjjapp4CkxElPDa/W6QSG4W72pMJhPGjBkDq9WKH/3oRx09nQ4zf/58JCcnY/To0cGvHTly\nBNOmTcOoUaMwffp01NfXd+AMO0ZL35clS5bAYDDAarXCarXC7XZ34AypPbRrWUdys3hXpNFoUF5e\njoqKCng8no6eTofJy8trVjorVqzAtGnTsHfvXtx4441YsWJFB82u47T0fdFoNLjvvvtQUVGBiooK\nzJgxo4NmR+2lXcs6kpvFuypejQKysrIwaNCgkK/98OGB3NxcbNy4sSOm1qFa+r4A/Jnpatq1rCO5\nWbwr0mg0mDp1Kmw2G1atWtXR00kotbW1wd+SJycni9+0tjNauXIlMjIykJ+f3yUvD3U17VrWibyi\nVUd6//33UVFRgddffx3/+Mc/uO5DK9paOKerWbBgAb788ktUVlYiJSUF999/f0dPieKsXctar9fD\n5/MFt30+HwwGQ3tOISGlpKQAAJKSkjB37twufd36QsnJyTh8+DAA4NChQxg6dGgHzygxDB06NPiX\n1x133MGfmS6gXcv6hzeLNzY2oqSkBA6Hoz2nkHBOnTqFhoYGAMDJkyfx5ptvhvzWv6tzOBxYu3Yt\nAGDt2rWYM2dOB88oMRw6dCj4+YYNG/gz0wW060Mxrd0s3pXV1tZi7ty5AICzZ8/i5z//OaZPn97B\ns+oY8+bNw9atW1FXVwej0YilS5di8eLFuPXWW/Hss8/CZDLh5Zdf7uhptrsLvy+PPPIIysvLUVlZ\nCY1GgxEjRgQfuKDOiw/FEBGpAJdIJSJSAZY1EZEKsKyJiFSAZU1EpAIsayIiFWBZExGpAMuaiEgF\nWNZERCrwf/ABtLhejQ12AAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11054cdd0>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "tanh\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAE4CAYAAACpJRM9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X9QlHUeB/D3CpRW/gCV5cfSgAGBioiD2S+T0gXRc9Pq\nErOJQTTGZDpPu+TqblJv8tZr5qqTriP7cXQV0TWnUNIWpHidRJSCdeUAphQ/9yIiUbqQlfvjHnfC\nBdn97C/28f2a2Rl39/k836/L8vbxu8/zWc3AwMAAiIjIp43x9gSIiMh5DHMiIhVgmBMRqQDDnIhI\nBRjmREQqwDAnIlIBhjkRkQutWbMGWq0WCQkJw27z4IMPIiYmBomJiaitrXXJuAxzIiIXysrKgslk\nGvb5srIyHD9+HI2NjXjuueewfv16l4zLMCcicqH58+cjMDBw2OdLS0uRmZkJAJg3bx66u7thNpud\nHpdhTkTkQa2trYiIiLDe1+l0aGlpcXq/DHMiIg+7sIuKRqNxep/+Tu9hGK6YHBHRUNzRUsqZzHJk\nPuHh4Whubrbeb2lpQXh4uHjs89wW5lJXX321qO7AgQOiuqioKFGdv7/spevt7R32ud/97nf47W9/\nO+RzY8eOFY0nJX1jnzhxQlQn/TncfPPNDtds3bpVNJZerxfVWSwWUd2111477HPffvstJk+ePORz\nR48eFY135ZVXiurOnTsnqpNy54GiZN+O/sNiMBiQn5+PjIwMVFdXY9KkSdBqtQ6PeyFxmJtMJmzc\nuBEWiwVr167Fli1bnJ4MEZE3uSLMV61ahYMHD6KzsxMRERHYtm0bzp49CwDIycnBkiVLUFZWhujo\naFx55ZV46aWXXDJ3UZhbLBbk5uaioqIC4eHhmDt3LgwGA+Lj410yKSIiX1VUVDTiNvn5+S4fV/QB\naE1NDaKjoxEZGYmAgABkZGSgpKTE1XO75Nxyyy3engL5iHHjxnl7Cqqk0Wgcvo0WojAf6tSa1tZW\nl03qUrVgwQJvT4F8xBVXXOHtKaiSL4e5aJllNP0FiIhcxZezTRTmF55a09zcDJ1O57JJERGdV1lZ\nicrKSo+MdcmFeXJyMhobG9HU1ISwsDAUFxfbtehPROSolJQUpKSkWO9v27bNbWNdcmHu7++P/Px8\npKWlwWKxIDs7m2eyEJHPu+TCHADS09ORnp7uyrkQEXnVJRnmRERqwzAnIlIBhjkRkQr4cphrBtzR\nfgz/f1EkDYakDaw83eynv79fVHfZZZeJ6qQ/Jumb89ixY6I66Qfh0nlKfg7S19LPz09UJ/27efpn\n7unxpHXnzp1zW9fE8ePHO1zX09Pjlvk4iv3MiYhUgMssREQKX15mYZgTESkY5kREKsAwJyJSAYY5\nEZEKMMyJiFSAYU5EpAK+HOY8z5yISOGqbxoymUyIi4tDTEwMdu7cafN8Z2cnFi9ejNmzZ2PmzJn4\n61//6vTcGeZERApXhPn5L7w3mUz44osvUFRUZHNFdX5+PpKSklBXV4fKykps3rxZfFX5eQxzIiKF\nK8Lcni+8Dw0NxalTpwAAp06dwuTJk8WtTM7jmjkRkcIVa+ZDfeH9Rx99NGibdevW4bbbbkNYWBh6\nenrwxhtvOD0uj8yJiFzInn8QduzYgdmzZ6OtrQ11dXXYsGEDenp6nBp31B2ZS7uPjRkj+3fJk936\nnOHprpDS16Wvr09UJ+1IeOLECYdrYmNjRWNJuoAC8k6gNTU1orrbbrtNVJebmyuqe+yxx0R1oaGh\norru7m5RnT3sed//+OOPF32f2/OF91VVVXj00UcBANdccw2ioqJQX1+P5ORk4cx5ZE5EZGXPGvnY\nsWMxYcIE6+1CP/3C+76+PhQXF8NgMAzaJi4uDhUVFQAAs9mM+vp6TJs2zam5j7ojcyIib3HFmvlw\nX3hfUFAAAMjJycEjjzyCrKwsJCYm4ty5c/jDH/6AoKAg58Z1euZERCrhqouGhvrC+5ycHOufp0yZ\ngrfeesslY53HMCciUvjyFaAMcyIiBcOciEgFGOZERCrAMCciUgGGORGRCjDMiYhUwJfDnFeAEhGp\nAI/MiYgUvnxkzjAnIlIwzIdRVVXlcI20O6C0w9x1110nqpNatGiRqE7aVVD65nz33XdFdampqaI6\n6TxPnz7tcI2nO3NKuy1K5yntKrhp0yZR3SOPPCKq6+rqEtVJfw72YJgTEakAw5yISAUY5kREKsAw\nJyJSAYY5EZEKMMyJiFSAYU5EpAIMcyIiFfDlMGdvFiIiFzOZTIiLi0NMTAx27tw55DaVlZVISkrC\nzJkzkZKS4vSYPDInIlK44sjcYrEgNzcXFRUVCA8Px9y5c2EwGBAfH2/dpru7Gxs2bMC7774LnU6H\nzs5Op8flkTkRkUKj0Th8u1BNTQ2io6MRGRmJgIAAZGRkoKSkZNA2r732Gu68807odDoAwJQpU5ye\nO8OciEjhijBvbW1FRESE9b5Op0Nra+ugbRobG9HV1YVbb70VycnJ+Nvf/ub03LnMQkSkcMUyiz37\nOHv2LI4cOYL3338fvb29uOGGG3D99dcjJiZGPK5bw1zSQW/79u2isbZs2SKq6+vrE9VJuxiWlpaK\n6saNGyeqW7BggahO2v3wsssuE9VJuzRefvnlDtdIuxFOmzZNVNfR0SGqCwgIENV9//33orrbb79d\nVCftBPrHP/5RVOdO9gRxT08Penp6hn0+PDwczc3N1vvNzc3W5ZTzIiIiMGXKFIwbNw7jxo3DLbfc\ngqNHjzoV5lxmISJS2LOsMmHCBISHh1tvF0pOTkZjYyOamprQ19eH4uJiGAyGQdvcfvvt+Ne//gWL\nxYLe3l589NFHmD59ulNz5zILEZHCFcss/v7+yM/PR1paGiwWC7KzsxEfH4+CggIAQE5ODuLi4rB4\n8WLMmjULY8aMwbp16xjmRESu4qqLhtLT05Genj7osZycnEH3H3roITz00EMuGQ9gmBMRWfnyFaAM\ncyIiBcOciEgFfDnMeTYLEZEK8MiciEjhy0fmDHMiIgXDnIhIBRjmREQqwDAnIlIBhjkRkQowzIdx\n+vRph2v8/WVTGj9+vKhO2uWvsLBQVLdq1SpR3blz50R10g6B0p+DtGOftCvkzJkzHa6RdFoEgLvv\nvltUN9zXho3khx9+ENWNGSM741gaZP39/aK60YhhTkSkApdkmEdGRmLChAnw8/NDQEAAampqXDkv\nIiKPuyTDXKPRoLKyEkFBQa6cDxGR11ySYQ7I12OJiEYjXw5zcW8WjUaDRYsWITk5Gbt373blnIiI\nvMIVX+jsLeIj80OHDiE0NBTffPMN9Ho94uLiMH/+fFfOjYiI7CQO89DQUADA1KlTsWLFCtTU1NiE\n+bZt26x/XrBgAVJSUqTDEdElqrKyEpWVlR4ZazQdaTtKFOa9vb2wWCwYP348zpw5g/feew+PPfaY\nzXZDPUZE5IiUlJRBB4I/PUh0NV8Oc9Gaudlsxvz58zF79mzMmzcPP/vZz5CamurquREReZSr1sxN\nJhPi4uIQExNz0YvGPv74Y/j7++Mf//iH03MXHZlHRUWhrq7O6cGJiEYTVxyZWywW5ObmoqKiAuHh\n4Zg7dy4MBgPi4+NtttuyZQsWL17skjMD+U1DREQKVxyZ19TUIDo6GpGRkQgICEBGRgZKSkpsttu1\naxfuuusuTJ061SVzZ5gTESlcEeatra2IiIiw3tfpdGhtbbXZpqSkBOvXr7eO6yy39mb58ccfHa75\n7rvvRGMFBgaK6iwWi6hO+uJLG2ZJG19Jl8Okr8uFb1p3jych/S/t0aNHRXVPPPGEqK6jo0NUN2XK\nFFGd5PcVkDcuO3v2rKjOnVwRqvbsY+PGjTAajdBoNBgYGHDJMgsbbRERKewJ4s7OTnz77bfDPh8e\nHo7m5mbr/ebmZuh0ukHbHD58GBkZGdb9vfPOOwgICIDBYBDOnGFORGRlT5hPnTp10Dp3Q0PDoOeT\nk5PR2NiIpqYmhIWFobi4GEVFRYO2OXHihPXPWVlZWLZsmVNBDjDMiYisXLJ27e+P/Px8pKWlwWKx\nIDs7G/Hx8SgoKAAA5OTkOD3GkOO6Za9ERD7IVRcNpaenIz09fdBjw4X4Sy+95JIxeTYLEZEK8Mic\niEjhy5fzM8yJiBQMcyIiFWCYExGpAMOciEgFGOZERCrAMCciUgGGORGRCjDMh3HhFVD2uOKKK9ww\nE9eTdjm7sOGOu8c7ePCgqG727Nmiuu3bt4vqHn30UVHdX/7yF4drNmzYIBprx44dorr+/n5RnfRn\n7ufnJ6qTBpn07yd9Pd2JYU5EpAIMcyIiFfDlMGdvFiIiFeCRORGRwpePzBnmREQKhjkRkQowzImI\nVIBhTkSkAgxzIiIV8OUw56mJREQKjUbj8G0oJpMJcXFxiImJwc6dO22ef/XVV5GYmIhZs2bhpptu\nwqeffur03HlkTkSkcMWRucViQW5uLioqKhAeHo65c+fCYDAgPj7eus20adPwz3/+ExMnToTJZML9\n99+P6upqp8blkTkRkcIVR+Y1NTWIjo5GZGQkAgICkJGRgZKSkkHb3HDDDZg4cSIAYN68eWhpaXF6\n7gxzIiKFK8K8tbUVERER1vs6nQ6tra3DjvnCCy9gyZIlTs/drcssK1ascLjmwIEDorHOnDkjqrNY\nLKK6xsZGUV1HR4eorr6+XlQXExMjqpN2FpQaO3asqK67u9vhmgceeEA01pYtW0R1nibttih17tw5\nUd1vfvMbF8/EefYss7S1taGtrc2pfZx34MABvPjiizh06JDdNcPhmjkRkQPCwsIQFhZmvX/kyJFB\nz4eHh6O5udl6v7m5ecjW159++inWrVsHk8mEwMBAp+fFZRYiIoUrllmSk5PR2NiIpqYm9PX1obi4\nGAaDYdA2X3/9Ne644w688soriI6OdsnceWRORKRwxdks/v7+yM/PR1paGiwWC7KzsxEfH4+CggIA\nQE5ODrZv347vvvsO69evBwAEBASgpqbGuXGdnjkRkUq46qKh9PR0m29ay8nJsf75+eefx/PPP++S\nsc5jmBMRKXz5ClCGORGRgmFORKQCDHMiIhVgmBMRqQDDnIhIBRjmREQqwDAnIlIBXw5zXs5PRKQC\nbj0y37Rpk8M1fX19orHGjJH9uzRhwgRRnbRLo7Sj3Q033CCq+/rrr0V148ePF9VJu1D+8MMPorqL\ntRZ1NekVe3v27BHVzZo1S1QXFxcnqpN25vTz8xPVSd8r0t91e/jykTmXWYiIFAxzIiIVYJgTEakA\nw5yISAUY5kREKsAwJyJSAYY5EZEKMMyJiFSAYU5EpAK+HOa8nJ+ISKHRaBy+DcVkMiEuLg4xMTHY\nuXPnkNs8+OCDiImJQWJiImpra52eO8OciMiFLBYLcnNzYTKZ8MUXX6CoqAjHjh0btE1ZWRmOHz+O\nxsZGPPfcc1i/fr3T4zLMiYgUrjgyr6mpQXR0NCIjIxEQEICMjAyUlJQM2qa0tBSZmZkAgHnz5qG7\nuxtms9mpuTPMiYgUrgjz1tZWREREWO/rdDqbhnBDbdPS0uLU3N36AWh/f79HagCgsbFRVCftfvjE\nE0+I6t58801RXVVVlahO+oFOXV2dqO7UqVOiukWLFonqDh8+7HCNtFuf1NatW0V177zzjqhO+jsk\nJe1imJaW5uKZOM+e35empiY0NTU5tQ/AtoOqsx++8mwWIiKFPYEaFRWFqKgo6/2DBw8Oej48PBzN\nzc3W+83NzdDpdBfdpqWlBeHh4dJpA+AyCxGRlSuWWZKTk9HY2Iimpib09fWhuLgYBoNh0DYGgwEv\nv/wyAKC6uhqTJk2CVqt1au4XDfM1a9ZAq9UiISHB+lhXVxf0ej1iY2ORmpqK7u5upyZARDRauCLM\n/f39kZ+fj7S0NEyfPh0rV65EfHw8CgoKUFBQAABYsmQJpk2bhujoaOTk5ODPf/6z03O/aJhnZWXB\nZDINesxoNEKv16OhoQELFy6E0Wh0ehJERKOBq84zT09PR319PY4fP45f//rXAICcnBzk5ORYt8nP\nz8fx48dx9OhRzJkzx+m5XzTM58+fj8DAwEGP/fSUmszMTOzdu9fpSRARjQauCnNvcPgDULPZbF3b\n0Wq1Tp8bSUQ0WoymcHaUU2ezjPQv009PyUpJSUFKSoozwxHRJejbb79FV1eXR8a6pMJcq9Wio6MD\nISEhaG9vR3Bw8LDbSs+vJSI6b/LkyZg8ebL1/pdffum2sXw5zB0+NdFgMKCwsBAAUFhYiOXLl7t8\nUkRE5JiLhvmqVatw4403or6+HhEREXjppZeQl5eH8vJyxMbGYv/+/cjLy/PUXImI3Eq1H4AWFRUN\n+XhFRYVbJkNE5E2jKZwdxcv5iYgUDHMiIhVgmA/Dz8/P4Zp58+aJxvrwww9FdX19faK6a6+9VlR3\n/PhxUZ2UtN/Dhc2D7DVp0iRR3YUd5NxJ2qFRurx4YftTe0mDRVon+X11RllZmahO2qXRHgxzIiIV\nYJgTEakAw5yISAUY5kREKsAwJyJSAYY5EZEK+HKY82vjiIhUgEfmREQKXz4yZ5gTESl8Ocy5zEJE\npHB318Suri7o9XrExsYiNTUV3d3dNts0Nzfj1ltvxYwZMzBz5kz86U9/smvfDHMiIoW7w9xoNEKv\n16OhoQELFy6E0Wi02SYgIABPPvkkPv/8c1RXV+OZZ57BsWPHRtw3w5yISOHuMC8tLUVmZiYAIDMz\nE3v37rXZJiQkBLNnzwYAXHXVVYiPj0dbW9uI++aaORGRwt1r5maz2dr8TqvVwmw2X3T7pqYm1NbW\n2tWA0K1h3t/f73BNQECAaKyvvvpKVHf11VeL6k6ePCmqy83NFdXl5+eL6latWiWqi4uLE9VJfuaA\nvGOfv7/jb+EDBw6Ixnr22WdFdUFBQaI66ZcYS14TALBYLKI6T3a8dDd7wry+vh4NDQ3DPq/X69HR\n0WHz+OOPP24z1sXGO336NO666y48/fTTuOqqq0acF4/MiYgU9oR5XFzcoIOdt99+e9Dz5eXlw9Zq\ntVp0dHQgJCQE7e3tCA4OHnK7s2fP4s4778S9995r9/csc82ciEjh7jVzg8GAwsJCAEBhYeGQQT0w\nMIDs7GxMnz4dGzdutHvfDHMiIoW7wzwvLw/l5eWIjY3F/v37kZeXBwBoa2vD0qVLAQCHDh3CK6+8\nggMHDiApKQlJSUkwmUwj7pvLLERECnd/ABoUFDTkN1aFhYVh3759AICbb74Z586dc3jfDHMiIgWv\nACUiIq/ikTkRkcKXj8wZ5kRECoY5EZEKMMyJiFSAYU5EpAIMcyIiFWCYExGpgC+HuWbATS3PNBqN\n6CqmNWvWiMZ78cUXRXWTJ08W1XV2dorqpB3tJK+lM3Xjx48X1Z0+fVpUN2aM7JIHyS9fX1+faCxp\nZ0dpnae7EUp/BocPHxbVzZkzR1Q3MDDgltdGo9Hgtddec7junnvuGRWdI3lkTkSk8OUjc4Y5EZGC\nYU5EpAK+HObszUJEpAI8MiciUvjykTnDnIhIwTAnIlIBhjkRkQr4cpjzA1AiIoW7vwO0q6sLer0e\nsbGxSE1NRXd397DbWiwWJCUlYdmyZXbtm2FORKRwd5gbjUbo9Xo0NDRg4cKFMBqNw2779NNPY/r0\n6XaPwTAnIlK4O8xLS0uRmZkJAMjMzMTevXuH3K6lpQVlZWVYu3at3a0CGOZERAp3h7nZbIZWqwUA\naLVamM3mIbf75S9/iSeeeMKhfjn8AJSISOGKD0D1ej06OjpsHn/88cdtxhpqvLfffhvBwcFISkpC\nZWWl3eO6NcwlHQJ7enpEY0k700lJf+gTJkwQ1XmyqyAA9Pf3i+reeustUV1QUJCorqury+Gayy+/\nXDSWxWIR1UlfS4PBIKorKysT1UnnKe0YOBo6DV7Int+Xzz77DJ999tmwz5eXlw/7nFarRUdHB0JC\nQtDe3o7g4GCbbaqqqlBaWoqysjL897//xalTp3Dffffh5Zdfvui8uMxCRKSwZ1ll1qxZWL16tfXm\nCIPBgMLCQgBAYWEhli9fbrPNjh070NzcjJMnT+L111/HbbfdNmKQAwxzIiKPycvLQ3l5OWJjY7F/\n/37k5eUBANra2rB06dIha+z93zXXzImIFO6+aCgoKAgVFRU2j4eFhWHfvn02jy9YsAALFiywa98M\ncyIihS9fAcowJyJSMMyJiFSAYU5EpAIMcyIiFWCYExGpAMOciEgFGOZERCrAMCciUgGGORGRCjDM\nhyHpwnbixAmPjQXIuy2OHTtWVHf27FlRnbRjn7Qz3YwZM0R1UVFRorrOzk5RneTvJ+nmCQC9vb2i\nuquuukpUN378eFGd9L0iDbKWlhZR3a233iqqO3DggKhO7XhkTkSk8OUj84t2TVyzZg20Wi0SEhKs\nj23duhU6nQ5JSUlISkqCyWRy+ySJiDzB3d805E4XDfOsrCybsNZoNNi0aRNqa2tRW1uLxYsXu3WC\nRESeotownz9/PgIDA20eH43fEEJE5CzVhvlwdu3ahcTERGRnZ6O7u9vVcyIi8opLKszXr1+PkydP\noq6uDqGhodi8ebM75kVE5HG+HOYOn83y0y8gXbt2LZYtWzbstlu3brX+OSUlBSkpKY4OR0SXuO++\n+85jKwCjKZwd5XCYt7e3IzQ0FACwZ8+eQWe6XOinYU5EJBEYGDjos7umpia3jaXaMF+1ahUOHjyI\nzs5OREREYNu2baisrERdXR00Gg2ioqJQUFDgqbkSEbmVu8O8q6sLK1euxFdffYXIyEi88cYbmDRp\nks123d3dWLt2LT7//HNoNBq8+OKLuP766y+674uGeVFRkc1ja9ascXD6RES+wd1hbjQaodfr8fDD\nD2Pnzp0wGo0wGo022/3iF7/AkiVL8Oabb6K/vx9nzpwZcd+is1mIiNTI3R+AlpaWIjMzEwCQmZmJ\nvXv32mzz/fff44MPPrAeOPv7+2PixIkj7pthTkTkIWazGVqtFgCg1WphNptttjl58iSmTp2KrKws\nzJkzB+vWrbOrLxDDnIhI4Yojc71ej4SEBJtbaWnpkGNdqL+/H0eOHMEDDzyAI0eO4MorrxxyKeZC\nbm20JemAGBsbKxrr3//+t6hO2mFO2nlPOt5TTz0lqlu3bp2orrq6WlQ31Ic57nT55Zc7XCPtsPmf\n//xHVCf9mUtJ/37S9/T5ZQNHvf/++6K6MWPcdwxqz7LJJ598gsOHDw/7fHl5+bDPabVadHR0ICQk\nBO3t7YNO9T5Pp9NBp9Nh7ty5AIC77rrL+2FORORL7AnzuXPnWoMWAHbv3m33/g0GAwoLC7FlyxYU\nFhZi+fLlNtuEhIQgIiICDQ0NiI2NRUVFhV0tqbnMQkSkcPcHoHl5eSgvL0dsbCz279+PvLw8AEBb\nWxuWLl1q3W7Xrl1YvXo1EhMT8emnn+KRRx4Zcd88MiciUrj71MSgoCBUVFTYPB4WFoZ9+/ZZ7ycm\nJuLjjz92aN8McyIihWqvACUiupQwzImIVIBhTkSkAgxzIiIVYJgTEamAL4c5zzMnIlIBHpkTESl8\n+cicYU5EpGCYExGpgC+HuWZgYGDALTvWaETdzaTf73f+e0kdNW7cOFHdjz/+KKqTdqbzFdK308aN\nG0V19jTtv5C0M+d9990nqpO+Jl9//bWobtq0aaI6SQdK4P9fuCwREBAgqhsYGBC/phej0WhQX1/v\ncN21117rlvk4St3JQkTkAF8+MmeYExEpGOZERCrAMCciUgGGORGRCjDMiYhUgGFORKQCvhzm7M1C\nRKQCDHMiIoW7v9C5q6sLer0esbGxSE1NRXd395Db/f73v8eMGTOQkJCAe+65x66LFBnmREQKd4e5\n0WiEXq9HQ0MDFi5cCKPRaLNNU1MTdu/ejSNHjuCzzz6DxWLB66+/PuK+GeZERAp3h3lpaSkyMzMB\nAJmZmdi7d6/NNhMmTEBAQAB6e3vR39+P3t5ehIeHj7hvhjkRkcLdYW42m6HVagEAWq0WZrPZZpug\noCBs3rwZV199NcLCwjBp0iQsWrRoxH3zbBYiIoUrzmbR6/Xo6Oiwefzxxx+3GWuo8b788ks89dRT\naGpqwsSJE/Hzn/8cr776KlavXn3Rcd0a5ufOnXO4JjIyUjTW2bNnRXV9fX2iOsnfDQCefPJJUd2G\nDRtEdZLOlYC8u6P0dfEkaYe7559/XlRXWVkpqgsLCxPVSd/T69atE9UdPXpUVOfn5yeq6+/vF9XZ\nw54wr6qqQlVV1bDPl5eXD/ucVqtFR0cHQkJC0N7ejuDgYJttPvnkE9x4442YPHkyAOCOO+5AVVXV\niGHOZRYiIoU9yyo33XQTfvWrX1lvjjAYDCgsLAQAFBYWYvny5TbbxMXFobq6Gj/88AMGBgZQUVGB\n6dOnj7hvhjkRkcLda+Z5eXkoLy9HbGws9u/fj7y8PABAW1sbli5dCgBITEzEfffdh+TkZMyaNQsA\ncP/994+4b66ZExEp3H0FaFBQECoqKmweDwsLw759+6z3H374YTz88MMO7ZthTkSk8OXL+RnmREQK\nXw5zrpkTEakAj8yJiBS+fGTOMCciUjDMiYhUgGFORKQCDHMiIhVgmBMRqQDDnIhIBRjmw7BYLA7X\nrFy50g0zGZ60M90333wjqpN2Fdy/f7+o7u233xbVSTvTSbstSt4rAOz6Oq0LXXHFFaKxpJ057733\nXlGdNFikr6W0i+Gzzz4rqpO+x6SdQO3BMCciUgGGORGRCjDMiYhUgGFORKQCvhzmbLRFRKQCPDIn\nIlL48pE5w5yISMEwJyJSAV8Oc66ZExEp3P2Fzn//+98xY8YM+Pn54ciRI8NuZzKZEBcXh5iYGOzc\nudOufXslzCsrK70x7Kg3MDDg7SmMSny/2OJr4h7uDvOEhATs2bMHt9xyy7DbWCwW5ObmwmQy4Ysv\nvkBRURGOHTs24r69EuYHDx70xrDkoxhctviauIe7wzwuLg6xsbEX3aampgbR0dGIjIxEQEAAMjIy\nUFJSMuK+ucxCRKRwd5jbo7W1FREREdb7Op0Ora2tI9aNug9Ao6KiPDpeQkKCqK67u1tUd7FGW21t\nbcM2/roAbFugAAADOUlEQVTmmmtE482ZM0dUJ+Xp8SS/TJ6eY2RkpEfHk5K+Lu5sfOVprghnvV6P\njo4Om8d37NiBZcuWuW8OA24CgDfeeOPNLbfRllmOSklJGTh8+PCQz3344YcDaWlp1vs7duwYMBqN\nI+7TbUfmA/wwj4h8iKcza7jxkpOT0djYiKamJoSFhaG4uBhFRUUj7k89/z8iIhrl9uzZg4iICFRX\nV2Pp0qVIT08H8P8l1qVLlwL4/3cC5OfnIy0tDdOnT8fKlSsRHx8/4r41AzyEJiLyeR4/MpecDK92\nkZGRmDVrFpKSknDdddd5ezpes2bNGmi12kEfSnd1dUGv1yM2NhapqaniD5592VCvy9atW6HT6ZCU\nlISkpCSYTCYvzpBGA4+GufRkeLXTaDSorKxEbW0tampqvD0dr8nKyrIJJaPRCL1ej4aGBixcuBBG\no9FLs/OeoV4XjUaDTZs2oba2FrW1tVi8eLGXZkejhUfDXHoy/KWAq13A/PnzERgYOOix0tJSZGZm\nAgAyMzOxd+9eb0zNq4Z6XQC+Z2gwj4a59GR4tdNoNFi0aBGSk5Oxe/dub09nVDGbzdBqtQAArVYL\ns9ns5RmNHrt27UJiYiKys7MvyeUnGsyjYe7LHcnc6dChQ6itrcU777yDZ555Bh988IG3pzQqueuK\nO1+0fv16nDx5EnV1dQgNDcXmzZu9PSXyMo+GeXh4OJqbm633m5ubodPpPDmFUSk0NBQAMHXqVKxY\nseKSXje/kFartV5N197ejuDgYC/PaHQIDg62/uO2du1avmfIs2H+05Ph+/r6UFxcDIPB4MkpjDq9\nvb3o6ekBAJw5cwbvvfeeuMWAGhkMBhQWFgIACgsLsXz5ci/PaHRob2+3/nnPnj18z5Bne7P89GR4\ni8WC7Oxsu06GVzOz2YwVK1YAAPr7+7F69WqkpqZ6eVbesWrVKhw8eBCdnZ2IiIjA9u3bkZeXh7vv\nvhsvvPACIiMj8cYbb3h7mh534euybds2VFZWoq6uDhqNBlFRUSgoKPD2NMnLeNEQEZEK8HJ+IiIV\nYJgTEakAw5yISAUY5kREKsAwJyJSAYY5EZEKMMyJiFSAYU5EpAL/A2OUICVq8aIfAAAAAElFTkSu\nQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11109e410>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Rectified Linear Unit\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAWsAAAEzCAYAAAD3myLiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHrlJREFUeJzt3X9QVWXix/HPLdh0wc3fgMCGCi4oCmxs5qaGFoqVrKbt\n4mTjIhrD5DZuNqPT7o7Q7LjQ7E6b4uz4ozU2N1anzdRGGbNCXVujDCzTiH6QiEaxxoK6JSLfP74n\nJuTXvU/3Xs6R92uGGS/3fM7zyI+Px+fe+1xXa2trqwAAtnZdb08AANAzyhoAHICyBgAHoKwBwAEo\nawBwAMoaABwgwFcndrlcvjo1gD7OF884/i6d5Y9nQPusrKWu//Ktra1d3ldXV2c01rBhw4xypsrL\ny41yiYmJXd6Xm5ur3NzcTu9bsGCB0Xj33nuvUa5///5GudmzZxvlXnnllS7vKyoq0qJFizq9b+LE\niR6PFRQU5HFG6v57151jx44Z5c6dO9flfQUFBVq5cmWn9w0ePNhovKNHjxrlbr75ZqOcKV9eCJqc\n218vVTFeBikpKVFsbKxiYmJUUFDgzTkBQK9wuVwef/iLUVm3tLRo2bJlKikp0YkTJ1RcXKyTJ096\ne24AAItRWZeVlSk6OlpRUVEKDAxURkaGdu7c6e259TkpKSm9PQVbSkhI6O0p2M5tt93W21O4Jl1z\nV9a1tbWKjIxsux0REaHa2lq38zz42DnKunOma8XXssmTJ/f2FK5Jdi5rowcYKVsA1yI7d5tRWYeH\nh6umpqbtdk1NjSIiIjocd/WjpHb+QgCwp9LSUpWWlvplLDt3lFFZJycnq6qqStXV1RoxYoS2bdum\n4uLiDsfZ+S8OwBlSUlLaLRHm5eX5bCw7d5ZRWQcEBKiwsFAzZ85US0uLsrKyFBcX5+25AYBfXXNl\nLUmzZs3SrFmzvDkXAOhV12RZA8C1hrIGAAegrAHAAexc1i5fvWGuy+VSS0uLx7lf/epXRuOtX7/e\nKGfK9Mu2Y8cOo1x1dbVR7uuvvzbKrVq1yii3evVqo9y8efOMcv58deMLL7xglBs6dKhR7uLFi0a5\njz76yCgXGhpqlAsODjbKhYSEGOWSkpJ8tuvegAEDPM41NTX5ZTMn9rMGAAdgGQQALHZeBuHKGgAs\n3tgbpKamRtOmTdO4ceMUHx+vtWvXdjimvr5eaWlpSkxMVHx8vJ555pke50ZZA4DFG2UdGBioJ598\nUu+9956OHDmi9evXd9hCurCwUElJSaqoqFBpaalWrFihy5cvdzs3yhoALN4o69DQ0LadIoODgxUX\nF6czZ860OyYsLEyNjY2SpMbGRg0ZMkQBAd2vSrNmDQAWb69ZV1dXq7y8vMPbzy1dulTTp0/XiBEj\n1NTUpO3bt/d4Lq6sAcDizf2sz58/r/nz5+upp57q8PTGNWvWKDExUWfOnFFFRYUeeughNTU1dTs3\nyhoALO6Uc3Nzsy5evNj20Znm5mbNmzdPCxcu1Jw5czrc//rrr+u+++6TJI0ePVojR45UZWVlt3Oj\nrAHA4k5Z33DDDQoODm77uFpra6uysrI0duxYLV++vNNxYmNjtX//fklSXV2dKisrNWrUqG7nxpo1\nAFi8sWZ9+PBhbd26VRMmTFBSUpKk/1/2OHXqlCQpOztbjz32mDIzM5WQkKArV67oiSee0ODBg7s9\nL2UNABZvlPXkyZN15cqVbo8ZOnSodu/e7dF5KWsAsPAKRgDAd+LTXfdMTr1r1y6j8a67zuzfnUOH\nDhnl8vPzjXIzZ840yu3bt88o52+mP07+vKJ5+eWXjXJ33nmnUc70zV6nTZtmlPv000+Ncj/84Q+N\ncqbfu9OnTxvlIiMjfbbrXlhYmMe5s2fP+mXXPZZBAMBi52UQyhoALJQ1ADgAZQ0ADkBZA4ADUNYA\n4ACUNQA4AGUNAA5AWQOAA1DWAOAAdi5r9gYBAAfgyhoALHa+sqasAcBi57K23a57/ubvXeIuXLhg\nlHvrrbeMcufPnzfK3X333Ua5wsJCo9yNN95olHvggQc8zpSUlBiNlZaWZpRzip42zPc2050yfdUt\nLpdLY8aM8Tj3wQcfsOseAPiTna+sKWsAsFDWAOAAlDUAOABlDQAOQFkDgANQ1gDgAJQ1ADiAncua\nvUEAwAG4sgYAi52vrClrALDYuaxZBgEAi8vl8vjjajU1NZo2bZrGjRun+Ph4rV27tsvx3nzzTQUE\nBOiFF17ocW5cWQOAxRtX1oGBgXryySeVmJio8+fP6+abb1Zqaqri4uLaHdfS0qKVK1cqLS3NrY2g\nfFrWH3/8sceZUaNGGY1lutvbsmXLjHKmgoKCjHI1NTVGublz5xrl9u7da5SbMWOGUc5ktzNTR48e\nNcolJycb5f773/8a5QICzH49b7rpJqOc6S54BQUFRrmVK1ca5XzJG2UdGhqq0NBQSVJwcLDi4uJ0\n5syZDmW9bt06zZ8/X2+++aZb52UZBAAs3lgG+bbq6mqVl5dr4sSJ7T5fW1urnTt3Kicnp23cnrAM\nAgAWbz7AeP78ec2fP19PPfWUgoOD2923fPly5efnt+3N3evLIADgJO6UdWNjoxobG7s9prm5WfPm\nzdPChQs1Z86cDvcfPXpUGRkZkqT6+nrt3btXgYGBSk9P7/KclDUAWNwp6xtvvLHdOxudOXOm3f2t\nra3KysrS2LFjtXz58k7P8e3H8zIzMzV79uxui1qirAGgjTeWQQ4fPqytW7dqwoQJSkpKkiStWbNG\np06dkiRlZ2cbnZeyBgCLN8p68uTJHr2f5ZYtW9w6jmeDAIADcGUNABY7v9ycsgYAC2UNAA5AWQOA\nA1DWAOAAlDUAOICdy9rV6s6L0k1ObL3m3VM7duwwGs90RzSTnQEl6fbbbzfKOYXpLoYPPfSQUW7D\nhg1GuW9esuuJL774wmis0aNHG+Vyc3ONcuPGjTPKffXVV34dz/R3z5Rpt7hz3pSUFI9zpaWlPpnP\n1biyBgCLna+sjcs6KipKP/jBD3T99dcrMDBQZWVl3pwXAPjdNVnWLpdLpaWlGjx4sDfnAwC95pos\na0l+WacBAH+xc1kb7w3icrl05513Kjk5WZs2bfLmnACgV3j7nWK8yfjK+vDhwwoLC9MXX3yh1NRU\nxcbGasqUKd6cGwDAYlzWYWFhkqRhw4Zp7ty5Kisr61DW337KUkpKitHTYgD0baWlpSotLfXLWHZe\nBjEq64sXL6qlpUUDBgzQhQsXtG/fPq1evbrDcabPLwWAb1x9oZeXl+ezsa65sq6rq9PcuXMlSZcv\nX9b999+vGTNmeHViAOBv11xZjxw5UhUVFd6eCwD0qmuurAHgWkRZA4AD2LmsbbeRk+l0tm7dapR7\n4IEHjHL+9uqrrxrlpk+f7uWZwFNvv/22Ue7HP/6xl2fiG8ePHzfKxcfHG+V8uZHTPffc43HupZde\nYiMnAPAnO19ZU9YAYKGsAcABKGsAcAA7l7XxRk4AAP/hyhoALHa+sqasAcBi57JmGQQALN7Yz7qm\npkbTpk3TuHHjFB8fr7Vr13Y61sMPP6yYmBglJCSovLy8x7lxZQ0AFm9cWQcGBurJJ59UYmKizp8/\nr5tvvlmpqamKi4trO2bPnj368MMPVVVVpTfeeEM5OTk6cuRIt+flyhoALN64sg4NDVViYqIkKTg4\nWHFxcTpz5ky7Y3bt2qVFixZJkiZOnKiGhgbV1dV1OzfKGgAs3n5br+rqapWXl2vixIntPl9bW6vI\nyMi22xERETp9+nS352IZBAAs7iyDfP755/r88897PO78+fOaP3++nnrqKQUHB3e4/+r9RHoam7IG\nAIs7ZR0SEqKQkJC22++9916HY5qbmzVv3jwtXLhQc+bM6XB/eHi4ampq2m6fPn1a4eHh3Y7r07K+\ncOGCx5nrrjNbmRk0aJBRzt96WpfqSnNzs5dn0r3du3cb5WbPnu3lmXjf5s2bjXJLliwxyvl797y/\n/e1vRrkxY8YY5W699VajnB154wHG1tZWZWVlaezYsVq+fHmnx6Snp6uwsFAZGRk6cuSIBg4c2O4f\ngM5wZQ0AFm+U9eHDh7V161ZNmDBBSUlJkqQ1a9bo1KlTkqTs7Gzddddd2rNnj6KjoxUUFKQtW7b0\neF7KGgAs3ijryZMn68qVKz0eV1hY6NF5eTYIADgAV9YAYLHzy80pawCwUNYA4ACUNQA4AGUNAA5A\nWQOAA1DWAOAAlDUAOABlDQAOQFkDgAP02bIOCgryOPPVV18ZjXXPPfcY5UxdvRetu755dwhPlZSU\nGOVM+Xv3vMrKSqPcj370I48z/fr1MxrLKRoaGoxypjtCmnrppZf8Op477FzW7A0CAA7AMggAWOx8\nZU1ZA4CFsgYAB6CsAcABKGsAcADKGgAcgLIGAAegrAHAAShrAHAAyhoAHICyBgAHsHNZszcIADiA\nT6+sDxw44HHm9ttv98FMuvbpp58a5d59912jnOmue2fOnDHKjRgxwij3+9//3ij329/+1ihnsnue\nKX/vuve///3PKGc6z4ULFxrltm/fbpQz5e+dMt1h5ytrlkEAwGLnsmYZBAAsLpfL44+rLV68WCEh\nIRo/fnyX45SWliopKUnx8fFKSUlxa26UNQBYvFHWmZmZ3b5ZSENDgx566CHt3r1bx48f1/PPP+/W\n3ChrALB4o6ynTJmiQYMGdTnGc889p3nz5ikiIkKSNHToULfmRlkDgMUbZd2TqqoqnTt3TtOmTVNy\ncrKeffZZt3I8wAgAFn88wNjc3Ky3335br7zyii5evKhJkybp1ltvVUxMTLc5yhoALO6U9aeffqpT\np04ZjxEZGamhQ4eqf//+6t+/v6ZOnapjx471WNYsgwCAxZ1lj6ioKE2dOrXtw1M/+9nP9K9//Ust\nLS26ePGi3njjDY0dO7bHHFfWAGDxxjLIggULdODAAdXX1ysyMlJ5eXlqbm6WJGVnZys2NlZpaWma\nMGGCrrvuOi1dupSyBgBPeKOsi4uLezzm0Ucf1aOPPurReVkGAQAH4MoaACx2frk5ZQ0AFjuXtau1\ntbXVJyd2ueSjU3fKdKzuXr/fnePHjxvlmpqajHIDBgwwyjlFY2OjUe7QoUMeZ+6++26jsUx9/vnn\nRrmjR48a5WbNmmWUM/0duvfee41yzz33nFHu+9//vk+6xeVyafXq1R7n8vLy/NJ1XFkDgMXOV9aU\nNQBY7FzW3T4bpLOt/s6dO6fU1FSNGTNGM2bMUENDg88nCQD+4I+9QUx1W9adbfWXn5+v1NRUffDB\nB7rjjjuUn5/v0wkCgL84tqw72+pv165dbW9NtWjRIr344ou+mx0A+JGdy9rjNeu6ujqFhIRIkkJC\nQlRXV+f1SQFAb7DzmvV3eoCxp39ZcnNz2/6ckpLi9tvXAMA3Dh48qIMHD/plrGuqrENCQvTZZ58p\nNDRUZ8+e1fDhw7s89ttlDQAmrt7dbs2aNT4by85l7fHeIOnp6SoqKpIkFRUVac6cOV6fFACgvW7L\nesGCBfrpT3+qyspKRUZGasuWLVq1apVefvlljRkzRq+++qpWrVrlr7kCgE859gHGrrb6279/v08m\nAwC9yc7LILyCEQAslDUAOECfLeuf/OQnHmdeffVVo7FMd6V79913jXL/+c9/jHJDhgwxypk6e/as\nUS4sLMwot3HjRqOc6S/J0KFDPc5cuHDBaKygoCCj3D//+U+jXGRkpFHOdAe4FStWGOVMf4f69+9v\nlPOlPlvWAOAklDUAOABlDQAOQFkDgANQ1gDgAJQ1ADiAncva471BAAD+x5U1AFjsfGVNWQOAhbIG\nAAewc1mzZg0AFm9skbp48WKFhIRo/PjxnY7x97//XQkJCZowYYJuu+02vfPOO27NjbIGAIs3yjoz\nM1MlJSVdjjFq1CgdPHhQ77zzjn73u9/pwQcfdGtuLIMAgMUbyyBTpkxRdXV1l/dPmjSp7c8TJ07U\n6dOn3TqvT8u6rKzM44y7/yW4WkJCglHO9Jvz0UcfGeUGDhxolLv++uuNct+8E72/LF261Ch3ww03\nGOWOHTvmcWbTpk1GYy1fvtwol56ebpR77bXXjHIFBQVGuT/96U9+zdmRv9esn376ad11111uHcuV\nNQBY/FnWr732mv7617/q8OHDbh1PWQOAxZ2yrqysVGVl5Xca55133tHSpUtVUlKiQYMGuZWhrAHA\n4k5Zx8bGKjY2tu327t27PRrj1KlTuvfee7V161ZFR0e7naOsAcDijWWQBQsW6MCBA6qvr1dkZKTy\n8vLU3NwsScrOztbjjz+uL7/8Ujk5OZKkwMBAtx7fo6wBwOKNsi4uLu72/s2bN2vz5s0en5fnWQOA\nA3BlDQAWO7/cnLIGAAtlDQAOQFkDgANQ1gDgAJQ1ADgAZQ0ADtBny9rkLz527FgfzKRrNTU1Rrlb\nbrnFKPfvf//bKPftbRU9cd11Zk+lN91VcNSoUUa56dOnG+UOHDjgccZ09zxT9fX1RrnAwECjXEVF\nhVFu586dRrk5c+YY5YqKioxyvtRnyxoAnISyBgAHoKwBwAHsXNbsDQIADsCVNQBY7HxlTVkDgIWy\nBgAHoKwBwAEoawBwAMoaAByAsgYAB6CsAcABKGsAcIA+W9YbN270OPPggw8ajfXyyy8b5Zqamoxy\nkZGRRrmIiAijXFlZmVHu0qVLRrnJkycb5Uw9/fTTRrmPP/7Y48ySJUuMxtq8ebNRLiEhwSgXExNj\nlCspKTHKme6eZ8r0d8+X+mxZA4CT2Lms2RsEAByAK2sAsNj5ypqyBgALZQ0ADmDnsmbNGgAsLpfL\n44/OlJSUKDY2VjExMSooKOhwf319vdLS0pSYmKj4+Hg988wzPc6NsgYAizfKuqWlRcuWLVNJSYlO\nnDih4uJinTx5st0xhYWFSkpKUkVFhUpLS7VixQpdvny527lR1gBg8UZZl5WVKTo6WlFRUQoMDFRG\nRkaHd44PCwtTY2OjJKmxsVFDhgxRQED3q9KsWQOAxRtr1rW1te1eNBcREaE33nij3TFLly7V9OnT\nNWLECDU1NWn79u09npeyBgCLO2VdUVGhY8eOfadzrFmzRomJiSotLdVHH32k1NRUHTt2TAMGDOgy\nQ1kDgMWdok1KSlJSUlLb7Weffbbd/eHh4aqpqWm7XVNT02Gbiddff12/+c1vJEmjR4/WyJEjVVlZ\nqeTk5C7HZc0aACzeWLNOTk5WVVWVqqurdenSJW3btk3p6entjomNjdX+/fslSXV1daqsrNSoUaO6\nnRtX1gBg8caadUBAgAoLCzVz5ky1tLQoKytLcXFx2rBhgyQpOztbjz32mDIzM5WQkKArV67oiSee\n0ODBg7ufW2tra+t3nl1nJ3a55KNTd8p0rFdeecUoFx4ebpS7cOGCUa67/x75wpdffmmU69evn1Gu\nf//+RjmT7/uvf/1ro7H+/Oc/G+UOHjxolIuKijLKme4I6e8XhJj+LgQHB/ukW1wul1577TWPc9Om\nTfNL13FlDQAWx76CcfHixQoJCdH48ePbPpebm6uIiIi2RXbTvXMBwG689QpGX+i2rDMzMzuUscvl\n0iOPPKLy8nKVl5crLS3NpxMEAH9xbFlPmTJFgwYN6vB5f65FA4C/OLasu7Ju3TolJCQoKytLDQ0N\n3p4TAPSKa6qsc3Jy9Mknn6iiokJhYWFasWKFL+YFAH5n57L2+Nkgw4cPb/vzkiVLNHv27C6Pzc3N\nbftzSkqKUlJSPB0OQB938OBBHTp0yC9j2fnZIB6X9dmzZxUWFiZJ2rFjR7tnilzt22UNACamTp2q\nqVOntt3+wx/+4LOxHFvWCxYs0IEDB1RfX6/IyEjl5eWptLRUFRUVcrlcGjlyZNurcgDA6Rxb1sXF\nxR0+t3jxYp9NBgB6k2PLGgD6EjuXNbvuAYADcGUNABY7X1nbbtc9d97ltzO//OUvjXKmf/1ly5YZ\n5davX2+Uc4qioiKj3KJFi4xyx48f9zgzbtw4o7Hs/Ivcm0x3zwsKCjLK+WpHT5fLpbfeesvjXHJy\nMrvuAYA/2fkfZMoaACyUNQA4AGUNAA5AWQOAA1DWAOAAlDUAOABlDQAOQFkDgAPYuazZGwQAHIAr\nawCw2PnKmrIGAAtlDQAO0GfL+i9/+YvHmZycHB/MpGvvvfeeUa6wsNAo98c//tEoV1tba5QbMmSI\nUe7rr782yt10001GuStXrhjl4uPjjXImTHb4k6R9+/YZ5R555BGj3Pvvv2+U69evn1EuKirKKLd/\n/36jnC/Zuax5gBEALC6Xy+OPzpSUlCg2NlYxMTEqKCjocrw333xTAQEBeuGFF3qcG2UNABZvlHVL\nS4uWLVumkpISnThxQsXFxTp58mSnx61cuVJpaWlu7YdNWQOAxRtlXVZWpujoaEVFRSkwMFAZGRna\nuXNnh+PWrVun+fPna9iwYW7NjbIGAIs3yrq2tlaRkZFttyMiIjo85lRbW6udO3e2PUbnzlo5zwYB\nAIs7pXnkyBEdOXLkO51j+fLlys/Pb3uLMneWQShrALC4U7STJk3SpEmT2m6vXbu23f3h4eGqqalp\nu11TU6OIiIh2xxw9elQZGRmSpPr6eu3du1eBgYFKT0/vclzKGgAs3njqXnJysqqqqlRdXa0RI0Zo\n27ZtKi4ubnfMxx9/3PbnzMxMzZ49u9uilihrAPCqgIAAFRYWaubMmWppaVFWVpbi4uK0YcMGSVJ2\ndrbZeb05SQBwMm+9KGbWrFmaNWtWu891VdJbtmxx65yUNQBY7PwKRsoaACyUNQA4AGUNAA7QZ8s6\nNDTU48zGjRuNxnrwwQeNcv7ctU2SHn30UaOcO0+a9ybT78OSJUu8PBP7GDlypFFu9OjRRjnT7/mm\nTZuMck888YRRzlR4eLhfx3NHny1rAHASyhoAHICyBgAHoKwBwAEoawBwADuXNftZA4ADcGUNABY7\nX1lT1gBgoawBwAEoawBwAMoaAByAsgYAB6CsAcAB7FzWrlYfbef2zVuse+rrr782Gu+GG24wyn32\n2WdGufr6eqPcW2+9ZZT75p2QPfXiiy8a5X7xi18Y5Z5//nmj3H333WeUM5GXl2eUW716tVHu0qVL\nRrnvfe97RjnTX+mdO3ca5fr162eUS0tLM8qZdos7521qavI4N2DAAL/sismVNQBY7HxlTVkDgIWy\nBgAHoKwBwAHsXNZs5AQADsCVNQBY7HxlTVkDgIWyBgAHsHNZs2YNABaXy+XxR2dKSkoUGxurmJgY\nFRQUdHrMww8/rJiYGCUkJKi8vLzHufVKWZeWlvbGsLb3/vvv9/YUbImfl474mviGN8q6paVFy5Yt\nU0lJiU6cOKHi4mKdPHmy3TF79uzRhx9+qKqqKm3cuFE5OTk9zo2ythHKunP8vHTE18Q3vFHWZWVl\nio6OVlRUlAIDA5WRkdHhpfy7du3SokWLJEkTJ05UQ0OD6urqup0byyAAYPFGWdfW1ioyMrLtdkRE\nhGpra3s85vTp093OzXYPMPp7gT8gwOxLYLp5zZAhQ7q8r3///l3eb/p1GTRokFHO1MCBA/06nomw\nsDC/jmfnB62+7cYbbzTKmW6iZkfe+F65e46rN3/qKefTsu5ucNOdz651//jHP3p7CrZkh5+X7Ozs\n3p5CO3b4mlxrvFHW4eHhqqmpabtdU1OjiIiIbo85ffq0wsPDuz2vz8raH1sGAoC3eKuzkpOTVVVV\nperqao0YMULbtm1TcXFxu2PS09NVWFiojIwMHTlyRAMHDlRISEi357XdMggAOFlAQIAKCws1c+ZM\ntbS0KCsrS3FxcdqwYYOk//8f2l133aU9e/YoOjpaQUFB2rJlS4/n9dmbDwAAvMfvzwZx58nifU1U\nVJQmTJigpKQk3XLLLb09nV6zePFihYSEaPz48W2fO3funFJTUzVmzBjNmDFDDQ0NvTjD3tHZ1yU3\nN1cRERFKSkpSUlKSSkpKenGG8Ae/lrU7Txbvi1wul0pLS1VeXq6ysrLenk6vyczM7FA6+fn5Sk1N\n1QcffKA77rhD+fn5vTS73tPZ18XlcumRRx5ReXm5ysvLjd8iC87h17J258nifRWrUdKUKVM6PNXw\n2y8eWLRokfF7SjpZZ18XiZ+ZvsavZe3Ok8X7IpfLpTvvvFPJycnatGlTb0/HVurq6toeJQ8JCenx\nVV59ybp165SQkKCsrKw+uTzU1/i1rJ3y4gB/O3z4sMrLy7V3716tX79ehw4d6u0p2VJ3G+f0NTk5\nOfrkk09UUVGhsLAwrVixorenBB/za1m782TxvuibV9QNGzZMc+fO7dPr1lcLCQnRZ599Jkk6e/as\nhg8f3sszsofhw4e3/eO1ZMkSfmb6AL+W9befLH7p0iVt27ZN6enp/pyC7Vy8eFFNTU2SpAsXLmjf\nvn3tHvXv69LT01VUVCRJKioq0pw5c3p5RvZw9uzZtj/v2LGDn5k+wK8viunqyeJ9WV1dnebOnStJ\nunz5su6//37NmDGjl2fVOxYsWKADBw6ovr5ekZGRevzxx7Vq1Sr9/Oc/19NPP62oqCht3769t6fp\nd1d/XfLy8lRaWqqKigq5XC6NHDmy7QUXuHbxohgAcAC2SAUAB6CsAcABKGsAcADKGgAcgLIGAAeg\nrAHAAShrAHAAyhoAHOD/AHE4J7qzEzbzAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x1135d1550>"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Backpropagation"
     ]
    },
    {
     "cell_type": "heading",
     "level": 6,
     "metadata": {},
     "source": [
      "Intro to backprop"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's say we want to maximize $f = g*z$ and we also have $g = x + y$  \n",
      "Previously, we saw the case for one layer (logistic regression), that we can minimize $f$ by following the gradient in both these directions $\\frac{\\partial f(q,z)}{\\partial g} = z$ and $\\frac{\\partial f(g,z)}{\\partial z} = g$   \n",
      "Now, what does it mean to follow $g$? We have $\\frac{\\partial g(x,y)}{\\partial x} = 1$ and $\\frac{\\partial g(x,y)}{\\partial y} = 1$  \n",
      "We want to know how we have to change $x, y, z$ so that $f$ increases. Just as previously, we will take a gradient step $x = x + \\lambda \\frac{\\partial f}{\\partial x}$; $y = y + \\lambda  \\frac{\\partial f}{\\partial y}$; $z = z + \\lambda \\frac{\\partial f}{\\partial z}$. For $z$, we already know the gradient, it's $g$, so it's $x+y$. So we update $z$ with $z = z + \\lambda \\times (x+y)$. What about for $x$ and for $y$? We just apply the chain rule: $\\frac{\\partial f(g,z)}{\\partial x} = \\frac{\\partial f(g,z)}{\\partial g}\\frac{\\partial g(x,y)}{\\partial x}$, so for $x$ we update it with $x = x + \\lambda \\times \\frac{\\partial f(g,z)}{\\partial g} \\times \\frac{\\partial g(x,y)}{\\partial x} = x + z \\times 1$, and for $y$ we update it with $y = y + \\lambda \\times z \\times 1)$ too.\n",
      "\n",
      "So if you have not understood the above, you can try reading more details about backprop [here](http://karpathy.github.io/neuralnets/). Otherwise, let's apply backprop to our logistic regression to which we had a layer:"
     ]
    },
    {
     "cell_type": "heading",
     "level": 6,
     "metadata": {},
     "source": [
      "Backprop on our logistic regression, with one additional hidden layer"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The forward pass of our logistic regression is currently: $$X \\rightarrow W_{logistic} \\cdot X \\rightarrow softmax$$\n",
      "\n",
      "Let's consider that we add a hidden layer with a non-linearity $\\sigma$ to our logistic regression, it becomes: \n",
      "$$X \\rightarrow W_{hidden} \\cdot X \\rightarrow \\sigma(W_{hidden} \\cdot X) \\rightarrow W_{logistic} \\cdot \\sigma(W_{hidden} \\cdot X) \\rightarrow softmax$$\n",
      "\n",
      "The intuition behind back-propagation is that we keep doing what we did for optimising our previous loss function but for all the layers. The cool thing is that thanks to the chain rule of derivatives, this is an efficient process! Previously our loss (error) was the sum of the errors for all points, and it's a function of our weights: $$E(W) = \\sum_{n=1}^N E_n(W)$$\n",
      "\n",
      "Now if we add a hidden layer with a non-linearity $\\sigma$ our predictions become $$P(y_i=j|x_n; W) = softmax(W_{logistic}, \\sigma(W_{hidden}, x_n))[j]$$\n",
      "\n",
      "Our loss function code changes to take this formula into account. And that's it for the forward pass. Let's note $a_j = \\sum_i w_{ji}z_i$ for the pre-activation units, also called pre-synaptic, and $z_i = \\sigma(a_i)$ are the activations (or the layer below in the previous $a_j$ formula), also called post-synaptic. Now if we keep using the gradient updates that we saw before, we can only apply them to the activated hidden units, and we don't know how we should change $W_{hidden}$. We do that through the chain rule: $$\\frac{\\partial E_n}{\\partial w_{ji}} = \\frac{\\partial E_n}{\\partial a_j}\\frac{\\partial a_j}{\\partial w_{ji}}$$\n",
      "\n",
      "That means, we can back-propagate the errors through the partial derivatives: we can compute how much each of the hidden units contributes to the error, and have a gradient w.r.t. its weights. In the code, I called *err_h* the errors at the hidden units, and *err_o* the errors are the output, *act_xi* the activations of the hidden units, *act_d* the activation derivatives, and *backprop_o* the backpropagation of the output error. The algorithm looks like:\n",
      "1. Do a forward pass to get the output errors $\\delta_k$ (*err_o*)\n",
      "2. Back-propagate the errors with $\\sum_k w_{kj} \\delta_k$ (*backprop_o*) and multiply these backpropagated errors by the derivative of the activations $\\sigma'(a_j)$ (*act_d*) to get the error at this level: $\\delta_j = \\sigma'(a_j)\\sum_k w_{kj} \\delta_k$ (*err_h*)\n",
      "3. Multiply by the activations to get the local gradient: $\\frac{\\partial E_n}{\\partial w_{ji}} = \\delta_j z_i$\n",
      "\n",
      "Everything I told you about back-prop now generalizes from 1 to K hidden layers, because the chain rule $\\frac{\\partial E}{\\partial W} = \\frac{\\partial E}{\\partial a_j}\\frac{\\partial a_j}{\\dots }\\frac{\\dots }{\\partial W}$ stays true. So if we had more hidden layers, we would repeat steps 2 and 3 as much as needed to get back to the input features ($z_i == x_{ni}$).\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Changing our Logistic-regression into a \"Neural\" Net"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def relu_u(W, x, clip=6):\n",
      "    \"\"\"\n",
      "    ReLU unit function: max(0, W.x)\n",
      "    In [1]: a = np.random.standard_normal((1000,1000))\n",
      "    In [2]: %timeit (a+abs(a))/2\n",
      "    100 loops, best of 3: 4.27 ms per loop\n",
      "    In [3]: %timeit a[a<0]=0\n",
      "    1000 loops, best of 3: 736 \u00b5s per loop\n",
      "    \"\"\"\n",
      "    #tmp = np.dot(x, W)\n",
      "    #tmp[tmp < 0] = 0\n",
      "    #return np.append(tmp, 1)  # for the biases of the next layer\n",
      "    return np.append(np.clip(np.dot(x, W), 0, clip), 1)\n",
      "\n",
      "def relu_d(W, x, clip=4):\n",
      "    #return np.asarray(np.dot(x, W) >= 0, dtype='float')\n",
      "    tmp = np.dot(x, W)\n",
      "    return np.asarray((tmp >= 0) & (tmp < clip), dtype='float')\n",
      "\n",
      "def tanh_u(W, x):\n",
      "    tmp = np.dot(x, W)\n",
      "    return np.append(np.tanh(tmp), 1)  # for the biases of the next layer\n",
      "\n",
      "def tanh_d(W, x):\n",
      "    tmp = tanh_u(W, x)[:-1]\n",
      "    return 1 - tmp**2\n",
      "\n",
      "\n",
      "class NeuralNet:\n",
      "    def __init__(self):\n",
      "        self.name = \"Neural Net\"\n",
      "        #self.act_u = relu_u  # try ReLUs, but with a small learning rate\n",
      "        #self.act_d = relu_d  # (think about why)\n",
      "        self.act_u = tanh_u\n",
      "        self.act_d = tanh_d\n",
      "        \n",
      "    def loss(self):\n",
      "        \"\"\" Negative log-likelihood of logistic regression using the hidden layer \"\"\"\n",
      "        l = 0.\n",
      "        for i, xi in enumerate(self.x):\n",
      "            p_i = self.pred(xi)\n",
      "            l += np.log(p_i[self.y[i]])\n",
      "        return - l / self.x.shape[0]\n",
      "            \n",
      "    def one_epoch_of_backprop_SGD(self, learning_rate=0.01):\n",
      "        \"\"\" applying the backprop equations explained above \"\"\"\n",
      "        for i, xi in enumerate(self.x):\n",
      "            act_xi = self.act_u(self.W_hid, xi)\n",
      "            p_i = softmax(self.W_log, act_xi)\n",
      "            err_o = np.zeros(p_i.shape)\n",
      "            err_o[self.y[i]] = 1\n",
      "            err_o = err_o - p_i\n",
      "            backprop_o = np.dot(err_o, self.W_log[:-1,:].T)\n",
      "            act_d = self.act_d(self.W_hid, xi)\n",
      "            err_h = act_d * backprop_o\n",
      "            self.W_hid += learning_rate * np.outer(xi, err_h)\n",
      "            self.W_log += learning_rate * np.outer(act_xi, err_o)\n",
      "    \n",
      "    def pred(self, xi):\n",
      "        return softmax(self.W_log, self.act_u(self.W_hid, xi))\n",
      "                    \n",
      "    def train(self, x_train, y_train, n_hidden=42):  # try different numbers of hidden units, e.g. 4, 64, 128, 200\n",
      "        self.x = np.hstack((x_train, np.ones((x_train.shape[0], 1))))\n",
      "        self.y = y_train\n",
      "        self.n_classes = len(set(y_train))\n",
      "        self.n_hidden = n_hidden\n",
      "        self.W_hid = np.random.random((x_train.shape[1] + 1, self.n_hidden)) - 0.5\n",
      "        self.W_hid *= 2 * 6. / (x_train.shape[1] + self.n_hidden)\n",
      "        self.W_log = np.zeros((self.n_hidden + 1, self.n_classes))  # +1 for biases\n",
      "        print(\"x shape:\", self.x.shape)\n",
      "        print(\"W_hid shape:\", self.W_hid.shape)\n",
      "        print(\"W_log shape:\", self.W_log.shape)\n",
      "        tmp_loss = self.loss()\n",
      "        print(\"training loss before training:\", tmp_loss)\n",
      "        print(\"prediction on x[0] before training:\", self.pred(self.x[0]))\n",
      "        # training with SGD\n",
      "        losses = [tmp_loss]\n",
      "        l_r = 0.01\n",
      "        for _ in xrange(10):  # 10 epochs\n",
      "            self.one_epoch_of_backprop_SGD(learning_rate=l_r)\n",
      "            losses.append(self.loss())\n",
      "            # play with the learning rate schemes\n",
      "        print(\"best loss:\", losses[-1])\n",
      "        print(\"training loss after training:\", self.loss())\n",
      "        print(\"prediction on x[0] after training:\", self.pred(self.x[0]))\n",
      "        print(\"true x[0] label:\", self.y[0])\n",
      "        plt.plot(np.arange(len(losses)), np.log(losses), linewidth=1)\n",
      "        plt.ylabel(\"log of the loss function\")\n",
      "        plt.xlabel(\"#epochs\")\n",
      "        plt.show()\n",
      "        plt.imshow(self.W_hid, interpolation='none')\n",
      "        plt.colorbar()\n",
      "        plt.show()\n",
      "        #plt.imshow(self.W_log, interpolation='none')\n",
      "        #plt.colorbar()\n",
      "        #plt.show()\n",
      "        \n",
      "    def score(self, x_test, y_test):\n",
      "        x = np.hstack((x_test, np.ones((x_test.shape[0], 1))))\n",
      "        y = y_test\n",
      "        c = 0\n",
      "        for i, xi in enumerate(x):\n",
      "            a = np.argmax(self.pred(xi))\n",
      "            if a == y[i]:\n",
      "                c += 1\n",
      "        return c * 1. / y.shape[0]\n",
      "    \n",
      "nn = NeuralNet()\n",
      "nn.train(x_train, y_train)\n",
      "print(\"Train set accuracy:\", nn.score(x_train, y_train))\n",
      "print(\"Test set accuracy:\", nn.score(x_test, y_test))\n",
      "#nn.train(x_train, y_train, n_hidden=10)\n",
      "#print \"Train set accuracy:\", nn.score(x_train, y_train)\n",
      "#print \"Test set accuracy:\", nn.score(x_test, y_test)\n",
      "print(\"you can also regularize a neural net, just change the loss function appropriately\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "x shape: (1527, 65)\n",
        "W_hid shape: (65, 42)\n",
        "W_log shape: (43, 10)\n",
        "training loss before training: 2.30258509299\n",
        "prediction on x[0] before training: [ 0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1  0.1]\n",
        "best loss:"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 0.0253889805665\n",
        "training loss after training: 0.0253889805665\n",
        "prediction on x[0] after training: [  3.83913679e-05   8.68270986e-01   3.32674656e-04   1.51426304e-06\n",
        "   5.77964865e-03   1.86531340e-05   2.65499370e-03   2.51086136e-05\n",
        "   1.22848931e-01   2.90992410e-05]\n",
        "true x[0] label: 1\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAFHCAYAAABals4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0VPXBxvFnIGHfdyFAoiwhkISwKhEMUASBYGWxgAoG\n3EDRal+X1hbRVxG0vlVQ1CqySVEQZFVUkKACEQpIWMJOJIBYFUEQIyS57x+3SQgBJ8vM/ObOfD/n\nzIEZwszTOcc+/O79LS7LsiwBAABHKmM6AAAAKDmKHAAAB6PIAQBwMIocAAAHo8gBAHAwihwAAAcz\nWuSjRo1S/fr1FR0dbTIGAACOZbTIk5KStHLlSpMRAABwNKNF3rVrV9WsWdNkBAAAHI175AAAOBhF\nDgCAg4WYDvBbXC6X6QgAAPhccY5B8fsRuWVZPLz4eOKJJ4xnCIYH3zPfcSA8+I598yguo0U+bNgw\ndenSRXv37lXjxo01Y8YMk3EAAHAco5fW582bZ/LjAQBwPL+/tA7vSkhIMB0hKPA9ex/fsffxHfsn\nl1WSC/I+4nK5SnS/AAAApypu9zEiBwDAwShyAAAcjCIHAMDB/L7Iz583nQAAAP/l90W+eLHpBAAA\n+C+/L/JXXzWdAAAA/+X3Rb5rl5SWZjoFAAD+ye+LfPRo6bXXTKcAAMA/+f2GMOnpltq1kw4flipX\nNp0IAADvCrgNYZo2leLjJbZlBwCgML8vckkaM8ae9Oa/1w4AADDDEUXeu7f044/Spk2mkwAA4F8c\nUeRlykj33CNNm2Y6CQAA/sXvJ7vlxvvuO6l5c+ngQalWLcPBAADwkoCb7Jarbl0pMVGaOdN0EgAA\n/IdjRuSStH69dPvt0u7d9uV2AAACTcCOyCXpmmukihWl1atNJwEAwD84qshdLmnsWPZfBwAgl6Mu\nrUvSmTNSkyZSaqoUFmYoGAAAXhLQl9YlqUoVadgw6Y03TCcBAMA8x43IJWnHDnuTmPR0KTTU97kA\nAPCWgB+RS1KbNtJVV0lLl5pOAgCAWY4scsnef52d3gAAwc6Rl9Yl6ddf7Ulvn30mtWzp42AAAHhJ\nUFxal6Ty5aXRo6XXXjOdBAAAcxw7IpfsyW7t20sZGVKlSr7LBQCAtwTNiFySwsPt3d7eecd0EgAA\nzHB0kUvs9AYACG6OL/LevaXvv5c2bTKdBAAA33N8kZctK919N6NyAEBwcvRkt1z/+Y+9BO3gQalm\nTR8EAwDAS4JqsluuevWkvn2lmTNNJwEAwLcCYkQuSV98Ya8r373bPu4UAAAnCsoRuSTFx9ubxHz6\nqekkAAD4TsAUucvF/usAgOATMJfWJen0aXv/9R07pEaNvBgMAAAvCdpL65JUtao0bJj05pumkwAA\n4BsBNSKXpO3bpT597H3YQ0O9kwsAAG8J6hG5JEVHSxER0rJlppMAAOB9AVfkEvuvAwCCR8BdWpek\nX3+1J719/rnUooUXggEA4CVBf2ldsteTJyVJr71mOgkAAN4VkCNySTp0SOrYUTp8WKpUycPBAADw\nEkbk/xURIXXuLL37rukkAAB4T8AWuWTv9MakNwBAIDNa5CtXrlRkZKSaN2+uyZMne/z9b7jBPuJ0\n82aPvzUAAH7BWJFnZ2frvvvu08qVK7Vr1y7NmzdPaWlpHv2MsmWlu+9mVA4ACFzGinzjxo1q1qyZ\nwsPDFRoaqqFDh2rJkiUe/5zRo6WFC6Uff/T4WwMAYJyxIj969KgaN26c9zwsLExHjx71+OfUq2df\nYp892+NvDQCAccaK3OVy+eyzcie9+e9COwAASibE1Ac3atRIGRkZec8zMjIUFhZW6OcmTJiQ9/uE\nhAQlJCQU+7OuvVYKCZHWrJF69ChJWgAAvCM5OVnJyckl/vvGNoTJyspSy5YttXr1ajVs2FCdOnXS\nvHnz1KpVq/xwpdgQ5mLTptlFvmCBR94OAACvcMyGMCEhIXr55ZfVu3dvRUVF6Q9/+EOBEve0W2+V\nVq2Sjh3z2kcAAOBzAbtF66Xcc4/UsKE0frzH3hIAAI8qbvcFVZFv2yb172/vwx5ibHYAAACX55hL\n6ybExtrHmy5fbjoJAACeEVRFLtlL0aZNM50CAADPCKpL65KUmWmPytetk5o39+hbAwBQalxad6NC\nBSkpSXr9ddNJAAAovaAbkUvSwYNSp05SRoZUsaLH3x4AgBJjRF4EV14pdewozZ9vOgkAAKUTlEUu\nSWPHcrwpAMD5grbI+/aVvvlG2rLFdBIAAEouaIu8bFnprrsYlQMAnC0oJ7vlOn5catXK3umtRg2v\nfQwAAEXGZLdiaNBA6t1bmj3bdBIAAEomqItcsnd6e+01yX+vSwAAcHlBX+Tdukkul7R2rekkAAAU\nX9AXucvF/usAAOcK6sluuU6dksLDpV27pCuu8PrHAQBwWUx2K4Hq1aWbb5amTzedBACA4mFE/l9f\nfSUlJtpL0UJCfPKRAAAUwoi8hNq2lcLCpBUrTCcBAKDoKPILsP86AMBpuLR+gcxMqXFjacMGqVkz\nn30sAAB5uLReChUqSLffLr3+uukkAAAUDSPyixw4IF19tZSRYRc7AAC+VNzuK1KRHz16VOnp6crO\nzpZlWXK5XOrWrVupghYpnIEil6Q+faRbbpFuu83nHw0ACHIeL/JHH31U7777rqKiolS2bNm815ct\nW1bylEUNZ6jIlyyRJk2y75UDAOBLHi/yFi1aaPv27SpfvnypwxWXqSLPypKuvFJautRelgYAgK94\nfLLbVVddpXPnzpUqlNOEhEh33cVSNACA/3M7Ih84cKC2bdumnj175o3KXS6XpkyZ4v1whkbkknT8\nuNSqlZSebm/hCgCALxS3+9xuRjpgwAANGDBALpdLkvImuwW6Bg2k66+X5syR7rvPdBoAAC6tSLPW\nf/31V+3du1eSFBkZqdDQUK8Hk8yOyCUpOVm6915pxw77uFMAALzN4/fIk5OT1aJFC91777269957\n1bx5c61du7ZUIZ3iuusky5I++8x0EgAALs3tiLxdu3aaN2+eWrZsKUnau3evhg4dqi1btng/nOER\nuSRNnSqtWye9847RGACAIOHxEXlWVlZeiUv2crSsrKySpXOgESOkjz6yJ78BAOBv3I7Ik5KSVLZs\nWd16662yLEtz585VTk6O3nrrLe+H84MRuSTdeacUHi49/rjpJACAQOfxDWEyMzP1yiuvaN26dZKk\nrl27auzYsT7ZIMZfinzLFummm6SDB6ULNrcDAMDjvLLXuin+UuSSfZDK449LiYmmkwAAApnHinzI\nkCFasGCB2rRpU2jduMvlUmpqaumSFiWcHxX5rFn2hLcPPzSdBAAQyDxW5MeOHVPDhg319ddfF3pD\nl8ulpk2bli5pUcL5UZH/8ovUpIn05Zf2PuwAAHiDx2atN2zYUJI0bdo0hYeHF3hMmzat9EkdpmJF\naeRI6fXXTScBACCf23vkcXFx2rp1a4HXoqOjtX37dq8Gk/xrRC5J+/ZJ8fHS4cNShQqm0wAAApHH\nRuSvvvqqoqOjtWfPHkVHR+c9wsPDFRMT45GwTtO8uRQXJ733nukkAADYLjsiP3XqlH788Uc99thj\nmjx5ct6/DqpWraratWv7JpyfjcglafFi6fnn7d3eAADwNI8vP9uwYYNat26tatWqSZJ++uknpaWl\nqXPnzqVLWpRwfljkWVlSRIS0fLkUG2s6DQAg0Hh8i9YxY8aoSpUqec8rV66se+65p2TpAkBIiL3T\n26uvmk4CAEARilySypTJ/7GyZcsqOzvba4Gc4I47pHfflX76yXQSAECwc1vkERERmjJlis6fP69z\n587ppZde0pVBvpC6YUPpd7+T5swxnQQAEOzcFvlrr72mdevWqVGjRgoLC1NKSor++c9/+iKbXxsz\nxr687me38AEAQcbIXusLFizQhAkTtHv3bm3atEnt2rW75M/542S3XJYltWolvfGG1LWr6TQAgEBR\n3O4LcfcD//nPf/TGG28oPT097xxyl8tVqmNMo6Oj9f777+vuu+8u8XuY5nLZo/Jp0yhyAIA5bov8\nxhtvVLdu3dSrV6+8SW8XH6JSXJGRkaX6+/5i5EhpwgTp22+l+vVNpwEABCO3Rf7LL79o8uTJvsji\nODVqSIMGSW+9Jf35z6bTAACCkdsi79+/v1asWKF+/foV64179eql48ePF3p94sSJSizGod4TJkzI\n+31CQoISEhKKlcPbxoyRBg6UHnlEKlvWdBoAgNMkJycrOTm5xH/f7WS3KlWq6OzZsypXrpxCQ0Pt\nv+Ry6ScPLKLu3r27XnjhBUdOdrtQp07S+PFS//6mkwAAnM7jO7udOXNGOTk5yszM1OnTp3X69GmP\nlHguJxS1O2PHstMbAMAMtyPyzz777JKvd+vWrcQf+v777+v+++/X999/r+rVqysuLk4ffvhh4XAO\nGZH/8ovUuLG0aZO9DzsAACXl8UNT+vfvnzdLPTMzUxs3blT79u316aefli5pUcI5pMgl6aGHpHLl\npEmTTCcBADiZx4v8YhkZGXrggQe0aNGiYocrLicV+d699nryw4el8uVNpwEAOJXH75FfLCwsTGlp\nacX9awGvRQupbVtp4kTTSQAAwcTt8rNx48bl/T4nJ0dfffWV2rdv79VQTjVzptS9u1ShAuvKAQC+\n4bbIO3TokP/DISEaPny44uPjvRrKqa64Qvr0UykhQSpTRnr0UdOJAACB7rJF3rNnT61evVo7d+7U\nc88958tMjtawobRmTX6ZP/yw6UQAgEB22SL/5ptvtH79ei1dulRDhw4t9OeX28QFUqNGBcv8T38y\nnQgAEKguO2t9wYIFmj59utatW1fg8nquNWvWeD+cg2atX0pGhn3P/N57pQcfNJ0GAOAEHl9+9tRT\nT2n8+PGlDlYSTi9yyV6O1r27dP/90gMPmE4DAPB3Xl9H7kuBUOSS9PXXdpk/+KB0wSIAAAAKKW73\nuZ21jtJr2tSezd69u33P/N57TScCAAQKitxHwsPzJ8C5XPZBKwAAlJbbnd3279+vzMxMSfYEtylT\npujkyZNeDxaIcst88mTptddMpwEABAK3RT5o0CCFhIRo//79uvvuu5WRkaHhw4f7IltAioiwy/zZ\nZ6V//tN0GgCA07m9tF6mTBmFhIRo0aJFGjdunMaNG6e4uDhfZAtYV15Z8J75HXeYTgQAcCq3RV6u\nXDn961//0uzZs7Vs2TJZlqXz58/7IltAu+qqgmU+apTpRAAAJ3J7af2tt95SSkqKHn/8cUVERCg9\nPV233XabL7IFvGbN7DJ/4glpxgzTaQAATlSsdeQnTpzQkSNHFBMT481MeQJlHbk7e/ZIPXtKzzwj\njRxpOg0AwCSPryO/7rrrtGzZMmVlZal9+/aqW7eu4uPj9Y9//KNUQZGvZUtp1Sq7zMuUkbjgAQAo\nKreX1k+dOqVq1app0aJFGjFihDZu3KhVq1b5IltQiYy0y/yxx6S33zadBgDgFG6LPDs7W998843m\nz5+vfv36SbKH/fC8Vq2kTz6RHnlE+te/TKcBADiB20vr48ePV+/evRUfH69OnTrpwIEDat68uS+y\nBaWoKLvMe/WyL7Nf4gRZAADycGiKn9qxwy7zF1+U/vAH02kAAL5S3O5ze2k9IyNDN910k+rWrau6\ndetq0KBBOnLkSKlCwr02baSPP5b++EdpwQLTaQAA/sptkSclJWnAgAE6duyYjh07psTERCUlJfki\nW9CLjpZWrrSPPl240HQaAIA/cntpPTY2Vtu2bXP7mjcE86X1C331ldSnjzRtmjRwoOk0AABv8vil\n9dq1a2vOnDnKzs5WVlaW3n77bdWpU6dUIVE8bdtKH35oH326eLHpNAAAf+J2RJ6enq5x48YpJSVF\nktSlSxdNnTpVTZo08X44RuQFbNki3XCDfWrajTeaTgMA8Ibidh+z1h1m82apb1/pzTelxETTaQAA\nnuaxIh83btxvfsiUKVOKn66YKPJL27RJ6t9fmj7d/hUAEDg8ttd6+/btL7mDm2VZ7OxmWMeO0rJl\ndonPmCH9d8M9AEAQ4tK6g335pX15fdYs+945AMD5PD5rHf6rc2dpyRL76NOVK02nAQCYQJE73DXX\n2EvSRoywd4IDAAQXijwAdOkivf++dOut9oErAIDg4bbI9+zZo549e6p169aSpNTUVD399NNeD4bi\niY+3t3EdPlxavdp0GgCAr7gt8jvvvFMTJ05UuXLlJEnR0dGaN2+e14Oh+Lp2tct86FDp009NpwEA\n+ILbIj979qw6d+6c99zlcik0NNSroVBy3bpJ771nH32anGw6DQDA29wWed26dbV///685++9956u\nuOIKr4ZC6Vx3nTR/vjRkiLR2rek0AABvcruO/MCBA7rrrru0fv161axZUxEREZo7d67Cw8O9H451\n5KXy6af2yHzhQnukDgDwf17ba/3nn39WTk6OqlatWuJwxUWRl96qVdKwYfas9muvNZ0GAOCOx4s8\nMzNTCxcuVHp6urKzs/O2aB0/fnypw7oNR5F7xCefSLfcYpd5fLzpNACA3+Lxnd1uvPFGLV26VKGh\noapcubKqVKmiypUrlyokfKtXL2nOHOmmm6T1602nAQB4ktsReZs2bbRjxw5f5SmAEblnrVxp7wC3\ndKl09dWm0wAALsXjI/IuXbooNTW1VKHgH/r0kWbOlAYMsA9cAQA432VH5NHR0ZKk7Oxs7du3TxER\nESpfvrz9l1wun5Q7I3LvWLFCSkqSli+XOnUynQYAcCGPTXZLT0+/7Bu6XC41bdq05CmLGo4i95pl\ny6TRo6UPPpA6dDCdBgCQy+Oz1m+77TbNmTPH7WveQJF719Kl0qhR0gMP2I9q1UwnAgB4/B75xRPd\nsrKytHnz5uInu8jDDz+sVq1aKTY2VgMHDtSpU6dK/Z4ongEDpA0bpH37pGbNpEmTpDNnTKcCABTH\nZYt84sSJqlq1qrZv366qVavmPerVq6cBAwaU+oOvv/567dy5U9u2bVOLFi307LPPlvo9UXzNm0uz\nZ9tbuX71lV3oL7wgnT1rOhkAoCjcXlp/7LHHNGnSJK+GeP/997Vw4UK9/fbbBcNxad3ntm+XJkyw\nR+qPPSbddZdUoYLpVAAQPLy2Ras3JSYmatiwYRo+fHiB1ylyc7ZulZ54wv71L3+x76X/d9ECAMCL\nitt9IV7Mol69eun48eOFXp84caISExMlSc8884zKlStXqMRzTZgwIe/3CQkJSkhI8EZUXCQuzp4M\nt2mTXeiTJ0t//as0cqTEKbYA4DnJyclKLsW505cdkR86dEgRERElfuOimDlzpt544w2tXr1aFS5x\n/ZYRuf9Yv94u9AMHpPHjpVtvlUK8+s9AAAhOHpu1PnjwYElSjx49Sp/qElauXKnnn39eS5YsuWSJ\nw7906WIfvjJjhv2IipLmzpWys00nA4DgdtkRedu2bTVkyBC9+uqreuihhwr868Dlcumhhx4q1Qc3\nb95c586dU61atSRJ11xzjaZNm1YwHCNyv2RZ9lnnf/ubdOqUPTlu0CCpjNvFjAAAdzx2j/ydd97R\n4sWLlZ2drdOnT+cdX5r7a2nt27ev1O8BM1wuqWdPqUcP6aOP7Evt//u/0pNPSr//vf3nAADfcDtr\n/YMPPlDfvn19lacARuTOYFn2vu3jx9sl/tRTUr9+FDoAlITHl5+dPHlSTz75pD777DNJ9szx8ePH\nq3r16qVLWpRwFLmj5ORIixfbk+IqVbIL/frrKXQAKA6Pb9E6atQoVatWTQsWLND8+fNVtWpVJSUl\nlSokAlOZMtLAgdK2bdJDD0l//KPUtat9Px0A4B1uR+SxsbHatm2b29e8gRG5s2VnS/Pm2ffOw8Ls\nEXrXrqZTAYB/8/iIvGLFivr888/znn/xxReqVKlSydIhqJQta683T0uTRoywH9dfL6WkmE4GAIHD\n7Yj8q6++0ogRI/JOJ6tZs6ZmzZql2NhY74djRB5Qzp2TZs6Unn5aio62R+qchQ4ABXltr/XcIvfF\nJLdcFHlg+vVX6c03pYkTpY4d7UL3wb8LAcARPH5pPVf16tV9WuIIXOXLS/feK+3fLyUkSH36SEOG\nSDt3mk4GAM7DXlwwpmJFe2b7/v1Sp072BjPDhkl79phOBgDOQZHDuMqVpYcftgs9Olq69lr7lLX9\n+00nAwD/5/Ye+cKFCwttyVq9enVFR0erXr163g3HPfKgdOqU9OKL0tSp9pavf/2rFB5uOhUA+IbH\nJ7v169dPGzZsUPfu3SXZ56a2a9dOhw4d0vjx4zVixIjSJf6tcBR5UPvxR+n//k+aNk26+WbpL3+R\nGjc2nQoAvMvjk93Onz+vtLQ0LVy4UAsXLtSuXbvkcrn05ZdfavLkyaUKC/yWmjXtw1j27JGqV7dn\nto8bJx05YjoZAPgPt0WekZGh+vXr5z2vV6+eMjIyVLt2bZUrV86r4QBJqlNHmjTJ3lgmNNS+j96n\nj/Svf0lnz5pOBwBmuS3y7t27q1+/fpo1a5ZmzpypAQMGKCEhQT///LNq1Kjhi4yAJKl+fftS+9Gj\n9i5xs2dLjRpJo0dLa9fah7YAQLBxe488JydHixYt0rp16yRJ8fHxGjRokEfOJHcbjnvkcOPYMWnu\nXGnWLOnnn6XbbrMfzZubTgYAJeOVnd2OHz+uTZs2SZI6d+7s9dnquShyFJVlSV99ZRf6vHlSs2b2\nqP3mm+177QDgFB6f7DZ//nx17txZCxYs0IIFC9SpUyctWLCgVCEBT3O5pLg4e9nakSPSn/8srVpl\nL1u7+WZp+XLp/HnTKQHA89yOyGNiYrRq1aq8Ufh3332nnj17KjU11fvhGJGjlH78UZo/3x6pHzgg\nDR9uj9TbtrXLHwD8jcdH5JZlqW7dunnPa9euTbnCMWrWlO6+W1q/XvriC6lqVemmm6SYGOnvf5e+\n+cZ0QgAoHbcj8ocffljbtm3T8OHDZVmW3n33XcXExOi5557zfjhG5PCCnBzp88/tWe+LFklXX22P\n0m+8UapUyXQ6AMHO45PdLMvSokWL9MUXX8jlcqlr16666aabSh20SOEocnjZ2bPS4sX2pfeNG6VB\ng+xSv/ZaqQwnEQAwwGvnkZtAkcOXjh61N5mZNcsu+NylbM2amU4GIJh4rMirVKly2bXiLpdLP/30\nU8kSFgNFDhMsS9q61b70nruUbeRIe/Y7eyAB8DZG5IAHnT8vrVxpl/rHH0u9e9ulfv319naxAOBp\nFDngJSdO2EvZZs+WDh7MX8oWG8tSNgCeQ5EDPrBvn13oc+ZI1arZo/Thw6UrrjCdDIDTUeSAD+Xk\nSJ99Zpf6++/bS9lGjrSXslWsaDodACeiyAFDzp61y3z2bGnTJmngQLvUr72WS+8Aio4iB/zA0aP5\np7L98ou9Pr1/f6lLFybJAfhtFDngR3JPZVuyxD645eBBe+Z7//5Snz5S7dqmEwLwNxQ54MeOHZM+\n+MAu9TVr7D3f+/eX+vWTWrfmEjwAihxwjMxMKTnZLvXly+3X+ve3HwkJUoUKJtMBMIUiBxzIsqRd\nu/JLPTVV6t7dLvW+faWGDU0nBOArFDkQAH74wd5Rbvly6aOPpCuvzB+tt2vHgS5AIKPIgQBz/rx9\nnnruaP3kSfueev/+0u9+J1WpYjohAE+iyIEAt3+/tGKFXeopKVJ8fH6xR0SYTgegtChyIIj89JP0\nySd2qX/wgVSnTv4l+GuukUJCTCcEUFwUORCkcnKkf/87/xL811/ba9Vz16zXrGk6IYCioMgBSJKO\nHMlfs56cLMXF5Y/WIyNZsw74K4ocQCG//GJvQJM7Wg8NzS/1bt2k8uVNJwSQiyIH8JssS9q+Pb/U\nd+60Z7/362evWW/QwHRCILhR5ACK5bvvpA8/tEv9k0+k5s3tUu/RQ+rcWSpXznRCILhQ5ABK7Px5\n6Ysv7Hvra9ZIe/bYZ6x3725vG9uxI6e3Ad5GkQPwmJMnpc8+s0t9zRr79LYuXexi797d3mWOJW6A\nZ1HkALzmhx8KFvvhw9K11+YXe9u2UtmyplMCzuaIIv/b3/6mpUuXyuVyqXbt2po5c6YaN25cOBxF\nDvi1776T1q7NL/ZvvrFnwSck2MUeE8O+8EBxOaLIT58+rapVq0qSpk6dqm3btunNN98sHI4iBxzl\n+HF7zXpusf/wg3Tddfkj9qgoih1wp7jdZ+TuVm6JS9KZM2dUp04dEzEAeFiDBtLQofZDko4ezS/2\nF1+UTp8uWOxsTAOUnrF75I8//rjmzJmjSpUqKSUlRTVq1Cj0M4zIgcBy+LBd6rnlnpmZfxm+e3d7\n6RvFjmDnN5fWe/XqpePHjxd6feLEiUpMTMx7PmnSJO3Zs0czZswoHI4iBwLaoUP5l+HXrLE3q8kt\n9e7d7dPcKHYEG78p8qI6fPiw+vbtqx07dhT6M5fLpSeeeCLveUJCghISEnyYDoCvWJZ9ROuF99jL\nlSs4Ym/a1HRKwPOSk5OVnJyc9/zJJ5/0/yLft2+fmjdvLsme7LZx40bNmTOncDhG5EDQsix7Q5rc\nUk9OlqpUyd+cpnt3KSzMdErA8xwxIh88eLD27NmjsmXL6qqrrtKrr76qevXqFQ5HkQP4L8uSdu3K\nL/a1a+2jWXNH6926SY0amU4JlJ4jiryoKHIAl5OTYx/+klvs69ZJlSpJ11yT/4iLY694OA9FDiAo\nWZa0b5+0YUP+Y/9+e7e5C8u9YUPTSYHfRpEDwH+dPi1t3Fiw3KtVK1jsbdtyEAz8C0UOAJeRkyPt\n3Vuw2A8dsi/BX1junMkOkyhyACiGU6cKjtpTUuxJdBcWe0wMo3b4DkUOAKWQk2Mve1u/Pr/cv/5a\nat++YLlfYqEN4BEUOQB42MmT0pdf5hf7l19KtWvbhd6li/1rdDRns8MzKHIA8LKcHCktreC99owM\nqUOHgqN2zoNCSVDkAGDAiRMFR+0bN9qX3y8s9jZtGLXDPYocAPxAdra9E92Fo/Zjx6SOHfOLvWNH\n7rWjMIocAPzUDz8UHLVv3mzvH9++fcFH/fqmk8IkihwAHMKypIMH7ULfvFn697+lLVukypULlztr\n24MHRQ5EjV+WAAALdUlEQVQADnZxuec+KPfgQZEDQICxLHsHuovLvWLFwuV+xRWm06K0KHIACAKW\nJaWn25fjKffAQpEDQJDKLfeLR+7lyxcud06B818UOQAgj2XZW8xePHIvV84u9A4dKHd/Q5EDAH5T\nbrlfPHIPDb30yN3lMp04uFDkAIBisyzp8OGCS+E2b7Z3oruw2GNjpaZNKXdvosgBAB5xcblv2SKl\npkpnztiHxMTE2I/YWHv72apVTScODBQ5AMCrvv9e2r7dLvXUVGnbNns72iuuyC/23JK/8kqpTBnT\niZ2FIgcA+FxWlrR/f8FyT021D5Np0ya/2GNi7NF8jRqmE/svihwA4DdOnswfveeW+44d9hGvF5Z7\nbKzUrJlUtqzpxOZR5AAAv5aTY29De2G5p6ZKx49LUVEFyz06Wqpd23Ri36LIAQCOdPp0wXvvuY9q\n1Qrfe2/Rwl4uF4gocgBAwMhd837hyD01VcrIkFq2LFjuMTGBcb47RQ4ACHg//yzt3Fmw3Ldts7ej\nvbDc27SRIiPtPeidgiIHAAQly5KOHClY7Dt32rPpGzWy779f+GjVyj4e1t9Q5AAAXCArSzpwwF7r\nvnOn/euuXdLevVL9+pcu+GrVzOWlyAEAKILsbPuc94sLfvduqVYtqXXrwiXvi/XvFDkAAKWQk2NP\nsMst9tyST0uzt6GNiipc8p5cIkeRAwDgBTk59j343IK/sOQrVChY7LlFX7du8Q+YocgBAPAhy5KO\nHStY8LklX6ZM4cvzrVtLDRpcvuApcgAA/IBlSd9+e+mCz8oqXPBRUVJYmFSmDEUOAIBf++67wgW/\na5e9Pv70aYocAABH+uEHqU4dihwAAMcqbvdx3DsAAA5GkQMA4GAUOQAADkaRAwDgYBQ5AAAORpED\nAOBgFDkAAA5GkQMA4GAUOQAADkaRAwDgYBQ5AAAOZrTIX3jhBZUpU0YnTpwwGQMAAMcyVuQZGRn6\n5JNP1LRpU1MRICk5Odl0hKDA9+x9fMfex3fsn4wV+UMPPaTnnnvO1Mfjv/gP0zf4nr2P79j7+I79\nk5EiX7JkicLCwhQTE2Pi4wEACBgh3nrjXr166fjx44Vef+aZZ/Tss8/q448/znuNM8cBACgZl+Xj\nFt2xY4d69uypSpUqSZKOHDmiRo0aaePGjapXr17BcC6XL6MBAOAXilPNPi/yi0VERGjz5s2qVauW\nyRgAADiS8XXkjLoBACg54yNyAABQcsZH5JezcuVKRUZGqnnz5po8ebLpOAEnIyND3bt3V+vWrdWm\nTRtNmTLFdKSAlZ2drbi4OCUmJpqOErBOnjypwYMHq1WrVoqKilJKSorpSAHn2WefVevWrRUdHa3h\nw4fr119/NR3J8UaNGqX69esrOjo677UTJ06oV69eatGiha6//nqdPHnS7fv4ZZFnZ2frvvvu08qV\nK7Vr1y7NmzdPaWlppmMFlNDQUP3jH//Qzp07lZKSoldeeYXv2EteeuklRUVFcRvJix544AH17dtX\naWlpSk1NVatWrUxHCijp6el64403tGXLFm3fvl3Z2dl65513TMdyvKSkJK1cubLAa5MmTVKvXr20\nd+9e9ezZU5MmTXL7Pn5Z5Bs3blSzZs0UHh6u0NBQDR06VEuWLDEdK6A0aNBAbdu2lSRVqVJFrVq1\n0rFjxwynCjxHjhzRBx98oDvuuINlll5y6tQpff755xo1apQkKSQkRNWrVzecKrBUq1ZNoaGhOnv2\nrLKysnT27Fk1atTIdCzH69q1q2rWrFngtaVLl2rkyJGSpJEjR2rx4sVu38cvi/zo0aNq3Lhx3vOw\nsDAdPXrUYKLAlp6erq1bt6pz586mowScBx98UM8//7zKlPHL/9QCwqFDh1S3bl0lJSWpXbt2uvPO\nO3X27FnTsQJKrVq19Kc//UlNmjRRw4YNVaNGDf3ud78zHSsgffvtt6pfv74kqX79+vr222/d/h2/\n/H8XLkH6zpkzZzR48GC99NJLqlKliuk4AWX58uWqV6+e4uLiGI17UVZWlrZs2aKxY8dqy5Ytqly5\ncpEuR6LoDhw4oBdffFHp6ek6duyYzpw5o7lz55qOFfBcLleR+tAvi7xRo0bKyMjIe56RkaGwsDCD\niQLT+fPnNWjQIN166636/e9/bzpOwFm/fr2WLl2qiIgIDRs2TJ9++qlGjBhhOlbACQsLU1hYmDp2\n7ChJGjx4sLZs2WI4VWD597//rS5duqh27doKCQnRwIEDtX79etOxAlL9+vXzdkX95ptvCm2Udil+\nWeQdOnTQvn37lJ6ernPnzundd9/VgAEDTMcKKJZlafTo0YqKitIf//hH03EC0sSJE5WRkaFDhw7p\nnXfeUY8ePTR79mzTsQJOgwYN1LhxY+3du1eStGrVKrVu3dpwqsASGRmplJQU/fLLL7IsS6tWrVJU\nVJTpWAFpwIABmjVrliRp1qxZRRpkeW2v9dIICQnRyy+/rN69eys7O1ujR49mFqqHrVu3Tm+//bZi\nYmIUFxcnyV5e0qdPH8PJAhe3jLxn6tSpuuWWW3Tu3DldddVVmjFjhulIASU2NlYjRoxQhw4dVKZM\nGbVr10533XWX6ViON2zYMK1du1bff/+9GjdurKeeekqPPfaYbr75Zk2fPl3h4eGaP3++2/dhQxgA\nABzMLy+tAwCAoqHIAQBwMIocAAAHo8gBAHAwihwAAAejyAEAcDCKHAggf/7zn5WcnKzFixf7bJvS\n8PBwnThxwiefBaAwihwIIBs3btTVV1+ttWvXqlu3bj75TJfLxV7ygEEUORAAHnnkEcXGxmrTpk26\n5pprNH36dI0ZM0ZPP/20Dhw4oBtuuEEdOnRQt27dtGfPHknS7bffrnvuuUcdO3ZUy5YttWLFCklS\nZmamkpKSFBMTo3bt2ik5OVmSlJ2drf/5n/9RdHS0YmNj9corr+R9/tSpU9W+fXvFxMTkvf/atWsV\nFxenuLg4tWvXTmfOnPHtlwIECwtAQNi0aZN1//33W+fPn7fi4+PzXu/Ro4e1b98+y7IsKyUlxerR\no4dlWZY1cuRI64YbbrAsy7L27dtnhYWFWZmZmdbf//53a/To0ZZlWdbu3butJk2aWJmZmda0adOs\nIUOGWNnZ2ZZlWdaJEycsy7Ks8PBw6+WXX7Ysy7KmTZtm3XHHHZZlWVZiYqK1fv16y7Is6+eff7ay\nsrK8/RUAQckv91oHUHybN29WTEyM0tLS8s4mOHPmjDZs2KAhQ4bk/dy5c+ck2ZfEb775ZklSs2bN\ndOWVV2r37t1at26d7r//fklSy5Yt1bRpU+3du1erV6/WmDFj8s5Wr1mzZt57Dhw4UJLUrl07LVq0\nSJIUHx+vBx98ULfccosGDhyoRo0aefkbAIITRQ443LZt23T77bfryJEjqlOnjs6ePSvLsvIui9eo\nUUNbt24t0nvlHuxiXeae9+VeL1++vCSpbNmyysrKkiQ9+uij6t+/v1asWKH4+Hh99NFHatmyZXH/\n5wFwg3vkgMPFxsZq69atatGihdLS0tSjRw99/PHH2rJli6pVq6aIiAi99957kuwiTk1Nzfv9ggUL\nZFmWDhw4oIMHDyoyMlJdu3bV3LlzJUl79+7V4cOHFRkZqV69eun1119Xdna2JOnHH3/8zVwHDhxQ\n69at9cgjj6hjx455984BeBZFDgSA7777TrVq1ZIk7d69W5GRkXl/NnfuXE2fPl1t27ZVmzZttHTp\nUkn26LtJkybq1KmT+vbtq9dff13lypXT2LFjlZOTo5iYGA0dOlSzZs1SaGio7rjjDjVp0kQxMTFq\n27at5s2bd8ksuaP6l156KW9iXLly5XTDDTd4+VsAghPHmAJBKikpSYmJiXn3twE4EyNyAAAcjBE5\nAAAOxogcAAAHo8gBAHAwihwAAAejyAEAcDCKHAAAB6PIAQBwsP8HAx8609gqaOIAAAAASUVORK5C\nYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110f38b50>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAREAAAE2CAYAAABGPaYAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXtwVHWWx78NxEEBYRBIgIDhFUICJIHEyPsREgQkgrCI\n42AWAXepslzcsQR3qsYtqxQcZ4dBYWZXRZaRHcCpUYwIGUDC2xDABBCQNxggoBiUAWGA0PsHlZ7+\nfX8n3R1uLtzg+VRRlUP3ffTtzsn9fft7zvH5/X4/FEVRbpI6t/sEFEWp3WgSURTFEZpEFEVxhCYR\nRVEcoUlEURRHaBJRFMURjpJIfn4+EhIS0KlTJ7z22ms1dU6KotQifDfrE6moqEDnzp2xZs0atG7d\nGunp6Vi8eDG6dOlS0+eoKIqHuek7kaKiInTs2BFxcXGIiorC+PHj8dFHH9XkuSmKUguod7Mbnjx5\nEm3atAnEsbGx2Lp1q/Ecn89382emKB7EDYO3k98TLxjObzqJRPrCZ8+ejfz8fDz00EP4+uuvjcfu\nuece6/lRUVEh48uXL1vbNGrUyIivX79uxHyhpX3ce++9AIBPPvkEI0aMwFdffWU83rx5c2ubCxcu\nGPHdd98d8rh33XWXtY9r164Zcd26dY34hx9+sLa5evUqAGDdunUYOHAgrly5YjweHR1txOfPn7f2\nwftt0KBB2HNlvv/+eyNu1qyZ9Zw6dW7c7FZ+Bs6ePWs8ztdV+lzx+VfusxJ+vwH7/Pn1Nm7c2Ii/\n+eabwM8bN25Ev3790LBhQ+M5v/zlL63j1BTSawgHX4fbxU0nkdatW6O0tDQQl5aWIjY21npefn4+\nDh06hPz8fNx33324//77b/aQinJLOXLkCI4ePXpLjuWFO4qb5aaTSFpaGg4ePIhjx46hVatWWLp0\nKRYvXmw976GHHqryTkRRvEz79u3Rvn37QLx27drbeDbe5aaTSL169TB37lwMHToUFRUVmDRpkvjN\nzJEjR3DPPffgyJEj1u1h8C1kJR07djTiw4cPG3HLli2tbfh2nrl48aIR87ID+MetYefOnVGnTh3r\ndrhyuRMMnz8vRU6fPm3EwR/ISnjZFB8fb8QtWrSocpu2bduioqICqampxuO8zJL2sX37diPmZQUv\nI4Eb72UwfDstLYHOnTsHAGjVqhX+9re/WXerZWVlRszvFQD87W9/M2K+9U9MTLS24f1+9913RsxL\n6eBrlJycjBYtWlifPTf5Ud6JAMCwYcMwbNiwsM+TljlehX+JvUxtWhq2a9fudp9CxPAfslvBjzaJ\nKIpSM2gSURTFEZpEQh2g3j8OcezYMeMx/ioSgKWb8Lpbui3++OOPjbhHjx5GXFhYaMQDBgyw9vHl\nl18aMa+7pa+j+dxYE+KvPPn1A/Z6/yc/+YkRB38DVglrHLzeZ93o0qVL1j6Yb7/91ojvu+8+6zks\njLMWlZGRYW1z5swZI965c6cRB38+AKB79+7WPvi94DguLs7apkmTJkbMGg9rQNJ7I+kzbqFJRFEU\nR2gSURTFEZpEFEVxRG1OIt7wzSrKjxy/31/tfxKRtufYtm0b6tWrhw8++MDxubt+JzJixIjAz3v2\n7DEek8xJLNaxKUrycaSnpxsxi2TDhw83YjaBATccuMHwubLgCdgmJ677YBFREoXZ0MRCqmSk++lP\nf2rEx48fN+JVq1YZMYuMgG1ImzJlihFv3rzZ2obNck2bNjVirmECbEE6KSnJiA8dOmTEktjO9UVv\nv/22Ef/Hf/yHtc3+/fuNuFOnTkZcWX9UiXTufK5uUhN3IhUVFXjmmWeM9hw5OTmW4F9RUYHp06fj\noYceqpHj6p2IoniAmrgTibQ9x5tvvomxY8eKRaU3gyYRRblDkNpznDx50nrORx99hKlTpwKomXYd\nKqwqigeIZFmxceNGbNq0qcrHI0kI06ZNw6xZs+Dz+UJqK9XB9SSye/fuwM+ff/658djAgQOt5/OF\n4D4XrJEA9np3yJAhRsxFbrweBmztgftPSMYj/j9e/3NhnGRo6tevnxF/+umnRty7d29rG9Z02BiW\nkpJixNw7A7DrmVhDYM0EsPWLRx991IgljWv9+vVGPG3aNCPm/iJS0Rtft5iYGCNm7QkATpw4YcRs\nUBs5cmTI8wRkg6FbRPLL3LdvX/Tt2zcQz5o1y3g8kvYcO3bswPjx4wHcuPYrV65EVFQUcnJybvrc\n9U5EUTxATdwRRNKeI7gSe+LEiRg5cqSjBAJoElEUT1Ajy4oq2nP8z//8DwDgX/7lXxwfQzyuK3tV\nFKVa1JTZTGrPUVXyWLBgQY0c0/UkElwc1qpVK+MxjgG78U2fPn2MePLkydY2v/vd74yYxSf2eAQr\n2JWwT+Czzz4zYumW7+9//7sRs67C/gaJ3/72t0bMb7ikM7AG0rlzZyNmzUfyxbz//vtGzH1nI/F8\nbNmyxYil95P/Ly8vz4jXrVtnxJJOxpoOfyZY8wLshlEVFRVGvG/fPiNmHQmA2KnPLWqzY1XvRBTF\nA2gSURTFEZpEFEVxhCYRRVEcoUkkBMGDh+rXr288tnfvXuv5/fv3N+K//OUvRiyJaCxocmczNrlJ\nRXynTp0yYi5ak7pnsfGNO46xOUv6oPC5sogoDa/i68YiKJuzJCG5vLzciPnc2QQHACtXrjTiyk7u\nlfz85z+3tuEu+WweZBOYJM6yoMsisFQox3UhbGLjLm2SyY3fPzepzUlEa2cURXGELmcUxQPU5jsR\nTSKK4gFqcxLx+V08e5/Ph1//+teBmNf7rFUA4QddSYVwPNGOzVZsxpIGerP2wMOqpe7nrM+w5sNz\nXNkkBdh6DndmlxoZcRMiNlJxB3nJOMbXgIvaJO2JTXuso3DHeMAexs2Dofg8+LUA9uthA5404Y81\nLj53NgpKnwnWVZ599llXftl9Pp9VMBgJsbGxnkg+eieiKB7AC8ngZtEkoigeQJOIoiiO0CQSguCL\nw9/N8zocsCeisY4iXWzWEVi/+OKLL4yYp+wBtibC58GT3AB7Ah5PvON9cAMewPZRsOYj6UbczJin\nu/H6X2oyHW4CnKSjMNy4SNIV+FpzMWDdunWNWHp/+bpyYaM0JZD3y6+HC/SkZtY6AS8y9E5EUTyA\nJhFFURyhSURRFEdoEglBsO+D16EFBQXW87nOg+szpCbLvO5mv0JCQoIRc+MjwB4IxXqG9CaH8yuw\nnsO+EcDWSfjcsrOzrW3YexFOm2DdBbC9JhkZGUYsDa9iT0vPnj2NWGpsHNyoG4A1SIn1DamRE+s1\nwY2uAFnjuv/++42Yh5E9+OCDRixpXgcOHLD+zy1qcxLR2hlFURwRNok89dRTiI6ORrdu3QL/V15e\njqysLMTHxyM7O9v6y6AoSvWoqVm8t4OwSWTixInIz883/m/WrFnIysrCgQMHkJmZac2/UBSletyq\ngd4fffQRkpOTkZqaip49e2Lt2rWOzz1sEunXr5+lF+Tl5SE3NxcAkJubi2XLljk+EUX5MVMTSaRy\noHd+fj727t2LxYsXWw2phwwZgp07d6K4uBj/+7//i6efftrxud+UsHrmzJmA4Sk6OloUpSoJLkrj\nLuw8ZR6wxUlu/MNiHmAXi3EDIRZa2ZwG2OJkuEJAwBZFWRRmQ9e2bdusffBxWNDljuoAkJiYaMTc\n2Kdp06ZGLBnHWPRl0ZT/cAC2wMnFctI14+7u/N7we9G2bVtrH/yZ4PdbElb53FhIbdmypRFLxjJp\ncqBb1MTyJHigN4DAQO9gMTu4KdSFCxcsI9/N4FhY9fl8NTIUWFF+zNTEnUgkA70BYNmyZejSpQuG\nDRuGN954w/G539SdSHR0NE6fPo2YmBiUlZWJpdiVLF26NPDz119/HfK5iuIljh8/LrZEcINI7kSK\niopQVFRU5eOR/jEfNWoURo0ahY0bN2LChAnWHObqclNJJCcnBwsXLsT06dOxcOFCjBo1qsrnPvbY\nY4Gfb+UwIEVxyv3332/4TSTvTE0RSRJJT09Henp6IP79739vPB7JQO9g+vXrh2vXruHbb78V++VE\nStgk8vjjj2P9+vU4e/Ys2rRpg5dffhkzZszAuHHjMH/+fMTFxVnT1IIJVn+5CW/w18aVsDGKMy83\n+gXsAi02Gu3YscOIpUIxbvbLgpR0XNYveF198OBBI5aaLrMWwYVgUvEcFzKG+wBwAx7A/qvF10i6\nY2RzHQvqzz//vLUNN1ViDYSbP0Xy15SL67gZFGAb0lgX4m0kTUQyz7nFrRroffjwYbRv3x4+ny9Q\n3OkkgQARJJGq7h7WrFnj6MCKovyDWzXQ+y9/+Qv++Mc/IioqCg0bNsSSJUucH9fxHhRFccytGuj9\nwgsv4IUXXqiRY1WiSURRPICXHKjVxfUkEqxPsBbBxVmArZtwwx1pwj1/519SUmLEDzzwgBFLajQ3\nUeZGRtJXZZmZmSGfw9qFpE289dZbRjxo0CAjnjRpkrUNN/Vln87OnTuNOC0tzdrHokWLjHj69OlG\nvHr1amsbbsIULPIBwJdffmltE06v4oFYDz/8sLUP1mtYV2ndurW1DRfc8bnx56x3797WPubNm2f9\nn2KjdyKK4gH0TkRRFEdoElEUxRGaRBRFcURtTiKuT8ALtr2z4Nm/f39rm/Xr1xsxd7qSJsBzARZ3\npGKjGE9UA2xTF4vA3GENAHr06GHELHCywUkSksNNq+NiO8AujmNTGxfkSa+XTX0sYEvd31h85H2w\nAArYxXKpqalGzKYw7nwG2IY8fn8l0ZvFdi5K5M8MXzPAvib/+q//6toEPC40jYS0tDRPJB+9E1EU\nD+CFZHCzaBJRFA+gSURRFEdoEglB8HqWm7xwkxvAXmdzsZXUz/Xuu+82Ym4OxM2DuJAMsAvfeN3N\nzXQA+/xZR2FzllRRyYVgbHKTigVZJ+IiN54Ix9cHsDUDNmdJjYy4QVLnzp2NmA1cgH3tudCNtSip\nEI71KP6MSA2UeBu+Bqx3SF3mpckCbqFJRFEUR2gSURTFEbU5iejcGUVRHOH6nUh5eXngZ/ZrSK3n\neFodP4cbAQEINFephLUIbggt6Qw8VY4LxaQ186lTp4yYi83KysqMmL0KgN1QmI/LjwO2FsPNc/i1\nsK4E2N4Rvu5SM2vWo/i4UnEk6zV8DbhRk+TH4ePGxMQYsXSNWGthDYS1F+lOQLpublGb70R0OaMo\nHkCTiKIojtAkoiiKIzSJRAjXJ0jNgNmfwWt19h1Iz2EdJZxXAbAb9xw/ftyIpYZCHTp0MGLWCFh3\nkJoQc4Mk3qc0mImbG7MGxD4KqZExN0zi9b80uImHcfH7KekorGGxTsT6lHSNWGtiDYQHjwH2NeJf\nUvYKSQ2xpebcbqFJRFEUR9TmJKJf8SqKB7hVA73/7//+D8nJyejevTv69OmDXbt2OT53vRNRFA9Q\nE3cilQO916xZg9atWyM9PR05OTlGe4X27dtjw4YNaNy4MfLz8/H000+jsLDQ0XH1TkRRPEBN3IkE\nD/SOiooKDPQOplevXgG9KyMjw2r6fTO4ficSLGKGK6QCbDGShTculAPsTt68XxY8pTdgw4YNRhwf\nH2/E0vS6cMJbSkqKES9fvtx6DpuvuLGPNCWQu9dzd3c+V8koF65Rk9Top2fPnkb87bffGrFkpmMB\nk01fPGmwbdu21j7YPMeNqaS/pB07djRiFnRZbJeKIzdt2mT9n1vUxJ2INNB769atVT5//vz5GD58\nuOPj6nJGUTxAJElk586d1h+MYCId6A0ABQUFePfdd2tkvrAmEUXxAJEkke7du6N79+6B+L333jMe\nj3Sg965duzBlyhTk5+eLbRSqi2oiinKHEDzQ+8qVK1i6dClycnKM53z11Vd49NFHsWjRImvJd7O4\nficS3LyYC6ek5kAs9IwdO9aIt23bZm0TXOQH2EVsH374oRHz2h4A1q1bZ8TcUFgqFuRCuKysLCNm\nU5t0u8lvMk+e+9Of/mRtww2S2GzFBjXpFpibTLM5K/gvXiV8XT/55BMjlv6qZWdnG/Hs2bONeNSo\nUUYsaU9sBOQp9tJfcdZWRowYYcTB82mr2ofUNNotbtVA75dffhnnzp3D1KlTAdzQqIqKipwd1/GZ\nK4rimFs10Pudd97BO++8UyPHqkSTiKJ4gNrsWNUkoigeQJNICIKLtPj7fcnzwb4BLozidTlgN61h\njYDX1FJx2ZAhQ4yYC+PYEwHYhX/8dRn7MyTNYMGCBUacmZlpxJL4xb4JbsrUt29fI5a0GB4SxkV8\n3MgZsP03XIDHRX0A0KdPHyN+6623jHjlypVGzE22pXPh65icnGxtw0WHrItxI2qpKbNU/OgWmkQU\nRXFEbU4iYb/iLS0txaBBg5CUlISuXbvijTfeAHDjG5GsrCzEx8cjOztbHOWgKEpk1FQB3u0gbBKJ\niorC7NmzsWfPHhQWFmLevHnYt28fZs2ahaysLBw4cACZmZmYNWvWrThfRbkjuaOTSExMTKAGpGHD\nhujSpQtOnjyJvLw85ObmAgByc3OxbNkyd89UUe5ganMSqZYmcuzYMRQXFyMjIwNnzpxBdHQ0ACA6\nOtowlQUzffr0wM9cDLR7927r+Vxwx53OpGKyTp06GfGiRYuM+Pe//70RS8YxLtLjpCgVwnXt2tWI\n2bTTunVrI5bESi4ee//994343//9361tuPvZ2rVrjZhNblIHOb7OLOjm5eVZ2/Tu3duIX3nlFSP+\nwx/+YG3DRrenn37aiN98800jZiMZYBvh+PVyR3nANsKxUM4isSTYS5MD3cJLSaG6RJxELly4gDFj\nxmDOnDmWsu3z+apV/KMoiskdn0SuXr2KMWPGYMKECQGbcnR0NE6fPo2YmBiUlZWJf+0A4L/+678C\nPzdp0sT6660oXmX//v3WHYxiEzaJ+P1+TJo0CYmJiZg2bVrg/3NycrBw4UJMnz4dCxcutGogKvnF\nL34R+DlUbwNF8RqdO3c2hpZ//PHHrh3rjr4T2bx5MxYtWoTu3bsHitJmzpyJGTNmYNy4cZg/fz7i\n4uKstXwlwYYk1iLYrAQA33zzjRFzD0jJoJaenm7ErEXwPtmcBQB79+41Ym7Wwo8D9lp99OjRRsyG\np/z8fGsfv/nNb4z4iSeeMOIPPvjA2mbkyJFGvH37diNesWKFEUuGtSlTphgxL1G5AFF6Dr9/PGkQ\nsAsqWXt68sknjXjAgAHWPlibYL1Kshc88sgjIc/16NGjRvzXv/7V2seDDz5o/Z9b3NFJpG/fvuIv\nOwCsWbOmxk9IUX6M3NFJRFEU99EkoiiKIzSJhGDPnj2Bn7nIi5sJAXaxWKtWrYyYdQgAOHz4sBFz\ngRr7USRfARd1cRGf1BWbt2H9JliUA+zmz4DteeCivv79+1vbHDx40IgnTZpkxMXFxUYsTXdjnwh7\nWCQ/DutC/I2c5BXiaXW8DTcP4kbVgF38yJ+bL774wtqGG01zMR1/RlhHA+zm1W6iSURRFEdoElEU\nxRGaRBRFcYQmEUVRHKFJJATBHbTYNCT5T3r16mXE9eqZp1i3bl1rGzaxsbmKp539/e9/t/bBQhsX\naHHxGWALjc2aNTNiFieliXksEvJxeLofYHfJZxGYO7WziArYYuvp06eNWJpmx6YuFjwrCzJDbcPn\nGjwnBbDfK8AWOLnbvdTZjH8puYt8JPNW+DPhJjWVRPLz8zFt2jRUVFRg8uTJRgEscOPzNHHiRBQX\nF+OVV14xHOU3i96JKModQiQDve+77z68+eabNdq6Q4dXKYoHuFUDvZs3b460tDRrJrITNIkoigeo\niSQiDfSWBrPXNK4vZ4IzHpvNpMIwXkNH0hiGNQJeV7OBTerszToJZ+pLly5Z27CBiYsDuciLdRbA\n1h64oRAXrAG28Y1fP6//WUMAbJ3h1KlTRsxGOcDWQFgDYl0FgDULlq8zG+ekpkR8Xfm9kIxx/P6x\nSfHKlStGLE0AkDQst4hEE9m3b5+okVVyu3r6qCaiKB4gkiSSkJBgOJpZ14h0oHdNo8sZRfEANbGc\niWSgd/Dxagq9E1EUD3CrBnqfPn0a6enpOH/+POrUqYM5c+Zg7969jgZ1uZ5EgtfvXIAmNZPhF8Pr\nXfaNALbXoGfPnkbMPhJpDX3kyBEjZj+K1MiX9Qv2SXADadZIAFsD4OPwhDjA1pLKysqMmIvcJD2H\nfSJcKCd9qFhH4GmFPXr0sLZh7YX1KNZ8uIEUYOsqrF9I58rXkd9fnlZ47tw5ax+SJ8ktbtVA75iY\nGOv3xSl6J6IoHkAdq4qiOEKTiKIojtAkEoJg/aF58+bGY5IHhGsreK0urbu5GRCv3bkGQhpvwaYc\nbrLM5w7Yfg1usMPrcEkD4qbLfE0KCgqsbdgXwpoP+0T4mgK2XhPcPAoAsrKyrG1YR+HmR1LDqMGD\nBxsxd0xnLULyayQlJRkxvxf8fgO2z4U9H5999pkRS6asqiYYuEFtTiL6Fa+iKI7Q5YyieIDafCei\nSURRPIAmEUVRHKFJJATjx48P/MyT5/7pn/7Jev7vfve7kM+RRFGemsaGtJSUFCOWptmxEMdvKheK\nSefC4h2bwlgABWzhmE1QLKICwIQJE4z4T3/6kxH369fPiN9++21rH3xd//znPxsxT9EDgD59+hjx\nww8/bMTSZEEWTtlwx0V7LGgDtomPixalbu88OZANWNwhf9OmTdY++DmTJ0+2nlNTaBJRFMURmkQU\nRXGEJhFFURyhSSQEwToBX6hVq1ZZzx8xYoQRs6GLJ7VJz2FzGRu6JOPY8ePHjZinxE+bNs3ahs+/\nsLDQiLkZME+3A4C8vDwjZjMdN9oFbENaYmKiEXNR29SpU619sFGMmztLxjjWDfi9kibtsfGNXx/r\nU9Jxt27dasT8eqVGRmwmmz17thFzMaj0S8yfATfRJKIoiiM0iSiK4ghNIoqiOEKTSAiCm9I88MAD\nxmOSF4Eb3fDadf/+/dY2/AZwY5stW7YYsdSAl30gGRkZRsx6B2A32GEfBRe+HT582NoHawTbt283\nYqm4jDUgfr3vv/++EUtaDA/N4oI7Lh4EbvTwDIb1DulcuQEUN4RmL400JIyHnHFTHUmLyc7ONuKv\nv/7aiNm/IjWZrs2/2LcSvRNRFA9QmxNWyCrey5cvIyMjAykpKUhMTMSLL74I4MZfsaysLMTHxyM7\nO1tU1BVFiZyaaNR8uwiZROrXr4+CggKUlJRg165dKCgowKZNmzBr1ixkZWXhwIEDyMzMxKxZs27V\n+SrKHckdm0SAf3gOrly5goqKCvz0pz9FXl4ecnNzAQC5ubk1OtdTUX6M1FQSyc/PR0JCAjp16oTX\nXntNfM6zzz6LTp06ITk52fIL3QxhNZHr16+jR48eOHz4MKZOnYqkpCScOXMmUBQVHR2NM2fOVLl9\nsFGIDU1SIVzfvn2NmIvWpKUTX1Duws6T2iTDGgu2LPjt3r3b2oY7ebE4x7E0VY7hzl7S9PoNGzYY\nMRcc8qS6Vq1aWfvgwj4ufOvfv7+1DXeNZ4OeVDzH7xeLzyxwSsI5vx5+f6XryteNxWiOuWM+YE9S\ndJOauLOIZKD3ihUrcOjQIRw8eBBbt27F1KlTxS8NqkPYO5E6deqgpKQEJ06cwIYNG6x2fT6f77aN\n71OUO4VbNdA7eBWRkZGB7777LuRNQCRE/O1M48aNMWLECOzYsQPR0dE4ffo0YmJiUFZWJpbnV7J+\n/frAz3Xr1rX6ZSqKVykrKxO/+nWDmrgTkQZ6c8mA9JwTJ05Y7RaqQ8gkcvbsWdSrVw9NmjTBpUuX\nsHr1arz00kvIycnBwoULMX36dCxcuDBkQ9vgXh+aQJTaRMuWLY2lb0lJiWvHiiSJHDlyRPTvVBLp\nioCP5XQlETKJlJWVITc3F9evX8f169cxYcIEZGZmIjU1FePGjcP8+fMRFxdnmZuCCV7PckdtyVhU\np465wmKjlPSXgU1fXKDFJi9pIhyv5/nNatSokbUNT7jnBkM8uU2aPMbFYzExMUZ8+fJla5tu3boZ\nMZu8gtfAgHzuXKTHZrtIphOyYU36ReD3hhsI8XshTZ1j0x5rWtztH7A1Ef5c8Wvh7veArPG4RSRJ\npF27dmjXrl0gXrt2rfF4JAO9+TknTpywTITVJWQS6datm9itqmnTplizZo2jAyuK8g9qYjkTPNC7\nVatWWLp0KRYvXmw8JycnB3PnzsX48eNRWFiIJk2aOFrKAOpYVRRPcKsGeg8fPhwrVqxAx44d0aBB\nAyxYsMD5cR3vQVEUzxBuoDcAzJ07t0aP6XoSCW4AxGtbLqwCwhd1Set73i9Pouf18bFjx6x98HN4\nbS59DcbNgXj9z7H0eqOiokIeR9qG4TUtF71JfgfWXlhck8Q29nTce++9RsyaEGBfa96GryF7XgD7\n/eXrKk3N49fMGhB/o8ifO8B+b9zESw7U6qJ3IoriATSJKIriCE0iiqI4QpNICLZt2xb4mYcdScaZ\n9u3bGzHXNPD0egBo27atEfNalgcmsc8CsD0cwa4+QG4GzLUzHHOzoOHDh1v74GvA3pkDBw5Y23DD\nZK6DYW2C9Q/A1gzYE8EDvwC7fogbSEl6Bl9r9muwViF5Pvga8XFYIwFsnSj4cwjYmo/0NadUT+MW\nmkQURXGEJhFFURyhSURRFEdoElEUxRGaREIQFxcX+JmFR0ngZOGNzUjc/R2wxbhevXoZMYuxXDgG\n2CIoi3dstAJsUxsbmFjwkwrwWABkkbSoqMjahoVTFpK5kZMkPLIYzYVx0oeaxcgmTZoYsVTYyK+H\nCyj5MyF1bk9LSzNiLirj9w6wiyP79etnxPyZkDrVcxMtN9EkoiiKI2pzEgnb2UxRFCUUeieiKB6g\nNt+JuJ5EgifSc28DqcEQ6waZmZlGzJoJYBfPhTOOnT9/3toHaxPcDEha73MxIBu4uDkQT7sD7OY/\nH374oRFLTZaDdSbAbtLDxjipITav93lK4MqVK61tuKnU888/b8SvvvqqtQ1fE9ZVBg4caMTSLxPr\nYPze8HQ7AFi6dKkRP/PMM0bMxj/pMyF91txCk4iiKI7QJKIoiiM0iSiK4ghNIiEILtriQjD2MwDA\n4MGDjZhGzhwrAAAcxUlEQVQ9EFIDYS6w42FHU6ZMMWIeTAUAhw4dMuIdO3YYcXCD3Ep4eBNPE2MP\nxOjRo619cJNrXqtLx+WJg9yUh4d1ScOduHM5Nz+SCg5Z32AtQipYe/DBB42YNR7Wc/7whz9Y+2Bf\nyM9+9jMjlrwlXNi3aNEiI+bJAzw0DbC9Jm6iSURRFEdoElEUxRG1OYmo2UxRPEBNDfSuivLycmRl\nZSE+Ph7Z2dmiLAAATz31FKKjo8WSlKrQJKIoHsDtJDJr1ixkZWXhwIEDyMzMxKxZs8TnTZw4Efn5\n+dXat+vLmWDTz9ChQ43HpAljbBzj6XUsmgJAhw4djJi7VHERmFSAxwVYnKnZ9AbYpjY2k3FRG4u1\ngF1Mx93QpdGNPNGeXx8XpLEwCdjXlY1x999/v7UNXzc2qEnHycjIMGIuqNy4caMRc2EgYL8+FsH5\n3AHg0UcfNeIvv/wy5LlytzjANim6idvLmby8vMBc7NzcXAwcOFBMJP369ROnIYRC70QU5UfAmTNn\nAn9co6OjxREoN4sKq4riASK5Ezlx4oRVehBMVlaWWEryyiuvGLHP53M8xDsYTSKK4gEiSSKtW7c2\nary4+fTq1aur3DY6OhqnT59GTEwMysrKrN43TnA9iQTfNnETH0kjGDNmjBHzuluagBc8ZQ+wJ75H\n0rWb14HccIe1CsAuBEtISDBiXodLk+h69uxpxNztXlqrcxMmNnCxJiIV/rGew1qFpG9s3brViE+d\nOmXE/fv3t7Zhwxa/Po6lBkPc7Z2HzLMmBtgmPf4Lzn+J2RgIyFPx3MJtTSQnJwcLFy7E9OnTsXDh\nQowaNarG9q2aiKJ4ALe/nZkxYwZWr16N+Ph4rF27FjNmzABw4w/BiBEjAs97/PHH0bt3bxw4cABt\n2rSJaOC3LmcUxQO4fSfStGlTrFmzxvr/Vq1a4ZNPPgnE3K4jEjSJKIoHqM2OVZ/fxbP3+Xx4/fXX\nAzF7Mfj7f8C+mFwYJq1deT3PXhOeKi+9ZN4vr5m5aRFgNyri9T9rM1LjG9Zerl27FnKfAFC3bl0j\n5iJF/vpOaq7DBWqs+bB/BbCvUdOmTUM+DtgNr/k9Z9+PNK2Prwl7ePgzAtjnz8flzwy/V4D9ef3F\nL37hyi+7z+fD008/Xe3t3nrrLU8kH70TURQP4IVkcLNEJKxWVFQgNTUVI0eOBBC5D19RlMhwW1h1\nk4iSyJw5c5CYmBi4xY/Uh68oSmTU5iQSdjlz4sQJrFixAr/85S/x29/+FkDkPnzA1A3Y4yE1P+b1\nPmsTUuNiXlezJsDrf9YQAKBBgwZGLK2Rw50rr/+51oTX9oCt1/BxJWfhxYsXjZj9J6yBSJ4P9s7w\n3aRU18Svj30yUnMgvta8X379Um0U6xdslJL0G75G4T5X0t20NNDKLbyUFKpL2DuR5557Dq+//rrx\nwXTTh68oP0Zq851IyCSyfPlytGjRAqmpqVWedE378BVFqV2EXM5s2bIFeXl5WLFiBS5fvozz589j\nwoQJ1fLhr127NvBzUlKS2N9UUbzIsWPHrD66buGlO4vqEjKJvPrqq4GBROvXr8dvfvMbvPfee3jh\nhRci9uEHN16W6l4UxavExcUZg8I2bNjg2rHu2CTCVC5bZsyYgXHjxmH+/PmIi4uzOpYHE9z8ho1V\n3BgHsJv/cIMh6Vi8XxbrevXqZcRSQdoHH3xgxCkpKUbMjXAAW5xkoZgFTUng5K7q27dvN2Kp2zsX\nvrEAyGKmVEjGTYf4mknNgc6dO2fEbFCrFNuDYeMfvx4unuPO/YD9evncJWGVO9zzRD8u/JOMcizO\nusmPIokMGDAAAwYMAFC1D19RlJvjR5FEFEVxD00iiqI4QpNICIKNQtyAh3UHAHjvvfeM+D//8z+N\nWDIFsWDL6+wlS5YYMa//Abvwi/cpTWZ78sknjZiNUpE07eFmSGzGktrh8Vo9MTHRiNkEtnfvXmsf\nfO7c6IgbSAG2vrFu3Toj/uyzz6xtWON65513jPj555834mAhsxJ+L9i0J/W84CbLjz32mBF/+umn\nRsyT+gBbA3ITTSKKojhCk4iiKI7QJKIoiiM0iYQgeBwff7//q1/9yno+F61x82PJ45GXl2fErG9w\n85zevXtb+9i9e7cRs/dA8gywm/HgwYNGzBpCenq6tY/NmzcbcZ8+fYxYajLNDaC5QI3PnQdIAcB/\n//d/hzw3qWEUF+CxF2Pu3LnWNv/8z/9sxKxpsU9GasK0b98+Ix42bJgRP/TQQ9Y2rCXxILHCwkIj\n7tu3r7WPZcuWWf/nFrU5iWijZkVRHKFJRFE8gBcGepeWlmLQoEFISkpC165d8cYbb0S0b00iiuIB\nvDDQOyoqCrNnz8aePXtQWFiIefPmWUtJCU0iiuIB3E4ieXl5yM3NBXCjkZik98TExAS8Ww0bNkSX\nLl0sr5OE68JqsJmKTVAPPPCA9XwupuOCNC7IA2zDEk9Iq6z5qUQqSGOxksVJNmcBtgmKC+GuXr1q\nxGy0AuzXy4Vg0kQ4nvj31VdfhXxcMqxNmjTJiF9++eWQjwM3utwFw8aw4cOHW9twr5nJkycbMZva\nNm7caO2DBWoudGSBF7BFYDaOsQmOnw/Y1yQnJ8d6Tk3htrBa3UZix44dQ3FxsSjKM/oVr6J4gEiS\nyDfffCOOYq2kpgZ6X7hwAWPHjsWcOXMiahOqSURRPEAkSaRZs2Zo1qxZIOa755oY6H316lWMGTMG\nP//5zyOe16uaiKJ4ALc1kcqB3gCqbCTm9/sxadIkJCYmYtq0aRHv2/U7kZ07dwZ+/vrrr43Hgrue\nVcI6AneTkqadcbHVo48+asTc/ZyNR9Jz+Dy4Gzxg6ybc6Ia7n2/bts3aBxun9uzZY8RSx3Eu0uMm\nRI0bNzZiab3PXfOTkpLCHpfNddyUSDLTFRcXG/HEiRONmA16kgbUvXt3I2bjn2TIYx2FXy83iOJi\nQkBuCOUWbmsiVTUSO3XqFKZMmYJPPvkEmzdvxqJFi9C9e3ekpqYCAGbOnCma+YLR5YyieAAvDPTu\n27ev+Ec6HJpEFMUD1GbbuyYRRfEAmkRCEKwCt2zZ0nhM+q46Pj7eiLnRDzfxAWz/Ca+H2TciNelh\nXYHX3dL35dyYmIvWWFeRGu5wcSAXoO3fv9/ahnUDHsPB2hPrPYB9zfi6S42b2CrN7w1rJIDt6ykp\nKTFi9slI2hP7XFgDkb5pYK8Mv5987tJ0QqkYULHROxFF8QB6J6IoiiM0iSiK4ghNIoqiOEKTSAiC\ni6NYRJMqBI8cOWLEbJRiAxdgC2AspIaboAbYxjGuGQg2zVXCpi4+D96n1C2MjW+8jfThYqNU3bp1\njZg7nUnX+eOPPw55btJUuU6dOhkxFzJKBjUWdblzHT/OXekAu4iPhVTJ28AFd/zZ42JC6TPBIrCb\naBJRFMURmkQURXGEJhFFURyhSSRCuAGNtIbmfgjc+IeL7QBYPRa4QI3X2dIEeD4X1mKkRkb33HOP\nEfP6njuZs8EJsNfzrKtIRio+FzZS8QdS0mJ4H6w7RKI9sUGPzXXSftmQxttIxjg+VzaksWEPsHUi\nNinyeye93lC9O2oaTSKKojhCk4iiKI6ozUlEmxIpiuII1+9EgtemvP7lQjHAnnDPBVyHDx+2tmFd\ngYvHuLkMF5sB9pR41iK4eBAIXwjG3gSpoIv1Gm5KxPsAbI2DryN7MSTvRTgi0Td4GiE31QZsbYk1\nLi7qk14v61X8eiQdhTURPg9uUiR9FqWCQreozXciupxRFA9wxyeRuLg43Hvvvahbty6ioqJQVFSE\n8vJyPPbYYzh+/Hig3Zr0V0RRlPDU5iQSkSbi8/mwbt06FBcXo6ioCEBkE7UURYkMtxs1u0nEyxk+\n6by8vEBTntzcXAwcOFBMJMHNb3idnZWVZT1/1apVRsyeDm78C9g1HTzMKD8/34i59gSw1+r8nD//\n+c/WNkOHDjVirtdgpKZElUm5EtYIuA4GsL0yvN9wTZkAWxdizwcPxALsa7R582brOQzXBnFj5v79\n+xsxe14kuCkT60iArd/wcRjJJ8LXxE28lBSqS8R3IkOGDEFaWhrefvttANWfqKUoStV4YaD35cuX\nkZGRgZSUFCQmJuLFF1+MaN8RJZHNmzejuLgYK1euxLx586xRh+EmaimKEhovDPSuX78+CgoKUFJS\ngl27dqGgoACbNm0Ku++IljOVX282b94co0ePRlFRUcQTtYLnecTGxool14riRQ4dOiRaCtzA7eVM\npPJDZTnAlStXUFFRIZYUMGGTyA8//ICKigo0atQIFy9exKpVq/DSSy8FJmpNnz69yolaADBw4MDA\nz5L3QFG8SseOHY0m2KzX1SReGeh9/fp19OjRA4cPH8bUqVMt35ZE2CRy5swZjB49GsCNjthPPPEE\nsrOzkZaWJk7UYoKL4Vjceuutt6zns+DHmZC7lEv/xwInNw+SGu6wyMsdxiUxlv9KccznIa0xefLa\ns88+a8RSY5xly5YZ8RNPPGHEu3btMmJJ0F2wYIERv/POO0YsiYrceZ5nwbL5DLCNfnxd2SgmGbxY\nfGbTnlSUyUV7LMjzdZ00aZK1j7lz51r/5xaRJJHz58/j/PnzVT5eEwO969Spg5KSEnz//fcYOnQo\n1q1bZ9wISIRNIu3atRM/yFVN1FIUpfpEkkQaNWpkfEPGHetqYqB3JY0bN8aIESOwffv2sElEa2cU\n5UdAJAO9z549G/jW5tKlS1i9enVgJm8oNIkoigdw+9uZGTNmYPXq1YiPj8fatWsxY8YMADfuZkaM\nGBH4efDgwUhJSUFGRgZGjhyJzMzMsPt2vXbmww8/DPzM67X27dtbz+e1OJvNeNobACtbcjEVryO5\naRFgr5H5OdIakrM5nys3wvnrX/9q7YOLywoLC41YWgMPGjTIiPlrOH6ci9EAoG3btkbMpjDpXFl7\n2bZtmxG3bt3a2oaNb9wM6IMPPjBinlwH2GYyfo5UbsGfCdaAuLBTKsBLSEgwYjcbN3thoHf37t1F\nY2I4tABPUTxAbXasahJRFA+gSURRFEdoEglB7969Az9zkVNSUpL1fB6qVCn6VCKt2dhbwOvdDRs2\nGLHUgJc9DjwlvrS01NqGG0+zF4ELA6VCMfafsMdDaobEDXVY/OLBWx999JG1j0ceecSI2XvBjZ0A\n4MCBA0bMzYHYAwIAR48eNWLWxR544AEjvnTpkrUP9rnwe8NeG8AuZOzXr58R82dA8g5lZ2cb8ZIl\nS6zn1BSaRBRFcYQmEUVRHKFJRFEUR2gSURTFEZpEQtCnT5/AzyxESpWE3bp1M2IW69i8BNiCLYuE\nbArjTmeAXejHHbYyMjKsbViw5dfH+2BBFLBFYO6gxqYwwBYFjxw5YsRseuNCOQAYP358yG2kDvGf\nffaZEffq1cuIudARsKf+8ethU5jUv4JFb+7M37dvX2sb/kyw4a5Lly5GzGKtdK5uUpuTiNreFUVx\nhC5nFMUD1OY7EU0iiuIBanMS8fldPHufz4d58+YZcTDctRuwp9lx/OCDD1rbsAGNNQLu/i519ubj\nsClK0mL49fC6mrWZZs2aWfvgKXqsm/A0O8DWUbhZEL+lUgEeN+3hgkNpqhzrCGxIkyb83XfffUbM\n15V7YkgGRD43viZ8DMDWp7jAjq+RVJTJ1+BXv/qVK7/sPp/P+oxGwv79+z2RfPRORFE8gBeSwc2i\nSURRPIAmEUVRHKFJJATBa29ed/KkdsBu0sM6AzfCAew1MX+/z8fhdThgF8uF0zsA4K677jJiLnzj\n7vZS4R+vu0+cOGHE3OgYsIvLwo3hiKS5dYMGDYxYmkTHWhN/8KXGTayb8H65kZGkxRw6dMiIWROR\nBjFxUabUADrccVk3chNNIoqiOEKTiKIojtAkoiiKIzSJhCB4Hc3+DGlAEjfHZR2BdQjArs/g5/Ca\nmbULwG6Gw3U9kseD33j2fPBxWO8B7GbO3GRZ+nDxUCyu6+H1vXSduekwe0ki0TfuvvtuI5Yab3NT\nIt6Ga3ak5kD83rB+I7037FlhPYeHpEneIUnDcgu3k0h5eTkee+wxHD9+PDBsTmpwDdx4D9LS0hAb\nG2s1CZPQ2hlF+REQyUDvSubMmYPExMQqp+QxmkQUxQO4PXcmLy8Pubm5AG4M9OZRrJWcOHECK1as\nwOTJkyM+hiYRRfEAbieRSAd6P/fcc3j99dfFr7yrQoVVRfEAkSSFy5cvi/VJlTgd6L18+XK0aNEC\nqampYvPrqnA9iQSLi2y+kjqKBw8sBoCVK1casdSAhrNqhw4djFgyIzFsJmNTVFpamrUNi4ZcoHb4\n8GEjlgROFl/5mkhCIxunuBM9C408dQ4ADh48aMQ8tJnfBwDYuXOnEXOxnDQBjwvfuPkT/1JIE/+4\nOJKF9J49e1rbsKGQGxtxESN3sgdsAdtNIkkiP/nJTwwBmIsMnQ703rJlC/Ly8rBixQpcvnwZ58+f\nx5NPPok//vGPIc9LlzOK4gHcXs5EMtD71VdfRWlpKY4ePYolS5Zg8ODBYRMIoElEUTyBFwZ6M5F+\nO6OaiKJ4AC8M9A5mwIABGDBgQET7dj2JBBd6se4g6QxsvmJDjDQ1/vjx40bMyjLvU2rSwyY3NjRx\nERhgF/6xJsBaDGsk0j54IhxrQoDdRJmPw5qQtLZPTk42Yv6rI53rsGHDjLigoMCIJW2CTV5s4OLG\nVCkpKdY+uICSXw9rJICtafF7zq+vY8eO1j4kfcYt1LGqKIojanMSiUgT+e677zB27Fh06dIFiYmJ\n2Lp1K8rLy5GVlYX4+HhkZ2dH9A2IoigybmsibhJREvm3f/s3DB8+HPv27cOuXbuQkJBQLRutoiih\nqc1JJOxy5vvvv8fGjRsDXw/Vq1cPjRs3Rl5eHtavXw/gho124MCBYiIJXovyhHtpHbpq1SojHjNm\njBFv3brV2qZNmzZVHhOwvQlSgyFeq/M+9+zZY23zs5/9zIi3b99uxNzoSNoHF8+x+CUVHHbv3t2I\nWZvgYjKpCRN7TVib2LJli7VN8CAyAIiLizNiLkAE7GFkO3bsMGL2a0hOynBFmNI2XOzIRXzsx5Ea\nEI0cOdKIZ86caT1HieBO5OjRo2jevDkmTpyIHj16YMqUKbh48WLENlpFUcJzR9+JXLt2DZ9//jnm\nzp2L9PR0TJs2zbrjqMpGC5ijF7t16yaWiyuKF/n888+tcSRu4aWkUF3CJpHY2FjExsYiPT0dADB2\n7FjMnDkTMTExYW20gDmvtarnKIoX6dGjh2GXf/fdd107Vm1OImGXMzExMWjTpk2gtmDNmjVISkrC\nyJEjw9poFUWJjDt6OQMAb775Jp544glcuXIFHTp0wIIFC1BRUYFx48Zh/vz5gU5JEsFmKhYrJTMP\ni6BsNJImlfHUOC5I422k47KAy13lpeIy3g8bmHgyXWJiorWPcKKoNHmPi8Uq7xIrYVFRer1cPPjw\nww8bsWRy46I9rhiVlrQsnHIRJnedl7rOsbnsq6++MmIuHpTOjU2KvE+eEADIBkO38FJSqC4RJZHk\n5GRxVINko1UUpfrc8UlEURR30SSiKIojNImEIDY2NvAzm5M+/fRT6/m8ZuZ1qWSv5212795txHxc\nqTtUuE7mUnMgXlfz+l7SMxguFuTu58HXr6ptuACNi8/YaAXYDW240FGaqseaBzdZkrbh18MNk1hr\n4vcSsJs9sW4iNV1inYhNimyc4+ZJwJ3V7d1N9E5EUTyAJhFFURyhSURRFEdoEglBsIbB3gRJ3+CL\nyVqEpBFw4RdrHuwrYI0EsHUF1gxOnjxpbcPNgTjm1yJ9UPi4/fr1M+KqppQFw2t39ppIWgUX5S1f\nvtyIJb9GuGZI3BwKsN8v9uzweQQ7nCvZt2+fEWdnZ4d8HLC1FS7K5EZV7GcBZC1JsdE7EUXxAHon\noiiKI7wyizcuLg733nsv6tati6ioKBQVFYXdt3Z7VxQP4HbtTKRNxHw+H9atW4fi4uKIEggA+Pwu\npkCfz2fUhrAvRGq6zL4Qbp7DTW4AYNOmTUbcrl07I+Z1d0lJibUPXovzGpm9J9JzuNkR18pIA5LY\nj8L1KaxDAPZ142bPXC0t+XG4sRF3/JYaRnHjnqysrLDH+eKLL4yYh0h17tzZiKU6H76ufE0KCwut\nbVjD4togbmzEOhJgN5l+4YUXXLlj8Pl8Vo1YJFy7di3i80lISMD69esDQ6wGDhyIL7/80npeu3bt\nsH37duszFYpbcici/dJ6FRbcvMytLBBzSnl5+e0+hYjhxHcr8MosXp/PhyFDhiAtLQ1vv/12RPu+\nJZpISUmJOArAi1SuGWsDhw4dEu8YvEh5eblVoe1V9uzZg65du97SY9bEHY7TWbwAsHnzZrRs2RLf\nfPMNsrKykJCQYH1jyKiwqigeIJIkEu45TmfxAv/og9y8eXOMHj0aRUVFtz+JNGzYEHfddRcaNmxo\n9f2Q1OFw/SWkehTuU8IXiLMu15YA/+hJ0qhRI7Rq1cr6qyn1m+DnsKeFH+dG1YDtm+BeINLatPIa\nNGjQAC1atEDjxo2Nx1lHknqh8LmxjsTvFWC/N5Ecp3Kb8vJydOzY0dovn4ekTfB15c+A5B3i/fB1\nZA0iWCOpX78+mjRpInpH3MLtb2cqZ/FOnz69yiZiP/zwAyoqKtCoUSNcvHgRq1atwksvvRR+534X\nAaD/9N8d9c9rvyeR8u233/ozMzP9nTp18mdlZfnPnTvn9/v9/pMnT/qHDx/u9/v9/sOHD/uTk5P9\nycnJ/qSkJP+rr74a0b5d/XZGUZQ7H/WJKIriCE0iiqI4wtUkkp+fj4SEBHTq1Amvvfaam4eqNk89\n9RSio6MN85pX5wuXlpZi0KBBSEpKQteuXfHGG28A8Ob5Xr58GRkZGUhJSUFiYiJefPFFAN4810oq\nKiqQmpoamHjn5XP1Iq4lkYqKCjzzzDPIz8/H3r17sXjxYrHa8nYxceJE5OfnG//n1fnCUVFRmD17\nNvbs2YPCwkLMmzcP+/bt8+T51q9fHwUFBSgpKcGuXbtQUFCATZs2efJcK5kzZw4SExMD3+J5+Vw9\nSXWV5EjZsmWLf+jQoYF45syZ/pkzZ7p1uJvi6NGj/q5duwbizp07+0+fPu33+/3+srIyf+fOnW/X\nqYXkkUce8a9evdrz53vx4kV/Wlqa/4svvvDsuZaWlvozMzP9a9eu9T/88MN+v7/2fA68gmt3IidP\nnjT8G7GxsWJPDi9RG+YLHzt2DMXFxcjIyPDs+V6/fh0pKSmIjo4OLMO8eq7PPfccXn/9daNvrVfP\n1au4lkSqstXWFkJZg28XFy5cwJgxYzBnzhw0atTIeMxL51unTh2UlJTgxIkT2LBhgzWgyyvnunz5\ncrRo0QKpqalVmr28cq5exrUk0rp1a2MSXWlpqegs9BKV1mAAIa3Bt4OrV69izJgxmDBhQsBt6OXz\nBYDGjRtjxIgR2LFjhyfPdcuWLcjLy0O7du3w+OOPY+3atZgwYYInz9XLuJZE0tLScPDgQRw7dgxX\nrlzB0qVLkZOT49bhaoRKazDgrfnCfr8fkyZNQmJiIqZNmxb4fy+e79mzZwPfZly6dAmrV69Gamqq\nJ8/11VdfRWlpKY4ePYolS5Zg8ODBeO+99zx5rp7GTcFlxYoV/vj4eH+HDh0ittDeKsaPH+9v2bKl\nPyoqyh8bG+t/9913q7QG3242btzo9/l8/uTkZH9KSoo/JSXFv3LlSk+e765du/ypqan+5ORkf7du\n3fy//vWv/X5/1bZrr7Bu3Tr/yJEj/X6/98/Va6jtXVEUR6hjVVEUR2gSURTFEZpEFEVxhCYRRVEc\noUlEURRHaBJRFMURmkQURXGEJhFFURzx/xa7N1IKtk2CAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x110ef2550>"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Train set accuracy: 0.998035363458\n",
        "Test set accuracy: 0.97037037037\n",
        "you can also regularize a neural net, just change the loss function appropriately\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Bonus:_ With more weights, this neural net is even more prone to overfitting. So you could regularize it."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "_Bonus:_ You can also add another hidden layer."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "4. Graduate to MNIST and Theano"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Intro to Theano"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Theano works by compiling symbolic expressions into native code (on CPUs or GPUs). There is a nice \"getting started\" [Theano tutorial](http://nbviewer.ipython.org/github/craffel/theano-tutorial/blob/master/Theano%20Tutorial.ipynb) and a more thorough tutorial here http://deeplearning.net/software/theano/tutorial/ Everything you should need for the moment is here http://deeplearning.net/software/theano/library/tensor/basic.html\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from theano import tensor as T\n",
      "A = T.fmatrix()\n",
      "B = T.fmatrix()\n",
      "C = T.dot(A, B)\n",
      "C"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "Using gpu device 0: GeForce GT 650M\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 118,
       "text": [
        "dot.0"
       ]
      }
     ],
     "prompt_number": 118
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "A = np.random.random((10,10))\n",
      "B = np.random.random((10,10))\n",
      "C = T.dot(A, B)\n",
      "C.eval()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 123,
       "text": [
        "array([[ 2.49987768,  3.46830327,  3.07533679,  3.28263693,  3.52006918,\n",
        "         2.81629526,  3.50309921,  2.48141154,  4.19651885,  3.27780272],\n",
        "       [ 1.70199903,  2.00788938,  1.74255066,  1.61437016,  1.83195921,\n",
        "         1.17366047,  1.54721088,  1.0543538 ,  1.85787448,  2.13366925],\n",
        "       [ 2.57912841,  2.53388569,  2.52165203,  3.13951882,  2.59604248,\n",
        "         2.49420427,  2.86811637,  1.95585705,  3.64772433,  2.71894285],\n",
        "       [ 1.65627613,  1.69303027,  1.29031643,  1.75713726,  1.45599642,\n",
        "         1.6585249 ,  1.23043086,  1.07044738,  1.84521083,  1.72626196],\n",
        "       [ 1.61517609,  2.49009226,  2.00571427,  2.26128717,  2.07083152,\n",
        "         1.6539112 ,  1.97799148,  1.57657089,  2.51517353,  2.21145721],\n",
        "       [ 1.65610118,  1.61789874,  1.7242132 ,  1.90119529,  1.96550927,\n",
        "         1.46273452,  1.55532116,  0.9811569 ,  2.17289324,  1.58394601],\n",
        "       [ 1.94265607,  3.00453935,  1.91916286,  2.93620479,  2.81513856,\n",
        "         2.02763359,  2.90217799,  2.03910296,  2.92258548,  3.04995599],\n",
        "       [ 1.32328403,  2.54435891,  1.09419405,  1.52412963,  1.90384414,\n",
        "         1.41099655,  1.27805053,  1.15345173,  1.67611008,  2.30179732],\n",
        "       [ 1.74330342,  3.39876145,  2.35130411,  2.57335844,  2.4635886 ,\n",
        "         2.55897828,  2.49926516,  2.11455371,  3.71868673,  2.75911756],\n",
        "       [ 2.13621873,  2.85752605,  2.03129205,  2.87908882,  2.70363845,\n",
        "         2.1858482 ,  2.81973067,  2.07834169,  2.94183379,  2.96044698]])"
       ]
      }
     ],
     "prompt_number": 123
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from theano import shared\n",
      "A = shared(A)\n",
      "B = shared(B)\n",
      "A"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 124,
       "text": [
        "<TensorType(float64, matrix)>"
       ]
      }
     ],
     "prompt_number": 124
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "C = T.dot(A, B)\n",
      "C.eval()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 125,
       "text": [
        "array([[ 2.49987768,  3.46830327,  3.07533679,  3.28263693,  3.52006918,\n",
        "         2.81629526,  3.50309921,  2.48141154,  4.19651885,  3.27780272],\n",
        "       [ 1.70199903,  2.00788938,  1.74255066,  1.61437016,  1.83195921,\n",
        "         1.17366047,  1.54721088,  1.0543538 ,  1.85787448,  2.13366925],\n",
        "       [ 2.57912841,  2.53388569,  2.52165203,  3.13951882,  2.59604248,\n",
        "         2.49420427,  2.86811637,  1.95585705,  3.64772433,  2.71894285],\n",
        "       [ 1.65627613,  1.69303027,  1.29031643,  1.75713726,  1.45599642,\n",
        "         1.6585249 ,  1.23043086,  1.07044738,  1.84521083,  1.72626196],\n",
        "       [ 1.61517609,  2.49009226,  2.00571427,  2.26128717,  2.07083152,\n",
        "         1.6539112 ,  1.97799148,  1.57657089,  2.51517353,  2.21145721],\n",
        "       [ 1.65610118,  1.61789874,  1.7242132 ,  1.90119529,  1.96550927,\n",
        "         1.46273452,  1.55532116,  0.9811569 ,  2.17289324,  1.58394601],\n",
        "       [ 1.94265607,  3.00453935,  1.91916286,  2.93620479,  2.81513856,\n",
        "         2.02763359,  2.90217799,  2.03910296,  2.92258548,  3.04995599],\n",
        "       [ 1.32328403,  2.54435891,  1.09419405,  1.52412963,  1.90384414,\n",
        "         1.41099655,  1.27805053,  1.15345173,  1.67611008,  2.30179732],\n",
        "       [ 1.74330342,  3.39876145,  2.35130411,  2.57335844,  2.4635886 ,\n",
        "         2.55897828,  2.49926516,  2.11455371,  3.71868673,  2.75911756],\n",
        "       [ 2.13621873,  2.85752605,  2.03129205,  2.87908882,  2.70363845,\n",
        "         2.1858482 ,  2.81973067,  2.07834169,  2.94183379,  2.96044698]])"
       ]
      }
     ],
     "prompt_number": 125
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you want to know more about \"shared-correctness\", you should read: http://deeplearning.net/software/theano/tutorial/aliasing.html"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "MNIST deep learning in one file"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Use `python dnn.py` from this repo https://github.com/SnippyHolloW/DL4H"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "5. LUCID words classification"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Use `python lucid.py` from this repo https://github.com/SnippyHolloW/DL4H"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "6. Some of the \"bonus\" questions solutions"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# solution for the first bonus question\n",
      "alpha = 0.001    \n",
      "    def loss_l2(self, W):\n",
      "        tmp_W = W.reshape((self.x.shape[1], self.n_classes))\n",
      "        return self.loss(W) + alpha*np.sum([np.linalg.norm(tmp)**2 for tmp in tmp_W])\n",
      "    \n",
      "    def loss_derivative_l2(self, W):\n",
      "        return self.loss_derivative(W) + 2*alpha*W    \n",
      "    \n",
      "    # in train:\n",
      "        tmp = fmin_l_bfgs_b(self.loss_l2, self.W.reshape((self.W.shape[0]*self.W.shape[1],)), fprime=self.loss_derivative_l2)\n",
      "        \n",
      "    \n",
      "# solution for the second bonus question\n",
      "    def loss_hinge_l2(self, W):\n",
      "        l = 0.\n",
      "        tmp_W = W.reshape((self.x.shape[1], self.n_classes))\n",
      "        for i, xi in enumerate(self.x):\n",
      "            tmp_max = -1.\n",
      "            good = np.dot(tmp_W[:,self.y[i]], xi)\n",
      "            for j in xrange(self.n_classes):\n",
      "                if j == self.y[i]:\n",
      "                    continue\n",
      "                delta = good - np.dot(tmp_W[:,j], xi)\n",
      "                if delta > tmp_max:\n",
      "                    tmp_max = delta\n",
      "            l += max(0, 1 + tmp_max)\n",
      "        return - l / self.x.shape[0] + alpha*np.sum([np.linalg.norm(tmp)**2 for tmp in tmp_W])\n",
      "\n",
      "    def loss_derivative_hinge_l2(self, W):\n",
      "        d = np.zeros((self.x.shape[1], self.n_classes)) # that's the Jacobian\n",
      "        tmp_W = W.reshape((self.x.shape[1], self.n_classes))\n",
      "        for i, xi in enumerate(self.x):\n",
      "            for j in xrange(self.n_classes):\n",
      "                if j == self.y[i]:\n",
      "                    pass\n",
      "                else:\n",
      "                    d[:,j] += -np.dot(tmp_W[:,j], xi) * xi               \n",
      "        d = d.reshape(W.shape) # and we flatten it\n",
      "        return - d / self.x.shape[0] + alpha*np.sum([np.linalg.norm(tmp)**2 for tmp in tmp_W])\n",
      "    \n",
      "    def score(self, x_test, y_test):\n",
      "        x = np.hstack((x_test, np.ones((x_test.shape[0], 1))))\n",
      "        y = y_test\n",
      "        c = 0\n",
      "        for i, xi in enumerate(x):\n",
      "            #a = np.argmax(softmax(self.W, xi))\n",
      "            a = np.argmax(np.dot(self.W.transpose(), xi))\n",
      "            if a == y[i]:\n",
      "                c += 1\n",
      "        return c * 1. / y.shape[0]\n",
      "    \n",
      "    # in train:\n",
      "        tmp = fmin_l_bfgs_b(self.loss_hinge_l2, self.W.reshape((self.W.shape[0]*self.W.shape[1],)),\n",
      "                            fprime=self.loss_derivative_hinge_l2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}