chapter02_supervised-learning/linear-regression-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear regression from scratch\n",
    "\n",
    "Powerful ML libraries can eliminate repetitive work, but if you rely too much on abstractions, you might never learn how neural networks really work under the hood. So for this first example, let's get our hands dirty and build everything from scratch, relying only on autograd and NDArray. First, we'll import the same dependencies as in the [autograd chapter](../chapter01_crashcourse/autograd.ipynb). We'll also import the powerful `gluon` package  but in this chapter, we'll only be using it for data loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd, gluon\n",
    "mx.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the context\n",
    "\n",
    "We'll also want to specify the contexts where computation should happen. This tutorial is so simple that you could probably run it on a calculator watch. But, to develop good habits we're going to specify two contexts: one for data and one for our models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_ctx = mx.cpu()\n",
    "model_ctx = mx.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear regression\n",
    "\n",
    "\n",
    "To get our feet wet, we'll start off by looking at the problem of regression.\n",
    "This is the task of predicting a *real valued target* $y$ given a data point $x$.\n",
    "In linear regression, the simplest and still perhaps the most useful approach,\n",
    "we assume that prediction can be expressed as a *linear* combination of the input features \n",
    "(thus giving the name *linear* regression):\n",
    "\n",
    "$$\\hat{y} = w_1 \\cdot x_1 + ... + w_d \\cdot x_d + b$$\n",
    "\n",
    "\n",
    "Given a collection of data points $X$, and corresponding target values $\\boldsymbol{y}$, \n",
    "we'll try to find the *weight* vector $\\boldsymbol{w}$ and bias term $b$ \n",
    "(also called an *offset* or *intercept*)\n",
    "that approximately associate data points $\\boldsymbol{x}_i$ with their corresponding labels ``y_i``. \n",
    "Using slightly more advanced math notation, we can express the predictions $\\boldsymbol{\\hat{y}}$\n",
    "corresponding to a collection of datapoints $X$ via the matrix-vector product:\n",
    "\n",
    "$$\\boldsymbol{\\hat{y}} = X \\boldsymbol{w} + b$$\n",
    "\n",
    "\n",
    "Before we can get going, we will need two more things \n",
    "\n",
    "* Some way to measure the quality of the current model  \n",
    "* Some way to manipulate the model to improve its quality\n",
    "\n",
    "\n",
    "### Square loss\n",
    "In order to say whether we've done a good job, \n",
    "we need some way to measure the quality of a model. \n",
    "Generally, we will define a *loss function*\n",
    "that says *how far* are our predictions from the correct answers.\n",
    "For the classical case of linear regression, \n",
    "we usually focus on the squared error.\n",
    "Specifically, our loss will be the sum, over all examples, of the squared error $(y_i-\\hat{y})^2$ on each:\n",
    "\n",
    "$$\\ell(y, \\hat{y}) = \\sum_{i=1}^n (\\hat{y}_i-y_i)^2.$$\n",
    "\n",
    "\n",
    "For one-dimensional data, we can easily visualize the relationship between our single feature and the target variable. It's also easy to visualize a linear predictor and it's error on each example. \n",
    "Note that squared loss *heavily penalizes outliers*. For the visualized predictor below, the lone outlier would contribute most of the loss.\n",
    "\n",
    "![](../img/linear-regression.png)\n",
    "\n",
    "\n",
    "### Manipulating the model\n",
    "\n",
    "For us to minimize the error,\n",
    "we need some mechanism to alter the model.\n",
    "We do this by choosing values of the *parameters*\n",
    "$\\boldsymbol{w}$ and $b$.\n",
    "This is the only job of the learning algorithm.\n",
    "Take training data ($X$, $y$) and the functional form of the model $\\hat{y} = X\\boldsymbol{w} + b$.\n",
    "Learning then consists of choosing the best possible $\\boldsymbol{w}$ and $b$ based on the available evidence.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Historical note\n",
    "\n",
    "You might reasonably point out that linear regression is a classical statistical model.\n",
    "[According to Wikipedia](https://en.wikipedia.org/wiki/Regression_analysis#History), \n",
    "Legendre first developed the method of least squares regression in 1805,\n",
    "which was shortly thereafter rediscovered by Gauss in 1809. \n",
    "Presumably, Legendre, who had Tweeted about the paper several times,\n",
    "was peeved that Gauss failed to cite his arXiv preprint. \n",
    "\n",
    "![Legendre](../img/legendre.jpeg)\n",
    "\n",
    "\n",
    "Matters of provenance aside, you might wonder - if Legendre and Gauss \n",
    "worked on linear regression, does that mean there were the original deep learning researchers?\n",
    "And if linear regression doesn't wholly belong to deep learning, \n",
    "then why are we presenting a linear model \n",
    "as the first example in a tutorial series on neural networks? \n",
    "Well it turns out that we can express linear regression \n",
    "as the simplest possible (useful) neural network. \n",
    "A neural network is just a collection of nodes (aka neurons) connected by directed edges. \n",
    "In most networks, we arrange the nodes into layers with each feeding its output into the layer above. \n",
    "To calculate the value of any node, we first perform a weighted sum of the inputs (according to weights ``w``) \n",
    "and then apply an *activation function*. \n",
    "For linear regression, we only have two layers, one corresponding to the input (depicted in orange) \n",
    "and a one-node layer (depicted in green) corresponding to the ouput.\n",
    "For the output node the activation function is just the identity function.\n",
    "\n",
    "![](../img/onelayer.png)\n",
    "\n",
    "While you certainly don't have to view linear regression through the lens of deep learning, \n",
    "you can (and we will!).\n",
    "To ground the concepts that we just discussed in code, \n",
    "let's actually code up a neural network for linear regression from scratch.\n",
    "\n",
    "\n",
    "To get going, we will generate a simple synthetic dataset by sampling random data points ``X[i]`` and corresponding labels ``y[i]`` in the following manner. Our inputs will each be sampled from a random normal distribution with mean $0$ and variance $1$. Our features will be independent. Another way of saying this is that they will have diagonal covariance.  The labels will be generated accoding to the *true* labeling function `y[i] = 2 * X[i][0]- 3.4 * X[i][1] + 4.2 + noise` where the noise is drawn from a random gaussian with mean ``0`` and variance ``.01``. We could express the labeling function in mathematical notation as:\n",
    "$$y = X \\cdot w + b + \\eta, \\quad \\text{for } \\eta \\sim \\mathcal{N}(0,\\sigma^2)$$ \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_inputs = 2\n",
    "num_outputs = 1\n",
    "num_examples = 10000\n",
    "\n",
    "def real_fn(X):\n",
    "    return 2 * X[:, 0] - 3.4 * X[:, 1] + 4.2\n",
    "    \n",
    "X = nd.random_normal(shape=(num_examples, num_inputs), ctx=data_ctx)\n",
    "noise = .1 * nd.random_normal(shape=(num_examples,), ctx=data_ctx)\n",
    "y = real_fn(X) + noise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that each row in ``X`` consists of a 2-dimensional data point and that each row in ``Y`` consists of a 1-dimensional target value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[-1.22338355  2.39233518]\n",
      "<NDArray 2 @cpu(0)>\n",
      "\n",
      "[-6.09602737]\n",
      "<NDArray 1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "print(X[0])\n",
    "print(y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that because our synthetic features `X` live on `data_ctx` and because our noise also lives on `data_ctx`, the labels `y`, produced by combining `X` and `noise` in `real_fn` also live on `data_ctx`. \n",
    "We can confirm that for any randomly chosen point, \n",
    "a linear combination with the (known) optimal parameters \n",
    "produces a prediction that is indeed close to the target value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[-6.38070679]\n",
      "<NDArray 1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "print(2 * X[0, 0] - 3.4 * X[0, 1] + 4.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the correspondence between our second feature (``X[:, 1]``) and the target values ``Y`` by generating a scatter plot with the Python plotting package ``matplotlib``. Make sure that ``matplotlib`` is installed. Otherwise, you may install it by running ``pip2 install matplotlib`` (for Python 2) or ``pip3 install matplotlib`` (for Python 3) on your command line. \n",
    "\n",
    "In order to plot with ``matplotlib`` we'll just need to convert ``X`` and ``y`` into NumPy arrays by using the `.asnumpy()` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3W2QXOWVH/D/mdYF9YBDi3jKhRoJaV2UKGShmfWE1ZZS\nqaC1ETZGjMFYpsBxKq4oH0yVpaW0GWLXSuziQvEsFlXZzSbK2mWnpOBhDcyC5ZTAlraoKJbt0c4A\nkpFiYhtBw5pxUGPQNFJPz8mH6Tu63X3fuvvevm//X5VK0y8z/YxmdPq55znPeURVQURE6dcX9QCI\niKg3GPCJiDKCAZ+IKCMY8ImIMoIBn4goIxjwiYgyggGfiCgjGPCJiDKCAZ+IKCOWRD0Aqw9+8IO6\natWqqIdBRJQox48f/62qDng9L1YBf9WqVZicnIx6GEREiSIir/p5HlM6REQZ0XXAF5EVInJERH4u\nIidF5Mv1+68UkedE5Bf1v5d1P1wiIupUEDP8OQD3q+r1ADYA+JKIXA9gFMCPVPVaAD+q3yYiooh0\nHfBV9U1V/Yf6x+8CeBlAEcDtAL5Tf9p3AIx0+1pERNS5QHP4IrIKwBCAnwD4kKq+WX/oHwF8KMjX\nIiKi9gRWpSMilwN4AsB2Vf2diCw+pqoqIrYnrYjINgDbAGDlypVdjWFiqoSxQ6fxRrmC5YU8dm5e\ng5GhYldfk4goLQKZ4YuIgYVgf0BVn6zf/RsRuar++FUA3rL7XFXdp6rDqjo8MOBZRupoYqqEB558\nCaVyBQqgVK7ggSdfwsRUqeOvSUSUJkFU6QiAbwJ4WVW/YXnoaQBfqH/8BQB/1+1ruRk7dBqVaq3h\nvkq1hrFDp8N8WSKixAgipbMRwOcBvCQi0/X7/gOAPQAeF5EvAngVwGcDeC1Hb5Qrbd1PRJQ1XQd8\nVf1fAMTh4T/q9uv7tbyQR8kmuC8v5Hs1BCKiWEvNTtudm9cgb+Qa7ssbOezcvCaiERERxUuseul0\nw6zGYZUOEZG91AR8YCHoM8ATEdlLTUqHiIjcpWqGHzVz41epXEFOBDVVFJlaIqKYyHzAD2p3rrnx\ny9wLUNOFjcXmBjAADPpEFKlMp3SC3J1rt/HLxA1gRBQHmQ74Qe7O9drgxQ1gRBS1TAf8IHfnem3w\n4gYwIopapgO+UxC2u39iqoSNew5j9ehBbNxzuCXtY7fxy8QNYEQUB5kO+H535/rJ9Y8MFfHwHetQ\nrL9Z5OrtoYuFPB6+Yx0XbIkocpmp0nGrxvGq0nHL9Vufy41fRBRnmQj4zSWTzaWSXkGanTiJKA0y\nEfC9Zuhetfi96MTJTVtEFLZM5PDdZuh+8vOdduL0Wui1Ps8cA9C6aYundhFREDIxw3ebofvJz/vJ\n9TdfJdx03QCeOF5yTCNZPfjMSc9NW5zlE1G3MhHwd25e05DDBy7O0HeMT9t+TvNVgVuu326N4MCx\nM2g+td0ueE9MlXB2tuo6fq4VEFEQMhHw3WboZt68md/8/MRUCfc//sJiGsbUHOxNZvC25uy9cNMW\nEQUhEwEfcJ6hu83+vZgz++Zg72Z5Id9yReCGm7aIKCipCPheVTZuj3dzUpZbwzRg4aBf61uBGby9\nPs9UyBvYvWXt4liC6uxJRNkk2sbsNGzDw8M6OTnZ1ufYzZbzRm5xd6vX491YPXrQMXWTN3K486NF\nHDk10xKgV40e9PX1jT7B5UuXoDxbRaHfwHvvz6E6f/EV2/k++GZBlF4iclxVh72el/gZvleVjd9d\nsp1wqv7JibgGYrPO3kt1XhcXdO0Wdv1+H14bz4goGxJfh++1C7abXbKdNEzLGzk88tn1roG0nZy/\nFz/fR5BtoDvld08CEYUn8TN8r12wne6S9TMrNv9+8JmTizPwS5f0NXyN5jQK0Jrb74afCh6/b3ph\npX14hUEUD4HM8EXkWyLyloicsNy3W0RKIjJd//PJIF6rmdcu2HZ3yZoz0e3j075nxe9X5xc/Lleq\n2D4+jY9/4+9td/A++MzJwIK9VwWP+b04vZ71zSLI07+axeEKg4iCm+F/G8BfAvjvTffvVdW/COg1\nbHlV2bRTheOnXLJ5VuxUcfOLt8613Fep1nxV5/hVm5/H7qdPYsf4dMv35fW9NL9ZtLPW0e6VAJvP\nEcVDIAFfVZ8XkVVBfK1OeHW89Nu22E+5ZHMKJaigtazf8Nxx2+xCTXGhsvA5zWkSt+/FrilbO2mf\ndtMzvWg+R0Tewl60vU9EXqynfJaF/FpdmZgqee56tUuhBBG0BAtVOOahKZ2qVGu4//EXsHr0oOf3\nsmN8umHx9Iq8Yfu85u+vk/RMp83niChYYQb8vwbwYQCDAN4E8Ijdk0Rkm4hMisjkzMxMiMNxZs5a\n3TidXBVE0DJz7EFU79RUPdcImvP0X514CecuzLU8z+iTlu+vk/SM9TQwAU8BI4pKaFU6qvob82MR\n+W8Avu/wvH0A9gELG6/CGo8bt/SH1+amkaEi/uR7L+BCLT4b2NpRqdbw2E9es32zuXzpkpbvu9P0\nDE8DI4peaDN8EbnKcvPTAE44PTdqbrNTPzPRr39mfdBD6imnK4vybLWlfv6m6waYniFKqKDKMh8D\n8GMAa0TkdRH5IoCvi8hLIvIigJsA7AjitcLgNDstFvK+ZqUjQ0Xcu2Fl0MOKXKHfaCnV3H/sDPoE\n6Dcu/upY9x4QUXwFVaVzt83d3wzia/dCNx0zTcPXXInxn77W0Osm6c5Xa6hY9hiYzl1oTH+VK1U8\n8ORLmHz1bdveQUQUD4nfaRuETjtmWuvR+3z2xwlSkDt27czaBHsnlWoN+4+dWbwd1G5aNn0jCk7i\nu2VGYWKq1NBOIQoigGpr0A/7TaBdxUIeR0c3dfS5YXY6JUoTv90ymXxtkxmEwgz2hbyBXJ99Tb6Z\nOjffp5uDe5yCPdDdxjS2ZCAKFgN+m/weXtKpYiGPyy5dgprNWsAlOUEbWZZY6BPpuB8PWzIQBYsB\nv01hB5tSueK4SzaJtf411Y6bsDlVT7ElA1FnGPDbxGDTvk7TMGzJQBQsBvw2Mdh0ppMrI7ZkIAoW\nyzLbNDJUxI7xadvFUQGw1MiFmuNPqk6vjNiSgSg4nOF3wCmTrsDijNRLrrvGmIlz9tz5jhdveTwi\nUTA4w+9A0aGBmNmKYWSoiImpEnZ+7wVUbRZa+wRI4PprV2ar89gxPo2/nTyDX/+/Ssuxj06bq3g8\nIlFwuPGqA343BA0++CzKleg2ZyWB0SeAoOWN0egDLl/qfChMNxu6iNLG78YrzvBdOG3rN4P67qdP\nLgb0pUZrduwdBntPTr2HqvNw3dzGWnyi9jGH78DPod7n5y7ugjo7W8WO8Wl8deLiQSos4QwP/22J\n2seA78BrW7/d4wrgwLEzi28KOzevWUhZUKDMWnwu5hK1hwHfgde2fqfHFVh8UxgZKmLsrvXI26R7\nqDNmLT4AzyswImrESOTAa1u/W0qhVK4sBp6RoSIuXZJzfK4dI2s1mz71AYvrKE5XYF956iXO+okc\nMOA78NrWv3PzGriFZXO2OTFVaqtSRwBcdgnX0u3M4+LVk9MV1rkLtYZZ//bxaQz92bMM/ERgwHfk\nta1/ZKiIezasdAz6Zr6/3R4ylyzpYymnCzPQX5E3fH/O2dkqAz8RWJbpymtb/0Mj6zB8zZXYPj5t\n+7hT10s31sofamW2W5YOsl5nZ6vctEWZxhl+m5orQwD4aqVAwaipYvv4dMcH0PAAFcoyBvw2ONXm\n33TdQEu+n+KrkysvojRgwG+DU2XIkVMzvpumUbj8/kL7yeWzzp/ShgG/DW61+SNDRRwd3eQr6LPo\nMjx+V0Cad0U387PTmihpGPDb4OfIPbtyzmbxaVeXXc27opvxAHVKo0ACvoh8S0TeEpETlvuuFJHn\nROQX9b+XBfFaUfJz5J61nNNLoY3SQgqedVd0Mx6gTmkU1Az/2wBuabpvFMCPVPVaAD+q30605mCe\nE1mc9VlnimZ659Gtg8g51A8WC3lM77qZQT9ipXLFNk/PA9QpjQKpw1fV50VkVdPdtwP4l/WPvwPg\n7wH8+yBeL0pm/bbXoRxmDrhmc96AtfkXN1lFz6zaMX+Ok6++jdkLcy3P4wHqlHRh5vA/pKpv1j/+\nRwAfsnuSiGwTkUkRmZyZmQlxOMHxk9+1ew6wcFVgbf5F8VKp1rD/2JmWOv9C3uAB6pR4Pdlpq6oq\nIrZrlaq6D8A+YOHEq16Mp1t+8rtOz5lXxchQERv3HOZh5wny7vtz2DE+jbFDpz2PZSSKqzAD/m9E\n5CpVfVNErgLwVoiv1VPLHc60teZ3vZ7Dxb9kMVNzpXIFO7/3AqAXT+viObuUFGGmdJ4G8IX6x18A\n8HchvlZP+anW8XqOn8W/fvbRj6VqTVuOZmTJJiVBUGWZjwH4MYA1IvK6iHwRwB4AHxeRXwD4WP12\nKnh10vTzHK/2ynkjh0vZriFReNVGcSdqU0USleHhYZ2cnIx6GD3z1YmXcODYmZaNWIW8gd1b1mLH\n+DQ3aSXIsn4DU396c9TDoAwSkeOqOuz1POYMIvTQyDrs3TrYcBXw6NZBTO+6GSNDRdZ8J0yM5k5E\nttgPP2JuPfd3bl7TUO9P8eZ3T8XEVIkVPhQJzvBjbqll4baQN3CvyylbFD2v5mpsykZR4gw/pszA\nYJ3dn5+bx/A1V2L/sTMRjozcjB063TBbb57Nnzs/57hpj7N8ChsDfky57eYt5A22ZIgpszfPG+UK\nrsgbOHdhDtXaxXp9J80VPkz7UBgY8GPKqcSvVK6gjzmdWDMDeztvytYF+uarO27soqAwhx9TThU6\nAmCe1SCpIrh4ZWDO7NmLn8LAgB9Tdjt1BTw8JS3MttjWn6k5k3dK/XBjF3WLAT+m7HbqMtinx/m5\neRTyRsvPtFKtOZ6hwH0Z1C3m8GOsuUZ/457Drgt/lByVas1xf0VNteVqjr34KQic4SeIn/NyKR2s\nwd6uVxNRJxjwE6Sd83IpHfL1jXc7xqcbjmAk6gQDfsKY5+VSNlSq89yVS4FhwE8op4U9SjeWZ1I3\nGPAT6u4/WBH1ECgiLM+kTjHgJ9RDI+tw74aVnOlnUJ8I0zrUER6AkhKrRw+yTj9D8kaOlTu0yO8B\nKKzDTwmnQ9Mpnay5fDZZI7+Y0kkJ1uhnj7UVA6t4yA+mdFLE2lK30G/gvffnUGWntczJiWBelTP+\nDPGb0mHATzHrG0CfCGox+llTb+SNHO78aBFHTs0w7ZNiDPjUYGKqhO3j01EPg2Ig1yf4wKVL8E6l\nyjeAlPAb8JnDz4iRoSKW9RtRD4M6FGT5bW1eUa5UmffPIAb8DJiYKmHjnsM4O1vlAegJJICvdFyn\nbwrcvZsdoZdlisivAbwLoAZgzs9lBwWn+bg8a9gQAWKU0SMHfn9ES40+nLtg33LZC3fvZkOv6vBv\nUtXf9ui1yMLuuDwTg326dBrsAR6ukhXceJVynLmRF/NwFWtVFxdz06kXAV8BPCsiCuC/quo+64Mi\nsg3ANgBYuXJlD4aTLdyBS83MKp1ypYqcCCrVGh585mTDvg1zMRcAg36K9GLR9p+r6u8D+ASAL4nI\nv7A+qKr7VHVYVYcHBgZ6MJxs8bMDlwu52WCejfzIXeuxe8ta5I3c4mLw2dlqyyY9LuamT+gzfFUt\n1f9+S0SeAnAjgOfDfl1aYM7Oxg6dRqlcaTkrFTa3KZ1+tefWxY837jnsuLZjZaYEme5Jh1Bn+CJy\nmYh8wPwYwM0AToT5mtTKPCXr0a2DKFhq8Tmzzy6/azvLC/nFSi/27Em+UHfaisjvAXiqfnMJgP+h\nql9zej532oanuTyTsqdYyC/O0M+dn0O5UnV9vnk1mHNoy1Es5HncZkzEoj2yqv4SwPowX4P8cSvP\ntNOc+skbOfz+yivwv//v2421/GBKKCnMxXunRXwjJ7jskoXFXOvP1WnTFyvAkoc7bTPC73/OvJHD\no1sHsXfrIIqF/OJC38N3rMOBf/uHLfffs2EljByTQ0lXyBsY+8x6TO+6GcVC3tebOGv3k4d1+Bnh\nVJ65rN9A/yVLGhbjAOdDNUaGii2LdcPXXMnGbAkncnGB38/kwKzdp2ThDD8j7Moz80YOu25bi6Oj\nm/CrPbcu5mPbXaBjY7bkOzt7MZ/vNXNf1m/weMWEYsDPiJGhIh6+Y11Lmqb5P61drt9PPfau29YG\nPWSKiNfejfer8z0cDQWJ/fCpgdth6NYqD7s67KE/e7ZhpkjJcu+GlXhoZB2Ai3X3Tgu8PFUrXmJR\npUPJ45TrFzRWedhtu99121r88ePT4KmKybT/2BnsP3YGxUIeN13nvuvdrNyx+13gJq34YkqHGthd\nztuVXtqleUaGigz2KVAqV7D/2BnfPZisvwvcpBVvDPjUwC7X7xTDWYdNJvN3odM1IOoNpnSoRXPp\n5cY9h21ne6zDJtMV+YUqLadJACcH8cAZPnlyKum0q8O+7BL76o5Ll/TB6OMGrbQ6d2EOE1Mlx0kA\nJwfxwIBPnvyUdJrn5tqdupTrE/zHO2/A2F3rWa+fUtWaYuzQ6bYmB9R7TOmQL3Y7bE12jdnMhd6i\nzU5dr5I/SqY3ypWGdtys0okf1uFT15xy/F7dFN1q/il57H7eLNHsDb91+EzpUNc6XahjXjc97NI2\nLNGMH6Z0qGtOm7W8AvrOzWvYoz8lOmnTwZl/73GGT13rdKHOXAym5Ns+Po2New43zN6drvDMmT5n\n/r3HgE9d89uYzelzi0ztpEJz4Ha6wsuJcHNWRLhoS5Hj8YvpY/bjeeJ4yffPVdB40Dr5x0VbSgzr\nFQKlQ6lcwfhPX8OdHy2ikPe394KL+OFjwKdYGBkq4ujoJgb9FKnOK/YfO+N5WDqwMLvn5qzwMeBT\nrLDnSjYpFqp2Vo8ebFn8peCwLJNixanEk9Kt+byFHePT2D4+jWX9BlSBdypVlm8GgDN8ihWv4/Uo\nfezOWzBvn52tolypsnwzIKEHfBG5RUROi8grIjIa9utRspkLuDmx76xZyBvM86dMO3WCLN/sTqgB\nX0RyAP4KwCcAXA/gbhG5PszXpOQbGSrikc+ut93MtXvLWhwd3QQ2Ws6uUrnCXH+Hwp7h3wjgFVX9\npapeAPBdALeH/JqUAl6budop4TP6hLnLlGGKpzNh/z8oAnjNcvv1+n1Ejsze+jvGpwEAe7cO4ujo\npobFOr+5/pwItt64AuKQIqJkY4qnPZFX6YjINgDbAGDlypURj4ai1rzr1pzFAWgI+M1916/IG3j3\n/BxqllPUjZxg6z9bgSeOl1CL0Y5yChZLef0Le4ZfArDCcvvq+n2LVHWfqg6r6vDAwEDIw6G4a+cQ\nbHOz1q/23IrdW9a2/jIrcPDFN9myIWGKhXxbJ6MpwHy+T2EH/J8BuFZEVovIJQA+B+DpkF+TEqzT\n3vpjh06jOt84i6/OK87Oeu/ypHjZuXkNdt22tiVlZ/QJjJx9ao75fH9CDfiqOgfgPgCHALwM4HFV\nPRnma1KydXoINjdrpYMZzu0W7cfuWo+xz6x3LMtlPt9b6Dl8Vf0BgB+E/TqUDnaHovjprZ8Tsc3T\nC4ClRo5pnYRQoGHNxm5X7chQ0fF4TObz3bFajWKl0976TouyCrhu5FrWbzg+RtGoVGu4//EXXGvt\nO70SzLrIq3SImjnN7NwUHXrwFAv5xa9l13Nf1fnNgqJj/kycqrQ6vRLMOs7wKRW8jlk0rxyaqz/K\nlSp37cacXW6+m1PWsowzfEqF5rp8u86KI0NFjB063VK5o7Bv4EXxYbZTWF4/SevIqZnFn/PerYMM\n9D4x4FNq+EkFOS3qKZzTQhQPZjuF/cfOLN7nlPIhe0zpUKY4LeoVC3kcHd2EX++5FY9uHeRCboKw\nHNM/BnzKFK9cP7AwU5x3Wcjt43tB7JhXbmYfJnbTtMeAT5nid7HPrbxvnsn+2FleyC/2YSqVK4vp\nnx3j0/jqxEtRDy82mMOnzPGT69+5eQ2217t1UryZV2h2fZgUwIFjZzB8zZXM8YMzfCJbI0PFthp4\nUTRyIrjzowtv4G4L8szxL2DAJ3Lgt4GXU0MvCl9NFU8cL2FiqoSCyxs0Wy4sYMAncuDVwGvxPpeG\nXhQ+s0rHbcN0nwgXcMEcPpErtwZezexaN1BveM3ga6qs1wdn+ESBMK8GWL8fjT4Rz53SrNdnwCcK\njFf9PoXHbwO8rOfymdIhCtByh/YMy/oNqC40a6PomPX6bj2X0owBnyhATm17d922tiGobNxzmH17\nekwA3HTdQMPPx+zFM/nq2zhyagalcmXxMJ1iCt8MmNIhCpDfnbx2LR4oPALgng0rceTUTMvCeqVa\nw4FjZxbfgJt78aepuoczfKKA+dnJa23nzJl+OAp5A+9Uqg1pm1WjB22f67QCYC70pmWWzxk+UURG\nhoo4Orop6mGk1vm5eezdOoijo5swMlTExFSpo8NuSuVKamb5DPhECWTkBHmD/33dNJdhjh063fEh\nN2lJ7fA3hihihby/nj3m7NTc3ft+dT68QaWEtQzTrSTTaz0lLTX8DPhEEdu9ZS0MH032zVO5zBSF\nWwtnWmD9N3L698qJoFKteW6aS0MNPwM+UYTMmvDqvC4GHLe+PNagw0ofb6VyBR9+4AdYNXoQ587P\n2Ta6M6tyaqrIGznHK640vMGGFvBFZLeIlERkuv7nk2G9FlESWQ/sAC4GnJ2b1zgGfWvQsSsB5fGM\nrcyAXq5UAV3YBCeA7b9TpVqDSGuKp/lUtKQKuyxzr6r+RcivQZRIdgd2mLlipw1czUHHrgSUB7c4\nq84rzs5WXQ+sL89WsXfrYCp347IOnygiTjnhN8qVhjr9doOOWzCjBaVyBQL7+nsF8MePTy8eZXnu\n/FwPRxausAP+fSLyrwBMArhfVc+G/HpEieHUd8dM2/jZwGXH7uqg15yCaZwonMdpPbe4XKli59++\nACD5rZW7yuGLyA9F5ITNn9sB/DWADwMYBPAmgEccvsY2EZkUkcmZmZluhkOUKHaLrkHkiq25/ajE\nPdibzMonL9V5TUVZpmgP2rmKyCoA31fVj7g9b3h4WCcnJ0MfD1FchN250VwY5sEs9gp5A9O7bsbq\n0YOeb1ICxDa3LyLHVXXY63mhpXRE5CpVfbN+89MAToT1WkRJ1Wnapp2vD1xcC7gib+DchTlUa43h\nTQSuRwSmlVmo45Resyr0G7adNoHkpHrCzOF/XUQGsXDV9GsA/y7E1yIiB81vKhNTJTz4zEmcnb3Y\nm181GXn3oJXr/wY7N6/BjvFpx+/f6BOowrGqKikBP7Q6fFX9vKquU9UbVHWLZbZPRBExU0jWYG8y\nFzG7lfOxazguzMPNR4aKuGfDStvvv9/ow9hd6/GOw+E1SdqBy522RBnRvNHLjsJ+Q1KzS2x2rAIL\nbxh337gi0gXjdpiHm09MlfDQyDrs3TrYspHt53/+CddWFuabRhIw4BNlhN1Gr2bFQt7XubwXavbP\nUQBHTs3g6OimQK4WulHIG/Cz6dhMy3gtoDu1srC+acQdAz5RRnilHsyS0EK/v+6dTkrlCjbuOdzW\n1wn6zSFv5PCp9Vdhic/0krkAWypXoLA/7cosd3VqyZCEsk0GfKKMcGv+ZT2KMYhqnVK5gvfe97dD\nNW/kcM+GlYE2gqtUa3jsJ6+1VCM5MTtmNn+N5iA+MlR0vAJKQi6fAZ8oI5w2ej1qORUKgOPiZLuq\n897BtpA38PAd6/DQyDo8fMc632cD+FHz+c6VN3KOz7UL4k5vnEnopsmAT5QRfg9Y71XgMjc9ma8/\nMlTEZZf2tr2XVy98u3+LsHZI9wKbpxFliJ+NXu324lkIfoqKzQlcy/oN2xJQwP5KotdpEWsv/GZG\nTnDu/BxWjx5sWMTtprFd1BjwiahBc0BzS4wUC3ncdN0Axn/2WstjRp/g1huuwv5jZ2w/12727GfH\na5hyIphXRaHfwHvvzy300Efrrlq7N86w22QEgSkdImoxMlTE0dFN2OtyoIp53OKRUzO2i6OXL12C\nI6fsGyIKYJsCsUuXGH1ie1JVGOZVsXfrIH5XmWtZg3CrxLHucXCq8okDBnwismUGMbt0hzVn7ZSG\nKc9WHR9T2PefsVtnGLtrPcY+s74nJ3mZ/XLaWcQF3A+ziROmdIjIltNGrZxIw2KvV19/u8fcduI6\nrTPsCPkkr7yRs+2XY2WXhpqYKjmmoeJWqskZPhHZcgpW86qeO1DNK4AgK1rCrB4y38TcSlLtxm1e\nBTlRABv3HI5NaocBn4hs+a03dyv39FsK6ofTm8e9XW7ayhs5PPLZ9a79cpqvakx+2lXEKZ/fkwNQ\n/OIBKETxYXd4St7IdRywgxqTXSWM9X6zwsbPxq9iUzVNu9+zn4NTrK91dHSTz2e3J/IDUIgo2eJY\nb+6U3zfvt7Z/zomgpopiIY9V/zSPY788i5oqciK4+w9W4KGRdbZfB/D/PbdTRhqHfD5n+ESUClFc\nkdi9ptNBMnGY4TOHT0SpEEVppN0ahV0juLi0XmBKh4hSwSllEnYqxS7NNHzNlbFKhZkY8IkoFbz2\nA/SSV8+iqNowMKVDRKmQlC6WUbZhYMAnolQIsuY/TFG2YWBKh4hSw0/756hFtdYAcIZPRNRTUZ6Y\nxYBPRNRDUa41dBXwReQuETkpIvMiMtz02AMi8oqInBaRzd0Nk4goHaxrDcDFYxbHDp0OfeG22xn+\nCQB3AHjeeqeIXA/gcwDWArgFwH8WkeCOpCciSrCRoeLiTN/svd+Lap2uAr6qvqyqdkvLtwP4rqqe\nV9VfAXgFwI3dvBYRUZpEUa0TVg6/CMB6yOXr9ftaiMg2EZkUkcmZGfvj0IiI0iaKah3PgC8iPxSR\nEzZ/bg9iAKq6T1WHVXV4YGAgiC9JRBR7UVTreNbhq+rHOvi6JQArLLevrt9HRBSoqNoUdGvn5jW2\n3T3DrNYJK6XzNIDPicilIrIawLUAfhrSaxFRRkXZpqBbUewM7mqnrYh8GsB/AjAA4KCITKvqZlU9\nKSKPA/g5gDkAX1JV93PAiIja5LbwmYRZfq93BncV8FX1KQBPOTz2NQBf6+brExG5ibJNQRJxpy0R\nJVaUbQoXeKz1AAAD5ElEQVSSiAGfiBIrKS2R44LdMokoseJ40HqcMeATUaIloSVyXDClQ0SUEQz4\nREQZwYBPRJQRDPhERBnBgE9ElBGi9eb7cSAiMwBejXocDj4I4LdRD8IHjjM4SRgjwHEGLYnjvEZV\nPdsNxyrgx5mITKrqsPczo8VxBicJYwQ4zqCleZxM6RARZQQDPhFRRjDg+7cv6gH4xHEGJwljBDjO\noKV2nMzhExFlBGf4REQZwYDfBhH5cxF5UUSmReRZEVke9ZiaiciYiJyqj/MpESlEPSY7InKXiJwU\nkXkRiV1FhIjcIiKnReQVERmNejx2RORbIvKWiJyIeixuRGSFiBwRkZ/Xf+ZfjnpMzURkqYj8VERe\nqI/xwajH5EZEciIyJSLfb+fzGPDbM6aqN6jqIIDvA/jTqAdk4zkAH1HVGwD8HwAPRDweJycA3AHg\n+agH0kxEcgD+CsAnAFwP4G4RuT7aUdn6NoBboh6ED3MA7lfV6wFsAPClGP57ngewSVXXAxgEcIuI\nbIh4TG6+DODldj+JAb8Nqvo7y83LAMRuAURVn1XVufrNYwCujnI8TlT1ZVU9HfU4HNwI4BVV/aWq\nXgDwXQC3RzymFqr6PIC3ox6HF1V9U1X/of7xu1gIVLHqZ6wL3qvfNOp/Yvf/GwBE5GoAtwL4m3Y/\nlwG/TSLyNRF5DcA9iOcM3+rfAPifUQ8igYoAXrPcfh0xC1BJJSKrAAwB+Em0I2lVT5NMA3gLwHOq\nGrsx1j0K4E8AzLf7iQz4TUTkhyJywubP7QCgql9R1RUADgC4L45jrD/nK1i4lD4QxRj9jpOyQ0Qu\nB/AEgO1NV8uxoKq1err2agA3ishHoh5TMxH5FIC3VPV4J5/PE6+aqOrHfD71AIAfANgV4nBseY1R\nRP41gE8B+CONsO62jX/LuCkBWGG5fXX9PuqQiBhYCPYHVPXJqMfjRlXLInIEC+sjcVsQ3whgi4h8\nEsBSAP9ERPar6r1+Ppkz/DaIyLWWm7cDOBXVWJyIyC1YuNzboqqzUY8noX4G4FoRWS0ilwD4HICn\nIx5TYomIAPgmgJdV9RtRj8eOiAyYFW0ikgfwccTw/7eqPqCqV6vqKiz8Xh72G+wBBvx27amnJF4E\ncDMWVsrj5i8BfADAc/Xy0f8S9YDsiMinReR1AH8I4KCIHIp6TKb6ovd9AA5hYYHxcVU9Ge2oWonI\nYwB+DGCNiLwuIl+MekwONgL4PIBN9d/J6foMNU6uAnCk/n/7Z1jI4bdV8pgE3GlLRJQRnOETEWUE\nAz4RUUYw4BMRZQQDPhFRRjDgExFlBAM+EVFGMOATEWUEAz4RUUb8fw+QjglvBwH4AAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f9378417748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(X[:, 1].asnumpy(),y.asnumpy())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data iterators\n",
    "\n",
    "Once we start working with neural networks, we're going to need to iterate through our data points quickly. We'll also want to be able to grab batches of ``k`` data points at a time, to shuffle our data. In MXNet, data iterators give us a nice set of utilities for fetching and manipulating data. In particular, we'll work with the simple  ``DataLoader`` class, that provides an intuitive way to use an ``ArrayDataset`` for training models.\n",
    "\n",
    "\n",
    "We can load `X` and `y` into an ArrayDataset, by calling `gluon.data.ArrayDataset(X, y)`. It's ok for `X` to be a multi-dimensional input array (say, of images) and `y` to be just a one-dimensional array of labels. The one requirement is that they have equal lengths along the first axis, i.e., `len(X) == len(y)`. \n",
    "\n",
    "Given an `ArrayDataset`, we can create a DataLoader which will grab random batches of data from an `ArrayDataset`.  We'll want to specify two arguments. First, we'll need to say the `batch_size`, i.e., how many examples we want to grab at a time. Second, we'll want to specify whether or not to shuffle the data between iterations through the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 4\n",
    "train_data = gluon.data.DataLoader(gluon.data.ArrayDataset(X, y),\n",
    "                                   batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've initialized our DataLoader (``train_data``), we can easily fetch batches by iterating over `train_data` just as if it were a Python list. You can use your favorite iterating techniques like foreach loops: `for data, label in train_data` or enumerations: `for i, (data, label) in enumerate(train_data)`. \n",
    "First, let's just grab one batch and break out of the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[-0.14732301 -1.32803488]\n",
      " [-0.56128627  0.48301753]\n",
      " [ 0.75564283 -0.12659997]\n",
      " [-0.96057719 -0.96254188]]\n",
      "<NDArray 4x2 @cpu(0)> \n",
      "[ 8.25711536  1.30587864  6.15542459  5.48825312]\n",
      "<NDArray 4 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "for i, (data, label) in enumerate(train_data):\n",
    "    print(data, label)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we run that same code again you'll notice that we get a different batch. That's because we instructed the `DataLoader` that `shuffle=True`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[-0.59027743 -1.52694809]\n",
      " [-0.00750104  2.68466949]\n",
      " [ 1.50308061  0.54902577]\n",
      " [ 1.69129586  0.32308948]]\n",
      "<NDArray 4x2 @cpu(0)> \n",
      "[ 8.28844357 -5.07566643  5.3666563   6.52408457]\n",
      "<NDArray 4 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "for i, (data, label) in enumerate(train_data):\n",
    "    print(data, label)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if we actually pass over the entire dataset, and count the number of batches, we'll find that there are 2500 batches. We expect this because our dataset has 10,000 examples and we configured the `DataLoader` with a batch size of 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2500\n"
     ]
    }
   ],
   "source": [
    "for i, (data, label) in enumerate(train_data):\n",
    "    pass\n",
    "print(i+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model parameters\n",
    "\n",
    "Now let's allocate some memory for our parameters and set their initial values. We'll want to initialize these parameters on the `model_ctx`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "w = nd.random_normal(shape=(num_inputs, num_outputs), ctx=model_ctx)\n",
    "b = nd.random_normal(shape=num_outputs, ctx=model_ctx)\n",
    "params = [w, b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the succeeding cells, we're going to update these parameters to better fit our data. This will involve taking the gradient (a multi-dimensional derivative) of some *loss function* with respect to the parameters. We'll update each parameter in the direction that reduces the loss. But first, let's just allocate some memory for each gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for param in params:\n",
    "    param.attach_grad()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural networks\n",
    "\n",
    "Next we'll want to define our model. In this case, we'll be working with linear models, the simplest possible *useful* neural network. To calculate the output of the linear model, we simply multiply a given input with the model's weights (``w``), and add the offset ``b``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    return mx.nd.dot(X, w) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, that was easy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function\n",
    "\n",
    "Train a model means making it better and better over the course of a period of training. But in order for this goal to make any sense at all, we first need to define what *better* means in the first place. In this case, we'll use the squared distance between our prediction and the true value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def square_loss(yhat, y): \n",
    "    return nd.mean((yhat - y) ** 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer\n",
    "\n",
    "It turns out that linear regression actually has a closed-form solution. However, most interesting models that we'll care about cannot be solved analytically. So we'll solve this problem by stochastic gradient descent. At each step, we'll estimate the gradient of the loss with respect to our weights, using one batch randomly drawn from our dataset. Then, we'll update our parameters a small amount in the direction that reduces the loss. The size of the step is determined by the *learning rate* ``lr``. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def SGD(params, lr):    \n",
    "    for param in params:\n",
    "        param[:] = param - lr * param.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute training loop\n",
    "\n",
    "Now that we have all the pieces, we just need to wire them together by writing a training loop. \n",
    "First we'll define ``epochs``, the number of passes to make over the dataset. Then for each pass, we'll iterate through ``train_data``, grabbing batches of examples and their corresponding labels. \n",
    "\n",
    "For each batch, we'll go through the following ritual:\n",
    "     \n",
    "* Generate predictions (``yhat``) and the loss (``loss``) by executing a forward pass through the network.\n",
    "* Calculate gradients by making a backwards pass through the network (``loss.backward()``). \n",
    "* Update the model parameters by invoking our SGD optimizer.",
    "     \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24.6606138554\n",
      "9.09776815639\n",
      "3.36058844271\n",
      "1.24549788469\n",
      "0.465710770596\n",
      "0.178157229481\n",
      "0.0721970594548\n",
      "0.0331197250206\n",
      "0.0186954441286\n",
      "0.0133724625537\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "learning_rate = .0001\n",
    "num_batches = num_examples/batch_size\n",
    "\n",
    "for e in range(epochs):\n",
    "    cumulative_loss = 0\n",
    "    # inner loop\n",
    "    for i, (data, label) in enumerate(train_data):\n",
    "        data = data.as_in_context(model_ctx)\n",
    "        label = label.as_in_context(model_ctx).reshape((-1, 1))\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            loss = square_loss(output, label)\n",
    "        loss.backward()\n",
    "        SGD(params, learning_rate)\n",
    "        cumulative_loss += loss.asscalar()\n",
    "    print(cumulative_loss / num_batches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing our training progess\n",
    "\n",
    "In the succeeding chapters, we'll introduce more realistic data, fancier models, more complicated loss functions, and more. But the core ideas are the same and the training loop will look remarkably familiar. Because these tutorials are self-contained, you'll get to know this ritual quite well. In addition to updating our model, we'll often want to do some bookkeeping. Among other things, we might want to keep track of training progress and visualize it graphically. We demonstrate one slighly more sophisticated training loop below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEICAYAAABbOlNNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8FPW5+PHPkwRy4SIQEKNIQkWtgoIStZ7Tlp56Q1AR\ntUCNFHtURNqftD32iIfTqlh+x1ZbK79yEVtarKh4QbSWHhVbRduiAqIF1ILcMQTkJgECJHl+f8xs\nMruZve+yu8nzfr32tbuzszPf3Z2dZ753UVWMMcaYgLxMJ8AYY0x2scBgjDEmiAUGY4wxQSwwGGOM\nCWKBwRhjTBALDMYYY4JYYMgwEakQERWRggTf31tEakUkP9VpSwURqRKRV1K9rmlJRL4iIh9nOh1+\nRORrIrI10+mIh4i8LiI3h3lNROS3IrJHRN45xun6k4iMTec+2mxgEJGNInJxptORLFXdrKodVbUh\n1dsWkd+JyE+S2YaqzlPVS1O9bmviHouH3AAfuP0qhvepiPQNPFfVN1X19DSlMeljoZX5MnAJ0EtV\nz0/XTkTkHhF53LtMVS9X1bnp2idAQlepJjuISIGq1rfV/bcyV6rq4kwnojURkfx0XDC5yoGNqnog\nTdvPLFVtkzdgI3BxmNduAdYBu4EXgRPd5QI8BOwAPgf+AfR3XxsKrAH2A9uAO8JsOx94EPgMWA98\nB1CgwC9dwD3A4+7jCnfdm4DNwBLPssD7XwfuA/7qpuUVoLtne98CNgG7gB+F+x6AccBR4AhQC/zB\nk747gQ+AwzgXF5OAT9z9rQFGeLZzI/CW57kC44G1wF5gOiAJrJsP/Nz9HjcA3/V+D7l0i3Is9gXe\nAPa5n3W+u3yJ+3kPuL/PKOBrwNaQ7f7Q/a0OAL8BegJ/cn+rxUBXz/rPANvdfS0B+kU5Fk4EngN2\nur/B7Z5tFQO/A/a4x8QPvWkL+YwzgQdDlr0A/MB9fCfOf2o/8DFwUZjt/M7d1iL3814MFOL83zYD\nNcAsoNhdvyvwkpv+Pe7jXp7tvQ7c7LOfm4A6oMH9Pu4NPXY9x29fT9qmA390P8fbwCmedfsBr+Kc\nc2qA/wKGuN/5UXc/74emC6fU579x/tM7gMeA40LOF2Pdz/8ZMDmmYzLTf4ps+zMCX3e/wHPdg+r/\nAUvc1y4DlgNdcILEGUCZ+1o18BXPAXdumP2OBz4CTga6AX8h/sDwGNAB589XQcvA8Alwmvv668D9\n7mtnugfYl4H2OH+Yo37fg+dg/onP97bSTX/gD/YNnJNEHs4J6oDne7mRlif7l9zvsDfOn3JIAuuO\nxznh9HK/78W0zsDwJDDZ/W6LgC+HfD99Pc+/RsvAsBQnGJyEc+JYAZzjbuvPwN2e9f8d6IRz3P8S\nWBnuWHDTsxz4sXssfQHnQucy9/X7gTdxjvGTgVWEDwxfBbbQHPS7AofcY+p097XAxVkFnhOqz/G6\nD/hXz/f1EM7FXTf3s/0B+B93/VLgWqDEfe0ZYKFne6/jExjCHKtBz0N/Hzdtu4DzcS6m5gFPua91\nwjl//Ieb5k7ABaH/f790ub/ZOvf77wgsAH4fcr54FOdcMADnYu6MaMdkm61jiKAKmKOqK1T1MHAX\ncKGIVOCcRDsBX8Q5iD9U1Wr3fUeBM0Wks6ruUdUVYbY/Evilqm5R1d3A/ySQxntU9YCqHgrz+m9V\n9Z/u608DA93l1+Fc7b2lqkdw/tSJDJY1zU3/IQBVfUZVP1XVRlWdj3OFH6nc9X5V3auqm3EC48AE\n1h0JPKyqW1V1D86JKJctFJG9ntst7vKjOMUWJ6pqnaq+Fed2/5+q1qjqNpwT9duq+p6q1gHP4wQJ\nAFR1jqrud4/7e4ABInJcmO2eB/RQ1SmqekRV1+OcgEa7r48EpqrqblXdAkyLkMY3cY7Dr7jPrwP+\nrqqf4lyVF+L8t9qp6kZV/STCtl5Q1b+qaiPOSXAc8H03HfuB/xtIo6ruUtXnVPWg+9pUYHCEbSfr\neVV9R53i13k0H8tXANtV9efub7xfVd+OcZtVwC9Udb2q1uKcr0aHNGa5V1UPqer7wPs4ASIiCwwt\nnYiTLQPA/bJ3ASep6p+BX+FkCXeIyGwR6eyuei1OcdImEXlDRC6MsP0tnuebwqwXyZYor2/3PD6I\ncyXRYt+qehDnsyW1fxH5loisDJzUgP5A9wTSF8+6od9jtO8k212tql08t0fd5f+Jkzt9R0RWi8i/\nx7ndGs/jQz7PO4JTHi8i94vIJyLyOU5uA8L/juXAid5ghlP80dN9PebjXJ3L26eAb7qLrsc5caKq\n64Dv4QSqHSLylIicGP7jBu2zB05uYLknjf/rLkdESkTkERHZ5H7mJUCXNLbwC3csn4yTy09E0PnK\nfVxA8+8Qab9hWWBo6VOcgx4AEemAk+XcBqCq01R1EE6xzGk4Zaeo6ruqOhw4HliIc6XupxrnQAjo\nHfL6AZyDOeAEn20kOiRuNU7RCwAiUozz2cIJt5+m5SJSjnOl+F2gVFW74BQbSIJpjFXQZyH4O201\nVHW7qt6iqicCtwIzvC2RUuh6YDhOufxxOMUQ0Pw7hh4LW4ANIcGsk6oOdV+PdpyHehK4zj2eLsCp\nu3B2rPqEqn4Z53+pwE8jbMebzs9wgl8/TxqPU9XAifE/cIqqLlDVzjhFWt7PHI+g/62I+P1vw9mC\nUxTkJ9p/Peh8hfM91xN8ARC3th4Y2olIkedWgHOAfltEBopIIU7W821V3Sgi54nIBSLSDudAqAMa\nRaS92wb/OFU9ilMx3Rhmn08Dt4tILxHpilNx67USJyvYTkQqcbLVqfIscKWI/IuItMe5Cov0J6gh\n/AEb0AHn4N0JICLfxskxpNvTwEQROUlEuuBUULY6IvINEQkEwD0433Xg2Irl94lVJ5yil104J7j/\nG/J66L7eAfaLyJ0iUuzmOPqLyHnu608Dd4lIVzf9/yfSzlX1PZwT+a+Bl1V1L4CInC4iX3f/i3U4\nJ/pw/63QbTbiXLQ8JCLHu9s7SUQu83zmQ8BeEekG3B3LdsN4H+jnnjeKcP5bsXoJKBOR74lIoYh0\nEpEL3NdqgAoRCXeufhL4voj0EZGOOL/bfE2ytWBbDwyLcA6MwO0edZoM/gjniqUaOIXmctPOOAfa\nHppb9jzgvjYG2OhmScfjlP35eRR4GedAWoFTWeT1I3efe3BaOzyR1Cf0UNXVOH/Qp3A+Wy1OheTh\nMG/5DU7Z7l4RWRhmm2twWgf9HecgPgunRVS6PYrT4uoD4D2c37Iep0w6F/0hpB/D8+7y84C3RaQW\npxJ1olueD87JZ677+4xMcv+P4RzT23Aq9ZeGvB50LKjTDPQKnHLyDTSf1AN1Eve629uA8zv9PoY0\nPIGTY/Ee84U49Uef4RSJHI9Tjh6rO3EqZ5e6/83FOLkEcCrYi91tL8UpZkqIqv4TmOJufy0Qc12Q\nW79xCXAlzmdcC/yb+/Iz7v0uEfGrt5yD890uwfmu64gShGMRaAVg2iD3CmMvcKqqbsh0epIhIpcD\ns1S1POrKxpiI2nqOoc0RkSvdSrcOOM1V/0FzRWPOcIsvhopIgYichFMM8Hy09xljorPA0PYMx6mw\n+hQ4FRituZltFJziij04RUkf4jS/NcYkyYqSjDHGBLEcgzHGmCA5OYhe9+7dtaKiItPJMK3U8uXL\nP1PVHsd6v3Zcm3SK57jOycBQUVHBsmXLMp0M00qJSCK90ZNmx7VJp3iOaytKMsYYE8QCgzHGmCAW\nGIwxxgTJyToGY0zrcPToUbZu3UpdXV2mk9JqFBUV0atXL9q1a5fwNiwwGGMyZuvWrXTq1ImKigpE\n0j0gb+unquzatYutW7fSp0+fhLdjRUnGZLnq/dUM/t1gttduj75yjqmrq6O0tNSCQoqICKWlpUnn\nwCwwGJPl7ltyH29tfospb0zJdFLSwoJCaqXi+7SiJGOyVPHUYurqm6/8Zi6bycxlMykqKOLQ5HCz\nuhqTPMsxGJOl1t++nuv7X09JgTMxWElBCVVnVbFhYk6PkJ518vPzGThwYNPt/vvDTx++cOFC1qxZ\n0/T8xz/+MYsXL046DXv37mXGjBlxv++ee+7hwQcfTHr/oSwwGJMkEZkjIjtEZJVn2T0iss2dC3ul\niAyNtA0/ZZ3K6FzYmbqGOooKiqhrqKNzYWdO6BjPrJGtzLx5UFEBeXnO/bx5SW+yuLiYlStXNt0m\nTQqdVLFZaGCYMmUKF198cdJpSDQwpIsFBmOS9ztgiM/yh1R1oHtblMiGaw7UMH7QeJbetJTxg8a3\nygromM2bB+PGwaZNoOrcjxuXkuDgZ9KkSZx55pmcffbZ3HHHHfztb3/jxRdf5Ic//CEDBw7kk08+\n4cYbb+TZZ58FnCFN7rrrLgYOHEhlZSUrVqzgsssu45RTTmHWrFkA1NbWctFFF3Huuedy1lln8cIL\nLzTt65NPPmHgwIH88Ic/BOCBBx7gvPPO4+yzz+buu5tnHZ06dSqnnXYaX/7yl/n444/T8tlR1Zy7\nDRo0SI1JF2CZxnlMAhXAKs/ze4A74tlGWzyu16xZE/vK5eWqTkgIvpWXJ5WGvLw8HTBgQNPtqaee\n0s8++0xPO+00bWxsVFXVPXv2qKrq2LFj9Zlnnml6r/d5eXm5zpgxQ1VVv/e97+lZZ52ln3/+ue7Y\nsUOPP/54VVU9evSo7tu3T1VVd+7cqaeccoo2Njbqhg0btF+/fk3bffnll/WWW27RxsZGbWho0GHD\nhukbb7yhy5Yt0/79++uBAwd03759esopp+gDDzzQ4jP5fa/xHNdW+WxM+nxXRL4FLAP+Q1X3ZDpB\nOW3z5viWxyhQlORVX19PUVERN910E1dccQVXXHFFTNu66qqrADjrrLOora2lU6dOdOrUicLCQvbu\n3UuHDh34r//6L5YsWUJeXh7btm2jpqamxXZeeeUVXnnlFc455xzAyWmsXbuW/fv3M2LECEpKSoL2\nl2pWlGRMeswETgEGAtXAz/1WEpFxIrJMRJbt3LnzWKYv9/TuHd/yJBQUFPDOO+9w3XXX8dJLLzFk\niF9JYUuFhYUA5OXlNT0OPK+vr2fevHns3LmT5cuXs3LlSnr27Onb50BVueuuu5rqPdatW8dNN92U\nmg8XAwsMxqSBqtaoaoOqNgKPAueHWW+2qlaqamWPHsd8CojcMnUquFfKTUpKnOUpVltby759+xg6\ndCgPPfQQ77//PgCdOnVi//79CW933759HH/88bRr146//OUvbNq0yXe7l112GXPmzKG2thaAbdu2\nsWPHDr761a+ycOFCDh06xP79+/nDH/6QxKcMz4qSjEkDESlT1Wr36QhgVaT1TQyqqpz7yZOd4qPe\nvZ2gEFieoEOHDjFw4MCm50OGDGHixIkMHz6curo6VJVf/OIXAIwePZpbbrmFadOmNVU6x/cRqrjy\nyis566yzqKys5Itf/CIApaWl/Ou//iv9+/fn8ssv54EHHuDDDz/kwgsvBKBjx448/vjjnHvuuYwa\nNYoBAwZw/PHHc9555yX12cPJyTmfKysr1SY0MekiIstVtTKO9Z8EvgZ0B2qAu93nAwEFNgK3egKF\nr7Z4XH/44YecccYZmU5Gq+P3vcZzXFuOwZgkqeo3fRb/5pgnxJgUsToGY4wxQSwwGGOMCWKBwRhj\nTBALDMYYY4JYYDDGGBPEAoMxpk0LDLvdv39/rrzySvbu3ZvwtioqKvjss89SmLrMsMBgjMkpqZ7q\nNDBW0qpVq+jWrRvTp09PyXZzWUoCg4gMEZGPRWSdiLQYzFxECkVkvvv62yJSEfJ6bxGpFZE7UpEe\nY0zrlc6pTi+88EK2bdvW9Dzc0NdXX301gwYNol+/fsyePTvl6ci0pDu4iUg+MB24BNgKvCsiL6rq\nGs9qNwF7VLWviIwGfgqM8rz+C+BPyabFGNN6pXuq04aGBl577bWmwepeeeUV1q5dyzvvvIOqctVV\nV7FkyRK++tWvMmfOHLp168ahQ4c477zzuPbaayktLU06DdkiFTmG84F1qrpeVY8ATwHDQ9YZDsx1\nHz8LXCTujNUicjWwAVidgrQYY1qpdE11Ghgr6YQTTqCmpoZLLrkECB76+txzz+Wjjz5i7dq1AEyb\nNo0BAwbwpS99iS1btjQtby1SERhOArZ4nm91l/muo6r1wD6gVEQ6AncC90bbiQ1PbEzblq6pTgN1\nDJs2bUJVm+oYwg19/frrr7N48WL+/ve/8/7773POOef4Dp2dyzJd+XwPzvSHtdFWtOGJjTHpnOq0\npKSEadOm8fOf/5z6+vqwQ1/v27ePrl27UlJSwkcffcTSpUtTloZskYpB9LYBJ3ue93KX+a2zVUQK\ngOOAXcAFwHUi8jOgC9AoInWq+qsUpMsY08osGLWg6fH0YalvPXTOOedw9tln8+STTzJmzBjfoa+H\nDBnCrFmzOOOMMzj99NP50pe+lPJ0ZFoqAsO7wKki0gcnAIwGrg9Z50VgLPB34Drgz+4cpF8JrCAi\n9wC1FhSMSa3q/dWMfm4086+bn3SxS2sUyBEEeCe/mThxIhMnTmzxnj/9yb+tzMaNG1OatkxJuijJ\nrTP4LvAy8CHwtKquFpEpIhKYkPQ3OHUK64AfAC2atBpjYhNvO/50Nu80rVNK5mNQ1UXAopBlP/Y8\nrgO+EWUb96QiLca0dt4T/YxhM8Kul+7mnab1ynTlszEmRsVTi5F7hZnLZtKojcxcNhO5VyieWuy7\nfrqad6ZaLs4imc1S8X1aYDAmR8R7ok9X885UKioqYteuXRYcUkRV2bVrF0VFRUltx6b2NCZHJHKi\nDzTvHDdoHLOXz6a6NuK002kXWhHeq1cvtm7divVNSp2ioiJ69eqV1DYsMBiTQ+I90ae7eWe8QutH\n2rVrR58+fTKdLBNCcjELV1lZqcuWLct0MkwrJSLLVbXyWO+3NR/XoRXhAVYRfuzEc1xbHYMxJu1y\npSLcOCwwGGPSLhcqwk0zCwzGmGMineMcmdSyymdjTFr5DcmRDRXhJjzLMRhj0sqG5Mg9FhiMaWNS\nPWdyOPH21DbZwwKDMW3MsbqCt5ZIucvqGIxpI471oHrWEil3WY7BmDYiE1fw1hIpN1mOwZg2IhNX\n8Nk2JIeJjeUYjGlD7ArexMJyDMa0IcfqCj7Qd2HakGnc/r+327SiOcZyDMaYlAu0fKpaUGV9GHKQ\nBQZjTMyi9YEI7buweudq68OQgywwGGNiFq0PRKDlU3FBcAAozi+2Pgw5xOoYjDFRxdoHItDy6XDD\nYfIlnwZtIF/yOdx42Pow5BDLMRiTJBGZIyI7RGSVZ1k3EXlVRNa6910zmcZYRComiqcPRKDl0+Dy\nwfTr0Y/B5YOtBVSOsRyDMcn7HfAr4DHPsknAa6p6v4hMcp/fmYG0xSx02k2vePpAeFs+eQUCj7VQ\nyn6WYzAmSaq6BNgdsng4MNd9PBe4+pgmKg6xDnaXbB8IG2U1d9icz8aESGTOZxGpAF5S1f7u872q\n2sV9LMCewPOQ940DxgH07t170KZNm5JMffyq91dzxyt3sPCjhRysP0hJQQkjzhjBg5c+mJIre5vv\nOTvYnM/GZBF1rr58r8BUdbaqVqpqZY8ePY5xyhzRiomSHabbRlnNPRYYjEmPGhEpA3Dvd2Q4PRFF\nKiaKVgQULXDYKKu5xyqfjUmPF4GxwP3u/QuZTU5kfkNlxNpE1a/SOnQ6z0DgGTdoHLOXz6a6tvoY\nfTKTCKtjMCZEvHUMIvIk8DWgO1AD3A0sBJ4GegObgJGqGlpBHSTbjutodQ+R6g6+PfDbPLL8EW4d\ndGuLFk4mM6yOwZhjSFW/qaplqtpOVXup6m9UdZeqXqSqp6rqxdGCQjaKVgTkV3eQJ3nU1df5tnA6\nVlOKmuSlJDCIyBAR+VhE1rlttkNfLxSR+e7rb7stOBCRS0RkuYj8w73/eirSY4xJjUh1D36B44az\nbghb0WzNVXNH0kVJIpIP/BO4BNgKvAt8U1XXeNaZAJytquNFZDQwQlVHicg5QI2qfioi/YGXVfWk\naPvMtiy3aV0Saa6aCrl4XF8z/xrKOpYF1R307NCT2Stm0z6/PUcajiAIDdrQ4r3WXPXYiue4TkXl\n8/nAOlVd7+78KZzOPWs86wwH7nEfPwv8SkREVd/zrLMaKBaRQlU9nIJ0GWNSKLRCGfwrra+Zf01Q\nRfOGvRvoWtTVt67CZKdUBIaTgC2e51uBC8Kto6r1IrIPKAU+86xzLbDCgoIx2SnSkBlefsHitpdu\ns+aqOSQrmquKSD/gp8ClEdbx9hA9RikzxsTabDUSa66aW1IRGLYBJ3ue93KX+a2zVUQKgOOAXQAi\n0gt4HviWqn4SbieqOhuYDU5ZbArSbYyJwfrb14dtthqrYzWlqEmNVLRKehc4VUT6iEh7YDRO5x6v\nQGcfgOuAP6uqikgX4I/AJFX9awrSYoxJsVzruWzNYpOXdGBQ1Xrgu8DLwIfA06q6WkSmiMhV7mq/\nAUpFZB3wA5whiHHf1xf4sYisdG/HJ5smY0xqJTKyaqZO0NYsNnnW89mYENZcNTUm/HFCUO9nv1ZN\nqWSjuEZmPZ+NMRkTbn6Hkx86Oa1X8jaKa+pYYDDGpFToCTqgQRvCTgSUimKnXKsLyWYWGIwxKVXW\nqYz8vHwO1h+kML8QQejbrW/EK/lU1QskO8uccWRFPwZjTOvy1ua3ALjq9KvoUdKDResW+V7Jp6KP\nhJc1i00NCwzGmJQJPdE/s+YZAPIkz7eDWyr6SJjUs8BgjEmZSCf6QFm/90re6gWyk9UxGGNSJpET\nvdULZB/LMRhjUirecZGsXiD7WGAwxqSUnehznxUlGWPSwsYsyl0WGIwxaWFjFuUuK0oyxqRUqvsm\nmGPPcgzGmJTKxjGLrFgrPhYYjDEp5ddkNV/yGfXsqIydmK1YKz4WGIwxKRfaN+HNzW9m5MQcbqRX\n7wB+piWrYzDGpFygyWqm6xtsyI3EWI7BGJM2ma5vsCE3EmOBwRiTNt4Tc2F+IQfrD3Kk/sgxrW+w\nITfiZ0VJxpi0CpyYdx7cyTNrnuG5D59DUaa8MYUZw2akff/WEzt+FhiMMWn1p3V/CqpnaKQRsP4N\n2cyKkowxabX+9vXkif+pJk/y2DBxg/UzyDIWGIwxaVXWqYyqs6p8X7v2jGsZ9ewo7nrtLutnkEUs\nMBhj0qp4ajG//+D3vq89s+YZlmxawtz35/r2MwjNSVjO4tiwwGCMSUisJ+lAk9VAcVJRfhGChF3f\n25w1tMdyvD2YLZAkxgKDMSYhsZ6kA01WAYoKijjSeIQxZ48J6t8ATn2DIHQu7Eyfh/v49liOtwez\nDYWRGGuVZIyJSyK9mf1mdevZoSd1DXXkSR6N2si1Z1xLj5IeVNdWt+ixXJxfTH5+PrVHapu2eWq3\nU1ny7SUpS6NpZjkGY0xcwvVmXnrT0rDFNgtGLWD6sOkMOGEA04dNZ8GoBU3BYsW4FUyonEB9Yz3T\nh01n+tDpjH5uNAV5BdQ1OCf3Qw2HgoICwNrdayn7eRnFU4tbFBllusd1rrPAYEwaichGEfmHiKwU\nkWWZTk8qhBtm4pHlj8RVbOMXLKC5+OfNzW8yftB4Xh3zKqd2O7XF+/PIazrZT1o8iSWbljBp8aSI\nabShMGIjqprpNMStsrJSly1rFf8xk4VEZLmqVqZoWxuBSlX9LNq6uXRcXzP/Gso6ljFu0DgGzR5E\ngza0WMev2KZ6fzWjnxvN/OvmtzhJhxb/eLdz44Abmb1iNoLQoA3kSz6KNj33e8/lfS9vSmOg+Mrb\nC7qtiee4thyDMSZu3qv9Ld/fEnOxTWhlsLcIKFLxT6DYaXD5YPr16Mfg8sGMHzQexf/CtlEbw+ZI\nTHRW+WxMeinwiogo8Iiqzva+KCLjgHEAvXv3zkDykhdLsU24yuDAlX9g3KRw2wk9qQdyHivGreC6\np69j3Z51Qa9/s/830/uhW7mU5BhEZIiIfCwi60Rkks/rhSIy3339bRGp8Lx2l7v8YxG5LBXpMSaL\nfFlVzwUuB74jIl/1vqiqs1W1UlUre/TokZkUpkC0EUxDcwMBDdoQ1Px01vJZdC/uzkvffCniSKiB\nnMcjyx+hXutbvD73/bk2IU8Sks4xiEg+MB24BNgKvCsiL6rqGs9qNwF7VLWviIwGfgqMEpEzgdFA\nP+BEYLGInKbqU2hoTA5S1W3u/Q4ReR44H/BvY5nDoo1gGpqrOFx/mL7d+rLt821NE+ic1Pkk1u5e\ny46DO3juw+d8R171y3kEfL3867yx+Q0atMEm5ElSKnIM5wPrVHW9qh4BngKGh6wzHJjrPn4WuEhE\nxF3+lKoeVtUNwDp3e8bkPBHpICKdAo+BS4FVmU1V5nhzFbdV3sbRxqNNzVEP1h9k7e61TeuG68Dm\nl/PoXNgZQdh+YDuKWiukFEhFHcNJwBbP863ABeHWUdV6EdkHlLrLl4a89yS/nbSGsljT5vQEnneu\ngSgAnlDV/81skjInNFdxzfxrGNp3KCO+OIJxL41jw97myup8yWf46cNb5D7KOpUxf/X8oJZInx/+\nHIA1nzmFFI3ayPhB46murU7nx2nVcqby2a20mw1Os74MJ8eYqFR1PTAg0+nIVt5AcdkplzFr+aym\n5w3aQM+OPX2v+C895VLW7V7H5r2bOdx4uGl5cUEx15xxDQ9e+iAndDyhqcWTX9NYE1kqipK2ASd7\nnvdyl/muIyIFwHHArhjfa4xppQIn7037NtGnSx9GnjmSK0+7ksL8Qjbu3ej7nkVVi7ioz0Uc1aPk\nSz7g5DAONxwOKj6ycZISl4rA8C5wqoj0EZH2OJXJL4as8yIw1n18HfBndXrWvQiMdlst9QFOBd5J\nQZqMMTkgcPKu6FLB+onrmf+N+fTq3IujjUep6FIR9n3efg2nl55Ox/YdGXP2GLbXbqfoJ0UJDbhn\nmiUdGFS1Hvgu8DLwIfC0qq4WkSkicpW72m+AUhFZB/wAmOS+dzXwNLAG+F/gO9YiyRiPefOgogLy\n8pz7efMynaLEeT5L8Y/yfE/esZ7QA53XXhv7Gl/v83X2H9lPSbsSFoxawKh+owCachM2TlL8bEgM\nY0Kkckg+sFBOAAAeZklEQVSMeLQ4rufNg3Hj4ODB5mUlJTB7NlT5z4iWtebNo/r7tzB62CHmP+v0\n+rtjaD4L++VzUI9QUlDCkL5DUJSX173c1IQ10OTUr44g3BAafm6rvM23+WtbYkNiGNMaTJ4cHBTA\neT55cmbSk4zJk7nv/EO81RumDIayWuh8oIG6xiNNzUt7duzZNBR3LE1OQ5uuFhcUU3FcBcX5Tg6j\nQAoY2ncoNw640SbqiVPOtEoyps3ZvDm+5VmqeGoxdd/2dEo7z7nlNcL4d2Hcb5Y2DXIHtJi3IZwW\nneYaDlPSroTDjYedCYEajlDepbzN5xQSYYHBmGzVuzds2uS/PIesv309d3ynLwtPPsjB9lByBEZ8\nBA++AieUloM7yF2o6cOmR21yGqiEHvHFEdzw/A38c9c/GXP2GL7/pe9HDCyRRnk1VpRkTPaaOtWp\nU/AqKXGW54p58yg760I67z5IXQEUHYW6Auh8GE5ojP5ZIo3GCs2V0As+cib+qdd6StqVRB1RNVpT\n1rY+V7RVPhsTImsqn8GpgJ482Sk+6t3bOZHmSsWzp/L8mpFOvcK4FTD7XKg+vpgFFz8a9rOEq1gO\njMZ666BbmTFsRsQK6ND5IKr3V3PSL07yHao7dN0Jf5zAI8sfadpPLLI9FxLPcW2BwZgQWRUYcllF\nhX9RWHk5bNwY8a3V+6uD5nwOpzC/kKGnDuXFj19sGibDO5yG9wQ94Y8TmLVsFqd0O4VPP//Ut+VT\npMmCos0VnUgwOZasVZIxJvMSqDwPFOGISFDFsiD07da3xSQ+G7+3kZ4degaNnRQ6nEbx1OKm/hGK\nsm73uqZgE9ryKZG5or3bby0d6iwwGGPSI1wleYTKc2/Zf+horPWN9b5NWWsO1DQNpzHyzJH06dKH\njXs3hp0ZLl/yObXbqSwes7jFnA+JzBWdSDDJdtYqyRiTHlOn+nfQ86lwDjfPQlFBEdOHTQ8ajTW0\nKatfBXOgWMdvZrgjDUe4+AsXc9EXLuLMHmcy+rnRbK/d3nTyDwSkWJrMQmLBJNtZYDCmNct05XVx\ncXNgKC2Fhx/23f/629cH1Sn4TbQTbTIgCD+FaB55jK9sebL35lAC9QKx7CdUvMEk21lgMKa1Ch1S\nY9Mm5zmkNzjMmwcTJ8KuXcHLD4WvvA131a2qcQ2dvb7XA9zx2p1NfSbyyaOBRsYMGNN0kp8+bHpT\nvUBAIIDEUsnsJ5Fgks2sjsGY1ioTQ2oEglFoUIhh337zRsc1dPa8eZRNuJPOuw9ysJ2zqIFGoOUc\n0K2xXiCVLMdgTGuViSE1/IKR16ZNTjNWnyIt71X3nJVzfIuEIl7Ru/uu6QBjV8KODvDqKVCfT4ui\nqdZYL5BKlmMwJpfEMwx3Aq2CkhZL0AkUaUVIe+CKPjB0NsCp3U6NfEXv7nvB0/C7F6B8HzSK29va\n58Tvl0MxDgsMxuSKQDHNpk2gGv0Em+ohNWIJSrEGnSjFSl+Y9gWeWPVEUP+EtbvXUvbzsvD9A0L2\nvek46FkLL712vO+JPzCcRrThM9oiCwzG5Ip46wyqqpy5G8rLQcS5HzvWWT/eiX9iDUp+wSicCLmL\n9bevp6xDGYIELb/mi9eEzzWE7LtiH9R0hOdGntXixN/Wx0KKxgKDMbkikTqDqipn+InGRufEOXdu\n7DkOr1iDUlWVE3zy3SKg/Hzo0MF/mxFyF2Wdyhj+xeEtxjXy9mhuwQ2Exf8tyD3O0N6NeTDz89da\n9ET2VmpbkGjJAoMxuSLZOoNkWinFGpTmzXOCT4NbBNTQAIcP+7936NCIu/Tr0Rz15F1Vxfr/3Ba2\nxZHf8BUn/uJElmxaErXlU1sKINYqyZhcEUdPYl/JtFIKNzdEt25OkVSgA11tbcvgU1/vv81FiyLu\nMtEy/xYtjuoP0fnpFzjhuidYf/qJ3HFzbxbWvd9icL6mznCSx7YfbGuRM/HrDNdaWY7BmFzhV2cQ\nz/zPyeQ4/OoO2reHzz8PLpry678QThqbzTa1OCr7EeNX5LFda0GVso+20fnNd6irP0RhfiFAU8un\nkoIS+nTpQ6M2MmnxpKZttcZB8qKxwGBMLvHWGWzcGF8P5kitlMK1OAosHzPGGd6itLQ5KHXqBEeP\nJv5ZAgEpnia4MWpqcTRlNtNfbGDB082v1RQ2MP7DDrx989v0696vqeXTwfqDbNjrVGx7O8S1xc5w\nVpRkTFsRCCKhYyeB/9AZf/2rU18QWL5rlxNIfv97Z1t5MV5Xtm/vFCc1NjYva9euOSClc9gOn1zJ\ngqcBOQBPDeC07qcxuGIwv37v1xxpONJiXVVtk53hLMdgTFvil+MIVyk9e3bkyupwRVClpc4toH17\nJ5fhFXie7mE7evemuiMMvhG2dwxeDs05i40TN9K3a9+gtx5XeBwbv7cRaHud4SwwGNPWhSvrb2jw\nXx5YP1zR1MiRwQPm1da23NaRI805l3jSFK+pU7nvonze6g1TBnvSGFJhX9apjHV71gUt23d4X1OH\nukid4VpjayULDMa0deGu/PPz/Zfn5TlFQOEqwxctijxeUkCgOCueNMUgcKIu+kkRsu4GZp7T4PRn\nOA/kHii+86hvMdXJnU8GIB/nc+dJXkx1CXEN9JcjLDAY09aFu/IfN86/F3NDQ3PHOL+iqViv9gN1\nHKkctoPmE/WofqP8K41/EJy+QKujLZ9vcT4eTu6mURsj1iW05tZKFhiMaevCXfnPmOHc++UcItUD\ndOsWfZ+Bk3+yTXA9Qk/Uj33wGE+seoKDRw9SVI/Tn8HnRB/a6qhAChjadyg3DrgxYvFQa26tZIHB\nGBO+GWxVVXBrIi+/nMG8eU7fhlAFBcFNXb0n/2Sa4Hq0OFEfhV574caVsPRRnP4MH77b4n2hrY4a\naaS8Szm/vfq3ETvZtebWStZc1RgTWbhez371AJMn+/dtOO44+Oyz1KfNI+hEXQ91+XDlWpjxR+f1\n6S82wPs74Y6W7010as7WNqVnE1XNudugQYPUmHQBlqkd180ef1y1pETV6d/s3EpKnOWhRILX896O\ngRFPjdAJL03QlSegE4aiI0aGpEEkPTt+/HHV8nJn++Xl/t9NDD79/FP96m+/qtX7q1OaPNX4juuk\nipJEpJuIvCoia937rmHWG+uus1ZExrrLSkTkjyLykYisFpH7k0mLMSZN4qkHCNeaSCT5Hs3hekhP\nmOAUVYkw/dYXWPW35+nZpRfTFxHU4zli+pJNVzzzZESQNS2cYo0gfjfgZ8Ak9/Ek4Kc+63QD1rv3\nXd3HXYES4N/cddoDbwKXx7LfrL2yMq0ClmNI3OOPR841xHM17b0K79Ch5bZEVM88M2jZbcPQvB+j\nt12Zp9quXcRcTsquzsvLw3/WGBX9pEi5hxa3op8UJZc2j3iO62Qrn4cDc93Hc4Grfda5DHhVVXer\n6h7gVWCIqh5U1b+4wekIsALolWR6jMkqIjJERD4WkXUiMin6O3JcVZVzWgxn0ya44QYnBxG46vfL\nCYRehR840HJbqrBmDQDFkwmeg2FQIzL5KMX/TdhcTsquzlPQSS/bWjglGxh6qmqgtmU70NNnnZOA\nLZ7nW91lTUSkC3Al8FqS6TEma4hIPjAduBw4E/imiJyZ2VQdA+Xlsa0XCBI33NCyGGbixNg6ybnW\nPwzXfwAl7nBHJUeg6gPY8Eua+0u4QSFq/4N4B/VLQSe9bGvhFDUwiMhiEVnlcxvuXc/NqkS4VAi7\n/QLgSWCaqq6PsN44EVkmIst27twZ726MyYTzgXWqut7NFT+Fk8tu3eKZ3tPPwYPxDd8NlNVC58NQ\nVwBFR537zofhhFpalPlHvDpPpL4gTCe96nvviGuojGwajylqYFDVi1W1v8/tBaBGRMoA3PsdPpvY\nBpzsed7LXRYwG1irqr+Mko7ZqlqpqpU9evSIlmxjskEsueXWd8Hjraw+hmo6wPhlsPTXzv1274yi\nng55ZS/+mc7PvkjdEZ+Ob4kM6hemcv6+7mviKqqKNB7TsSYaqTww2ptFHgB2qer9bvlpN1X9z5B1\nugHLgXPdRSuAQaq6W0R+ApwBfENVw/SiaamyslKXLVuWcLqNiURElqtqZQq2cx1OfdrN7vMxwAWq\n+l2/9VvlcT1vnlNUlA1EnCHDx43jmisOUlYL45bD7Avyqf7KOUy/9UVG334i8591cxqh7w3X0S9E\n8dRi6urrWiwvKiji0ORDPu84NuI5rpOtY7gfuERE1gIXu88RkUoR+TWAqu4G7gPedW9T3KDQC5iM\nU/a6QkRWisjNSabHmGwSLbfc+lVVQYcO0deLJnQo70Tk5TlB6uBBFjwN0xfBgBqn49uCX+10KqPL\n4c6Lww/THYtsq0hORFI9n1V1F3CRz/JlwM2e53OAOSHrbAVCBmk3plV5FzhVRPrgBITRwPWZTVIG\nxFGJ7Eukudd0RYV/L+xYhBlGvGgyHG63CZbNBIHHBjrLe30f6u8j7kH9Eq5Injev5SRKqZisKAE2\nVpIxaaKq9cB3gZeBD4GnVXV1ZlOVAeGuttu3Dz+0d7j3T53actIf8J+MJ0ajVuE0mwkpVW/I9x+m\nO5b5F+KuSE5hJ7lUSKqOIVNaZVmsyRqpqmOIV6s9rkOn7wwoKHCm/IwkUC/gvXKeMAFmzQrqLzFh\nGDwyCG5d3jw2UjTFk6GuXfjXSwpKGHHGCB689EFOeOG1pqv5CSM78MgZB7i1cjwzhs2IbWfRhMsJ\nlZc7AwumwLGsYzDGtEWhbf0nTAjf9r+qCsaObXmlHy0ogHPyDy1OmTHDCRb5+S07tgUm4wk0Iiop\nCVs3sf5hGLEG8j0lTOLGGkGai4BeeA3GjaP4hk3I3crMM2ppRFM7/0K6Z7KLkwUGY0x8/Io9Zs6M\nXAyyaFHkHtHhhBtjqaoK5s5l/bQwHdsedterq3P6RPjMOV1WCz0PQEMeTUVJCvTr0Y/3bn2vuQjI\nbcLaohNdvVDV8V/Y8Hj32DvDhZOGmeySYYHBGBMfv7b+oULb/id65avavJ3QXApQtj9CxzZobmKq\n2hwcysth/HgoKaGmA/TZAyNXw8hV0GcfnPbGagZ8aTjT9/6L05fATXuLTnR5Suc33+aEj7YmXy+Q\nhpnskmHzMRhj4hPrSd67Xrg5HWKxaRN07+5MABSY6yFwEi4tpabDLsYvc/skDHIqon2pOkFh6tSm\n4NZi9NXmncK3v90i7YFOdOOWw+xKqO4Q0tIpEBDjbU0UWD9LWiVZ5bMxIazyOYpYm4x6K07DVUAn\nq7QUDh2Kb7slJbGv36EDPPJIy7RH2kYcneGOJat8NsakTyxjIYUWg4QOG5Equ3cHbzda89f8/PiC\nSGBUV3cf1Z1g8PhCts/4WfghP3r3jn8gvixjgcEYEx+/sYFuuy3yRD6hnbfC9WIuLY1vjKXevYPn\njJ471+kf4ad9+7Cd3CIKFA1t3Mh9T97GW2VHmdJ9dfh6gaFDs6pPQiKsKMmYEFaUlGJ+xUjt2zsn\nTe/80O3aQefOTi4gLy/6SbykxH8mue7d/UdnLS2Fjh3jr+sQofi+Qv/xj6Qdh+acGFwvMHky1bs2\nMfo6gsddSmGfhERYUZIxJnv4tWI6csQJAoFcRmmpc79rlxMwwgWFDh2iTy+6e7f/e3fvTmxI8N69\nW4x/BDD01KFs+MHm5tzKxo1OejZv5r7B8FZvmDLYs50M9UlIhAUGY0x6hTsh7t7dfFLt2NEJFtEc\nPep0bguchP2Ea/uf557uQovBInHrSrzjH4k7xNvft/y9xfhHxVOLnU5wfh3uMtQnIREWGIwx6RVL\n561Yr6aPHIk8NwKEzxU0NDhFWhB8lR8uOOTnUz3jpww+OpvttduZtXwWjdqIuoMq7anbg9wryL3N\nlenrb1/P9R0vpMQtIWvqcDe7OGN9EhJhgcEYk16xdN6K52o6WhAJVI77tVDym3QnXPrmzg2abOe9\nce9RflxwEKnoUsH7499vel7WqYzOpw+grp04kwAVQOd2HTnhoUcz1ichERYYjDHpFWaGs6ATZTxl\n/5GCSKCZ6Jgx4espNm8Obk46ebIzlpMnfcV3HkXW3RA0L/Q5s89h877goNShXQfO7nl20LKaAzWM\nr7yNpd9ZyfjzJ7B9xCU5FRTAWiUZ04K1SsoQb5PWSJ3EHn/c/0Qbayc6v05xIS2cqvdXc8crd7Dw\no4UcrD/YNNLqq6v/QI/PDvLj1+qZclEBu3t05NP/3pPgBz62rFWSMSb3ePsjPPaYf3+E224Lf/Ud\nyxhOgVyJ37zOgRFgCwoo63yiMy90/aHmyXY2bafm/npWTatn5GpYNa2eT//nSFz9E2KZyyEbWGAw\nxmSfqiqYMye4+Onxx50ht8OJVPfgLcIK15w1UPTk3tc01jLmH3mcmXcCY84ew/YP/uYfUELrLCL0\ner5vyX1NdRbZzIqSjAlhRUk5KlzHttCOZXFMDzphGDxSCbeedxszrpzlP3S4t9jLrzirpITiO49S\np0dbvLWooIhDkw/FlJZkWVGSMaZtmTcP9u9vubxdu5bNRGOo6A6aAEhwJuW5W5snAPLyVob7FWcd\nPMj6J493mrHWO01bm+ZymLgh+mfLAAsMxpjcN3myfwe5zp1b1klEas7qajEpj7SnapU0TwAUkJ8f\nHHjCFGeVfbSNzm++Q12eBs/l8MJrMXy4GKVw4D4LDMaY3Bepd7WfqqqIQ2MHTcoj7ahrPELnQ9o8\n7lFAaJPYcE1p8/OpKWxg/DJY+mtnTofthQ3RO+vFym9WvSQG7rPAYIzJfYlMjRmlU11Nj2LGd7mI\npePeZfy7sL1DmBUnTmx+HK6zXEMDC56G6YtgQI1zv+BpUjd+UpgirEQDjwUGY0zuS2RqzKlTnTqI\nUO3bw+OPs+DiR5n+8DoGnHgO01/ODz/bm7fCO1xnvkhzN6RCuACTYOCxqT2NMbkvkakxA69NnNh8\nci8thYfdigRv66J45nGoqvLfr98scKkaPync1KkJBh7LMRhjcpt3GAyIPvqqV1UVfPaZUy6v6jyu\nqoqts1xAuEmHQvcTbViQZCSSY4rAcgzGmNwV2m8gUOkKyZ10Yy2Cad++OYcRTbicRCokkmOKwHIM\nxpjcleJK1ybhimACU48GrvrnzMmeAfK8Q4rEmmMKwwKDMSZ3pbjStcnQof7LR45M2ck3m1lgMMbk\nrkSaqcZi0aL4licihR3SUs0CgzEmd6W40rVJunIiASnukJZqSQUGEekmIq+KyFr3vmuY9ca666wV\nkbE+r78oIquSSYsxpg1KV2ufdOVEAtJUN5KqYb2TzTFMAl5T1VOB19znQUSkG3A3cAFwPnC3N4CI\nyDVAaEdzY4yJTQorXZukKycSkKYcSaqG9U42MAwH5rqP5wJX+6xzGfCqqu5W1T3Aq8AQABHpCPwA\n+EmS6TDGmNRJd7+DFOdIiqcWI/dK0FSkcq9QPLU4oe0lGxh6qmq1+3g70NNnnZOALZ7nW91lAPcB\nPwei9iQRkXEiskxElu3cuTOJJBtjTAzSkRMJcHMk1R1h8I2wvSPx5UhCKq7X93qA6/tfT0mBk8sp\nKSih6qyqhIf1jhoYRGSxiKzyuQ33rqfOjD8xz/ojIgOBU1T1+VjWV9XZqlqpqpU9evSIdTfGGJN9\n3BzJfcM68lZvmDKsY+w5Ep+K67IJd9J583bqGuqapyIt7MwJHU9IKHlRez6r6sXhXhORGhEpU9Vq\nESkDdvistg34mud5L+B14EKgUkQ2uuk4XkReV9WvYYwxrVjx1GLq6uvgDOf5zDNqmbnuBoqm3hx9\nRrcwFdc1H/yd8deNZ9ygccxePpvq2mr/98cgqak9ReQBYJeq3i8ik4BuqvqfIet0A5YD57qLVgCD\nVHW3Z50K4CVV7R/Lfm0KRJNONrWnSbfq/dXc8codLPxoIQfrD1JSUMKIM0bw4KUPRr/Kz8uLPsWo\nj2M5tef9wCUisha42H2OiFSKyK8B3ABwH/Cue5viDQrGtEYico+IbBORle4tTFda0xaVdSqjc2Hn\nxIp+0t2UliQH0VPVXcBFPsuXATd7ns8B5kTYzkYgptyCMTnkIVV9MNOJMNmp5kAN4wclUPQzdWp6\nh/DGRlc1xpiMWDBqQdPj6cOmx/7GFI+k6seGxDAmfb4rIh+IyJwIowJYM2wTv3Q2pcUCgzEJi9KU\neyZwCjAQqMbpr9OCNcM22ciKkoxJUKSm3F4i8ijwUpqTY0zKWI7BmDRw+/UEjABskEiTMyzHYEx6\n/Mzt3a/ARuDWzCbHmNhZYDAmDVR1TKbTYEyirCjJGGNMEAsMxhhjglhgMMYYE8QCgzHGmCAWGIwx\nJgelan5nPxYYjDEmB6Vqfmc/1lzVGGNySNMkP66Zy2Yyc9lMigqKok/yEyPLMRhjTA5Zf/v6lM7v\n7McCgzHG5JCkJvmJkQUGY4zJMYFJfpbetJTxg8anvALa6hiMMSbHJDzJT4wsx2CMMSaIBQZjjDFB\nLDAYY4wJYoHBGGNMEAsMxhhjglhgMMYYE0RUNdNpiJuI7AQ2hXm5O/DZMUxOONmSDsietGRLOiBy\nWspVtcexTAxEPa5TLZt+i1SwzxNdzMd1TgaGSERkmapWWjqaZUtasiUdkF1pyYTW9vnt86SWFSUZ\nY4wJYoHBGGNMkNYYGGZnOgGubEkHZE9asiUdkF1pyYTW9vnt86RQq6tjMMYYk5zWmGMwxhiTBAsM\nxhhjguREYBCRbiLyqoisde+7hllvrLvOWhEZ61n+uoh8LCIr3dvx7vJCEZkvIutE5G0RqUhnWkSk\nRET+KCIfichqEbnfs/6NIrLTk8abw2x3iPtZ1onIJJ/Xw34mEbnLXf6xiFwW6zYjfBcJpUVELhGR\n5SLyD/f+6573+P5WaUpHhYgc8uxrluc9g9z0rRORaSIisX4vuUJEHnCPxQ9E5HkR6ZLpNMUr0WM3\nG4nIySLyFxFZ454fJmYsMaqa9TfgZ8Ak9/Ek4Kc+63QD1rv3Xd3HXd3XXgcqfd4zAZjlPh4NzE9n\nWoAS4N/cddoDbwKXu89vBH4VZd/5wCfAF9z3vw+cGctnAs501y8E+rjbyY9lm2lIyznAie7j/sA2\nz3t8f6s0paMCWBVmu+8AXwIE+FPgd2pNN+BSoMB9/FO/Yzmbb4keu9l6A8qAc93HnYB/Zurz5ESO\nARgOzHUfzwWu9lnnMuBVVd2tqnuAV4EhcWz3WeCiGK4ME06Lqh5U1b8AqOoRYAXQK8r+vM4H1qnq\nevf9T7npieUzDQeeUtXDqroBWOduL5ZtpjQtqvqeqn7qLl8NFItIYUzfQArTEW6DIlIGdFbVper8\nSx/D/3fOaar6iqrWu0+XEt+xmA0SPXazkqpWq+oK9/F+4EPgpEykJVcCQ09VrXYfbwd6+qxzErDF\n83wrwV/qb93igh95TgpN73H/IPuA0mOQFtxs+5XAa57F17rZ+mdF5OREtkv4zxTuvbFs008yafG6\nFlihqoc9y/x+q3Slo4+IvCcib4jIVzzrb42yzdbm33FyRrkk0WM367nFnecAb2di/1kztaeILAb8\nZrOe7H2iqioi8baxrVLVbSLSCXgOGINzFZiJtCAiBcCTwDRVXe8u/gPwpKoeFpFbca5wvx5uG62B\niPTDKcK41LM4rt8qSdVAb1XdJSKDgIVumlqNSMeyqr7grjMZqAfmHcu0GX8i0hHn2P+eqn6eiTRk\nTWBQ1YvDvSYiNSJSpqrVbjZ/h89q24CveZ73wimvRlW3uff7ReQJnCzoY+57Tga2uifr44Bd6UyL\nazawVlV/GVigqrs8r/8apy7Db7venEQvd5nfOkGfKcp7o23TTzJpQUR6Ac8D31LVTwJviPBbpTwd\nbjHRYXd/y0XkE+A0d31vsUqs30nWiXQsg9PoAbgCuMj9PnJJLL99ThGRdjhBYZ6qLoi2ftpkusIl\nlhvwAMEVvj/zWacbsAGnkrer+7gbTvDr7q7TDqeMebz7/DsEV0o+nc60uK/9BOeHzwt5T5nn8Qhg\nqc92C3AqsvvQXNnWL2Qd388E9CO48nk9TuVd1G2G+R6SSUsXd/1rfLbp+1ulKR09gHz38RdwTiqB\n3ym08nlopv8HafhfDQHWAD0ynZYE05/QsZutN/dYewz4ZcbTkukExPiFleKUxa8FFnv+vJXArz3r\n/TtOpeo64Nvusg7AcuADnIrOhz0ngyLgGXf9d4AvpDktvQDFqVRa6d5udl/7Hzd97wN/Ab4YZv9D\ncVorfIJTHAAwBbgq2mfCKQr7BPgYTysbv23G+LsklBbgv4EDnu9gJXB8pN8qTem41t3PSpyGAFd6\ntlkJrHK3+SvcUQJa0839PrZ4foNZmU5TAp8hoWM3G2/Al93zwwee3yQjFyQ2JIYxxpggudIqyRhj\nzDFigcEYY0wQCwzGGGOCWGAwxhgTxAKDMcaYIBYYjDHGBLHAYIwxJsj/BzGqsf2PNfcNAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f92804281d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, batch 2499. Mean loss: 16.9325145943\n",
      "Epoch 1, batch 2499. Mean loss: 6.24987681103\n",
      "Epoch 2, batch 2499. Mean loss: 2.31109857569\n",
      "Epoch 3, batch 2499. Mean loss: 0.858666448605\n",
      "Epoch 4, batch 2499. Mean loss: 0.323071002489\n",
      "Epoch 5, batch 2499. Mean loss: 0.125603744188\n",
      "Epoch 6, batch 2499. Mean loss: 0.0527891687471\n",
      "Epoch 7, batch 2499. Mean loss: 0.0259436405713\n",
      "Epoch 8, batch 2499. Mean loss: 0.0160523827007\n",
      "Epoch 9, batch 2499. Mean loss: 0.0124009371101\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXl8lOXV978nYUlYlSgQtsSiKIgsEqw+WqlVK+61KqIB\nl1rzYGtL+9a12uoD8ta32ipWwNLWuqWIC1oXrEufWkpb1GBBAUGQNRhCiOx7kvP+cd2TTCYzyeyT\nmZzv53N/Zu5l7uvMzD2/+5pznescUVUMwzCMtkNWqg0wDMMwkosJv2EYRhvDhN8wDKONYcJvGIbR\nxjDhNwzDaGOY8BuGYbQxTPgTjIgUioiKSLsoXz9ARPaISHa8bYsHIlIsIm/H+1ijKSLyNRFZlWo7\ngiEiXxeR8lTbEQki8p6IfDfEPhGRP4rIdhH5IMl2vSki1yWyjYwVfhFZLyLnpNqOWFHVjaraRVVr\n431uEXlSRO6P5RyqWqqq34z3sZmEdy3u927gvuWxMF6nInKsb11V/6GqxyfIxpivhQzjDOBcoJ+q\nnpKoRkTkPhF51n+bqp6vqk8lqk2AqHqhRnIQkXaqWtNW288wLlbVd1NtRCYhItmJ6BB5FADrVXVv\ngs6fWlQ1IxdgPXBOiH03AWuAL4FXgT7edgEeBrYCu4BPgKHevguAFcBuYDNwa4hzZwMPAduAtcD3\nAQXaBbMLuA941nte6B17I7ARWOC3zff694CpwD89W94GjvI737XABqAa+FmozwEoAQ4Dh4A9wGt+\n9t0BfAwcxHUO7gQ+99pbAVzmd57rgYV+6wpMAlYDO4AZgERxbDbwK+9zXAfc4v85pNPSwrV4LPB3\nYKf3Xud62xd473ev9/1cBXwdKA84723ed7UX+APQC3jT+67eBY70O/4FYIvX1gLgxBauhT7AS0CV\n9x380O9cucCTwHbvmrjN37aA9zgLeChg25+B/+M9vwP3m9oNrALODnGeJ71zzffe7zlAR9zvbSNQ\nCTwO5HrHHwm87tm/3Xvez+987wHfDdLOjcABoNb7PP4n8Nr1u36P9bNtBvCG9z7eBwb6HXsi8A5O\ncyqBnwJjvc/8sNfO0kC7cF6Ze3C/6a3A00D3AL24znv/24C7w7omU/2jSPaPDfiG9wGd7F00vwEW\nePvOAxYDR+BuAoOBfG9fBfA1vwvq5BDtTgJWAv2BHsDfiFz4nwY6435chTQV/s+BQd7+94AHvH1D\nvAvoDKAD7gdxONjn4Hex3h/kc1vi2e/7AV2JE4EsnADt9ftcrqepmL/ufYYDcD+6sVEcOwknKP28\nz/tdMlP45wB3e59tDnBGwOdzrN/612kq/ItwYt8XJwwfASO9c/0vcK/f8d8BuuKu+0eAJaGuBc+e\nxcDPvWvpK7iOzHne/geAf+Cu8f7AMkIL/5nAJhpu6kcC+71r6nhvn6/zVYifYAa5XncCp/t9Xg/j\nOm89vPf2GvAL7/g84HKgk7fvBeAVv/O9RxDhD3GtNloP/H4826qBU3CdpVLgOW9fV5x+/MSzuSvw\n1cDffzC7vO9sjff5dwHmAc8E6MXvcFowHNdZG9zSNZmxPv5mKAaeUNWPVPUgcBdwmogU4kSyK3AC\n7iL9VFUrvNcdBoaISDdV3a6qH4U4/zjgEVXdpKpfAr+Iwsb7VHWvqu4Psf+PqvqZt/95YIS3/Qpc\nb22hqh7C/WijScb0qGf/fgBVfUFVv1DVOlWdi+uhN+f3fEBVd6jqRtyNb0QUx44DpqtquapuxwlN\nOvOKiOzwW27yth/GuRX6qOoBVV0Y4Xl/o6qVqroZJ8Tvq+p/VPUA8DLuJgCAqj6hqru96/4+YLiI\ndA9x3tHA0ao6RVUPqepanMCM9/aPA6ap6pequgl4tBkb/4G7Dr/mrV8B/FtVv8D1qjviflvtVXW9\nqn7ezLn+rKr/VNU6nMiVAD/27NgN/F+fjaparaovqeo+b980YEwz546Vl1X1A3Xu0VIaruWLgC2q\n+ivvO96tqu+Hec5i4NequlZV9+D0anxAsMj/qOp+VV0KLMXdAJqlLQp/H9zfJgC8D7Ma6Kuq/ws8\nhvvLtlVEZotIN+/Qy3Hung0i8ncROa2Z82/yW98Q4rjm2NTC/i1+z/fhegJN2lbVfbj3FlP7InKt\niCzxiRYwFDgqCvsiOTbwc2zpM2ntfEtVj/Bbfudtvx337/IDEVkuIt+J8LyVfs/3B1nvAs4fLiIP\niMjnIrIL928BQn+PBUAf/5sVzj3Ry9sf9nWurnv6HHC1t+kanDCiqmuAH+FuRFtF5DkR6RP67TZq\n82hcb36xn41/8bYjIp1E5LcissF7zwuAIxIYIRfqWu6P+5ceDY30ynvejobvobl2Q9IWhf8L3EUN\ngIh0xv0l3Aygqo+q6iic22QQzneJqn6oqpcCPYFXcD3tYFTgvmgfAwL278VdrD56BzlHtClTK3Cu\nEQBEJBf33kIRqp367SJSgOvp3QLkqeoRuL/1EqWN4dLovdD4M80YVHWLqt6kqn2A/wZm+kfyxJFr\ngEtxfvHuODcBNHyPgdfCJmBdwM2qq6pe4O1v6ToPZA5whXc9fRU3duAaVv2Tqp6B+10q8P+aOY+/\nndtwN7cT/Wzsrqo+4fsJzpX0VVXthnM5+b/nSGj0uxWRYL/bUGzCuWqC0dJvvZFe4T7nGhrf4CMm\n04W/vYjk+C3tcBfgDSIyQkQ64v4avq+q60VktIh8VUTa477oA0CdiHTwYtC7q+ph3MBvXYg2nwd+\nKCL9RORI3MCoP0twf9Xai0gR7m9vvHgRuFhE/ktEOuB6Uc1d5JWEviB9dMZdnFUAInIDrsefaJ4H\nJotIXxE5AjcAmHGIyJUi4rvBbcd91r5rK5zvJ1y64lwj1TgB+78B+wPb+gDYLSJ3iEiu949hqIiM\n9vY/D9wlIkd69v+gucZV9T84of498Jaq7gAQkeNF5Bveb/EATshD/bYCz1mH65Q8LCI9vfP1FZHz\n/N7zfmCHiPQA7g3nvCFYCpzo6UYO7rcVLq8D+SLyIxHpKCJdReSr3r5KoFBEQmnxHODHInKMiHTB\nfW9zNcZou0wX/vm4L9633KcupO5nuB5HBTCQBr9lN9yFtJ2GyJgHvX0TgfXeX8ZJON9bMH4HvIW7\nUD7CDcb48zOvze24aIE/xfQO/VDV5bgf4HO497YHN+B3MMRL/oDzre4QkVdCnHMFLrrm37iL9CRc\nRFGi+R0uYulj4D+477IG5xNOR14LiON/2ds+GnhfRPbgBikne/50cOLylPf9jIux/adx1/Rm3KD5\nooD9ja4FdWGSF+H81OtoEG3fmMD/eOdbh/uengnDhj/h/nH4X/MdceM323Aui544P3a43IEb/Fzk\n/TbfxfXywQ1g53rnXoRzA0WFqn4GTPHOvxoIeyzGG184F7gY9x5XA2d5u1/wHqtFJNi44RO4z3YB\n7rM+QAs32XDwjbIbGYjXQ9gBHKeq61JtTyyIyPnA46pa0OLBhmE0S6b3+NscInKxN6jVGRfO+QkN\nA3lpg+deuEBE2olIX9zf9Jdbep1hGC1jwp95XIobEPoCOA4Yr+n5t05w7oTtOFfPp7jwVMMwYsRc\nPYZhGG0M6/EbhmG0MVpM0iYiT+BG97eq6lBv21waRs6PAHaoapPZmSKyHpe3ohaoUdWicIw66qij\ntLCwMJxDDSNiFi9evE1Vj052u3ZdG4kkkus6nOycT+Jmsz7t26CqV/mei8ivcPkzQnGWqm4Lxxgf\nhYWFlJWVRfISwwgbEYlmNnXM2HVtJJJIrusWhV9VF3h5bII1JLicHd8It0HDMAwjtcTq4/8aUKmq\nq0PsV+BtEVksIiUxtmUYhmHEgVgLsVyNm1IcijNUdbM3nfodEVmpqguCHejdGEoABgxoKe2HYRiG\nES1RC7+X9+bbwKhQx3ipYlHVrd4U9VNwU4+DHTsbmA1QVFRkMaaGkQEcPnyY8vJyDhw4kGpTMoac\nnBz69etH+/btoz5HLD3+c4CVqhq0wLI3czRLVXd7z7+Jy3VhGEYboby8nK5du1JYWIgbEjRiQVWp\nrq6mvLycY445JurztOjjF5E5uARdx4tIuYjc6O0aT4CbR0T6iMh8b7UXsFBEluIy/b2hqlEnSTKM\nTKVidwVjnhzDlj1bWj44zThw4AB5eXkm+nFCRMjLy4v5H1Q4UT1Xh9h+fZBtX+CKleBlGGyxEoxh\ntGlKS5n62iQWHr+HKd89jpkXPw7FoRK/picm+vElHp9n+szcXbMG7rkHyoN6lgwj7cid0gFZM4FZ\ng/dQlwWzBu9B1kwgd0qHVJtmZDjpI/xbtsC0abB8eaotMYy4sHZOT675GDodcuudDkHxx7BuTq/m\nX2hERHZ2NiNGjKhfHnggdPnmV155hRUrVtSv//znP+fdd9+N2YYdO3Ywc+bMiF9333338dBDD8Xc\nfiDpI/y+qe7r16fSCqMNIiJPiMhWEVnmt+0+EdksrhbxEhG5oLlzBCN/1Rd0OwgH2kHOYffY7SD0\nXrU5vm8gnSgtdb/1rCz3WFoa8ylzc3NZsmRJ/XLnnYFF8RoIFP4pU6ZwzjnnxGxDtMKfKNJH+PPz\noV072JCS2fZG2+ZJYGyQ7Q+r6ghvmR9kf/MMGEBlZ5hUBot+7x63dHbb2ySlpVBS4n7jqu6xpCQu\n4h+MO++8kyFDhjBs2DBuvfVW/vWvf/Hqq69y2223MWLECD7//HOuv/56XnzxRcCl3LjrrrsYMWIE\nRUVFfPTRR5x33nkMHDiQxx9/HIA9e/Zw9tlnc/LJJ3PSSSfx5z//ub6tzz//nBEjRnDbbbcB8OCD\nDzJ69GiGDRvGvfc2VIWcNm0agwYN4owzzmDVqlUJee+oaqtbRo0apUH5yldUr746+D7DCBOgTCO8\nJnHFyZf5rd8H3BrJOZpc188+q9qpk6qTObd06uS2ZwgrVqwI/+CCgsafhW8pKIjJhqysLB0+fHj9\n8txzz+m2bdt00KBBWldXp6qq27dvV1XV6667Tl944YX61/qvFxQU6MyZM1VV9Uc/+pGedNJJumvX\nLt26dav27NlTVVUPHz6sO3fuVFXVqqoqHThwoNbV1em6dev0xBNPrD/vW2+9pTfddJPW1dVpbW2t\nXnjhhfr3v/9dy8rKdOjQobp3717duXOnDhw4UB988MEm7ynY5xrJdR3rzN3kUlBgPX6jNXGLiFwL\nlAE/UdXtEb3aF71z992wcaPr6U+blnFRPWGzcWNk28PE5+rxp6amhpycHG688UYuuugiLrroorDO\ndckllwBw0kknsWfPHrp27UrXrl3p2LEjO3bsoHPnzvz0pz9lwYIFZGVlsXnzZiorK5uc5+233+bt\nt99m5MiRgPunsHr1anbv3s1ll11Gp06dGrUXb9LH1QPO52c+fqN1MAsYiCtGXoErSN8EESkRkTIR\nKauqqmp6QHGxu6br6txjWxV9CO3iSoDrq127dnzwwQdcccUVvP7664wdG8yT15SOHTsCkJWVVf/c\nt15TU0NpaSlVVVUsXryYJUuW0KtXr6Ax96rKXXfdVT/usGbNGm688cYmxyWK9BL+ggKoqICDB1Nt\nidHGUdVKVa1V1Trgd7h0JMGOm62qRapadPTRSS8BkF5MmwZeT7eeTp3c9jizZ88edu7cyQUXXMDD\nDz/M0qVLAejatSu7d++O+rw7d+6kZ8+etG/fnr/97W9s8DwUgec977zzeOKJJ9izZw8AmzdvZuvW\nrZx55pm88sor7N+/n927d/Paa6/F8C5Dk16unsJC5/XbtAmOPTbV1hhtGBHJV9UKb/UyYFlzxxth\nkCDX1/79+xkxoqFO1NixY5k8eTKXXnopBw4cQFX59a9/DcD48eO56aabePTRR+sHdSN7C8VcfPHF\nnHTSSRQVFXHCCScAkJeXx+mnn87QoUM5//zzefDBB/n000857bTTAOjSpQvPPvssJ598MldddRXD\nhw+nZ8+ejB49Oqb3HopWWXO3qKhIgxaseO89OOssePddOPvspNtlZAYisljDrAbnHT8H+DpwFFAJ\n3Outj8ClHl8P/LffjSAoIa/rDObTTz9l8ODBqTYj4wj2uUZyXadfjx/Mz28kFQ2etuQPSTfEMOJE\nevn4+/Z1EztM+A3DMKImvYS/fXvo189COg3DMGIgvYQfXGSP9fgNwzCiJv2Ev7DQevyGYRgxkH7C\nX1DgUjMfPpxqSwzDMNKS9BP+wkI3y3FzG85gaBhG2PjSMg8dOpSLL76YHTt2RH2uwsJCtm3bFkfr\nUkN6Cj+Yn98wMpR4l6L05epZtmwZPXr0YMaMGXE5bzqTfsJfUOAezc9vGBnJ1AVTWbhxIVP+PiXu\n5z7ttNPY7OctCJUa+Vvf+hajRo3ixBNPZPbs2XG3I9Wk1wQugP79QcR6/IaRYeROy+VATUNCs1ll\ns5hVNoucdjnsv3t/zOevra3lr3/9a30ytLfffpvVq1fzwQcfoKpccsklLFiwgDPPPJMnnniCHj16\nsH//fkaPHs3ll19OXl5ezDa0FtKvx9+xoyvKYj1+w8go1v5wLdcMvYZO7Vyitk7tOlF8UjHrJq+L\n6by+XD29e/emsrKSc889F2icGvnkk09m5cqVrF69GoBHH32U4cOHc+qpp7Jp06b67ZlCi8IfS9k5\nERkrIqtEZI2IhK53FimWntkwMo78rvl069iNA7UHyGmXw4HaA3Tr2I3eXXrHdF6fj3/Dhg2oar2P\nP1Rq5Pfee493332Xf//73yxdupSRI0cGTa2czoTT43+SKMrOiUg2MAM4HxgCXC0iQ2Ixth4ryGIY\nGUnl3komjZrEohsXMWnUpLgN8AJ06tSJRx99lF/96lfU1NSETI28c+dOjjzySDp16sTKlStZtGhR\n3GxoLbTo41fVBSJSGMW5TwHWqOpaABF5DrgUWNHsq8KhsBBeeAFqayE7O+bTGYbROph31bz65zMu\njH/0zciRIxk2bBhz5sxh4sSJQVMjjx07lscff5zBgwdz/PHHc+qpp8bdjlQTy+BuS2Xn+gKb/NbL\nga+GOpmIlAAlAANaqrhTUAA1Na4oS79+UZhuGJlDxe4Kxr80nrlXzI3ZLZKJ+Hr0PvyLm0yePJnJ\nkyc3ec2bb74Z9FzrM8TFHO3gblhl5yIhokpFFstvZCqlpVQM7seYG4Qtg/tDaWmLx0+9aRAL1y1g\nynePa/l4wyBK4Q+z7NxmoL/fej9vW+xYLL+RiZSWQkkJUwduZuEAmDKwHEpKQop57pQOyJoJzBq8\nh7osmDV4D7JmArlTOiTZcCPdiEr4RSTfbzVU2bkPgeNE5BgR6QCMB16Npr0m+ITfevxGBpG7ciJy\n+z5mjcYJ+WiQ2/eRu3Ji0OPXzunJNR9Dp0NuvdMhKP4Y1s3plUSrW6Y1VvlLZ+LxeYYTzjkH+Ddw\nvIiUi8iNwC9F5BMR+Rg4C/ixd2wfEZnvGVcD3AK8BXwKPK+qy2O2GCA3F3r2tB6/kVGsna7BhfyR\n4D/0/FVf0O0gHGgHOYfdY7eD0HtV68ljlZOTQ3V1tYl/nFBVqqurycnJiek84UT1hF12TlW/AC7w\nW58PNAn1jAsWy29kGPk9Cuh2cENTIc8rCP6CAQOo7LyBSWVQshhmj4KKLm57SigtpeL+Oxh/6mbm\nLupH73seoN+4cZSXl1NVVZUamzKQnJwc+sUY1JJ+KRt8FBTAkiWptsIw4se0aVS+cS2TyuoahLx7\nFkybFvL4eSUlsG8fADPmA506wewQxycS3/jEWfvc+ERVOTNLSmgPHFNcnHx7jGZJv5QNPgoLYeNG\nl6LZMDKB4mLmXfg0M5YXMHyrMGN5AfMufBpCCWdxMcye7TpBIu5x9uzQxyeQSMcnjNSS3sJ/8CBU\nVqbaEsOIH8XFzoVZV+ceWxLxSI9PEJGOTxipJX2F30I6DaPV4MYnggw0hxqfMFJK+gq/TeIyjNbD\ntGlUdstiUhks+j1MKoMt3ZoZnzBSSvoKv/X4DaNVULG7gjGHZzPzvEfDH58wUkr6RvV06QJ5edbj\nN4xUUlrK1NcmsfD4PUxZ9REzpz1uYp8GpG+PHyw9s2EEEmmunxiwlBHpS3oLv03iMowGIsz1Eyvp\nkjLCaEp6C7+vx2/TwQ0j6bH06ZAywghOegt/YaGbtbhtW6otMYyUk/RY+gEDqOxM40iezqQuZYQR\nNuk7uAuNI3tayuFvGBlOxLl+YqU1pYwwIiL9e/xgfn7DgOTH0reilBFGZGROj98w2jrFxcwDuPtu\n2LqRGcsHONGPsxDXl3qUcegDv2D8Wb5snPFvy0gM6S38RxwB3btbj98wfBQXJ1Z8/eP2Fy+AgTTK\nxllvg9GqSW9XD1gsv2HEixbmADSJ2x+NZeNMU9Jf+C2W3zBiJ4w5AIFx+9l1kF3rnls2zvQic4Tf\nYvkNI2rCmQMQGLdfK1CbZdk405H0F/6CAti9G3bsSLUlRoYiIk+IyFYRWea3rYeIvCMiq73HI1Np\nYzhU7K5gzJNj2LJnS5N9Yc0BCIjbP2a7WywbZ/qR/sJvIZ1G4nkSGBuw7U7gr6p6HPBXb731UlrK\n1JsGsXDdAqZ897gm/vuw8ulPm8a81zsxYz4Mr4S1j8La3wjDK+Gef2ZR3bGOLfffmdD8QEZ8aFH4\nQ/R2HhSRlSLysYi8LCJHhHjtehH5RESWiEhZPA2vx0I6jQSjqguALwM2Xwo85T1/CvhWUo2KgLCS\nqYUzByBY3P6kSdCpE1PPqEtKfiAjPoTT43+Spr2dd4ChqjoM+Ay4q5nXn6WqI1S1KDoTW8B6/EZq\n6KWqFd7zLUDQzGQiUiIiZSJSVlVVlTzr/AgrmVq49X4DSj3m5j1utXbTkBaFP1hvR1XfVtUab3UR\n0C8BtoVHjx7QubP1+I2UoaoKBI0uUNXZqlqkqkVHpyitSEvJ1Op9/5eeHXH9Xqu1m57Ew8f/HeDN\nEPsUeFtEFotISXMnibpnJGIhnUYqqBSRfADvcWuK7QlNc8nUmvP9h5Hb32rtpicxzdwVkbuBGiCU\nQ+8MVd0sIj2Bd0RkpfcPogmqOhuYDVBUVBRZd8EmcRnJ51XgOuAB7/HPqTWnGUIkU8u9rZwDaybA\nYHfYrMF7mLVmAjlTbmD/wD+6uP6z9jWdmQtU3H8H40/dzNy6I6nsKkwqU0oWw+xRUNHdontaO1H3\n+EXkeuAioNj7q9sEVd3sPW4FXgZOiba9ZrEev5FARGQO8G/geBEpF5EbcYJ/roisBs7x1lsnIZKp\nrZ3bO6TvP3Rc/4TGE72GbWfevPbMeD/Pau2mEVH1+EVkLHA7MEZV94U4pjOQpaq7veffBKZEbWlz\nFBTA9u2waxd065aQJoy2i6peHWLX2Uk1JBaC5PDJnziRbgOD+/7XTlduPRdeOQH2dXA3hctWwotD\n3A3Ah0vbcIiONdV8tRzmLqqld7LfmxEx4YRzBuvtPAZ0xblvlojI496xfURkvvfSXsBCEVkKfAC8\noap/Sci78EX2mLvHMMKnGd9/KN/9ukcI+i9h/CdYOGca0WKPP0Rv5w8hjv0CuMB7vhYYHpN14eIf\ny3/SSUlp0jDSnhYKqVS+cS2Tyuoa+e7zOx5Jt4PV9TeEfe2hdFjDKd0/gH3krJzIfszd01pJ/5m7\nYLH8hhEN/r5/nLCPGbfPzb6F4HH906c3muh13RLot8PCOdON9M7H76NnT8jJMVePYUSKz+9fUsLU\nMwIieGbPDtqZ8i/28uTrWdw8tpbZoyycM53IjB6/L1LBevyGETHhZOZshP/s3aeeSm65RyMuZIbw\ng3P3WI/fMCImptm34aZ6MFoVmSP81uM3jKiIefZtQP6eRJd+bGk2sdEymSP8hYVQVQV796baEsNI\nL8LJzBlIKgQ4jCphRnhkjvD7Qjo3bkytHYaRbkTqrgkmwN/5DhUDeiT0RhDxWIQRkswRfgvpNIzo\nicBdE1SAf3qI/jdsT2hP3DKBxo/MEX4ryGIYSSFQgH0JqWuzQvTE4+QWskyg8SNzhD8/H9q3tx6/\nYSSY/B4FZNe5Wbsda0AUjt0WoiceT798NGMRRlAyYwIXQFaWyy9uPX7DSCwXXMDC9rMAuGQlHL0P\n5h8bvCeeu3IiB25vcMXElNKhuLjR5LEZywc40bfQ0YjJHOEHS89sGAkmd0oHDvQ6XL/+wlD3mFXn\neuCBOfnX3jwhaJbPh95WmBqFAUGyjBqRkzmuHrCCLIaRYELV7938xyODRgWZX751knk9/ooKOHDA\n5e4xDCOu5K/6IngO//IdUPdl0xdMmxY0y6f55VNLZgm/L7Jn0yY47rjU2mIYmciAAVR23tDYrdPF\nbQ+K+eVbJZkl/P6x/Cb8hhF/WsjhHxTzy7c6Ms/HD+bnN4xE4ZfDv6IrjJnUkS0zf2nCnmZkVo+/\nb1/IzrbIHsNIJJ7IT31tEgt77mHKm3cys90RJv5pRGb1+Nu1g/79rcdvGAkkd0oHZM0EZg3e42bq\nDt6DrJlA7pQOqTbNCJPMEn6w9MyGkWBChXSum9MrNQZZquaICUv4ReQJEdkqIsv8tvUQkXdEZLX3\neGSI117nHbNaRK6Ll+EhsYIshpFQ8ld90SQ2P7sWrjq1nC17tiTXGEvVHBXh9vifBMYGbLsT+Kuq\nHgf81VtvhIj0AO4FvgqcAtwb6gYRNwoKYPNmOHSo5WMNw4icAQOo7EyjnDn/KMAJ73ePS6roWqrm\n6AhL+FV1ARA4O+NS4Cnv+VPAt4K89DzgHVX9UlW3A+/Q9AYSXwoLXWrZ8vKENmMYbZZp05j3eidm\nzIdTvwszT4F1PUiJv99SNUdHLD7+Xqpa4T3fAgRz8PUFNvmtl3vbmiAiJSJSJiJlVVVV0VtlIZ2G\nkVj8QjrXTiel/n5LCREdcRncVVWlPit31OeYrapFqlp09NFHR38iK8hiGInHK9ySv1fqhbfjYZeq\n+ZB4/v5kDLRaquaoiEX4K0UkH8B73BrkmM1Af7/1ft62xNGvH4hYj98wkoGfv/+SVW7TS0PgH8ka\naI20bKQBxCb8rwK+KJ3rgD8HOeYt4JsicqQ3qPtNb1vi6NDBTeSyHr9hJJ5p03jzOOfnf2EoIFCX\nDZrMgdbGMF3fAAAeQ0lEQVQIykYajnDDOecA/waOF5FyEbkReAA4V0RWA+d464hIkYj8HkBVv8Rl\n3f7QW6Z42xKLpWc2jORQXMza6S4ffzCy6txAa8XuCsY8OSb54Z5GUMJK2aCqV4fYdXaQY8uA7/qt\nPwE8EZV10VJYCP/8Z1KbNIy2Sn7HPIo/ruaZ4d4GoX7E7/LlcNUVcMy1fVg4zIV7zrz4ceuVp5jM\nm7kLrsdfXg41Nam2xDAyntybq3lmBE7wxdvoPb4wFBYUwlMjgoR7Bs64/d73bAZukshM4S8sdKL/\nxReptsQw0pMI0iD4Qjqzat16zmFXgL3RjcCHeuGev+/SdMbtrFnhz8C1NA0xkZnCb7H8hhE9EaZB\n8MXSI070D2XDxKWN4/tR5+8XdXH2x1y7vemM2/sIbwaupWmImcwUfovlN4yoiTgNQpBY+t0dqI/v\n9w38Xr4Cbi6DLZ1pMvEr9xB0OUDDbCCF47YFn4FraRpiJzOF31cGznr8hhExodIgLJqtwV0rwWLp\n826uvxl89Fv43odQk+Uqds14E8ZfAe1q3Y0Bhf0dYE8OjcYIVh8F+bdCzs+kUbuWpiF2MqsQi4+c\nHOjd23r8RsIRkfXAbqAWqFHVotRaFDvOdbOhSRqE3472ErFVlTOzpMQd7IvOCVJecV7p6a7WbuUG\nZryVDbW1IMLUM5WFA2BTN/fv4LJP4XsXwuo8GiKCxI0ZXL3c3SCeHg4n9yjno0kTyNfOdDu419I0\nxIC4bAuti6KiIi0rK4vtJKedBl26wDvvxMcoI2MQkcXxEmhP+ItUdVtLx8bluk4GpaV8+41ryd9Z\nR8liGFUCtdlND8upEfZPbRzAX7G7gvEvjWfuFXPp3aV3o325P8viQLumepNzGK5f4gq3C1ArkF0H\ntVk0HRz2jj//cyF/lzYUfO+e1eZn7EZyXWemqwesIIthREuA62bTwyESsQW6VkpLmXrTIBauW1Cf\nntl/4lZIF8106t1CY9bBiVthzAa4bglBM4AdaA9vHquWpiEGMtPVA26A9+WX3TTurMy9vxkpR4G3\nRUSB36rqbP+dIlIClAAM8I09pQN+rpv8wsKgrh9/10rulA4c0MMw2K3PGryHWWsmkP2Q+4CmfPc4\nZnbIo9vB6qDnmXfhNJg8GaqrAajo4sYBrlgOL57oNeK5gY6rhgVPArvXJ+nDyDwyVxELClwxli02\nRdxIKGeo6snA+cD3ReRM/51xyzqbSsLIgBlYjtHXU6/N8pu49YNqHi+Ca5cGOU9xMWzbBjff7MYB\nxrjxhMV9qPf5A40Gfa3Gb/RkrvBbSKeRBFR1s/e4FXgZV2kuswgjA2ZgOUbxeua+G0F2LfU3g9wa\nGF4JM97Pa3Ke3LzHkXu1PlRzXQ+cSqlL+wxu0DelNX4zgMwVfpvEZSQYEeksIl19z3HZZ5c1/6o0\npaUMmAHlGG8ug8Pi8vODNzjszeT1TdbKvfnLJucJHAdAodt+9/Sgd1NBPBfRqsRmeM9kMl/4rcdv\nJI5ewEIRWQp8ALyhqn9JsU2pwa8c4/BKF68/cosboP3G514KB4/sWvj2iuBx9/k9Cph7IuzzeXEE\nduVSf9M40N493dKZhvk6RsRkrvB37gxHHWU9fiNhqOpaVR3uLSeqatst++RXjhERyMtj3isdePLP\nMOhLz8vj1emrzYJee0PE3U+bxjc/d7N2fa6dehfRYefiKf81zHulAxV1uyxXT5RkrvCD8/Nbj98w\nkoO/O2jbNrjxRiq6wgtDYMBOGLcMxi13z58fAlvqdjUV7OJi5r+Vx9nr4HC2NzaAi+s/mO25ePZl\ngSpTh223XD1RktnCbwVZDCM1lJbCU08x9UzYngsXroa5L8HcF93z7bkwZdj24II9fXp9FNGp5c7f\nf+pGLwqoq5BzVx1y92HL1RMDmS38hYVO+Fvh7GTDyGSCJlK7L8wMnMXFzDviv5nxpjBsq/PrD6ty\n4wbznheuWgZow78By9UTOZkv/Pv3Q1VVqi0xjIwmsLRisFm6314Bl60IL7laYFhn/Y3j53U8PRKQ\nhjQS+9pbrp5IyWzht8gew0g8QVI1+HL0+8/S7bXXLeEkVwvnxtGuFi74zOX5CZxQZjRP5qZsgIZJ\nXBs2wCmZN6/GMFJNqFQNOddnc/5nWUwqq2tIpNbNTb+dVNY4uVowwQ6WIbTXXhfg49t2KBsKdsLM\nv2TDU09Zrp4IiLrHLyLHi8gSv2WXiPwo4Jivi8hOv2N+HrvJEWA9fsNIKIGpGurdN3Pzm872vegZ\n5l30TMO29/OYsbA7Y96d0DQk0y9NxGt/gl57YFUPFyE0MTDlQ6DoW1nGFola+FV1laqOUNURwChg\nH27KeiD/8B2nqlOibS8qunWDI4+0yB7DSBCBqRrq3TerNgef7evb9swzsH9/k5DMiqcec2MFl57t\nbhzv5zFvCFR2ccv2XOjUTMqHsMoy2o0hbj7+s4HPVbX1KaylZzaMxBGQqmFSWXizakOVT+y/9gf1\nYwW5q69DflBdf8zyXo0HehulfPDEPOuzCc2XZYy2Xm+G3SziJfzjgTkh9p0mIktF5E0ROTHEMYhI\niYiUiUhZVTyjcHwhnYZhxJ8gqRrmvd6pxYHWYDl5oHE2zwNSS1adm7Hrf0xuYDSQn5gDHLstdORQ\nVPV6M7C4e8zCLyIdgEuAF4Ls/ggoUNXhwG+AV0KdJ2Hpa309fovlN4z4E5iqoaDArQcbaPXrNYtk\nNcnmGUywJ3zsZuxme4W+smtdsjb/aCB/MdcsWHOUl+tHm0YORVOvNxOLu8ejx38+8JGqVgbuUNVd\nqrrHez4faC8iR8WhzfApLIQ9e2D79qQ2axhthpYyd0LTXvPX6ppk86zJajpWsLsDTSpzTSqD9d1h\nzFkb2DK4P2sfaSzm2XUu18+7TzWtHRAszLSlOQCZWNw9HuGcVxPCzSMivYFKVVUROQV3o6mOQ5vh\n4x/Z06NHUps2DMORu3IiB25vEMpZo91jzmGY8aYw460svt2llgvW0BDq2QXmvSBB/61/70L47Siv\n8PsqodtBbRTmec46OHs9DNmexfjL6thy/530Bhct9Ma1jcNMQ4SU+ghVfD6dJ4zF1OP3cpCfC8zz\n2zZJRCZ5q1cAy7y0tY8C4zXZ1d2tIIthpJzm6u1SVwdPPRV8rGDSpEZupNx7gqR9uFeZPSpggLmr\nQPv2TD2jrrFfHlosKtOEMCqQpRsx9fhVdS+QF7Dtcb/njwGPxdJGzFhBFsNIOaF6zZqdxZgbhLmL\n+tH7uutg/nzYuNFFBflKMvqxtptw67nwygnOj59d6waEJy51/xxQV4Q9d+IG5O7D9a+bNRpmjd5H\nzsqJ7J9aF9lkr+Ji17O9+27YupEZy4Pblk5kdsoGcHH8Xbtaj98wUkmwXnNnGvfIn3rKCWozYwU+\nH31gZa+nRrqef+49AuvXs3Y68fXLhzOOkUZkvvD7Ig2sx28YqSOgbu8TI+HlIcEydU5o/jzeDeS6\nJXD+Zy5fDzQV9mgGcdsSmS/8YAVZDKM14Ndr9vXIfamVUReJs+4Rmo+P924gT76eTcFOqJMQwp6B\nfvl4ktlJ2nwUFsLCham2wjAMj69Mdnn26xFYfRTk34rzw9O8K6WiTzdeGLKdiUvhx4uCROdkoF8+\nnrSNHn9BAezYATt3ptoSwzCAtdMhfydInd9GDV2EvR7ffIBh20Pn7fFNFHt3Altya11eoAzwy8eT\ntiH8/umZDcNIOfl5BVz6GahQX4QdminC7hGqsld93p4g6RUqbp6QMTl24kXbEH5Lz2wYrYtp06js\nKhyz3RVgH7cMjtnuxd8344dvaRZtsBtDn5/AgoJmcuxkWAK2cGg7Pn6wHr9htBZ8PvjJk6Ham8yf\nlwfTpzfrkmlpFu3a6doozr8eaYjlz/psApsH30nvex5w+0pKmHrWPvcPoaqcmd5Er0x2DbWNHv9R\nR0FuLqxbl2pLDMPwUVwM27a5lAyq7nlLYttCtI5/GGfHGpoUZT+m2kUC3Xm86/3nrmwhjXOG0jaE\nXwRGjID33ku1JYZhxELAfIAmKRf8bgzv/84ldqvNAtT9A1iXR8OEr9v3oRrniV5pQtsQfoArroD/\n/AfWrEm1JYZhxEJzs2j9bwyVMKgavvchdKgNcS6hTU70alvCD/BCsLIBhmGkJcEGZn03BlXmXfIs\nM5YXsP4ROLaa+ughFLrvh/WP0CYnerUd4R8wAE491YTfMDKFcCpjeTeB/D2wpgcg3naBnbluwtib\ngyS06yhDI37ajvADjBvn3D2rV6faEsMwoqRidwVjnhxDTrgDs6WlIEL/nTQa7M2q9fz5c/ODu44y\nsOSij7Yl/ObuMYz0prSUqTcNYuG6BVz1SXgDs7krJyL3KpuOAMTL6Im7WXQ7CL1XbQ7aVCaWXPTR\ntoS/f3847TQTfsNIQ3KndEDWTGDW4D3UZcHTI+FPw1yK5uYGZgMnfbWrhQs+g+uXuNTQDBgQtL1M\nLLnoo20JP8CVV8KSJebuMYw0Y+2cnk2EuN8OJ+DNDcwGpmiuEyjYCX/8s1flK8RAbiandm57wm/u\nHsNIS/JXfdFEiC9e7QS82TKKIYrAkJcHs2eHnjSWyamdVbXVLaNGjdKEctppqsOHJ7YNo9UClGkm\nXteZTkGBXjYO/d4F6JJe7vGycagWFLT82mefdceJuMdnnw2vzWhfF+JcX5zQV8+8Hq04oV9s5wpC\nJNd128jVE8i4cfDjH8Nnn8GgQam2xjCMligtpaJuF9WdYOZ86L3HFWSnUyeYHUYPvLg4utw70b4u\nEF+EUCvJCRSzq0dE1ovIJyKyRETKguwXEXlURNaIyMcicnKsbcaMuXsMI33wy8G/cABMGeNtb85V\n08ri71tbhFC8fPxnqeoIVS0Ksu984DhvKQFmxanN6OnXD/7rv0z4jZgRkbEissrr2NyZansykRZz\n8AfSCuPvW1uEUDIGdy8FnvbcUIuAI0QkPwntNs+VV8LSpc7dYxhRICLZwAxc52YIcLWIDEmtVZlH\npKLZ2nrX0PoihOIh/Aq8LSKLRaQkyP6+wCa/9XJvW2oxd48RO6cAa1R1raoeAp7DdXSMOBKpaCal\ndx2pK6mVRQjFQ/jPUNWTcb2e74vImdGcRERKRKRMRMqqqqriYFYL+Nw9zz+f+LaMTKXFTk3Sr+tM\nJELRTHjvOhpXUkvppJNMzMKvqpu9x63Ay7hekD+bgf5+6/28bYHnma2qRapadPTRR8dqVniMGwcf\nfwyrViWnPaPNkZLrOtOIVDSD3Si6ChV1u+Iy2Bu1K6m5dNJJJibhF5HOItLV9xz4JrAs4LBXgWu9\n6J5TgZ2qWhFLu3Hj8svdo7l7jOgIq1NjxIFIRDPwRvF+HvNealcfFXTHCeWMeXcCWwb0iOoG0NoG\naqMh1jj+XsDLIuI7159U9S8iMglAVR8H5gMXAGuAfcANMbYZP/r1g9NPd8J/zz2ptsZIPz4EjhOR\nY3CCPx64JrUmGUBD/H1pKTmfTuBg+4ZdT49wj/2u305NFLH0LdX9TQdi6vF7g1rDveVEVZ3mbX/c\nE328aJ7vq+pAVT1JVZvE+qeUK680d48RFapaA9wCvAV8CjyvqstTa1WGEUs8fmkp3HADVy3DhaAE\ndMhrswNcNOG21coGaqOh7eXqCcSie4wYUNX5qjrI69ikzy8/HYgxHj935UTk7sM8PRJXgEUa72/k\noomkrVY2UBsN4lI8tC6Kioq0rCyJfwzOOAN27XI9fyPjEZHFISYbJpSkX9dpTu7PsjjQrqk+5dQI\n+6fWtfj6im7C98+HV49vyMEv6jr+ou4+8N+LYeayAnInboyprdZAJNe19fjBRfd88gmsXJlqSwzD\n8Ih1EDW/RwG99kJtFvWuHgVO3Ar/+W1jF00mDNhGggk/WHSPYbRCYo7HnzaNyq7CMdth3HIYtwyO\n2Q6DqmF4JY1cNK1tZm2iaZvZOQPp29e5e55/Hn72s1RbYxgGOOF+41omldVRshhmj4KK7hEMohYX\nMw9g8mSornbb8vJg+nSYGyRnfyxtpRvh5m9O5pKSvOXTp6uC6ooVyW/bSCpYPv70IZ758FtTWwkg\nkuvaXD0+Lr8cRMzdYxitiSTPdq3IrWHMdcqW3NqEtpNqTPh9+Nw9JvyG0fZohamcE4kJvz9XXgnL\nlsGnn6baEsMwkkhE+XdaWZGXaDDh98fcPYbRJvGFc+YedutZdfDtFUHCOTPkn4EJvz99+jRE9xiG\n0WbwhXPubwco1AmsyoPee6WRqLfGIi/RYMIfyLhxsHw5rFiRaksMw0gkfi6bnIkbeHw0DakdBJb3\nArlXG4l6pkz0MuEPxNw9hpH5BLhsrvoELlsB2X7ZGbJrm7p7MmWilwl/IPn58LWvmfAbRgYT6LJ5\neiS8PARqhfr0DrVZ0GtvgKhnQGZOMOEPzpVXmrvHMDKYYC6bfjug/w6X2mHccpfeYUtnGot6BmTm\nBBP+4Ji7xzAymmAum4tXw8bpMPclmPsirP2NMC/v5qaiHmpSWRqFeZrwB8Pn7rHoHsPITLwEbhOX\nwpCtMHGpq8tLXp7r9BUUwDPPwMyZ4Z0vzcI8TfhDMW6cc/Ust4JKhpGJzHupHZ1qYEk+dKpx60yf\nHlV6iHQL8zThD4W5ewwjY/FV52ok1Hcfjlqo0y3M04Q/FL17w5lnmrvHMDKQeAt1uoV5mvA3x5VX\nurw95u4xjIwi7kKdZmGeUQu/iPQXkb+JyAoRWS4ik4Mc83UR2SkiS7zl57GZm2R87h7r9RtGZhFv\noU6zMM9YKnDVAD9R1Y9EpCuwWETeUdXA4Pd/qOpFMbSTOnzunhdegPvuczcBwzDSH191rrvvhq0b\nmbF8gBP9WIS6uLjVCn0gUff4VbVCVT/ynu8GPgX6xsuwVsO4cebuMYxMJMlFXloTcfHxi0ghMBJ4\nP8ju00RkqYi8KSInNnOOEhEpE5GyqqqqeJgVH779bYvuMQwjo4hZ+EWkC/AS8CNV3RWw+yOgQFWH\nA78BXgl1HlWdrapFqlp09NFHx2pW/OjdG8aMgTlz4NChVFtjGIYRMzEJv4i0x4l+qarOC9yvqrtU\ndY/3fD7QXkSOiqXNlHDLLbB6NXznO+5voWEYRhoTS1SPAH8APlXVX4c4prd3HCJyitdedbRtpozL\nL3cDP6WlcOutoK1zUoZhGEY4xBLVczowEfhERJZ4234KDABQ1ceBK4CbRaQG2A+MV01T1bzrLtiy\nBR5+2OXyue22VFtkGIYRFVELv6ouxNWqae6Yx4DHom2jVSECjzwClZVw++3Qsydcd12qrTIMw4gY\nm7kbCVlZ8PTTcPbZcOON8MYbqbbIMIy2QJxTPpvwR0rHjjBvHgwf7lI6LFqUaosMw8hkEpDy2YQ/\nGrp1g/nzoU8fuPBCN8HLMAwjASQi5bMJf7T06gVvvQXt2sF550F5eaotMgyjNREn90wiUj6b8MfC\nwIHwl7/Ajh0wdixs355qiwzDaA3E0T2TiJTPJvyxMnIkvPKKm+B18cWwf3+qLTIMI8XE1T2TgJTP\nJvzx4BvfcPU5//UvGD8eampSbZFhGCkkru6ZBKR8NuGPF+PGwaOPwquvwqRJNrs3wxGR+0Rks1+t\niQtSbZPReoi7eybOmURjmblrBHLLLW6C1/33u+Ru99+faouMxPKwqj6UaiOMVsi0aVS+cS2Tyuoo\nWQyzR0FF99ZTkcuEP95MmeJSO0yb5iJ/fvCDVFtkGEaySUShlzhiwh9vRGDWLKiqgsmTXWqHq65K\ntVVGYrhFRK4FynDV6JqEdYlICVACMGDAgCSbZ6SUVlyRy3z8iaBdO5e///TTYeJEePfdVFtkRIGI\nvCsiy4IslwKzgIHACKAC+FWwc7TaOhNGm8Z6/IkiN9cN9J55Jlx2GfziF+4m0L17qi0zwkRVzwnn\nOBH5HfB6gs0xjLhhPf5EcuSRboLXiSc6X3+fPnDTTbB4caotM2JERPL9Vi8DlqXKFsOIFBP+RNO3\nr0vk9uGHcM018Kc/QVERnHIKPPEE7N2baguN6PiliHwiIh8DZwE/TrVBhhEuJvzJoqgIfvc72LwZ\nfvMb2LfPpXbu2xd++ENYvjzVFhoRoKoTVfUkVR2mqpeoakWqbTKMcDHhTzZHHOHi/T/5BP7xD5fd\n87e/haFD3XjAnDlw8GCqrTQMI4Mx4U8VInDGGS5pU3k5/PKX8MUXzh3Urx/ccQesXZtqKw3DyEBM\n+FsDRx/tavh+9plL9fy1r8GvfuWyf44dC7Nnw9//7iaGWSoIwzBixMI5WxNZWfDNb7pl82b4wx/c\nuMBbbzUc07UrDBoExx/f+PG449w+wzCMFohJ+EVkLDAdyAZ+r6oPBOzvCDwNjAKqgatUdX0sbbYZ\n+vaFn/8c7rkHNm1y/wZWrWp4/Ne/3HiA/z+APn2a3hS+8hUXVtqtm5tbIJK692QYRniUllJx/x2M\nP3Uzcxf1o/c9D8R1FnDUwi8i2cAM4FygHPhQRF5V1RV+h90IbFfVY0VkPPD/AMtfEAlZWVBQ4JZz\nz228b/9++PzzhhuC76bw4otQXd30XNnZ7gbgW7p3b7weuL1rV8jJgfbt3dKhQ8PzUNv817PMk2gY\nEeMr4nLWPlfEpaqcmSUlbl+cxD+WHv8pwBpVXQsgIs8BlwL+wn8pcJ/3/EXgMRERVXNUx4XcXBcN\nNHRo033V1e5GsG4d7NoFO3e6x8ClstIVkfGtx7OQjIgT/2gWkYbFd65Inl99tfvHZBhpRu7KiRy4\nvUEiZ42GWaP3kbNyIvtJvfD3BTb5rZcDXw11jKrWiMhOIA/YFngyS2YVZ/Ly4LTT3BIJhw/D7t0N\nN4qDB922w4fh0KGG5+Fsq6lxrqi6uugWX/9ANfLnffrE/hkaRgpYO1259Vx45QTY18EVcblsJTz0\ntsLU+LTRagZ3VXU2MBugqKjI/hGkivbtoUcPtxiGkXRcEZcNca2xG0gsTtjNQH+/9X7etqDHiEg7\noDtukNcwDMMIRgJq7AYSS4//Q+A4ETkGJ/DjgWsCjnkVuA74N3AF8L/m3zcMw2iGJBRxiVr4PZ/9\nLcBbuHDOJ1R1uYhMAcpU9VXgD8AzIrIG+BJ3czAMwzCaI8FFXGLy8avqfGB+wLaf+z0/AFwZSxuG\nYRhGfLFAa8MwjDaGCb9hGEYbw4TfMAyjjWHCbxiG0caQ1hhdKSJVwIYgu44iyKzfJGFtZ067Bap6\ndILOHZJmrutEkMprJhHY+2mZsK/rVin8oRCRMlUtsrYzv+1UvudMINM+P3s/8cVcPYZhGG0ME37D\nMIw2RroJ/2xru820ncr3nAlk2udn7yeOpJWP3zAMw4iddOvxG4ZhGDFiwm8YhtHGSBvhF5GxIrJK\nRNaIyJ1JbLe/iPxNRFaIyHIRmZystr32s0XkPyLyepLbPUJEXhSRlSLyqYhEWMorprZ/7H3Wy0Rk\njojkJKvtTEJEHvS+v49F5GUROSLVNkVKqn73iSDVWuJPWgi/X2H384EhwNUiMiRJzdcAP1HVIcCp\nwPeT2DbAZODTJLbnYzrwF1U9ARieLBtEpC/wQ6BIVYfiUn5bOu/oeAcYqqrDgM+Au1JsT0Sk+Hef\nCFKtJfWkhfDjV9hdVQ8BvsLuCUdVK1T1I+/5bpwA9k1G2yLSD7gQ+H0y2vNrtztwJq6eAqp6SFV3\nJNGEdkCuV7WtE/BFEtvOGFT1bVWt8VYX4arkpRMp+90nglRqSSDpIvzBCrsn/QMTkUJgJPB+kpp8\nBLgdqEtSez6OAaqAP3pupt+LSOdkNKyqm4GHgI1ABbBTVd9ORtsZzneAN1NtRIS0it99IkiBljQi\nXYQ/5YhIF+Al4EequisJ7V0EbFXVxYluKwjtgJOBWao6EtgLJMW/KiJH4np1xwB9gM4iMiEZbacj\nIvKuNxYSuFzqd8zdODdDaeosNXwkW0uCEVMFriQSTmH3hCEi7XFfVKmqzktSs6cDl4jIBUAO0E1E\nnlXVZIhgOVCuqr7eyIskSfiBc4B1qloFICLzgP8Cnk1S+2mFqp7T3H4RuR64CDg7Detdp/R3nwhS\npCVNSJcef31hdxHpgBvsezUZDYuI4Hzdn6rqr5PRJoCq3qWq/VS1EPd+/zdJoo+qbgE2icjx3qaz\ngRXJaBvn4jlVRDp5n/3ZpGZwO+0RkbE4V+Elqrov1fZEQcp+94kgVVoSjLTo8Ycq7J6k5k8HJgKf\niMgSb9tPvXrDmcwPgFLvB7cWuCEZjarq+yLyIvARzj3xHzJvun6yeAzoCLzjNIdFqjoptSaFT4p/\n94mg1WiJpWwwDMNoY6SLq8cwDMOIEyb8hmEYbQwTfsMwjDaGCb9hGEYbw4TfMAyjjWHCbxiG0cYw\n4TcMw2hj/H8d6GtrAUFKvQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f928051e8d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "############################################\n",
    "#    Re-initialize parameters because they \n",
    "#    were already trained in the first loop\n",
    "############################################\n",
    "w[:] = nd.random_normal(shape=(num_inputs, num_outputs), ctx=model_ctx)\n",
    "b[:] = nd.random_normal(shape=num_outputs, ctx=model_ctx)\n",
    "\n",
    "############################################\n",
    "#    Script to plot the losses over time\n",
    "############################################\n",
    "def plot(losses, X, sample_size=100):\n",
    "    xs = list(range(len(losses)))\n",
    "    f, (fg1, fg2) = plt.subplots(1, 2)\n",
    "    fg1.set_title('Loss during training')\n",
    "    fg1.plot(xs, losses, '-r')\n",
    "    fg2.set_title('Estimated vs real function')\n",
    "    fg2.plot(X[:sample_size, 1].asnumpy(),\n",
    "             net(X[:sample_size, :]).asnumpy(), 'or', label='Estimated')\n",
    "    fg2.plot(X[:sample_size, 1].asnumpy(),\n",
    "             real_fn(X[:sample_size, :]).asnumpy(), '*g', label='Real')\n",
    "    fg2.legend()\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "learning_rate = .0001\n",
    "losses = []\n",
    "plot(losses, X)\n",
    "\n",
    "for e in range(epochs):\n",
    "    cumulative_loss = 0\n",
    "    for i, (data, label) in enumerate(train_data):\n",
    "        data = data.as_in_context(model_ctx)\n",
    "        label = label.as_in_context(model_ctx).reshape((-1, 1))\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            loss = square_loss(output, label)\n",
    "        loss.backward()\n",
    "        SGD(params, learning_rate)\n",
    "        cumulative_loss += loss.asscalar()\n",
    "\n",
    "    print(\"Epoch %s, batch %s. Mean loss: %s\" % (e, i, cumulative_loss/num_batches))\n",
    "    losses.append(cumulative_loss/num_batches)\n",
    "            \n",
    "plot(losses, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion \n",
    "\n",
    "You've seen that using just mxnet.ndarray and mxnet.autograd, we can build statistical models from scratch. In the following tutorials, we'll build on this foundation, introducing the basic ideas behind modern neural networks and demonstrating the powerful abstractions in MXNet's `gluon` package for building complex models with little code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "[Linear regression with gluon](../chapter02_supervised-learning/linear-regression-gluon.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}