ee046746_tut_03_04_convolutional_neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "---\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 03-04 - Convolution & Deep Learning\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_conv_cnn.png\" style=\"height:200px\">\n",
    "\n",
    "* <a href=\"https://mc.ai/how-does-convolutional-neural-network-work/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "\n",
    "* [2D Convolution](#-2D-Convolution)\n",
    "* [Convolution-based Classification](#-Convolutuion-as-Feature-Extractors-for-Classification)\n",
    "* [Convolutional Neural Networks (CNNs)](#-Convolutional-Neural-Networks-(CNNs))\n",
    "* [Regularization](#-Regularization---Preventing-Overfitting)\n",
    "* [Data Augmentation](#-Data-Augmentation)\n",
    "* [CIFAR-10 Classification with PyTorch](#-Building-a-CNN-Classifier-for-CIFAR-10-with-PyTorch)\n",
    "* [The CNN Story](#-The-CNN-Story)\n",
    "* [Other Applications of CNNs](#-CNNs-Applications-in-Computer-Vision)\n",
    "* [The Problem with CNNs](#-Are-CNNs-the-Holy-Grail?-The-Problem-with-CNNs)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import os\n",
    "\n",
    "# pytorch\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/layers.png\" style=\"height:50px;display:inline\"> 2D Convolution\n",
    "---\n",
    "\n",
    "Mathematically, 2D convolution is defined as: $$ f[n, m] * h[n,m] = \\sum_{k=-\\infty}^{\\infty}\\sum_{l=-\\infty}^{\\infty} f[k, l]\\cdot h[n-k, m-l] $$\n",
    "\n",
    "Convolution is moving a window or filter across the image being studied. This moving window applies to a certain neighborhood of nodes as shown below – here, the filter applied is (0.5 × the node value):\n",
    "<img src=\"./assets/tut_xiv_filter.jpg\" style=\"height:200px\" />\n",
    "\n",
    "* In our course, we will treat 2D convolution as *cross-correlation*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Numerical Example\n",
    "\n",
    "<img src=\"./assets/tut_conv_num_ex.PNG\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_anim.gif\" style=\"height:400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/color/96/000000/tweezers.png\" style=\"height:50px;display:inline\"> Convolutuion as Feature Extractors for Classification\n",
    "---\n",
    "\n",
    "* Convolution is useful since it helps us find interesting insights/features from images.\n",
    "* For example, the gradient/derivative filter helps us detect **edges** (low-level features).\n",
    "\n",
    "<img src=\"./assets/tut_conv_derv_filt.PNG\" style=\"height:150px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Recall that in classification tasks we need good features for better classification performance.\n",
    "* In *image classification*, we usually want to classify images into categories.\n",
    "* Imagine that we have a filter for each class, and that by applying this filter, we get a **probability** for the input image to be from this filter's class. <img src=\"./assets/tut_conv_image_class.jpg\" style=\"height:150px\">\n",
    "    * <a href=\"https://www.mathworks.com/solutions/deep-learning/convolutional-neural-network.html\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* What are features? Consider the following illustrative example - classifying *cats* and *dogs*.\n",
    "    * How do we tell the difference between cats and dogs? One can look at the length of the tail, shape of the paws, pattern of the fur and etc...\n",
    "    * So humans can usually tell these just by looking at the sample. But what do computers see?\n",
    "* In classification tasks, we need *good features* to learn a function that maps from samples to labels.\n",
    "* **Raw pixels** are usually not expressive enough features! That is because raw pixels do not capture the *spatial relationship* in the image.\n",
    "* Extracting features from raw pixels can be done using a deep learning network (which is a complex, non-linear function of the input).\n",
    "    * Using just linear layers (multi-layer Perceptron) might work for simple images (e.g. MNIST), but they have a lot of parameters! See tutorial 1.5 (Deep Learning and PyTorch basics) for more details.\n",
    "    * Using convolution, we can capture spatial structures (e.g., pixels the shape a tail)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/mind-map.png\" style=\"height:50px;display:inline\"> Convolutional Neural Networks (CNNs)\n",
    "---\n",
    "\n",
    "* Convolutional Neural Networks (CNNs) are deep neural networks that contain layers of stacked convolution layers or filters. \n",
    "* They are mainy used for image datasets, but are also useful for other areas such as Natural Language Processing (NLP).\n",
    "* In the convolutional part of the CNN, we can imagine a moving filter sliding across all the available nodes / pixels in the input image. This operation can also be illustrated using standard neural network node diagrams: <img src=\"./assets/tut_xiv_layer.jpg\" style=\"height:200px\" />\n",
    "* The first position of the moving filter connections is illustrated by the blue connections, and the second is shown with the green lines. The weights of each of these connections, as stated previously, is 0.5 (in this example)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### CNNs Properties\n",
    "---\n",
    "\n",
    "### Feature mapping and multiple channels\n",
    "---\n",
    "* Since the weights of individual filters are held constant as they are applied over the input nodes, they can be trained to select certain features from the input data. \n",
    "* In the case of images, it may learn to recognize common geometrical objects such as lines, edges and other shapes which make up objects. \n",
    "* This is where the name *feature mapping* comes from. Because of this, **any convolution layer needs multiple filters which are trained to detect different features**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pooling\n",
    "---\n",
    "It is a sliding window type technique, but instead of applying weights, which can be trained, it applies a **statistical function** of some type over the contents of its window. The most common type of pooling is called **max pooling**, and it applies the $max()$ function over the contents of the window.\n",
    "There are two main benefits to pooling in CNN's:\n",
    "1. It reduces the number of parameters in your model by a process called *down-sampling*\n",
    "2. It makes feature detection more robust to object orientation and scale changes\n",
    "\n",
    "\n",
    "* Max-pooling can be seen as \"zoom-out\", allowing to detect bigger objects with smaller convolutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_pooling.jpeg\" style=\"height:200px\">\n",
    "\n",
    "* <a href=\"https://medium.com/@duanenielsen/deep-learning-cage-match-max-pooling-vs-convolutions-e42581387cb9\">Image Source </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Other pooling operators:\n",
    "<img src=\"./assets/tut_conv_other_pool.PNG\" style=\"height:150px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "* Pooling generalizes over lower level, more complex information. \n",
    "* Let’s imagine the case where we have convolutional filters that, during training, learn to detect the digit “9” in various orientations within the input images. \n",
    "* In order for the Convolutional Neural Network to learn to classify the appearance of “9” in the image correctly, it needs to in some way “activate” whenever a “9” is found anywhere in the image, no matter what the size or orientation the digit is (except for when it looks like “6”, that is). \n",
    "* Pooling can assist with this higher level, generalized feature selection. An example can be seen <a href=\"https://adventuresinmachinelearning.com/convolutional-neural-networks-tutorial-in-pytorch/\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Strides and down-sampling\n",
    "---\n",
    "* In the pooling diagram below, you will notice that the pooling window shifts to the right each time by 2 places. \n",
    "* This is called a **stride of 2**, which should be considered both in the $x$ and $y$ direction. \n",
    "    * In other words, the stride is actually specified as $[2, 2]$. \n",
    "* One important thing to notice is that, if during pooling the stride is greater than 1, then the output size will be reduced. \n",
    "* As can be observed below, the 5 x 5 input is reduced to a 3 x 3 output. This is a good thing – it is called down-sampling, and it reduces the number of trainable parameters in the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_xiv_max_pool.jpg\" style=\"height:200px\" />\n",
    "(images from adventuresinmachinelearning.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Padding\n",
    "---\n",
    "* In the pooling diagram above there is an extra column and row added to the 5 x 5 input – this makes the effective size of the pooling space equal to 6 x 6. \n",
    "* This is to ensure that the 2 x 2 pooling window can operate correctly with a stride of [2, 2] and is called *padding*. \n",
    "* These nodes are basically **dummy nodes** – because the values of these dummy nodes is 0, they are basically invisible to the max pooling operation. \n",
    "* Padding will need to be considered when constructing our Convolutional Neural Network in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The FC Layer\n",
    "---\n",
    "* The fully connected layer can be thought of as attaching a standard classifier onto the information-rich output of the network, to “interpret” the results and finally produce a classification result. \n",
    "    * That is, the output of the convolutuinal layers is the new \"input features\" for the classifier.\n",
    "* In order to attach this fully connected layer to the network, the dimensions of the output of the Convolutional Neural Network needs to be *flattened*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_cnn.png\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/layers.png\" style=\"height:50px;display:inline\"> Low Level (Shallow) and High Level (Deep) Features\n",
    "---\n",
    "* It is quite common to observe the features (outputs of the convolutional filters) at different levels of the network.\n",
    "* **Low Level** - also called shallow features (first layers), which include lines, corners and edges.\n",
    "* **Mid Level** - the middel level features, usually object parts.\n",
    "* **High Level** - also called deep features (final layers), which include whole objects (global)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_low_high_level_feat.png\" style=\"height:350px\">\n",
    "\n",
    "<a href=\"https://medium.com/analytics-vidhya/the-world-through-the-eyes-of-cnn-5a52c034dbeb\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Calculating a convolutional layer output size\n",
    "---\n",
    "We define the following parameters of a *convolutional layer*:\n",
    "* $W_{in}$ - the width of the input\n",
    "* $F$ - filter size\n",
    "* $P$ - padding\n",
    "* $S$ - stride\n",
    "\n",
    "The output width:\n",
    "$$W_{out} = \\frac{W_{in} - F + 2P}{S} + 1$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Consider the input images of size $28\\times 28$, filter size of $5\\times 5$, padding of 2 and a stride of 1, the output of the convolutional layers, just before the FC:\n",
    "$$W_{1, out} = \\frac{28 - 5 + 2*2}{1} + 1 = 28 \\rightarrow MaxPooling(2x2) \\rightarrow 28 / 2 = 14$$\n",
    "$$W_{2, out} = \\frac{14 - 5 + 2*2}{1} + 1 = 14 \\rightarrow MaxPooling(2x2) \\rightarrow 14 / 2 = 7$$\n",
    "So the input to the FC layer is $7x7=49$ (because we have 7 for the width and 7 for the height)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Non-Linear Activations\n",
    "---\n",
    "* The key change made to the Perceptron that brought upon the era of deep learning is the addition of **activation function** to the output of each neuron. \n",
    "* These allow the learning of non-linear functions. A (1-layer) neural network without an activation function is essentially just a linear regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_activation.PNG\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Batch Normalization\n",
    "---\n",
    "* Batch normalization is a technique for improving the speed, performance, and stability of deep neural networks.\n",
    "    *  The reasons behind its effectiveness remain under discussion\n",
    "* It is used to normalize the input layer by adjusting and scaling the activations.\n",
    "* Formally:\n",
    "    * **Input**: $x \\in \\mathcal{R}^{N \\times D}$\n",
    "    * **Learnable Parameters**: $\\gamma, \\beta \\in \\mathcal{R}^{D} $\n",
    "    * **Intermediates**: $\\mu, \\sigma \\in  \\mathcal{R}^{D}, \\hat{x} \\in \\mathcal{R}^{N \\times D}$\n",
    "    * **Output**: $y \\in \\mathcal{R}^{N \\times D}$\n",
    "* In CNNs, we work with inputs of shape $[N, C, H ,W]$, where $N$ is the batch size, $C$ is the number of channels and $H, W$ are the height and width of the feature map respectively. BatchNorm in this case is performed **channel-wise**, i.e., on the channel dimension $C$ such that  $\\gamma, \\beta \\in \\mathcal{R}^{C} $."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_bn_algo.png\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### CNN Vs. Fully Connected\n",
    "---\n",
    "* Fully connected networks with a few layers can only do so much – to get close to state-of-the-art results in image classification it is necessary to go deeper. \n",
    "    * In other words, lots more layers are required in the network. \n",
    "* However, by adding a lot of additional layers, we come across some problems. \n",
    "    * First, we can run into the vanishing gradient problem. However, this can be solved to an extent by using sensible activation functions, such as the ReLU family of activations or using residual blocks (ResNets). \n",
    "    * Another issue for deep fully connected networks is that the number of trainable parameters in the model (i.e. the weights) can grow rapidly. \n",
    "        * This means that the training slows down or becomes practically impossible, and also exposes the model to overfitting. CNNs try to solve this second problem by exploiting correlations between adjacent inputs in images. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/officel/80/000000/rope.png\" style=\"height:50px;display:inline\"> Regularization - Preventing Overfitting\n",
    "---\n",
    "* A common phenomenon in machine learning is that even though the training error keeps decreasing (training loss keeps going down, training accrucay goes up), the validation/test error goes down but then at some point it starts going up! (which is bad...)\n",
    "* This is called **overfitting**. Although it's often possible to achieve high accuracy on the training set, what we really want is to develop models that generalize well to a testing set (or data they haven't seen before).\n",
    "* If you train for too long though, the model will start to overfit and learn patterns from the training data that don't generalize to the test data. \n",
    "    * We need to find a balance!\n",
    "* To prevent overfitting, the best solution is to use more complete training data. The dataset should cover the full range of inputs that the model is expected to handle. Additional data may only be useful if it covers new and interesting cases.\n",
    "* A model trained on more complete data will naturally generalize better. When that is no longer possible, the next best solution is to use techniques like **regularization**. \n",
    "* **Regularization** places constraints on the quantity and type of information your model can store. If a network can only afford to memorize a small number of patterns, the optimization process will force it to focus on the most prominent patterns, which have a better chance of generalizing well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The opposite of overfitting is **underfitting**. Underfitting occurs when there is still room for improvement on the test data. \n",
    "    * This can happen for a number of reasons: If the model is not powerful enough, is over-regularized, or has simply not been trained long enough. This means the network has not learned the relevant patterns in the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_overfit.png\" style=\"height:200px\">\n",
    "\n",
    "<a href=\"https://www.geeksforgeeks.org/underfitting-and-overfitting-in-machine-learning/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Regularization usually comes in form of placing constraints on the parameters, or in the case of neural networks, constraints on the weights of the layers.\n",
    "* It introduces a cost term for bringing in more features with the objective function. Hence it tries to drive the coefficients of many variables to zero and hence reduce cost term.\n",
    "    * Common regularizations are $L_2, L_1$ regularizations: $$ \\text{New Loss}_{L_2} = \\text{Original Loss} + \\lambda \\mid \\mid w \\mid \\mid^2$$\n",
    "* For deep neural networks (and CNNs) a common regularization technique is **Dropout**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Dropout Regularization\n",
    "----\n",
    "* First presented in <a href=\"http://jmlr.org/papers/v15/srivastava14a.html\">Dropout: A Simple Way to Prevent Neural Networks from Overfitting</a>, 2014.\n",
    "* Dropout is a regularization method that approximates training a large number of neural networks with different architectures in parallel.\n",
    "* During training, some number of layer outputs (i.e. neurons) are randomly ignored or “dropped out” with some probability $p$. This has the effect of making the layer look-like and be treated-like a layer with a different number of nodes and connectivity to the prior layer.\n",
    "* Dropout has the effect of making the training process noisy, forcing nodes within a layer to probabilistically take on more or less responsibility for the inputs.\n",
    "* Dropout is activated **only during training** (`model.train()`). In test time, it is turned off (`model.eval()`).\n",
    "\n",
    "Read more - <a href=\"https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/\">A Gentle Introduction to Dropout for Regularizing Deep Neural Networks</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_dropout.png\" style=\"height:250px\">\n",
    "\n",
    "<a href=\"https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/ch04.html\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/variation.png\" style=\"height:50px;display:inline\"> Data Augmentation\n",
    "---\n",
    "* Data augmentation is a common technique to improve results and avoid overfitting.\n",
    "* How do we get more data when there is limited number of samples? We can perform data augmentation.\n",
    "* Data augemnetation enriches the dataset by adding variations of the original samples.\n",
    "    * And as you know, deep learning flourishes when there is A LOT of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Popular augementation techniques:\n",
    "    * **Flip** - Flip images horizontally and/or vertically.\n",
    "    * **Rotation** - Rotate the images at certain degrees. This may change the size of the image, thus, cropping or padding is a common fix for that.\n",
    "    * **Scaling** - The image can be scaled outward or inward. This may also change the size of the image, thus, resizing (also stretching) is often followed.\n",
    "    * **Cropping** - Randomly sample a section from the original image. Then resize this section to the original image size. This is called Random Crop.\n",
    "    * **Translation** - Move the image along the X or Y direction (or both). This forces the neural network to look everywhere.\n",
    "    * **Noise** - Over-fitting usually happens when the network tries to learn high frequency features (patterns that occur a lot) that may not be useful. Gaussian noise, which has zero mean, essentially has data points in all frequencies, effectively distorting the high frequency features. This also means that lower frequency components (usually, your intended data) are also distorted, but your neural network can learn to look past that. Adding just the right amount of noise can enhance the learning capability (e.g., add Salt and Pepper).\n",
    "    \n",
    "Read More - <a href=\"https://nanonets.com/blog/data-augmentation-how-to-use-deep-learning-when-you-have-limited-data-part-2/\">Data Augmentation | How to use Deep Learning when you have Limited Data</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_augment.png\" style=\"height:300px\">\n",
    "\n",
    "<a href=\"https://towardsdatascience.com/machinex-image-data-augmentation-using-keras-b459ef87cd22\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/clouds/100/000000/dog.png\" style=\"height:50px;display:inline\"> The CIFAR-10 Dataset\n",
    "---\n",
    "\n",
    "* The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.\n",
    "    * There is also CIFAR-100, with 100 classes.\n",
    "* The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. \"Automobile\" includes sedans, SUVs, things of that sort. \"Truck\" includes only big trucks. Neither includes pickup trucks.\n",
    "* <a href=\"https://www.cs.toronto.edu/~kriz/cifar.html\">Official Site</a>\n",
    "\n",
    "<img src=\"./assets/tut_conv_cifar10.PNG\" style=\"height:300px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# ------------------------- <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> CODE TIME -------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# define pre-processing steps on the images\n",
    "# also called \"data augementation\" (only done for the train set)\n",
    "\n",
    "transform_train = transforms.Compose([\n",
    "    transforms.RandomCrop(32, padding=4),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),\n",
    "])\n",
    "\n",
    "# Normalize the test set same as training set without augmentation\n",
    "transform_test = transforms.Compose([\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),\n",
    "])\n",
    "\n",
    "# load dataset\n",
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', \n",
    "           'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(\n",
    "    root='./datasets', train=True, download=True, transform=transform_train)\n",
    "\n",
    "testset = torchvision.datasets.CIFAR10(\n",
    "    root='./datasets', train=False, download=True, transform=transform_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n",
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n",
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n",
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqUAAACpCAYAAAAShURgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO2dfXwV1Z3/P5N4TbxNuN5wm5gGYoANpmCKRoSiFKGKRRC1Pj9Uu1pbbWufW213tw+7rftzW1ur29badmt9qNaK1oeqLEoRRNhoirCRLJAG0kAaiJfEcOMl6SWZ3x9Qyed7pglgYAL5vF+vvupn5syZMzNnzj0ZPt/v8XzfhxBCCCGEEGGSFXYDhBBCCCGE0KRUCCGEEEKEjialQgghhBAidDQpFUIIIYQQoaNJqRBCCCGECB1NSoUQQgghROhoUmrwPO8Ez/Ne8zwv5XneZ8NujxD7i+d5v/I87ztht0OIfUFjrjhc8Tyv0fO8swK2f8DzvPX7WZfGbQBHhd2AIcjNAF70ff/ksBsihBDDAI254ojC9/2XAJwQdjsOR/Sl1OV4AGuDdniel32I2yLEkMDzPP0BKw4WGnPFsEFjaf9oUtoHz/P+AGAWgB95ntfped5Dnufd7Xnes57nvQVglud5Mc/z7vc87w3P8/7sed6/eJ6Xtef4bM/zvu95XtLzvE2e593keZ6vTigOJp7nnex53qo9//z5CIDcPvvO9Txvted5b3qet8LzvPf12fcez/Me29OXN/X9p1PP877led4Cz/Me9DxvB4B/PKQXJYYFGnPFEcCpnufVeZ7X7nnevZ7n5XqeN9PzvC1/K7Dnn/lv8TzvfwG85XneUf2N28MZTUr74Pv+BwG8BOAm3/fzAPwVwJUAbgWQD2A5gP8EEAMwFsAZAK4BcO2eKj4O4BwAJwGoAnDBoWy/GH54nnc0gCcAPACgAMCjAC7as68KwC8B3ABgJIB7ADzleV7Onh/1pwGsAVAC4EwAn/c870N9qj8fwAIAxwL49SG5IDGs0JgrjgCuAvAhAOMAjAfwL3+n3BUA5mH3eJqFvzNuD3c0KR2YJ33ff9n3/V4AGQCXAfia7/sp3/cbAXwfwNV7yl4K4E7f97f4vt8O4LZQWiyGE+8HEAHwQ9/3M77vLwDw6p59Hwdwj+/71b7v9/i+fx+A7j3HnArg3b7v/5vv+3/1fX8jgJ8DuLxP3St933/C9/1e3/d3HrpLEsMcjbnicOJHvu9v9n2/Dbv/mLri75S7a0+5neh/3B7W6J84BmZzn/9OADgawJ/7bPszdn9pAoD3mPJ9/1uIg8F7ADT7vu/32fa3/nk8gI96nveZPvuO3nNMD4D3eJ73Zp992dj91epvqP+KMNCYKw4n+va5P2N3nxyoXH/j9rBGX0oHpm+nSWL3X+7H99lWCqB5z3+3ABjVZ9/og9s0IdACoMTzPK/PttI9/78ZwK2+7x/b539R3/cf3rNvk9mX7/v+3D719O37QhwqNOaKw4m+fa4UwF/+Trm+/bq/cXtYo0npfuD7fg+A3wK41fO8fM/zjgfwRQAP7inyWwCf8zyvxPO8YwHcElJTxfBhJYBdAD67xzx/IYApe/b9HMCNnudN9XbzLs/z5nmelw/gFQA79pjvj9kTMHKi53mnhnQdQjhozBWHAZ/2PG+U53kFAP4JwCP7cEx/4/awRpPS/eczAN4CsBG7TfgPYXcwCbB7ErAIwP8CeA3As9jd8XoOfTPFcMD3/b8CuBC7o+Pbsdt/9/iefTXY7Sv90Z59f9pT7m8/9vOxO0BkE3Z/kfoFdgeUCDGU0JgrhjIPYXcf3LjnfwMmwO9v3B7ueGxpEIOJ53nnAPip7/vHD1hYCCHEO0JjrhCHN/pSOojs+WfQuXs+x5cA+CaA34XdLiGEOBLRmCvEkYW+lA4inudFASwFUAFgJ4BnAHzO9/0doTZMCCGOQDTmCnFkoUmpEEIIIYQIHf3zvRBCCCGECJ39Sp7veZ4+q4r+SPq+/+6wG2FRvxUDMCT7LTA4ffd9p3DMT4/fzefocU/R2+ORTr3Fx7S98Sbptzr3v5lZ5pNIb+9+VyGO8L4rDj6RHNaZbrdMdnb/dfQcWK6LwL6rFZ3EYKIVKcThyBHdb5+t4aW4U12bSGenM84x6fYI6Rdf3Uj64bufIF29rGu/2/WuPNYpuUAPhCO674qDT6FJ2d/c4JbJO9ZsMH9AdrQf0KkD+67++V4IIYQQQoSOJqVCCCGEECJ09M/34ojnlFNOQU1NTQhnXmv0RFIPf+MLvDs6CpYpM2aQ/u4XriG9rmkd6c6024oe88+iLWb/J6bwv9+0pjudOh58vY104XG8v3Mr6yvOr3TqqDf/LrSwhhtbYcs7NWiZngMhD3HSXRn+5/oXV652jnnkzudJP//8JqfMQNilwWacyfrMq7lEqstdTOzllUnSC+8L6OBCDDrjjd4QSisOBc1BA62hY/vBb8ff0JdSIYQQQggROpqUCiGEEEKI0NGkVAghhBBChM4ge0pzA7bJBbZ/2IRgQfdvoIR+Yf2t4aaWGQqkktvx4i/ue1uXlfH+utol7kGmK7d1NZEur5hFujRhHZHADY7/k59L/eaB/XFfnH8B6ZJxfJ4NmUbS0YSbmue4Atbnp9nbWlrEdabfYJ8qAFxRyfl7Wnv5fqSMUfWKqy9x6nhl5SrSnS2cVqis0BxQ51SBeAnrlRvdMoJZ8OiTpH/186dIL3++46Ccd+5o1iXGZpwTZR0vdj2lhaPG8TE5bIB78mdbDrh94kggGrDtnfuOoyMSXOMO66k2gyq2veNzit3oS6kQQgghhAgdTUqFEEIIIUToaFIqhBBCCCFCR5NSIYQQQggROoMc6HQgQTnDnYH+LnCDh/JzeF3qHBOV097NRu+eAwo2swFXhy/JN97Az37+s7f1bbfeQvuzeuP2EKypNcn2o60ktzdzkE5m6jynjskfqCLd838cHDR5KgcYtTfatPZAVxef98s3f5P06LUcCFLXsdKpo8DEj3ysgq9/RAGb9n+7/CGnjsQ4jkLKyuME+421nBh/2kWXOXXkRLmONLiOEaadE85y+35PNvf9lbe94JQRzPWXPnDQz1F+jLstfi7r4jH8gHtSxaTTWe6Y05PLQVjTzuB3tbubA50W3odQKDG/pDt2uWVSh6Ypw4wDCWrKN9oNDi0u4r6ZVXwe6fr1r5ojipw6suM8vve0L93nFg5n9KVUCCGEEEKEjialQgghhBAidDQpFUIIIYQQoTPInlKx/1jPLetJY12vysXnsFcxZhII161lH9ZPX6wOOK91OB05HlLLm+m38PtXVryt5yzmJO5nTj/bPSjGyZMzeJ10OtlG+nePsccUAMpOHUV61Jgc0u8/ayrpRMRNHr5+0RrSyRZO0jy9ajrp1mXsWwWAZF0j6WNOKiW9Od1Ouj7FGgCaNrD/c8apJ5PubOXztq5tdup4cRXf98V/Wks6GuU+WFxU5tSR3JJ0tolDj13oYNZFNpk4sCvB41BnVzfp1jb2IeeYdw4App3B72ZhJffdmZMvJH3F/EanjoULefxb8BgvDpEx3T06wqkC/3gG62tPYj2ZX21g/ERYkmle+eE/7lxE+vbX3POKA6G4373xkezDLyh2y0+onMA1vof7XXklj3870jw+AsDY97KnNPmXuaSffZi9/WI3+lIqhBBCCCFCR5NSIYQQQggROpqUCiGEEEKI0JGndL8YKN+n9WVaoxFgPaOlx+SRnlzFvsLRpewXBYBkC3uxWlL8t0XFP7BXcX7DJKeOpzevcbYxR47nNJZ7NM7+h+Pe1gufepL29xivGwBUnsaeonghP5fOOOcUvf+BHzh1zDybn8O4cvYubUlxHe0R18s5ooT7R90Gfm6XTb2G9IWT2GMHAG1l7HfKj3M7XqpeRvq4hNvn2tvZy5mV4jITyqaRXrzMzR/amd5KeuZpfH9aO9iDmJfv+hST7a53Sxx8istY57I9Ej1RswHAlPIzSWeDc+5m5XIe2h1t7njZwDZkzLn+3/pt59Tx7rZrLuI8lHf//FHSaxoeJN1bzV5PADhhCeviP5kCxmOKM0936kgUsM/0e0k+z9nmtTv7ZacKsQ8UjmYvZyzBcRmzP8Tezooq9/exsJjH+0iEc5vGojw2LXvmGaeOCdN4TCydyMfIUxqMvpQKIYQQQojQ0aRUCCGEEEKEjialQgghhBAidOQpHVSsF89dU7c0zrn4vv3Vc0hfeh57s3YYnx0A1FRznr1kktf/LcxjXZLnGq3qfsh5JBuwxSnDRAK2WY/tQJ7bcCguHImvfeqjb+vF/72c9r+04nHnmEZjZisfX0Z61gWzSFeVu76k3/+IfUaX3sB+z+5m6yl1qkBxcSXp5mbOB9qw9BVu5znumvOWNUvZy9ZWX0M6kcc5WAEgEWFP8ZpFbHhrNPlTu7PdfKnnz2APaXYvX/DiBr4fPR2u1/fEEn5/3CcnBoPyMtbZbG1G3FjM6/5Y69TR28V9oGzceKPnkM5ku6vDl5VVONv2n1xSMUwmXZFgj37NatdT2voc627ziiziIRnjc3c6dcz89BTecMEnSc7uupv09JVOFVhu01oLh4JR7CG95oYbSX/so5x01qTcDcT2zPWbWM+eP885ZpJJVZtv9uePvojPsfmxfWjJkY++lAohhBBCiNDRpFQIIYQQQoSOJqVCCCGEECJ0NCkVQgghhBCho0Cn/cImkLeBTXaOn4FlzixOXH7NzbeZEhzYlJu+16ljbAOXmTuHjfuIcQBJpslNhn7vEt7WsMZtKxP098vQDGyy9Px1F1JNeyMTCqOcGLm8yF0YoGY5B0P9+mkOfljwmwWkX292g4MajD65loMfKiqmczuz3fs55/qvkF54x49JP/yL+0j/y1kBgU4ZfrZZ3ayL4qNJ5xWwBoDnn3ue9L2PPEV66ukcgDThpFFOHa1NnDw9bWL4Gtc2ko4XcZ0AEM8OWpBCvBPK3bz3mM1dE+tNIE+PieGseJ9bR7sJ0lz+6quk65o4sG1CBQfCAcDk6Re4Fb9juG8Wxj5COm/Mi84RmdKlpNMmOuaG18wBrz3gnvZzvO3R6zkg5+L5fP0f/mijU8Xye7c524Y18XHOprGVY0nfYAKb3CU5BsYGKZXwo0Ox+xM7IHfcw4Ft189VoBOgL6VCCCGEEGIIoEmpEEIIIYQIHU1KhRBCCCFE6MhT+o5wkz33JTdg24Rx7GdE+kHWWe2s1yWdOvIjpbyh13giizghf6TI9TtuT/6X2WIzt9sszUGe08Mjk3NvTy92pvb6OWPHslcxbe8fgEsvOY90z0O/IZ2dYlOdu0yCyyd//EvSN05h497FH3YTMKOL3UxNtY2k62qN2a92jVtH9BjWqU6Sa1ZtJL3y1RVOFevWcyL0+aePIT3lLE5ynu52342Odr5LjX/mvn7Ce3kBgvaUu3BEpuPw6HOHEwGPCjN4zQacfSrrLWatjVZ3mEJehI12hTEeQzIRriQeZb89ADQ0sJe7ZKJNdf5e98QD4qQxJzXj+u84Rzz9mw+Qnm+Gwzfns17sdl10mITrU/5o/KHlfBNnXjnNrUSeUmLy5JOcbRdewOPogXhIB+JAPKSWk6ve/c4rOQLRl1IhhBBCCBE6mpQKIYQQQojQ0aRUCCGEEEKEzjDylA5GTk3rPbSazzFlgptnce4sk4uvgb16qdR23p8Z6dRRWMye0qa1daRLo+zDql3O+QABoMnJq2kdsNZoFnT/XC/mUMQ/ykMmsffvr+ZW9jJGEtZPC3Qbl+iMWexdmj1rDh8QcU1Gv1/KuU6/voCfy09f4f1NTcZPDOBXv3yE9IL61dyOEexRfvy5J5w6Xly6jPTzr3G+SONKDfRCf2P+aaRLSvl6O1rZHxjNce/p9jTrjgxviEf5mJ43zAEAMl3uNvHOaNrhblvPaTlx7rmsp53FuqbWrWPlavOs2MqMjNENNdwvAaC7k82b7R180LTTLiRdCM5JeUBEprvbetjv+vkaHlM/YYpfOCWg3jFG25S7r/IYW3mRiT8AcNfDE0l/9oq1AScaPqxZudzZdtWiBQElhx6vrXxq4ELDEH0pFUIIIYQQoaNJqRBCCCGECB1NSoUQQgghROgMI0/pQFh/5L7M19kDlwv2UF04zyT3AzCygOtNd3WTZgX0ZlwP3Y72RtKJAuM9SpSR7Ohxk+Z1OHlHXQ/gwPsPj5yRWcdkI1oZf1vnmJyK+XD9oD1m3e5ik0MxmmD/V1mlzZ8IdOTzGtv/c96XSC9c3Ew6mXLb8YcXniNt86Eu3sHtbPzh9506Gow2KShxzST2KJ9+jrsG+fgY98wRMfZLN5ocjKWlrp+63bxihZzaFC1bzIPpdft+bmSgfioGg+ZG1puN3XMHW+Ex0l2CHB+ezs9qewePORtaufzWgHypmS3soa7rYO9yspHNrDNmXebUUV78EbPlXe6JBmD+8r+QnvzLb3G7Hvoh6XVL3TF3nHm9E8a6OuJcvomRQrPAOoDJEzudbcOZTGfAGBFCO4KoaVhCunoF53/emeH3Izbamo6Bjs2bnG1HOvpSKoQQQgghQkeTUiGEEEIIETqalAohhBBCiNDRpFQIIYQQQoTOERroFDTXtpmK+098DycQCHADe9hkPXksB8NMGF8S0DSut8Mkgy4sHs2tHMd1AkDHOk6WHx1jzhM1hvmYDXWxLT9Q7H0ejAUKBh/f60FX9t7E9OMq4rQ/k3afdTyf73siVk46YgJu6rrcBQpyK/k8sRyOdKjKPYH0yhVuEMOJp3Iy8A+CAyjuXMyBUAE5zHHZaA64Ov80vpaTL5lBujPm3o/CKAc6pU1M0uSyMtLxIjcgqb6ND0q3c2RLTpz7U7TYhnUBhUUm+ONxm/pfDAZLeIjB+8wQk9rGOtdoAMiJcD/KNsNnzKzfERDXhm6Oa8IxRdyHoikOJmnpcfvDtqJnSReP4/6eXziJdE6+O27H8F6u47pvGX096Y7af3XqSK5bRborzu9lW0UZ6Qn5NiQRKC6xY+xSp8xwYtrMM8NuAgAgk/ofZ9u6VZwcv6iYx676Zn4/Kk6qcuqoVqCTEEIIIYQQhx5NSoUQQgghROhoUiqEEEIIIULnCPWU7ktSd+ubc/1rFuuSm3YcJ7uddhL7ELPgegS7u/nvgN5s1tkxTmSOgnynjphNTF46ypRg32Ek2/pngTi43nbHD+oe43KYJM/3gGifh1daxIsNZAKuo82Y5pp72PDWbXpDh131AMBxZexNqzMZydsifM9TGbe/TJh4IukbL5tDureD25mVca/lxs9+hXS6hROSZ5v+0dBc79TRXsx+2KZVXGZKBbezN6DPRTJmwYFC7sct0V2kE+PdjOztXUoefiiwPeDO51lfy7ZkjA2w4KdNF+g1Q2ynScAfNJzEzVAXNWWyTI76lmbX241i9vbt2vIM6RFjOIt9d5S93gCAPB7rC4smkB6XYP9nrPLzThWxSntX+Yb0mHG7G+61RPLdBTaGMytffCKkM79JKt290ykxdSrHA1Sv5diOktIy0idMdF+A6qcfO8D2Hb7oS6kQQgghhAgdTUqFEEIIIUToaFIqhBBCCCFC5wj1lO7LXLt/D2ll3M2zeMUlF5CectI00t1p9oxUjHP9P7E8Nl91drKPJNXSTjq/0ySEBIBCu8H6QXNJlRe6efemmpyqCzdu4PMaT2mW46gFOo0vt2eI5in1srOQE897Wzem+J5mIq7/sSfC9zBpcmrm5vH9iOSxTxUAtrazD/XkKs6ptzGHPWbNa01ySADlCTbVJapOIn3XUwv4gGL2vgXScSXJxb+4jXRBofE1AxhhXqkZp3H/iXTx/enq2OrU0bmF73vZRM4PmTT5YrOzXKNiNOP2Q3HwsW7IamN3jAZYHUeabbkmVXSPsdBlAobkTpP/1HQz1/pu01EDyDf5UNvaeJzato5zfeaPc3N/jp06j3RvG7/b60wO3oJxBU4dR2Vz46JmTO0Fm2xXrg3wS/aq/w9Ec/KvpEsSRx+Es3DHiyXKnBLr1q4hXTaOfci/eoRz7N5/208Hp2mHOfpSKoQQQgghQkeTUiGEEEIIETqalAohhBBCiNA5Qj2l+5JjkznzOPYE3vLPVzplZt/0MdLfvO4HpAuNrfDESa6vKGJ8VIXG79mS5Fx1+UEmKce+aL13vOZ6vsn1BwC3fe1C0letYj9jpMfkU82499Q0FZ99fGjmVHtr5068UrvX39Obz37RnhzXp5U0f69t7+Z7XGi8nmMTrqd0u82ZuNY8fONTnTOLPZYA0NPB+WRbGti3XDxxvHPMgJimnvmRW0g/fDfnNQWAiPG/9Yzj6y+Zynkao3nW/AdkzP1IbWNfXusG3p9jvIAAUFw01t0oDgKcIza38BLSr3T8mPSEqPu8zZCKtHme+XlGB/hSk8ZTmjFjjn11ewOG/jR3M0SNhjnvloDlxlONnNu0bNIyLpDPfsFMxn0v07k81vdm8Yk7M+zcbWl14wmiPbbxwnLt1fzbtui53x+Es/CzS6YCDNERLvPfi9mr3NTCnflT//FDp4qf3PIxZ9uRjr6UCiGEEEKI0NGkVAghhBBChI4mpUIIIYQQInQ0KRVCCCGEEKFzhAY6uaZjG8py8SROMn7teZzYfHTMTTj/lYu+Svrhx6tJf+La+aQ3NrqRGtkm+XE80026uZkTKHd3uEE4ZRU2IIaDEpoXcdDSg48ucuq45fbvkJ40j03X9SvYyF9Xa9NnA8nqbc62oUjk6AgKS0e9rTMZDlra3NRkD0F3Nt/30WWcUD6SMUE6a2qcOnJzOEitK4sDnYqyeH+ejcgAsLGOFzWIRspIO4FOm7Y4dby4iPvpylX8LL92D/frskS5U0e6o5XPW8yBTS09HOjyev2rTh3lcY58iZqAsxOqTiSdCbgfO1p2OdvE4FM2jZPFb93G70xLKz/vgKEO080QGrNra3AMH+IBgU7FJlhqwzrWOztZR0zwFACkTbzQNhMsFTXtyHZjFtHBa5rglVdNYFcOv2NZx7MGgF0cf4pjEhNJR4rKSM+Y8wmnjgjsWPUzp8xw5/mFzwxc6B3D3/M6071Oie1J7iPTpp9F+sSpI0l/91Ylzwf0pVQIIYQQQgwBNCkVQgghhBCho0mpEEIIIYQInSPSU5oPN1v8hybxtvnT2Yc5e/4s0rd//yGnjtsff4G0dXv2mA0jCt3k+Rmwz6Q9zToeZ1NUVq/roUuvep101CQy/86dT5D+6bOup/R3S9mcNba0mPTmTbx/hDVeAdjaYpP2D016AXT37jW0rVqygvZvX+t6Sueedx7pGeNOJr2qhj239bW1Th2jJ7E3M2L6R1lBBe9PBvyN2MH3uCRRSLqn9k9cfJubcHvmPPY6z7zhaPc8fZh25fXOtpbqxaRjpdNYZ/N5X2tY5dRRt42T4x8XYU9pIsHvi/UCAsCEijHuRrFfVJ7IffvL377LKXNC1fGkb7r+F6RrNnL5mvXueSYYj2glDzHO+9Dr2udRbizTSbMeRYPxesbNOQCgw1iTd5lhq9vYQ3uC8qAbP2yeaWunadcf18Bhi32989eSrDiDza7XXnmvW4mDPKUD86bRxw5CnR6pEUXunGN0N4//L1bz7/aTz/FiM6uWLBmEdh3+6EupEEIIIYQIHU1KhRBCCCFE6GhSKoQQQgghQmcIekptMjsAyB7gGDYv5SDXKRGLstmosMjk9mzmnJtPrnDzTlqsozKTbZLm5brmpByTqy6Wz17NdJprzcm496Ori8/z/AMLSf/62YG9KdX164y2eUg5h2Y04BlEjwp6VkMPr7sXWQ17jWV1S9hDOinL9f5eWjSF9GuLXyOdNn/OTZt3vlNHNMO5Pbs2NJKO984g3dDs5uUcXcG+0x1p9p1l53L/KaiscupAUf8eUocC91kXT53JG7JtnaNIXXzq1506li+5j48o5hy96DTXFuApRW9LwEaxP8RMHs5UW6tTZsc29pRWvo99yQ3rvkK6fvP3nDq+zq8MKoyeXcZ6gpsaGmnzaprUvig246nNawoANt1tpxmms2x+1ACrfLfpdibVMSImTWWW8boCgDktMmYMqUlxjuHbSy506khnBvotFC4vGz0vsNQ7YaftEABat3EnKC7kDn7F5ZeTvirAy/+56zhXbWqzm//5SENfSoUQQgghROhoUiqEEEIIIUJHk1IhhBBCCBE6Q8BTui++RFsmYJHkPiThrv9dv43zO/52aQPppgfYh7l8c9C67tbPw+1Kp1kXFLtexaYWXpe+In8C6WiEk+ZF467fC2Omk1z8fc4zmAoyRQ2ITRLI15qGWTAaQNom/BuixI95Fy6r3JtXc/rN7NPMpNw+mF3GeefGVnKfi0TYDxkvcw1x8WQz6fw89jGnWtgPur3FNaIdF+Pn0NDICSInjeP+4zzGAHqSfyWdndgHz2nB/vlSG1asdbb9cSl7eUeVlJJOm7WiJ5SaJJUAkNv/uy8GZvnLT/WrAaD0ZM5lOvkDt5Cef92XSN9/m0lcCgDdnIfRLFuPdY2s840GgElGn2I8pJVlrNMB/d+OUmaYRsZ4Tk2qaABAjvGMWg+p9YeWjnXrqOSl7pExba3ZxPqndy136kjlOJvEANSueoZ0ZdXge0pLIqOcbZtibEROt/Nv+bIX+PkuXMS5r4Hh4SG16EupEEIIIYQIHU1KhRBCCCFE6GhSKoQQQgghQkeTUiGEEEIIETpDINDpQHCs66SCQiFqN3KA0ZqNq/o9Q0AOZnSbwCYbHtO9jY3NZaVlTh05aQ6gyuoygU0x46AfM9WpI7WhjfTDz9gQArt4QNDfHuY8zj11E/+7HB6JnHt7Mkh37H02ZRUccNRigpYAoLWMr62jm+9Xe9IEfrW69zg3w1EJ3Z3cY+rqNpCOBPTczhZ+1muaOFioJI8jPxLlZU4dgAlS6uAALKRslAYvNAEAyJh7ZJL2W5qb3aCt2tUcDJMTY33+LBPWMiYgm3pTs7tNDDpN6zj46fyr/p306WcXkS4sftSp4/YbbcSQuzhEX1IB22yoz3LbrYwuhYsdy0canTABR8UBw1qheTVjJuCo0MTkVZ3l1jF2MuuIOe/FJoMLgDwAABS/SURBVLv+ShP4BACPPMf6yc1uGcEseMIGOg3+OYJ+LZuTHNjUuo3H7oTpRFNPLXfqeHrNCtLZ5pUaaTp36xEwPOpLqRBCCCGECB1NSoUQQgghROhoUiqEEEIIIUJnCHhK+09IHwz7H8eNYGPFFee4PszmTZwsf9IplaTjcfZhrlvrJq1Nd7Dpp7mV21HC+daBmOtMLZ5kzEeN3C7HaASTLRrAj+56hHSy23q1rKe0fy/XkY5/VDZ6Ens9kMkk3/PmYjfjdmcre3+PSbGnNJ5hk1l6i+uhXLymkcts4v7S2cbnHZFr3W5AQQ+/H90pU0c7e1s71q1x6kh2c9sKo+xl2r6NvU9t1e7CEZlePm80ajKB59jFBdyFFc4853TS8YJ6U4Ux7tms5gAaOoIWtRCDzk72DD/4wEOkX371bNJp807tZh9WchhkmgYu4rJzAL0PFK5nnftMQCFzO8p5PRdkmf3rG90qmmwogHCYPoN1/ZaW4IKDSNBj2ZpMkl63jhcU6WzdTnr2mabhAKImAf+2DdWkX1vhLmpzuKMvpUIIIYQQInQ0KRVCCCGEEKGjSakQQgghhAidIeAptQTlvmSfaTSLvWdf/tyVpLubapwaogXscSsv4zrm3vwVc0RAxrsk+/VWrXiBdCxm2x7g5cw1ntmo8d6N+gjrVKNTRaSTzUeXTWafak+UvSyZFjf/ZWuSz9vYzpnWWpy8pYcxvR56u/bes0RpBe1OFLret2SSvTyxbOOZzOf+0VjnejmfforzxzZv4v4w40OzSW9NmUSFAHYYX1KWeT9aTd7STJfrn2psbySdk+GMkJ3pHVxHxM2619HFbatft4B0r/n79lNf+JJTx+zJ7ON2rqVxC+nXm91EjYkAn7bYP0pNrsMmt9vBZg1tX/Pv/epDxaQp/0W6zvj6M4037kMto4y232aScOnfl99qNwSZDE2q36YBcox+5ooznG1X3TaX9PuPv6X/SoYhecU8Vv9+kX12bxj97nd8zuaet5xtp89iD33tGo5TeeTXvyH95HOsAeCD54whvQvsIS00XTkn6o6PiWL2/9ev43E2s+OAnNgHDX0pFUIIIYQQoaNJqRBCCCGECB1NSoUQQgghROgMQU9pUJ5SzruZ7mXDzuIX2EN68Tm8tjkATJ7D/o6NDZxY7r++/GnSH/unu9xmJGaRrDpvlinwF3uAWwcWkurJY19dNk7i4gGp/i796CdIXxVhz0xOHutItuuPbd3CvqnfP7GY9LLltaSfrlvl1JHZp5yy4XPUUREkCvqso55jfDfG6wUACZOTFkm+1rYGvj8rl7j3JxHhNeQbO9ljmmUSE/ZEXD91SwfnGI1FjiGdTrPHqCxq/XIAWtkf+FoD+1/HlnM74wl3XfsJ4yeQnnsJnyca5WspHe+uW1+3gc9bv4773OiisVxn3PVCd/UeQV7nQ4R9EnHTtYeWo8zC/Spu+mF5N/uh6/DJgDpMDuFCs8Z46/L+ywcwfc7nSX/uS/+P9CWzzUL3AADOU2mzSd/z0nWkr5k+Gy7FAdtEXxZWm3HUrAf/le+yH/p7N9/xjs+5rcNxFaPXxGXkmXF19tU8f0jkuz9EvV0bSNueWVTCv2VTT3V9yNH4iVzmA3x/Mmn+Nrl8xWtOHY1rnnK2HSz0pVQIIYQQQoSOJqVCCCGEECJ0NCkVQgghhBCho0mpEEIIIYQInSEY6BSEjfZhA/GClZyUduYsNsPvPoJNxCOjHPwxaRyHAyy865+cOoorpvMxF0zlArnGQI+jnToANiZnF7ht5TonOZtKZ5lgKBNw1LH6cdLPPvOoU0ckw/cjK83hDqefyvcjVuwGbd27+AWzZYgGoexMA2v7BMNNncb7x3BSdwDAcg7KqVvNbvl0dDTp2hY3OKJiPCfp7351NenX19aRjha7QQx5x3J/6XiDk+Nv6+DnGGtxzfLxCL8/ZZO4XblRPqZ0fKFTR7SwgHRnhgPlYkXcPzJZJroAQEkpBy4dN+pM0pubGkhnB/SnTFBSctEvlebTQ6e5h0GhM+4SDCGRxcEhu1oeIZ2xSe1zOIAVAFDIgR7xKK8e0N4aEOnowO/MKVP59+Hiszhs6V8f/k+nhuW1nyE975M8xpaP2kY6lXHfofxIhbNtWJPjBnZOmMXjed1jS0nffssPSc86i3/XAWBu1UX71YzxBWOcbQ0pDnzOzuIXsbOb37Lz581x6hgV42C3JUteJv3yEg6eW/d/7pvb3sn9qriY2zoyfhzporibgL/RGSUO3gihL6VCCCGEECJ0NCkVQgghhBCho0mpEEIIIYQInSHoKQ3IFr+fSdrvuNv1UM76yVdJl43h8zQ3sYewcqKbcP7x59jP9J0f/YKPOfU00lnZ7py/fkM96Qsvmkv6XONT7U5xcnQAaKlnr9HmWvbijTCJzMvjbh3bW7iOyonsGWnrYV9hayTId3V4GPze2voGqm+7+2094QxOYp//za+7BxWy36u6mn26uWZ9hqJSt7/EC9mrEy9lL8/zL1eTHlnmLvowYRL7hyMZ9ll2G92SZK8nAJxcyV7nKVXcjrp1nDy8I+n6Y0snsbc5J8Zev/bWLaQXL1zk1FFWzPe0ooo9VOs38f2oqWXPLQC83jC0U70PRbLMa9pg9h+IO8y632oCfkmSNo+3PXHjPpxoVhufZwv7AT9gukNJwHolG8exp7DpvabAVUa/FNCO53nMuPMOTnx+2XX8Dn3jcruwCvAfl/Nd+zy+T3qu8e1Nj7jevlRXrbNtqHLsyHx88Ny9iwh8aN7ZtP+11e61PPwEL4TTneExcXTFfNK5iS6njrwirgPZaadMX+5/4HvOtrlVtoe/i1QXfNIj4Tl1FOa/h3T0uo+STqd5fCsd5fpSf3orzzmefXyFU4YZ+G1uWFM9YJkw0ZdSIYQQQggROpqUCiGEEEKI0NGkVAghhBBChM4Q8JTaeXGQp9R6F/vPh9nQ7npIbrj5Z6Tv+MYVpCfPYg9Jqo09cgBQWsn+lifvfYr0gpf33+92/9OcY7VsBOe7qyjNcY654sPsXS02eSgzadZVVZxTEwAyY8pIr1zLPsJfP8q+nMdXsqdqN/vn9Q2L7KyjkJ937Nt6zWo2t4378YPOMcWTTiAdjXKOzY0Nm0hnRV3/V26UfZexBHsqo3H2f+bGXEPcSyYH7wenTiTdbV6FaH7cqSOWKCKdn839oyCXry0ekC91zUruDyUV/C40N/A9XfAr19cdj/B5/vVmvmfTKjh/bG1No1MHet33QfTPdqMHI8NgF3dlJD8ZUOifB6jkLKOvDijDNjx0sXUTz99hygel8bRpnS8ZoF1BpIzO5/yQpzXwO/PF5Dynih8kHuv3FM+aJ/MsHnIL5bqbhiqRiIfCkr1jzfgJPL4lSqbZQ5Du5TGieYsZz3J57Fr8B9cfec0X2M+buJ7jIVY+9RzpWILHaQBIgce7fEwmnWs8pkFYt+voCL80F19+JemaF9xrGdhDeuShL6VCCCGEECJ0NCkVQgghhBCho0mpEEIIIYQInUPgKR3Id7gvvkTrKbXHWB+q60tdvpn9Oud8/AekP/NpXof70vPcNednTmcPzPlTjI/uFfYZ7Zs/lq9l+w7+O2HS+CqnhtI89t2Ul7JndEQBewjbAzy4y6u57d+95xnS1c0232XQtRweZB+VjXjB3rXb69ax93fjf7/gHHNVyUjS+Wa97JUvsLmtvPJUp46utPE297K3qamdfanHJFxPacrkqW3eZnyoOXxMyzbrIATW1LEfODvGeUuXLFpMeuKEMqeOLUlux6olK0lPrjLvS4fb5+qTxjO3+EXSF1/7CdIzzuQcvgBQ//jjZovylg5E2tpw92Gp96j5ZUiYJcZ3TDAHnIb951tGu0uQu9gyhUaPP4B27Auu7ZAZxw7CH6wK8I8G5FA9kul8K4WV1Uve1pOm8hhREpCXM7mJx6o8k6d0XCl3xPZxbm7nK866nvScCtbLb+LxfmcX5+wGgHzkmS32pRnYU2p/MbPNlliUO+/9v3R9+MMRfSkVQgghhBCho0mpEEIIIYQIHU1KhRBCCCFE6GhSKoQQQgghQucQBDplGz1QYFP/ifGD67TaBhMBdv6dNOf54S842KM8IOik4hI2VU8/hTM1u4FOQe54e/0cQDJ3Bjv5L5431akhp7uedDLN13L/Mk7Cu+A5DkoBgFWN2wLa1peo0UHPZX+fbThEoseguGpvZMYHR5XS/pe2uMEy6+vXk47m8/0oG8N9oafXvfaNDWzaL45xxMnsEzmIrTHpPpPe7k4u02QXdeAAi1SHG5DWneHM3/llfC2tbRw81b3UTa9ePr6Sy3Tw9Rb2cGDAjR93s6kvXL6EdI1ZgODkZr6215vcAIQFv3UTTIv+abTrKbQaXQkHG6PXZJLlt77HHBCQ5x23Gm3Xl7gn4BjLA0bbeMInjL4soA43nubg48bf7D9LArbNCtg2RMkfMQIzz977e3bVeZ/n/bCdCIjdxDfu6d9wAG7FeO5EhRXujc6OOCsdkIoWl5EujrDe0xJS9tfPjrJBv3ydRueYeUmyhce3ujoFbQL6UiqEEEIIIYYAmpQKIYQQQojQ0aRUCCGEEEKEziB7SoO8nNZ9Yb2KFusHCcI6OAb2MtpWVBpf3Q1XXk56yvixTh21y9jLuXBp7YDndbE+TKani50orUn3fkSy2ZtYV8vtWvBYDelVrW370C57h6yLJqjdQc976LGr8y20rdjrRdwZyaX9ZQU2AzcQL2QTXd1f2B/akWRjXmlpASzxKHuZlq1+lXSiqJh0Jtf1gy7byoscJLfys8zFRNJbe9mjDAC5CX6WtR18/QVV7I1+v/FcAcAfHmVv18gYL9AQzedrrd/EfRIA8o1Pe3NDHemNTaybmtw6yir4PI1b92W8GN6kbjIbrA3vcrjcZ/RHWXZZ+3zQ8FBh9BlG70My+an1XHH11WasN0n8ywIsx42D4Sm922jXMs1Y/+wB8Lm8ic62OzvsjR+6xAviuPjyvSbfIA+p5eRJvEDNGR/40H6fd9x8Nh5PnsoL4+Tl87g7baprqm6t4PlBYYw7q32Fgrq/2wXM704J15kdsD7N0IzSOLjoS6kQQgghhAgdTUqFEEIIIUToaFIqhBBCCCFC5xDkKe0aQA9MSR77yMqNF2NMESfiKx/DfjcAKCtjh8fpp04i3d7G7Xp9i80yBnz12z8jva7T+tlyjQ7K7WldIuxGWfAKm6Iiue7fDZ+5YT7pE4x75dufZY/MytXsSwSA5as4d+nmds5V2dbFBpf2nfv/3IYKf+3OYGP93tybNSb9ZXyCzf0JNJvnUrd2A+kXjU/33CzOuQkAq7o42eOy19mD3Gg0Z/oMxqSPxPKtA3vM0jV8fSNzuY+deQYnP4yMcz22TRn2qo4oGEV6c4pzrNY1sgcXAJrf4DJHHc3vR92qZaTbt7l9blfv4dsPQ+OfD+AY4yGF7d42BXMpBuYn+9+M6q8O4KozQ+zWZEAZ+9I8Z7TNfRp0LVUB2/ojqJvan4cBuHN8wLs9CF7VQ4Xv++jO7H1ATWm+ntKo65n9YwPnC7/28/NI129gL39Ds5vbueFp9u43LFlNunLWJaRf/A2fEwDaTO7ma6+7hvTY8cZVGuAHnVzBOcZjER6H12/hMbInqO8OQ/SlVAghhBBChI4mpUIIIYQQInQ0KRVCCCGEEKHj+b6/74U9b98L76HEGGkqTmC/2olV7M0oKHANPV2d7O/MvMlrc5ePMjnERh3n1JEdYdNHYTF74n78y0dI/2Qx+1CCscYq6yENMhbZjGY5AWX60u1smTaa7+GnLmJP4Nzp7JfNjlhTFdCZaSe9qYUNLds7+X4tWdHo1PHsCvZZ1m9v+aPv+5OdgiFzQsTz7+mTRjRlHlN7QOrcTA4/p/Y0e9sWbeXyRgZiV5Q/VBai8+eyLjuF35dIhPtxZ9LtL90pvv6eDr6JsRxTR7dbR/0m9uGmTBFjFUdxwjXQvVzL3tZ1G50iB8KQ7LfAgY25DgdSg+2s1mZsh7Eg+3yAz474jdFB+VIHoOQOc0o3nACNc1h/kVPu4gecthI4a//bgSeMfiygzNlGX220tUcGXIuDN3T7bvnE0f6dv9273n2mm38Pe7rcTpNO86BQ38AveGsTv/8nVvBvHQAUjOJxo82MVX98lWMs7v33h5w6BsQEAMTi7sOafwb7UE+o4Bfidy/8J+mWDjfnctqknTZTIfTYWxiQtjnXDKNd7aZAeOnGA/uuvpQKIYQQQojQ0aRUCCGEEEKEjialQgghhBAidDQpFUIIIYQQoTOoyfMnHTfK2XbZWezuLsznwJ4Hf/co6bpkjVNH664A924/5AdkKc4Y133XPgUlWaxz3x5jHcM2GiAIe0xA1I1h5WaOQmi5m5P/vtLAwVE5Xe796zXJ0PMK+NpiCW7HxWe7hvJPnzeF9PiPf+/vtDhkekAG8HzzWILueCbbBKBlc38ZcxTr9bvcOuxdt/7yg0FQbEnF+ziCqKuD3fL/s5pDrta5awnAxAogZrrthWfzu99iFhcAgJSJ2WsxwTSN6805sozLH0BOwtkkBuLjRv+f0acFHPOq0Tb4Z7zR1XC5wGiTBz+2iHVHUOTfTUavYlneyHpUARwiJjd6njnPNF7DAiv5J2k3dvEA+1N3H8uSaW4VzcvNBhtM1mr07QHtOIzIifgYW7T3oefHeaRtaGx0jsmYn9SiotGk42fMJF2WqHDqeGUtR7KVT+JIn4px5VwnznHqaG7mYKhYjL/fdZlA4ZnTL3PqmFHFL0BHiuvsjnBy/Zp17gIsnakm0iPfzQNgopB1TbXpzABetzHb5gevtILnS7E8d5Dd0sQvTfvGg7eIib6UCiGEEEKI0NGkVAghhBBChI4mpUIIIYQQInT2N3n+GwD+fPCaIw5zjvd9/91hN8KifisGYEj2W0B9VwyI+q44XAnsu/s1KRVCCCGEEOJgoH++F0IIIYQQoaNJqRBCCCGECB1NSoUQQgghROhoUiqEEEIIIUJHk1IhhBBCCBE6mpQKIYQQQojQ0aRUCCGEEEKEjialQgghhBAidDQpFUIIIYQQofP/AcWEY+sJELIVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x180 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's see some of the images\n",
    "def convert_to_imshow_format(image):\n",
    "    # first convert back to [0,1] range from [-1,1] range - approximately...\n",
    "    image = image / 2 + 0.5\n",
    "    image = image.numpy()\n",
    "    # convert from CHW to HWC\n",
    "    # from 3x32x32 to 32x32x3\n",
    "    return image.transpose(1,2,0)\n",
    "\n",
    "trainloader = torch.utils.data.DataLoader(trainset, \n",
    "                                          batch_size=4,\n",
    "                                          shuffle=True)\n",
    "dataiter = iter(trainloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "fig, axes = plt.subplots(1, len(images), figsize=(12,2.5))\n",
    "for idx, image in enumerate(images):\n",
    "    axes[idx].imshow(convert_to_imshow_format(image))\n",
    "    axes[idx].set_title(classes[labels[idx]])\n",
    "    axes[idx].set_xticks([])\n",
    "    axes[idx].set_yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/fire-element.png\" style=\"height:50px;display:inline\"> Building a CNN-Classifier for CIFAR-10 with PyTorch\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_conv_cifar_arch.PNG\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "class CifarCNN(nn.Module):\n",
    "    \"\"\"CNN for the CIFAR-10 Datset\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        \"\"\"CNN Builder.\"\"\"\n",
    "        super(CifarCNN, self).__init__()\n",
    "\n",
    "        self.conv_layer = nn.Sequential(\n",
    "\n",
    "            # Conv Layer block 1\n",
    "            nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(32),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "\n",
    "            # Conv Layer block 2\n",
    "            nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(128),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "            nn.Dropout2d(p=0.05),\n",
    "\n",
    "            # Conv Layer block 3\n",
    "            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(256),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(kernel_size=2, stride=2),\n",
    "        )\n",
    "\n",
    "\n",
    "        self.fc_layer = nn.Sequential(\n",
    "            nn.Dropout(p=0.1),\n",
    "            nn.Linear(4096, 1024),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Linear(1024, 512),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.Dropout(p=0.1),\n",
    "            nn.Linear(512, 10)\n",
    "        )\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"Perform forward.\"\"\"\n",
    "        \n",
    "        # conv layers\n",
    "        x = self.conv_layer(x)\n",
    "        \n",
    "        # flatten\n",
    "        x = x.view(x.size(0), -1)\n",
    "        \n",
    "        # fc layer\n",
    "        x = self.fc_layer(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 256, 4, 4])\n",
      "torch.Size([1, 4096])\n",
      "num trainable weights:  5852170\n"
     ]
    }
   ],
   "source": [
    "# calculating the output size of the convolutional layers, before the FC layers\n",
    "dummy_input = torch.zeros([1, 3, 32, 32])\n",
    "dummy_model = CifarCNN()\n",
    "dummy_output = dummy_model.conv_layer(dummy_input)\n",
    "print(dummy_output.shape)\n",
    "dummy_output = dummy_output.view(dummy_output.size(0), -1)\n",
    "print(dummy_output.shape)\n",
    "# calculating the number of trainable weights\n",
    "num_trainable_params = sum([p.numel() for p in dummy_model.parameters() if p.requires_grad])\n",
    "print(\"num trainable weights: \", num_trainable_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/flat_round/64/000000/presentation.png\" style=\"height:50px;display:inline\"> Training the CNN Model\n",
    "---\n",
    "* So we have the model, but how do we train it to output the correct class of the input image?\n",
    "* As you have probably noticed, the output of the final fully-connected layer is a vector of length 10, which is exactly the number of classes we have!\n",
    "* We mentioned that we want entry $i$ of the final vector to be the probability of the input to be from class $i$.\n",
    "* But how do we force this vector to output probability and not just some numbers?\n",
    "* We consider the final output vector to represent *scores*, which we will normalize to be probabilities using the **Softmax** function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/s.png\" style=\"height:50px;display:inline\"> The Softmax Function\n",
    "---\n",
    "* The Softmax function is defined as: $$ Softmax(x_i) = \\frac{e^{x_i}}{\\sum_{j=1}^M e^{x_j}}, i \\in [1,...,M], x \\in \\mathcal{R}^M  $$\n",
    "* This forces the output vector to sum to 1, just like probabilities.\n",
    "* <img src=\"./assets/tut_conv_softmax.PNG\" style=\"height:150px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/plasticine/100/000000/unicorn.png\"  style=\"height:50px;display:inline\"> Making Predictions\n",
    "---\n",
    "* OK great, we have an output vector of probabilities, so how we predict the label of the input image?\n",
    "* Simple! Just take the $argmax$: $$ \\hat{y} = Softmax(CNN(x)) $$ $$ c_{pred} = argmax_i (\\hat{y})  $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/bearish.png\" style=\"height:50px;display:inline\"> Loss Function - Cross Entropy \n",
    "---\n",
    "* In order to train the model in an end-to-end fashion, we need to define a loss function which we can minimize using optimization techniques.\n",
    "* Let us assume that our model output (after softmax) is $\\hat{y}$ and the real label (the real class, given to us) is $y$.\n",
    "    * <img src=\"./assets/tut_conv_x_y.PNG\" style=\"height:150px\">\n",
    "* **Insight**: $$ argmax(\\hat{y}) = argmax(y) $$\n",
    "* Ideally, this is what we would want from our model, so what loss function would drive $\\hat{y}$ to be as close as possible to $y$?\n",
    "* As $y$ is a *one-hot vector* and $\\hat{y}$ represents probabilities, the **Cross Entropy** loss function fits right in!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Let $W$ denote the weights of the CNN, and $W^{*}$ the optimal weights.\n",
    "* In this case, the optimal weights are: $$ W^{*} \\leftarrow argmin_{W} \\left(-\\sum_{x, y} 1 \\cdot \\log (p_c) \\right) $$ $$ p_c = \\hat{y}_{c} $$\n",
    "* $c \\in [1,...,M]$ is the correct class, $\\hat{y}_{c}$ is the $c^{th}$ entry in the output vector $\\hat{y}$.\n",
    "* So the **Cross Entropy** loss function is: $$\\mathcal{L} = -\\log (p_c)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's analyze this loss fucntion, which represents how bad we are currently doing:\n",
    "$$ p_c = 0 \\rightarrow \\mathcal{L} = -\\log (0) = \\infty $$\n",
    "$$ p_c = 0.1 \\rightarrow \\mathcal{L} = -\\log (0.1) = 2.3 $$\n",
    "$$ p_c = 0.9 \\rightarrow \\mathcal{L} = -\\log (0.9) = 0.1 $$\n",
    "$$ p_c = 1 \\rightarrow \\mathcal{L} = -\\log (1) = 0 $$\n",
    "\n",
    "**The larger the loss, the worse the prediction**. We want to minimize it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/compress.png\" style=\"height:50px;display:inline\"> Minimizing the Loss Function with Gradient Descent\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_conv_gd.gif\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In order to perform Gradient Descent, we need to calculate the dervatives with respect to the network's weights.\n",
    "* Due to memory reasons, when the dataset is large, we cannot compute the gradients on the whole dataset, and thus we train in **mini-batches** of the data. The `batch_size` is a *hyper-parameter* which needs to be tuned. Usually the sizes are 32, 64, 128...\n",
    "* In order to propagate the gradients through all of the layers, we need to use the chain rule when calculating the gradients. This is called **backpropagation**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/plasticine/100/000000/serial-tasks.png\" style=\"height:50px;display:inline\"> Backpropagation\n",
    "---\n",
    "* Denote the output of the $k^{th}$ layer as $f(Z^{(k)}) $ and the input to the next layer $Z^{(k+1)}$.\n",
    "* **Forward Pass**: $Z^{(k+1)} = f(Z^{(k)}) $\n",
    "* **Backward Pass**: $\\delta^{(k+1)} = \\frac{\\partial E}{\\partial Z^{(k+1)}}$\n",
    "* Applying the **chain rule** for a single layer: $$ \\frac{\\partial E}{\\partial Z^{(k)}} = \\frac{\\partial E}{\\partial Z^{(k+1)}} \\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = \\delta^{(k+1)}\\frac{\\partial Z^{(k+1)}}{\\partial Z^{(k)}} = \\delta^{(k+1)}\\frac{\\partial f(Z^{(k)})}{\\partial Z^{(k)}} $$\n",
    "* The **gradient with respect to layer parameters** (if it has any): $$ \\frac{\\partial E}{\\partial W^{(k)}} = \\frac{\\partial E}{\\partial Z^{(k+1)}} \\frac{\\partial Z^{(k+1)}}{\\partial W^{(k)}} = \\delta^{(k+1)} \\frac{\\partial Z^{(k+1)}}{\\partial W^{(k)}}  $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_conv_backprop.jpg\" style=\"height:400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/64/000000/discount.png\" style=\"height:50px;display:inline\"> Learning Rate\n",
    "---\n",
    "* As we use Gradient Descent, we also have the **learning-rate** *hyper-parameter* which needs to be tuned.\n",
    "* <img src=\"./assets/tut_conv_lr.PNG\" style=\"height:250px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# ------------------------- <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> CODE TIME -------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# time to train our model\n",
    "# hyper-parameters\n",
    "batch_size = 128\n",
    "learning_rate = 1e-4\n",
    "epochs = 20\n",
    "\n",
    "# dataloaders - creating batches and shuffling the data\n",
    "trainloader = torch.utils.data.DataLoader(\n",
    "    trainset, batch_size=batch_size, shuffle=True, num_workers=2)\n",
    "testloader = torch.utils.data.DataLoader(\n",
    "    testset, batch_size=batch_size, shuffle=False, num_workers=2)\n",
    "\n",
    "# device - cpu or gpu?\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# loss criterion\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# build our model and send it to the device\n",
    "model = CifarCNN().to(device) # no need for parameters as we alredy defined them in the class\n",
    "\n",
    "# optimizer - SGD, Adam, RMSProp...\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# function to calcualte accuracy of the model\n",
    "def calculate_accuracy(model, dataloader, device):\n",
    "    model.eval() # put in evaluation mode\n",
    "    total_correct = 0\n",
    "    total_images = 0\n",
    "    confusion_matrix = np.zeros([10,10], int)\n",
    "    with torch.no_grad():\n",
    "        for data in dataloader:\n",
    "            images, labels = data\n",
    "            images = images.to(device)\n",
    "            labels = labels.to(device)\n",
    "            outputs = model(images)\n",
    "            _, predicted = torch.max(outputs.data, 1)\n",
    "            total_images += labels.size(0)\n",
    "            total_correct += (predicted == labels).sum().item()\n",
    "            for i, l in enumerate(labels):\n",
    "                confusion_matrix[l.item(), predicted[i].item()] += 1 \n",
    "\n",
    "    model_accuracy = total_correct / total_images * 100\n",
    "    return model_accuracy, confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 | Loss: 1.5168 | Training accuracy: 56.276% | Test accuracy: 57.460% | Epoch Time: 61.60 secs\n",
      "Epoch: 2 | Loss: 1.0972 | Training accuracy: 65.480% | Test accuracy: 66.850% | Epoch Time: 82.20 secs\n",
      "Epoch: 3 | Loss: 0.9108 | Training accuracy: 70.738% | Test accuracy: 70.790% | Epoch Time: 85.79 secs\n",
      "Epoch: 4 | Loss: 0.8017 | Training accuracy: 71.934% | Test accuracy: 72.200% | Epoch Time: 82.55 secs\n",
      "Epoch: 5 | Loss: 0.7141 | Training accuracy: 77.510% | Test accuracy: 77.880% | Epoch Time: 90.47 secs\n",
      "Epoch: 6 | Loss: 0.6533 | Training accuracy: 78.744% | Test accuracy: 77.530% | Epoch Time: 80.15 secs\n",
      "Epoch: 7 | Loss: 0.6037 | Training accuracy: 80.628% | Test accuracy: 79.630% | Epoch Time: 83.42 secs\n",
      "Epoch: 8 | Loss: 0.5586 | Training accuracy: 83.364% | Test accuracy: 81.740% | Epoch Time: 89.09 secs\n",
      "Epoch: 9 | Loss: 0.5291 | Training accuracy: 83.362% | Test accuracy: 81.300% | Epoch Time: 81.84 secs\n",
      "Epoch: 10 | Loss: 0.4992 | Training accuracy: 84.848% | Test accuracy: 82.760% | Epoch Time: 83.53 secs\n",
      "Epoch: 11 | Loss: 0.4725 | Training accuracy: 85.310% | Test accuracy: 83.250% | Epoch Time: 88.62 secs\n",
      "Epoch: 12 | Loss: 0.4396 | Training accuracy: 86.168% | Test accuracy: 83.710% | Epoch Time: 92.91 secs\n",
      "Epoch: 13 | Loss: 0.4224 | Training accuracy: 86.700% | Test accuracy: 84.120% | Epoch Time: 79.77 secs\n",
      "Epoch: 14 | Loss: 0.4053 | Training accuracy: 87.594% | Test accuracy: 84.380% | Epoch Time: 80.96 secs\n",
      "Epoch: 15 | Loss: 0.3809 | Training accuracy: 88.878% | Test accuracy: 85.430% | Epoch Time: 80.96 secs\n",
      "Epoch: 16 | Loss: 0.3620 | Training accuracy: 88.470% | Test accuracy: 84.940% | Epoch Time: 82.44 secs\n",
      "Epoch: 17 | Loss: 0.3496 | Training accuracy: 89.608% | Test accuracy: 85.490% | Epoch Time: 80.95 secs\n",
      "Epoch: 18 | Loss: 0.3281 | Training accuracy: 89.878% | Test accuracy: 85.800% | Epoch Time: 83.10 secs\n",
      "Epoch: 19 | Loss: 0.3210 | Training accuracy: 90.690% | Test accuracy: 86.050% | Epoch Time: 80.14 secs\n",
      "Epoch: 20 | Loss: 0.3026 | Training accuracy: 88.678% | Test accuracy: 84.950% | Epoch Time: 82.72 secs\n",
      "==> Saving model ...\n",
      "==> Finished Training ...\n"
     ]
    }
   ],
   "source": [
    "# training loop\n",
    "for epoch in range(1, epochs + 1):\n",
    "    model.train()  # put in training mode\n",
    "    running_loss = 0.0\n",
    "    epoch_time = time.time()\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # get the inputs\n",
    "        inputs, labels = data\n",
    "        # send them to device\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "\n",
    "        # forward + backward + optimize\n",
    "        outputs = model(inputs)  # forward pass\n",
    "        loss = criterion(outputs, labels)  # calculate the loss\n",
    "        # always the same 3 steps\n",
    "        optimizer.zero_grad()  # zero the parameter gradients\n",
    "        loss.backward()  # backpropagation\n",
    "        optimizer.step()  # update parameters\n",
    "\n",
    "        # print statistics\n",
    "        running_loss += loss.data.item()\n",
    "\n",
    "    # Normalizing the loss by the total number of train batches\n",
    "    running_loss /= len(trainloader)\n",
    "\n",
    "    # Calculate training/test set accuracy of the existing model\n",
    "    train_accuracy, _ = calculate_accuracy(model, trainloader, device)\n",
    "    test_accuracy, _ = calculate_accuracy(model, testloader, device)\n",
    "\n",
    "    log = \"Epoch: {} | Loss: {:.4f} | Training accuracy: {:.3f}% | Test accuracy: {:.3f}% | \".format(epoch, running_loss, train_accuracy, test_accuracy)\n",
    "    epoch_time = time.time() - epoch_time\n",
    "    log += \"Epoch Time: {:.2f} secs\".format(epoch_time)\n",
    "    print(log)\n",
    "    \n",
    "    # save model\n",
    "    if epoch % 20 == 0:\n",
    "        print('==> Saving model ...')\n",
    "        state = {\n",
    "            'net': model.state_dict(),\n",
    "            'epoch': epoch,\n",
    "        }\n",
    "        if not os.path.isdir('checkpoints'):\n",
    "            os.mkdir('checkpoints')\n",
    "        torch.save(state, './checkpoints/cifar_cnn_ckpt.pth')\n",
    "\n",
    "print('==> Finished Training ...')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test accuracy: 84.950%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAFzCAYAAAA0dtAgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZxldX3n/9e7G8ImAtK4DkwLAQkCIt3EBdCG4WFUBBcGFTeIS4uOuIxEzTBjOiYoE0xmxqBo40RwZ5OHBGcAIbIqyNpNoyz5CbjGoRVQVqX78/vjnJZrWVXcaurWrVvn9Xw86lHnfs9yP/fUqfu+Z7nfk6pCkiTNffOGXYAkSZoZhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehPIslFSRYPu465KsnCJKvGaf9skl36mH9JknMGU930aGt8/iyoY1mSo4ddx/qY7bUneXeS7yf50rBrmchE/2ujLMntSRaM035wkg8NqaYtk7xzmpY1kPc3Q18AJNlg2DWsU1VvrarvjW1PMn8Y9TxGS4Chh/50mE3byCzzTuClVfX6dQ1zaV2N2mupqrOr6rghPf2WNNvD75lN712GPr/7FHxTklOSrExyRpJNx0xzYpKrk9yY5K972m9P8tdJrk1yQ5Kd2/bNkvxTkquSXJfk5TP4et7Uvo4VSb6Q5KAkV7Z1XJDkSe10y5IsT3I+8PmZqm+MDcau994jLEnuTfKRJFcCz0vy4vZvdRnwqiHV3Nc6TrIQOBJ4X5Lrk+w7wzUek+TmJBcAz2jbdkhybpJrklzas71uk+TMdnu9KsnebftQtpEJat8jyRXtej8ryVZt+15t23eSHD+Te7RJPg1sD5yd5J7edZVk4ySfa98XrkuyXzvPpklOa2s+td1uZuKI4vwkJ7XvYecn2WSSdXpRko8muRh4T5JDk6xqt/dL2mnmt+v7qnb+tw+q8Pb99Bvt869K8pp21FHjvPcekeSEdvjkJJ9ut/VbkrxsUDW2jgN2aP/fr0ryrSRfBm7ImKMtSY5Osqwd/uP2fWNF+3p2GPP692q3oe0fc4VV1fkfYCFQwN7t438CjgYuAha3bU9of89v23dvH98OHNUOvxP4bDv8UeAN7fCWwC3AZjPwWp4J3AwsWFc3sBWQ9vFbgb9vh5cB1wCbzOL1XsCr2+GNgR8BOwIBTgPOGULdU13HRw+hxkXADcCmwOOBf23X7YXAju00zwH+pR3+MrBPO7wd8P1hbSOT1L4SeGE7zUeA/9kOrwKe3w4fB6ya4XV9O7Bg7LoC3g98rh3eGfhhuw0fDXymbd8VeHjd9j7AGhe2z7NH+/g04A2TrNOLgE/1zH8D8LR2eMv291Lgv7bDGwFXA08fUP2HACf1PN6Cid97jwBOaIdPBs6l2cHdEfgxsPGA1/OqdngJcN+6ddI7rn18NLCsHb4SeGU7vHG77S8BzqE5UngNsN101Oie/iN+VFWXt8NfBPYZM/7VSa4FrqN50+895/y19vc1NH9YgBcBH0pyPc0/0MY0b6aDtj9wRlWtBqiqXwL/DjgvyQ3AX9DUv87ZVfXADNQ1kUdb72uAM9vhnYHbqurWav47vjhDNY411XU8DPsCZ1XV/VX1K+Bsmm3w+cDp7Xb5GeAp7fQHACe07WcDj0+yeTtupreR8WrfjCZsLm6nOQV4QZItgc2r6ttt+5dnsM7x9K6rfYAvAFTVTcAdwE5t+1fb9lU0wTsTbquq69vha4AdGGed9kx/as/w5cDJSd5Gs+MDzXvcm9pt5kpga5pgHYQbgAOS/Pck+1bVPW37eO+9Y51WVWur6lbgBzTvIzPlu1V122QTtP9nT6uqswCq6sGqur8d/SfAcuCgqvrhdBQ0UudqBmzsTQh+9zjJ02k+le1VVXclOZnmDXSdh9rfa3hknQY4pKpuHky5Ewp/+Fr+EfiHqjo7yRKaPZJ17puhuiYy4XpvPVhVayYZPwxTXcfDMrbGecDdVbXHONPOA543NtyTwHC2kX7/zhloFVPXu64mqm1YNT/UM7yG5gjkZH73WqrqyCTPAQ4Erk+yB83rOKqqzpv2SseoqluSLAJeCnysPYUC47/3/sHsj/J4kHq3h4f5/VPq6zJksu3hZ+10zwZ+Oh0Fuaf/iO2SPK8dPgy4rGfc42n+ePekOR/+kj6Wdx7N+aYAJHn2dBY7iQtpjkps3T7vE2gOhf2kHX/4DNXRr8nW+1g3AU/vOd912EArm9hU1vGvgc2ZeZcAr2zP224OHATcD9yW5FCANJ7VTn8+8K51M7dv6sMyXu33AXflkesi3ghcXFV3Ab9O8ty2/bUzX+6ELgFeD5BkJ5ojfTfTbOOvbtt3AXYbUn33MM46HW/CJDtU1ZVV9WFgNbAtzXvcO5Js2E6zU5LNBlFokqcC91fVF4GPA3tOYfZDk8xr3ze2p/kbDMpk/+8/B56YZOskGwEvA2iPZv04ySsAkmyUR64pu5vmg9ZH252Jx8zQf8T3gcOTrKQ5R3viuhFVtYLmsP6NNOedLx93Cb/vb4ANgZXtxRt/M+0Vj6OqbgSOBS5OsgL4B5q9ztOTXErzDzubTLjex6qqB2nOI34jzYV8d8xMiX9Qx1TW8T/TBNiMXshXVdfSHJ69nub0yKXtqNcDb2nrvhFYd4Hpu4HF7QVZ36O5AHEoJqn9cOD4dlvZg+YcNMBbgOVJvkOz13QPs8OnaC6eu4Hm9RxRVQ+17du0r+ODNIf3h1XzROt0rOPbi+VW0XyYWQF8FvgecG3b/hkGd/R4N+C77amEY4C/ncK8N9N8mPm/wJHt+8hAVNUvgMvb9XH8mHG/pVm/V9Kcq7+pZ/QbgXe3f4dvA0/ume/nNB98P9kebXlM1l141GlprrI+p6p2HXIpkqYoyeOq6t52+EPAU6rqPUMua0Jpvr61YVU92O59XgjsVFW/GXJpc057Kvacqjpj2LXMFp7TlzTqDkzylzTvZ3fQXL09m20KfKs9LB7gHQa+Zop7+pIkdYTn9CVJ6ghDX5Kkjpjx0I83sZEkaSjc019PSZYOu4apsubBG7V6wZpnwqjVC9Y8E4ZR78BCP3PsJjbjGKmNq2XNgzdq9YI1z4RRqxeseSbMndBvPQNYXlW7A7/iD285eExVLQZ2B16YZPeecaurak+azlrW3Uv7GJobhOwF7EfTYcRAeoCSJGmuGdhX9toOby6pqu3ax/vT9Pq1Jc1dx65OciTNJ50NaG78cVRVfTXJ7TR3XvtJ2wPRsVV1QJKrafohfrh9micAf1ZV3x/z3EtZ9wlq/kaLsvmTmW710K/JRoPpXXWP7RcMZLmrV9/JggXbDGTZg7L6zjtZsM3o1Dxq9cKIbhcDqnlQneLfufpOthnQOh7Ul67dLh4xqHX8izvvZOsBvF/88I7b+cXq1eNuzoPunGcoN7GpquU0dyZi3lYLa6P9/9t6v4BhuOwrbx52CVOyZu3o9fWQ2Xablj7MG7GiR2+rgPnzRmsdA6wdsf+/0aq2MWrvcS/c+08nHDfow/tz5SY2kiSNvEGH/py4iY0kSXPBoA/vr62qsXfrWrJuoKqOGG+mqlrYM3z1unnae32/fZprlCSpE/yeviRJHTGwPf2quh3wVrWSJM0S7ulLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHTHoG+4M3R7bL+Dyr47W/emf8IoThl3ClNz19aOGXYLUWfPmZdglTEnVaN2bHuCeBx8edglTsnbtxOvYPX1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOmLkQz/JBsOuQZKkUTCrAjPJm4CjgQJWAqcB/xX4I+AXwOur6udJlgFPBRYCq4HXDaNeSZJGyawJ/STPBI4B9q6q1UmeQBP+z62qSvJW4APA+9tZFgH7VNUDw6lYkqTRMmtCH9gfOKOqVgNU1S+T7AacmuQpNHv7t/VMf/ZEgZ9kKbAUYNvtthts1ZIkjYjZdE4/NHv2vf4ROKGqdgPeDmzcM+6+iRZUVcuranFVLV6wYJvpr1SSpBE0m0L/QuDVSbYGaA/vbwH8pB1/+LAKkyRpLpg1h/er6sYkxwIXJ1kDXAcsA05P8hPgCuDpQyxRkqSRNmtCH6CqTgFOGdP89XGmWzYjBUmSNIfMpsP7kiRpgAx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOmJW3XBnUB5eU8MuYUru+vpRwy5hShb91fnDLmHKLj/mPwy7hClbs3a0tuMHfrtm2CVM2RabbDjsEqYsGXYFU7PB/NHb19xi09HaLubPm3ijGL21L0mS1ouhL0lSRxj6kiR1hKEvSVJHGPqSJHWEoS9JUkcY+pIkdYShL0lSRxj6kiR1hKEvSVJHGPqSJHWEoS9JUkcY+pIkdYShL0lSRxj6kiR1xNBCP8nCJKvGaf9skl36mH9JknMGU50kSXPPBsMuYKyqeut47UnmV9Wama5HkqS5YtiH9zdIckqSlUnOSLJpkouSLAZIcm+SjyS5EnhekhcnuSnJZcCrhlu6JEmjZdih/wxgeVXtDvwKeOeY8ZsBq6rqOcDVwEnAQcC+wJMnWmiSpUmuTnL16jvvHEzlkiSNmGGH/o+q6vJ2+IvAPmPGrwHObId3Bm6rqlurqtrpx1VVy6tqcVUtXrDNNtNetCRJo2jYoV+P8vjBMefxx46XJEl9Gnbob5fkee3wYcBlk0x7E/D0JDv0TC9Jkvo07ND/PnB4kpXAE4ATJ5qwqh4ElgLfaC/ku2NmSpQkaW4Y2lf2qup2YLzv4y/pmeZxY+Y5l+bcviRJmqJh7+lLkqQZYuhLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEUO74c5M2mB+hl3CnHblhw8YdglT9mefmOwuzrPT/3nX3sMuYUp+vvqhYZcwZVtssuGwS5iyqmFXoFHinr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BEjGfpJliR5/rDrkCRplIxk6ANLAENfkqQpmFWhn+RNSVYmWZHkC0kOSnJlkuuSXJDkSUkWAkcC70tyfZJ9h1u1JEmjYYNhF7BOkmcCxwB7V9XqJE8ACnhuVVWStwIfqKr3J/k0cG9VfXyCZS0FlgJsu912M/QKJEma3WZN6AP7A2dU1WqAqvplkt2AU5M8Bfgj4LZ+FlRVy4HlAHsuWlwDqleSpJEymw7vh2bPvtc/AidU1W7A24GNZ7wqSZLmiNkU+hcCr06yNUB7eH8L4Cft+MN7pv01sPnMlidJ0mibNaFfVTcCxwIXJ1kB/AOwDDg9yaXA6p7J/xl4pRfySZLUv9l0Tp+qOgU4ZUzz18eZ7hZg9xkpSpKkOWLW7OlLkqTBMvQlSeoIQ1+SpI4w9CVJ6ghDX5KkjjD0JUnqCENfkqSOMPQlSeoIQ1+SpI4w9CVJ6ohHDf0kZyY5MIkfECRJGmH9BPmJwOuAW5Mcl2TnAdckSZIG4FFvuFNVFwAXJNkCOAz4ZpIfAScBX6yq3w64Rs1y85JhlzBl//eofYZdwpRt++YvDbuEKbntpMOGXcKUzZs3etvyg79dM+wSpmTDDUbvoPH8EdsuJqu2r7Xf3uP+COCtwHXA/wL2BL75WIuTJEkz41H39JN8DdgZ+AJwUFX9rB11apKrB1mcJEmaPpOGfnvx3vVV9arxxlfV4oFUJUmSpt2kh/erai3wkhmqRZIkDVA/5/TPT3JIMoJXa0mSpN951HP6wH8GNgPWJHmA5sLAqqrHD7QySZI0rfr5yt7mM1GIJEkarH729ElyMPCC9uFFVXXO4EqSJEmD0E83vMcB7wG+1/68p22TJEkjpJ89/ZcCe7RX8pPkFJoOej40yMIkSdL06rc/xC17hrcYRCGSJGmw+tnT/xhwXZJv0Vy5/wLgLwdalSRJmnb9XL3/lSQXAXvRhP4Hq+rfBl2YJEmaXv30vb9nO/jj9vdTk2wG3FFVDw+sMkmSNK36Obz/KZo76q2k2dPftR3eOsmRVXX++j55kmXAvVX18fVdhiRJ6k8/F/LdDjy7qhZX1SLg2cAq4ADg7wZYW1+S9NXXgCRJXddP6O9cVTeue1BV36P5EPCD9XnCJMckuTnJBcAz2rYdkpyb5JoklybZuW3fJsmZSa5qf/Zu25clWZ7kfODz61OHJEld089e8s1JTgS+2j5+DXBLko2A307lyZIsAl5Lc7RgA+Ba4BpgOXBkVd2a5Dk0pxT2B/4X8D+q6rIk2wHnAX/SLm4RsE9VPTCVGiRJ6qp+Qv8I4J3Ae2nO6V8GHE0T+PtN8fn2Bc6qqvsBkpwNbAw8Hzi950Z+G7W/DwB26Wl/fJJ19wI4e6LAT7IUWAqw7XbbTbFESZLmpn6+svdAkk8B51TVzWNG37sez1ljHs8D7q6qPcaZdh7wvLHh3n4IuG/CJ6haTnP0gD0XLR77fJIkdVI/fe8fDFwPnNs+3qPdQ18flwCvTLJJu8d+EHA/cFuSQ9vlJ8mz2unPB97VU8t4HwwkSVIf+rmQ76+APwXuBqiq64GF6/NkVXUtcCrNh4gzgUvbUa8H3pJkBXAj8PK2/d3A4iQrk3wPOHJ9nleSJPV3Tv/hqrqn57z6Y1JVxwLHjjPqxeNMu5rmwsGx7cumpRhJkjqkn9BfleR1wPwkO9LsfX97sGVJkqTp1s/h/aOAZwIPAV8G7gHeM8iiJEnS9OtnT//AqjoGOGZdQ3vR3ekDq0qSJE27fvb0x7uNrrfWlSRpxEy4p5/kJcBLgacl+UTPqMcD3l1PkqQRM9nh/Z8CVwMH03SVu86vgfcNsihJkjT9Jgz9qloBrEjy5aqaUh/7kiRp9unnQr6FST4G7ELTTz4AVbX9wKqSJEnTrp8L+T4HnEhzHn8/mlvZfmGQRUmSpOnXT+hvUlUXAqmqO9re8PYfbFmSJGm69XN4/8Ek84Bbk7wL+AnwxMGWJUmSpls/e/rvBTal6X53EfAG4PBBFiVJkqbfZN/T3xjYvKquapvuBf48yZNouuKVAJg3b3puxjSTfvObNcMuYcp+evIbhl3ClCw49DPDLmHK7jpz9G7kudEG/ey76bGoqmGXMCWTVTvZ1vIJYN9x2g8A/sdjqkiSJM24yUJ/n6r62tjGqvoS8ILBlSRJkgZhstCf7Jitx5MkSRoxk4X3/0vyp2Mbk+wF3Dm4kiRJ0iBM9pW9vwBOS3Iyj/S9vxh4E/DaAdclSZKm2YR7+lX1XeBPaQ7zH9H+BHhOVV05E8VJkqTpM2nnPFX1/4C/mqFaJEnSAHlBniRJHWHoS5LUEYa+JEkdMVk3vP/MJL35VdXBA6lIkiQNxGQX8n18xqqQJEkDN2HoV9XFM1mIJEkarEm/sgeQZEfgY8AuwMbr2qtq+wHWJUmSplk/F/J9DjgReBjYD/g88IVBFJNkWZKjB7FsSZK6rp/Q36SqLgRSVXdU1TJg/8GWJUmSpls/of9gknnArUneleSVwBOnq4AkxyS5OckFwDPatj2SXJFkZZKzkmzVtu/Vtn0nyfFJVk1XHZIkzXX9hP57gU2BdwOLgDcCh0/HkydZRHPznmcDrwL2akd9HvhgVe0O3MAjXQF/Djiyqp4HrJmOGiRJ6opHvZCvqq5qB+8F/nyan39f4Kyquh8gydnAZsCWPd8eOAU4PcmWwOZV9e22/cvAy8ZbaJKlwFKAbbfbbppLliRpNPVz9f63GKeTnqqarvP6E3YANLaUvhdYtRxYDrDnosX9Ll+SpDntUUMf6L2afmPgEJor+afDJcDJSY5razkI+AxwV5J9q+pSmtMJF1fVXUl+neS5VXUFzWkBSZLUp34O718zpunyJNPScU9VXZvkVOB64A7g0nbU4cCnk2wK/IBHTiu8BTgpyX3ARcA901GHJEld0M/h/Sf0PJxHczHfk6ergKo6Fjh2nFHPHaftxvbiPpJ8CLh6uuqQJGmu6+fw/jU0591Dc1j/Npo97mE4MMlf0tR9B3DEkOqQJGnk9BP6f1JVD/Y2JNloQPVMqqpOBU4dxnNLkjTq+vme/rfHafvOdBciSZIGa8I9/SRPBp4GbJLk2TzylbnH03TWI0mSRshkh/f/jOac+b8D/p5HQv9XwH8ZbFmSJGm6TRj6VXUKcEqSQ6rqzBmsSZIkDUA/5/QXtV3gApBkqyR/O8CaJEnSAPQT+i+pqrvXPaiqu4CXDq4kSZI0CP2E/vzer+gl2QQYylf2JEnS+uvne/pfBC5M8jmaTnreTHPrW0mSNEL66Xv/75KsBA6guYL/b6rqvIFXJkmSplU/e/pU1bnAuQBJ9k7yyar6TwOtTJIkTau+Qj/JHsBhwGto+t7/2iCLkiRJ02+yHvl2orln/WHAL2j6vE9V7TdDtU2LAEkedTqtv7Vra9glTNmG80dvm6garfX8s6+8bdglTNlW/3H5sEuYsl+ePnrrWcMz2Z7+TTT3tz+oqv4VIMn7ZqQqSZI07Sb7yt4hwL8B30pyUpL/wCNd8UqSpBEzYehX1VlV9RpgZ+Ai4H3Ak5KcmORFM1SfJEmaJo/aOU9V3VdVX6qql9HcfOd64EMDr0ySJE2rfnrk+52q+mVVfaaq9h9UQZIkaTCmFPqSJGl0GfqSJHWEoS9JUkcY+pIkdYShL0lSRxj6kiR1hKEvSVJHGPqSJHWEoS9JUkcMLfSTvDvJ95N8aVg1SJLUJZPdWnfQ3gm8pKpuW9eQZIOqeniINUmSNGcNZU8/yaeB7YGzk9yTZHmS84HPJ9k4yeeS3JDkuiT7tfNsmuS0JCuTnJrkyiSLh1G/JEmjaCh7+lV1ZJIXA/sB7wIOAvapqgeSvL+dZrckOwPnJ9mJ5sjAXVW1e5Jdae72N64kS4GlANtut92AX40kSaNhtlzId3ZVPdAO7wN8AaCqbgLuAHZq27/atq8CVk60sKpaXlWLq2rxNgu2GWjhkiSNitkS+vf1DGeCaSZqlyRJfZgtod/rEuD1AO1h/e2Am4HLgFe37bsAuw2rQEmSRtFsDP1PAfOT3ACcChxRVQ+17dskWQl8kObw/j3DK1OSpNEytK/sVdXCdnDZmPYHgSPGmeVB4A1V9WCSHYALac73S5KkPgzze/pTtSnwrSQb0pzff0dV/WbINUmSNDJGJvSr6teA38uXJGk9zcZz+pIkaQAMfUmSOsLQlySpIwx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI6YmS64V1fBVTVsMuYkiTDLmFK5s0brXoBNvTz7sCtrbXDLmHK7jpj6bBLmLKtDvz7YZcwJXd94/3DLmHKRixCJuU7nyRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEQML/SQLk6wa1PIlSdLUzMo9/SQbDLsGSZLmmkGH/vwkJyW5Mcn5STZJskeSK5KsTHJWkq0AklyU5KNJLgbek+TQJKuSrEhySTvN/CTHJ7mqnf/tA65fkqQ5Y9ChvyPwyap6JnA3cAjweeCDVbU7cAPwVz3Tb1lVL6yqvwc+DPxZVT0LOLgd/xbgnqraC9gLeFuSp4990iRLk1yd5OrVq+8c2IuTJGmUDDr0b6uq69vha4AdaIL94rbtFOAFPdOf2jN8OXBykrcB89u2FwFvSnI9cCWwNc0Hi99TVcuranFVLV6wYJvpezWSJI2wQZ87f6hneA2w5aNMf9+6gao6MslzgAOB65PsAQQ4qqrOm/ZKJUma42b6Qr57gLuS7Ns+fiNw8XgTJtmhqq6sqg8Dq4FtgfOAdyTZsJ1mpySbzUDdkiSNvGFcJX848OkkmwI/AP58gumOT7Ijzd79hcAKYCWwELg2SYA7gVcMvGJJkuaAgYV+Vd0O7Nrz+OM9o587zvRLxjx+1XiLBf5L+yNJkqZgVn5PX5IkTT9DX5KkjjD0JUnqCENfkqSOMPQlSeoIQ1+SpI4w9CVJ6ghDX5KkjjD0JUnqCENfkqSOMPQlSeoIQ1+SpI4Yxl32ZtTaKu598OFhlzElj9t4tP4szQ0PR8sIljxy63nTjUZrOx5Vd33j/cMuYUq2es3/HnYJU/bTLxwx7BKmpGrice7pS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdMWtCP8ntSRaM035wkg8NoyZJkuaSWX/D66o6Gzh72HVIkjTqhrKnn2SzJN9IsiLJqiSvaUcdleTaJDck2bmd9ogkJ7TDJyf5dJJLk9yS5GXDqF+SpFE0rMP7LwZ+WlXPqqpdgXPb9tVVtSdwInD0BPMuBF4IHAh8OsnGgy5WkqS5YFihfwNwQJL/nmTfqrqnbf9a+/samnAfz2lVtbaqbgV+AOw8doIkS5NcneTqX6xePd21S5I0koYS+lV1C7CIJvw/luTD7aiH2t9rmPh6g3qUx1TV8qpaXFWLt17wB9cGSpLUScM6p/9U4P6q+iLwcWDPKcx+aJJ5SXYAtgduHkSNkiTNNcO6en834Pgka4HfAu8Azuhz3puBi4EnAUdW1YODKVGSpLllKKFfVecB541pXtgz/mpgSTt8MnByz3SXV9X7BlqgJElz0KzpnEeSJA3WrO+cp1dVHTHsGiRJGlXu6UuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRhr4kSR1h6EuS1BGGviRJHWHoS5LUEYa+JEkdYehLktQRI9X3/vqYl/C4jUfrZa6tYVcwNfMz7AokjYq7Tn3LsEuYsq32+cCwS5iSh2758YTj3NOXJKkjDH1JkjrC0JckqViComcAAAaBSURBVCMMfUmSOsLQlySpIwx9SZI6wtCXJKkjDH1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI6wtCXJKkjBh76SbZM8s5pWtaSJOdMx7IkSeqamdjT3xL4g9BPMn8GnluSJLVmIvSPA3ZIcn2Sq5J8K8mXgRuSLEyyat2ESY5Osqwd/uMkFyRZkeTaJDv0LjTJXkmuS7L9DLwGSZJG3gYz8BwfAnatqj2SLAG+0T6+LcnCSeb7EnBcVZ2VZGOaDyjbAiR5PvCPwMur6oeDLF6SpLliJkJ/rO9W1W2TTZBkc+BpVXUWQFU92LYD/AmwHHhRVf10gvmXAkvbh/du+kfzbp6m2nstAFYPYLmDZM2DN2r1gjXPhFGrF6x5Jgyq3n8/0YhhhP59PcMP8/unGDZuf2eS+X/WTvdsYNzQr6rlNB8MBibJ1VW1eJDPMd2sefBGrV6w5pkwavWCNc+EYdQ7E+f0fw1sPsG4nwNPTLJ1ko2AlwFU1a+AHyd5BUCSjZJs2s5zN3Ag8NH2dIEkSerDwEO/qn4BXN5esHf8mHG/BT4CXAmcA9zUM/qNwLuTrAS+DTy5Z76fAwcBn0zynMG+AkmS5oYZObxfVa+bZNwngE+M034rsP+Y5h8AF7Xjfwg8c/qqnLKBnj4YEGsevGmtN8ka4Aaa/9XvA4dX1f3ruawlwNFV9bIkBwO7VNVxjFNzki2B11XVp6b4HMuAe6vq4+OMexPwAZrTdwH+abzpeqZ/BXBLVX1vnNGd3i5miDUP3ozXm6qa6eeU1Kck91bV49rhLwHXVNU/9IwPzf/x2j6WtYQ29PuYdiFwTlXtOsV6lzFO6Cd5CXAs8LKq+mn7jZw3VtVJkyzr5LaGM6ZSwxTrnV9Vawa1fGm2sRteaXRcCvxx27/F95N8CrgW2DbJi5J8p+3T4vQk6z4ovDjJTUkuA161bkFJjkhyQjv8pCRntX1irGi/Etvbv8bx7XR/0fa1sTLJX/cs65gkNye5AHjGBLX/Jc0Hjp9C842cdYGf5G3tclckOTPJpm0NBwPHtzXs0P6cm+SaJJcm2bmdf4ckV7TL+EiSe9v2JDk+yaokNyR5Tdu+ZEx/IX+T5D09r+fYJO9+rH8saVaqKn/88WeW/tDsNUNzeP/rwDuAhcBa4LntuAXAJcBm7eMPAh+m+ZbLj4AdaQ6nn0az5wxwBHBCO3wq8N52eD6wRfscq3rqeBHNocjQ7CycA7wAWERz+mFT4PHAv9KE+9jX8Utgiwle49Y9w38LHNUOnwz8x55xFwI7tsPPAf6lHT4HOKwdPrJnnR0CfLN9TU8Cfgg8BVhC8y2ip7fTLQSubYfnAf9fb03++DOXfobxlT1J/dskyfXt8KXA/waeCtxRVVe07c8FdqG5YBbgj4DvADsDt1VzfQxJvsgj/Vf02h94E0A1h7rvSbLVmGle1P5c1z5+HM2Hic2Bs6q9ziDJ2evxGndN8rc0XXY/Djhv7ATtkYvnA6e3rxFgo/b384BXtMNfBtadWtgH+Er7mn6e5GJgL+BX9PQXUlW3J/lFkmfTfDi4rpoLkKU5x9CXZrcHqmqP3oY29Hr7uwjwzao6bMx0ewDTddFOgI9V1WfGPMd7+3yOG2mOCvzLOONOBl5RVSuSHEGzJz7WPODuseuij5onct+Yx5+lOfrxZOCfpvAc0kjxnL40+q4A9k7yxwDtOfGdaL4C+/Se+1YcNsH8F9KcNiDJ/CSP5w/71zgPeHPPtQJPS/JEmtMKr0yySduT5kETPMfHgL9L8uR2/o16zptvDvwsyYbA63vm+V0N1fTdcVuSQ9v5k+RZPa//kHb4tT3zXwK8pn1N29CcjvjuBPWdBbyY5kjAHxxpkOYKQ18acVV1J81e6lfafi2uAHaupvvqpcA32gv57phgEe8B9ktyA3AN8Mzq6V8jyfFVdT7NofPvtNOdAWxeVdfSXBNwPXAmzSmI8Wr8P8AngQuS3Ng+z7ojjf+Npq+Ob/L7fXV8FfiLNDfW2oHmA8FbkqygOXLw8na69wL/Ocl3ac7Z39O2nwWsBFbQHGH4QFX92wT1/Qb4FnBaeTW/5jC/sidppKXprfOBqqokr6W5qO/ljzbfmGXMo/kmxKHrroGQ5iLP6UsadYuAE9o+C+4G3jyVmZPsQvMNgLMMfM117ulLktQRntOXJKkjDH1JkjrC0JckqSMMfUmSOsLQlySpIwx9SZI64v8H3jKE/PHbJMsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load model, calculate accuracy and confusion matrix\n",
    "model = CifarCNN().to(device)\n",
    "state = torch.load('./checkpoints/cifar_cnn_ckpt.pth', map_location=device)\n",
    "model.load_state_dict(state['net'])\n",
    "\n",
    "test_accuracy, confusion_matrix = calculate_accuracy(model, testloader, device)\n",
    "print(\"test accuracy: {:.3f}%\".format(test_accuracy))\n",
    "\n",
    "# plot confusion matrix\n",
    "fig, ax = plt.subplots(1,1,figsize=(8,6))\n",
    "ax.matshow(confusion_matrix, aspect='auto', vmin=0, vmax=1000, cmap=plt.get_cmap('Blues'))\n",
    "plt.ylabel('Actual Category')\n",
    "plt.yticks(range(10), classes)\n",
    "plt.xlabel('Predicted Category')\n",
    "plt.xticks(range(10), classes)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (2): ReLU(inplace=True)\n",
      "  (3): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (4): ReLU(inplace=True)\n",
      "  (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (6): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (8): ReLU(inplace=True)\n",
      "  (9): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (10): ReLU(inplace=True)\n",
      "  (11): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (12): Dropout2d(p=0.05, inplace=False)\n",
      "  (13): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (14): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  (15): ReLU(inplace=True)\n",
      "  (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (17): ReLU(inplace=True)\n",
      "  (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      ")\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAADaCAYAAABzYwkwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAeQUlEQVR4nO3debQlVZWg8W9DAimkKaKCAkoqCiIsSAptqnQhWEyN1eKQSEOhUNaimwZRUUR0WQgIKs4DkwtBZFRmLTHVRpmknKCkQJEsSiBBkBQomWdx9x8Rqaey890H55oRIff7rXVXrnfjnjz77Xci7tk3Is6NzESSJEmS1Fiu7wAkSZIkaUgskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVOi8SIqI1SLivIh4ICJuioi/7zqGIYqIfSPiioh4JCK+0nc8QxIRK0XECe14uS8iroyIHfqOaygi4tSIuC0i7o2I6yJiz75jGpqIeElEPBwRp/Ydy1BExMVtTu5vH//ed0xDEhG7RMS17XvV9RGxRd8xDUExXhY/Ho+II/uOawgiYk5EzI+IuyJiUUQcFREz+o5rCCJig4i4MCLuiYhfRcQb+46pL6PmexGxdUQsiIgHI+KiiFinpzB7MVVuImLFiDg7IhZGREbEVl3E08eZpKOBR4E1gN2AYyNiwx7iGJrfAIcDX+47kAGaAfwa2BJ4BnAQcGZEzOkxpiH5GDAnM2cDOwKHR8RmPcc0NEcDl/cdxADtm5mz2sf6fQczFBGxLfBx4G3A04FXAzf0GtRAFONlFs37+EPAWT2HNRTHALcDzwPm0rxn7dNrRAPQForfAM4HVgP+N3BqRKzXa2D9Wep8LyKeDZxLM8dZDbgCOKPz6Po1ai58GfAWYFFXwXRaJEXEKsA84KDMvD8zLwP+GXhrl3EMUWaem5lfB/6z71iGJjMfyMxDMnNhZv4hM88HbgQsBIDMvCYzH1n8Y/tYt8eQBiUidgHuBr7fdyz6i3Eo8OHM/HF7zLk1M2/tO6gB2ommKPhB34EMxAuBMzPz4cxcBHwH8ENgeCmwJvDZzHw8My8E/oUJnfuNmO+9CbgmM8/KzIeBQ4BNIuKlXcfYl6lyk5mPZubn2rrh8a7i6fpM0nrA45l5XfHcVXgQ0ZMQEWvQjKVr+o5lKCLimIh4EFgA3AbM7zmkQYiI2cCHgf37jmWgPhYRd0bEv3R1+cLQRcTywMuB57SXBd3SXjb1tL5jG6A9gJMzM/sOZCA+D+wSEStHxFrADjSF0qSLKZ7bqOtABm5Dmjkx0HxADFyPc+TedF0kzQLuWeK5e2guZ5CmFRErAKcBJ2Xmgr7jGYrM3IdmP9qC5nT9I6NbTIzDgBMy89d9BzJABwIvAtYCjgO+GRGegWwuIVuB5izJFjSXTW0K/FOfQQ1NRLyA5nKyk/qOZUAuoZnQ3gvcQnO51Nd7jWgYFtCccTwgIlaIiO1oxs7K/YY1OM6RB6brIul+YPYSz80G7us4Dv0FiojlgFNo7mnbt+dwBqe9jOEyYG1g777j6VtEzAW2AT7bdyxDlJk/ycz7MvORzDyJ5vKX1/Yd1wA81P57ZGbelpl3Ap/B3Cxpd+CyzLyx70CGoH1/+i7Nh1SrAM8Gnklzb9tEy8zHgDcAf0dzP8n+wJk0haT+xDnywHRdJF0HzIiIlxTPbYKXTWkaERHACTSf8s5rD7pauhl4TxLAVsAc4OaIWAS8F5gXET/rM6gBS5Z+WcxEycy7aCZvXkI22u54Fqm0GvB84Kj2g4f/BE7E4hqAzLw6M7fMzGdl5vY0Z7F/2ndcA3MNzZwY+ON9/OviHLk3nRZJ7fWV5wIfjohVIuJVwOtpzg5MtIiYEREzgeWB5SNipkuH/hfHAhsAr8vMh6Z78aSIiNXbpYpnRcTyEbE9sCtwYd+xDcBxNG8wc9vHF4FvAdv3GdQQRMSqEbH94uNMROxGs4Lbd/uObSBOBN7R7l/PBPajWZlLQES8kuYyTVe1a7VnHG8E9m73qVVp7tm6anTLyRARG7fHm5Uj4r00KwB+peewejFivncesFFEzGu3fwi4epJuLRg1F47m62Bmti9dsd22TD/Y62MJ8H2Ap9Fcn/pVYO/MtEpurnd/CHg/zRKHD+E18AC03xOwF81Ed1Hx/Ry79RzaECTNpXW3AHcBnwL2y8xv9BrVAGTmg5m5aPGD5lKGhzPzjr5jG4AVaJZZvQO4E3gH8IbM9LuSGofRLBl/HXAtcCXwkV4jGpY9gHMz08uA/qs3Af+dZr/6FfB74N29RjQcb6VZVOh2YGtg22JV1kmz1Ple+940j+ZYcxewObBLX0H2ZNRc+N/bn9ei+UDvIWCZfo9UuCiNJEmSJP1JH2eSJEmSJGmwLJIkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVJh5PfwfDKieum7A75Uv2pefLN+2fP8RnbyZYhvHOOLBs9lu+p+Iy6obkt2k5sY60sYx/kKpJWrWybd5AYgoj4/h15Z3++OR9X/inOP7yg/x9Yfc2Lv+m535q3Vbc/g5G72qzGOx28dY5ecn/dWt72T2d2MmzH2qXECHO+QOvzj8TgRzhmj8cKuchPn1u8YOa+66fW8q7rti/hcN/vUfxvjoPHT+hAvGaPbLTv6Muwbz6g/Fr/of44xPTr7/uqmudOsjo7Fbxpjnzqvuukrxhg3l48YN55JkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgoWSZIkSZJUsEiSJEmSpIJFkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKkRmTrnxuRFTb5zGolnvq20K9328vi3EOI2fsG03qc7NX3/v6upuf8yPq9uSm3eTm/+oHzex3v3V3R7y3FnVbQ++LbvJDRBBdX5WHqPfvxpj1/hBdpSf+WOMnddWN2Wcw0bS1dipz80nx4hwpbvq275z1eHn5tGH68fNSruOMW7O6yY3wQurf8Er9/9f1f3O/cwYR6vcr5PcvH+MY/ERY/z5LuLF1W1fw390M26O+2B1bnKvj9T3y99Wt828sJvjzRhzY97/neqm6x6xfXXb67ObuXGsWr9PPf+x+hBvfqC6KYx4D/dMkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVLBIkmSJEmSChZJkiRJklSwSJIkSZKkgkWSJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKkZlTbvwhTL1xGq9cJWqbwovqm/LzHKPjJy5eVp+bvLY+xBjx95q2X+gmN2OMm4z6EL/29/W52eW0bnIDwD/W54cv/xnjeHIGP3ZW+9G3q/v93d+8trpt0s0x5+1Rn5ujxwhxvzGG6+c6Gje8YIwgb67vNsY4XmV29F41xj51zRh/vjeM8dtd9xeQm/zH+n7nnFj/6y3sKDezrz2+Ojf3bfD26n5Xj0eq2/42u3qfivrjDf9Q3fKT8ZXqtu/taNzcNMb71JzNvl7db17xxuq2jHgP90ySJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVLBIkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgqRmX3HIEmSJEmD4ZkkSZIkSSpYJEmSJElSwSJJkiRJkgoWSZIkSZJUsEiSJEmSpIJFkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVLBIkmSJEmSChZJkiRJklRYpkVSROwbEVdExCMR8ZXi+b+OiAsi4ncRcUdEnBURz1uWsQzNiNy8rH3+rvbxvYh4WY+h9mKq/CzxmoMjIiNim47D69WIsTOnzcf9xeOgHkPt3KhxExErR8QxEXFnRNwTEZf2FGYvRoyb3ZYYMw+242izHsPt1DTjZueIuDYi7ouIX0bEG3oKsxfT5GbPiPhVO26+ExFr9hRmLyJipYg4ISJuasfHlRGxQ7F964hY0O5TF0XEOn3G26VRuYmIFSPi7IhY2B5rtuo53E5Nk5uJnh9Pk5vO58fL+kzSb4DDgS8v8fwzgeOAOcA6wH3Aics4lqGZKje/AXYCVgOeDfwz8LVuQxuEqfIDQESsS5On27oMaiBG5gZYNTNntY/DOoxrCEbl5jia/WqD9t93dxjXECw1N5l5WjFeZgH7ADcAP+shxr4sNTcRsRZwKvAeYDZwAHB6RKzeeYT9mSo3WwIfBV5Psz/dCHy18+j6NQP4NbAl8AzgIODM9gOrZwPnts+tBlwBnNFXoD2YMjft9suAtwCL+giuZ6NyM+nz41G56Xx+PGNZ/ueZeS5ARLwcWLt4/tvl6yLiKOCSZRnL0IzIzd3A3e22AB4HXtxHjH2aKj+Fo4ADgWO6jGsInkBuJtZUuYmI9YEdgbUz89726X/tPsL+PIlxswdwcmZmJ4ENwIjcrA3cXbxnfSsiHgDWBW7vNsp+jMjN64CzMvOadvthwK0RsW5mXt99pN3LzAeAQ4qnzo+IG4HNgGcB12TmWQARcQhwZ0S8NDMXdB1r10blJjMXAp8DiIjHu4+uX9Pk5pzytZM2P34Cuel0fjyUe5JeDVzTdxBDEhF3Aw8DR9J8WqdWRLwZeDQz5/cdy0DdFBG3RMSJ7aeZgs2Bm4BD28vtfh4R8/oOamjay4FeDZzcdywDcQVwbUTsGBHLt5faPQJc3XNcQxDto/wZYKMeYhmEiFgDWI9mPrMhcNXibe3k7/r2+YmzRG5UmCY3Ez0/Xlpuupwf914kRcTGwIdoLmNQKzNXpTnVuC9wZc/hDEZEzKLZKfbrO5YBuhN4Bc0p+s2ApwOn9RrRcKxNM3m7B1iTZr86KSI26DWq4dkd+EFm3th3IEOQmY/TFIyn0xRHpwN7tRPeSTcf2DkiNo6Ip9G8jyewcr9h9SMiVqA53p7UnimaRXO8Kd1Dc1yeKEvJjVqjcjPp8+OpctPl/LjXIikiXgx8G3hXZv6gz1iGqH0j/iJw8oRdAz/KocApTuL+f5l5f2ZekZm/z8zf0hxAtouI2X3HNgAPAY8Bh2fmo5l5CXARsF2/YQ3O7sBJfQcxFNEsCvMJYCtgRZrr5I+PiLl9xjUEmfl94GDgHJqztAtp7p+4pcewehERywGnAI/SHHcB7qe5j600myZHE2OK3IjRuZn0+fF046ar+XFvRVJ7Wcf3gMMy85S+4vgLsBzNJ3Nr9R3IQGwNvDMiFkXEIuD5NDf1HdhzXEO0+J6SGPmqyeDlUdOIiFfRnGU7u+9YBmQucGn74cMfMvNy4CfARK2oOZXMPDozX5KZq9MUSzOAX/QcVqfaeyNOANYA5mXmY+2ma4BNitetQnMv28RcOjUiNxNvVG4mfX78JMbNMp8fL+slwGdExExgeWD5iJjZPrcWcCFwdGZ+cVnGMFQjcrNtRGzaXv8+G/gMcBdwba8Bd2yq/NAUSRvRTF7m0qx2shdwdG/BdmzE2Nk8ItaPiOUi4lnAF4CLM3PJSz6eskaMm0uBm4EPtK95Fc3Zge/2F223RuRmsT2AczJzoj7phpG5uRzYYvGZo4jYFNiCCSq6RxxvZkbERtF4Ac2KXJ/PzLv6jbhzx9KsmPm6zHyoeP48YKOImNfm70PA1RN2udlUuVm81PPM9scV2/E0SR/oLTU3zo+BqXPT/fw4M5fZg2aFilzicQjNKfqkOR39x8eyjGVojxG5eTOwoM3JHTTXfW/cd7xDyc9SXrcQ2KbveIeQG2BXmmV4H6BZGv1k4Ll9xzuE3LTbNgR+1Obnl8Ab+453QLmZSbNq0NZ9xznA3OwL/IrmMqkbgP37jncIuQFWpSkWH6BZxvljwPJ9x9txbtZp8/HwEvOZ3drt27Tv5w8BFwNz+o55QLlZuJRxNRH5GZUbJnx+PE1uOp8fRxuUJEmSJIkBrG4nSZIkSUNikSRJkiRJBYskSZIkSSpYJEmSJElSYcaojUFUr+qw+gfrF4S4/SPfr26bbN3NEpKP1ueGlcZYLOPsf6tvO29uJ7mJPepzk98cIzevXKG+7fmPdbb06NaveWb1L3nhxXdX9zt/jEVadujsu5bqx86mMa+613973jnVbfM32c1+tT31f8D/O0bHRzxa3TQPXLGb3Jw6xjHnLXtX9/tgHFvdduXsZtxA/biJC+pDzG3HWhSqm3HDDdVB3sS61f2+oLolQEfj5mv1+9T5u9Z3+7ox3qeyo3HzjqjPzVFj9bx+dcvMBd2MmxjjeHNWfbe504n1jXnblLnxTJIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgoWSZIkSZJUsEiSJEmSpIJFkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVLBIkmSJEmSCpGZI7bGiI2jrVHbELid6m7JJMbo+kmoz8076pty5KL3V7dljSM6ys3a1b9gXHNrfbcbPbO6aebvOsoNcPEYA+A19d0GZ1a3zXxzJ/l5YIz9apUx+v3+GH+SrTs65hwa9QfGg8cJcdR7xPQ6yU1Qn5uMMUJ84Ri5uaGj3Bz/4+ogN97z+dX9XhVrV7cls5vj8avGmGz8cJwQxxg3HR1vxtunbq/u94r3bFbd9uWf/nU34ybuGGOSunp9t7ypvlvO6SY3fzvGm+lF9d1e9/b6tusdNfXxxjNJkiRJklSwSJIkSZKkgkWSJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVLBIkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpEZk69EabeOI3kwNqm7Pw/Pl7d9szzierGT0IQ1bkhPjRGzx+ubpmZneTmyjHGzWljRPip+r8I0M24Afjh/vX5edWn6/udOcav+BDdjB1iuercXDXiWDadTapbAh3lJvha/S/44l2rm+aZY+xYf9XNfvWhg06sDvJHh7+tut8LnjHGr3d3V+NmjPcqjqhvuu1Hq5vmBfd0lJurx3gfrz9qvDq3r257Cd/pJjfxeHVubmX56n7XGmO3yI6OxbeOMcdZixWr+42/e7S6bX6rm2Px/Kg/3lx9WH2/Kx+0ZnXbd+atU+bGM0mSJEmSVLBIkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgoWSZIkSZJUsEiSJEmSpIJFkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKkRm9h2DJEmSJA2GZ5IkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVLBIkmSJEmSChZJkiRJklSwSJIkSZKkgkWSJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUmGZFkkRsVJEnBARN0XEfRFxZUTssJTXHRwRGRHbLMt4hmRUbiJiTpuP+4vHQX3H3JXpxk1ErBwRx0TEnRFxT0Rc2me8XZpm3Oy2xJh5sB1Hm/UddxeewLjZOSKubbf9MiLe0Ge8XXoCudkzIn7VjpvvRMSafcbbtYg4NSJui4h7I+K6iNiz2LZ1RCxo96eLImKdPmPtw1T5iYgVI+LsiFjYHmu26jlUSfqzWdZnkmYAvwa2BJ4BHAScGRFzFr8gItYFdgJuW8axDM20uQFWzcxZ7eOw7kPszXS5OQ5YDdig/ffd3YfYmylzk5mnFeNlFrAPcAPws/7C7dSUuYmItYBTgfcAs4EDgNMjYvW+gu3YqNxsCXwUeD3N/nQj8NW+Au3Jx4A5mTkb2BE4PCI2i4hnA+fS5Gs14ArgjP7C7M1S89Nuuwx4C7Cor+AkaVmIzOy2w4irgUMz85z2528DRwLHAHtm5vc6DWhAFucG+FeaicoKmfn7fqMahiI3vwAuB9bOzHv7jWoYltyniucvAi7OzEP7iax/xbi5BfhmZq5ebLsD2DEzf9RXfH0qcvM3wNMy8+3t82sCtwIvzszrewyxFxGxPnAx8C5gVeAfMvOV7bZVgDuBTTNzQW9B9qjMT2aeWTx/C/CWzLy4p9Ak6c+q03uSImINYD3gmvbnNwOPZub8LuMYoiVz07opIm6JiBPbTzQn0hK52Ry4CTi0vdzu5xExr9cAezTFuKG9JOjVwMl9xDUES+TmCuDaiNgxIpZvL7V7BLi6zxj7skRuon38cXP770Zdx9Wn9hLeB4EFNFc2zAc2BK5a/JrMfAC4vn1+okyRH0l6yuqsSIqIFYDTgJMyc0FEzKK5xGO/rmIYqiVzQ/NJ5SuAdYDNgKe32yfOUnKzNs3k7R5gTWBf4KSI2KC/KPuxlNyUdgd+kJk3dh9Z/5bMTWY+TlMwnk5THJ0O7NVOeifKUsbNfGDniNg4Ip4GfAhIYOUew+xcZu5Dc6zdguYSu0eAWTTHmtI97esmyhT5kaSnrE6KpIhYDjgFeJRmUgvNZR6nTOokbrGl5SYz78/MKzLz95n52/b57SJido+hdm6KcfMQ8BhweGY+mpmXABcB2/UTZT+myE1pd+CkToMaiKXlJppFYT4BbAWsSHNvzvERMbenMHsxxfHm+8DBwDk0Z2kXAvfRXKI4UTLz8cy8jObDmL2B+2nuYSvNpsnPxFlKfiTpKWuZF0kREcAJwBrAvMx8rN20NfDOiFgUEYuA59PcSHzgso5pKEbkZkmLbxyLKbY/5YzIzUReHlWabtxExKtozrKd3UN4vRqRm7nApe2HD3/IzMuBnwCTtKLmlOMmM4/OzJe092ydQ7PQwy/6iXQQZgDr0lyOuMniJ9t7khY/P8kW50eSnrK6OJN0LM0qZK/LzIeK57emuWxqbvv4DbAXcHQHMQ3FUnMTEZtHxPoRsVxEPAv4As0N+Ete9vFUNtW4uRS4GfhARMxoC4KtgO92H2JvpsrNYnsA52TmJH7aPVVuLge2WHzmKCI2pblsaJKK7qmONzMjYqNovIBm9cjPZ+ZdfQXapYhYPSJ2iYhZ7f1q2wO7AhcC5wEbRcS8iJhJcyni1ZO0aMM0+Vm8vPzM9uUrtuNpYj7Qk/TUtUxXt2tvHl9Ic+1yuUrbXpl52hKvXcgErW43KjfAH2ju11oduBe4AHhfZk7EEqvTjZuI2BA4HtiY5vKgD2bmeZ0H2oMnkJuZNEvxzmsvo5oYTyA3+9LcA7kGcAdwdGZ+uvNAezDN8eZbNB8+rEtzGdmJwD+193E95UXEc2jOum5C88HhTcAXMvNL7fZtgKNo7hH9Cc1qdwv7ibZ7TyA/C2lyU3rhJOVI0lNT50uAS5IkSdKQdboEuCRJkiQNnUWSJEmSJBUskiRJkiSpYJEkSZIkSYUZoza+Nk6sXtVhfr6ttinjrB6amd0sPRpRv+LFOItljPPbZTffs/S7NepzM/f2+n5vZr/6xvnZzpasfRn1+bn2FfX95vz6tjynm/0q/vSdYBV2rm+60fOrm+YvPt1JblY7oz432/2f26r7/dpdz6tuS0ff7faVMY7HB+dnqvu9Od5T3baz96px9qktxuj1sjF+ve5yI0lT8kySJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVLBIkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgozRm2cz83V/3F8qbop5Bhtu3JXfdO9flrf9rgxctNVWu/5+KLqtht+dY3qtpcsiOq2W/LZ6rZP1i/H+EvEFfX9nn3y3dVtd9q/vt8nI39b/zfkufVNI4d/0Llr7frcnL//FmP0vNkYbbvZr952eX3bD/Oe6rYHDX/YEPGz6raZ9X/7uHqH+n6rW0rSn49nkiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVLBIkiRJkqSCRZIkSZIkFSySJEmSJKlgkSRJkiRJBYskSZIkSSpYJEmSJElSwSJJkiRJkgoWSZIkSZJUiMyceuPeMfXGaTx2bG1LmPHV6m5hV6K+8RMXn6I+yAPqQ8z3/1N1Wz52WCe5eSn1udmHM6v7fRc7V7dNuhk3AEH9fvW9Mfo9YIwh+7Pu8lMd5PpjRHjqGL/eKzK7OeZ8oH7ccMSJ1U3fl4dUt/04CzsaN++rzk3EJ6p7zTdWN4VzO3qvOnKMcfOOcXqu77bL47EkTcUzSZIkSZJUsEiSJEmSpIJFkiRJkiQVLJIkSZIkqWCRJEmSJEkFiyRJkiRJKlgkSZIkSVLBIkmSJEmSChZJkiRJklSwSJIkSZKkgkWSJEmSJBUskiRJkiSpYJEkSZIkSQWLJEmSJEkqRGb2HYMkSZIkDYZnkiRJkiSpYJEkSZIkSQWLJEmSJEkqWCRJkiRJUsEiSZIkSZIKFkmSJEmSVPh/DSFGWJw1hLcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x2304 with 32 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize filters - more methods in the appendix to this tutorial\n",
    "# observe available layers, in our case, the stacked layers are called \"conv_layer\"\n",
    "print(model.conv_layer)\n",
    "# extracting the model features at the particular layer number\n",
    "layer = model.conv_layer[0]  # to plot other layers, see the appendix tutorial\n",
    "# get the weights\n",
    "weight_tensor = layer.weight.data.cpu()\n",
    "\n",
    "# get the number of kernals\n",
    "num_kernels = weight_tensor.shape[0]    \n",
    "\n",
    "#define number of columns for subplots\n",
    "num_cols = 12\n",
    "# rows = num of kernels\n",
    "num_rows = num_kernels\n",
    "\n",
    "#set the figure size\n",
    "fig = plt.figure(figsize=(num_cols, num_rows))\n",
    "\n",
    "# looping through all the kernels\n",
    "for i in range(weight_tensor.shape[0]):\n",
    "    ax1 = fig.add_subplot(num_rows, num_cols, i+1)\n",
    "\n",
    "    #for each kernel, we convert the tensor to numpy \n",
    "    npimg = np.array(weight_tensor[i].numpy(), np.float32)\n",
    "    #standardize the numpy image\n",
    "    npimg = (npimg - np.mean(npimg)) / np.std(npimg)\n",
    "    npimg = np.minimum(1, np.maximum(0, (npimg + 0.5)))\n",
    "    npimg = npimg.transpose((1, 2, 0))\n",
    "    ax1.imshow(npimg)\n",
    "    ax1.axis('off')\n",
    "    ax1.set_title(str(i))\n",
    "    ax1.set_xticklabels([])\n",
    "    ax1.set_yticklabels([])\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/book-shelf.png\" style=\"height:50px;display:inline\"> The CNN Story\n",
    "---\n",
    "\n",
    "* **1996** - Lenet-5 - core of CNR check reading system, used by US banks.\n",
    "<img src=\"./assets/tut_conv_lenet5.gif\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **2012** - ILSVRC - Imagenet Large Scale Visual Recognition Challenge\n",
    "    * Imagenet data base: 14M labeled images, 20K categories.\n",
    "    * <img src=\"./assets/tut_conv_imagenet.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **2012** - AlexNet wins the challenge by a significant margin!\n",
    "    * <img src=\"./assets/tut_conv_ilv_12.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **2013** - thanks to deep CNNs, the results only keep improving.\n",
    "    * <img src=\"./assets/tut_conv_ilv_13.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Today** - <a href=\"https://paperswithcode.com/sota/image-classification-on-imagenet\">Link</a>\n",
    "    * <img src=\"./assets/tut_conv_ilv_20.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/clouds/100/000000/lightning-bolt.png\" style=\"height:50px;display:inline\"> CNNs Applications in Computer Vision\n",
    "---\n",
    "* **Object Detection**\n",
    "    * <img src=\"./assets/tut_conv_obj_det.PNG\" style=\"height:200px\">\n",
    "    * <a href=\"https://medium.com/better-programming/real-time-object-detection-on-gpus-in-10-minutes-6e8c9b857bb3\"> Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Semantic Segmentation**\n",
    "    * <img src=\"./assets/tut_conv_segnet.png\" style=\"height:200px\">\n",
    "    * <a href=\"https://missinglink.ai/guides/computer-vision/image-segmentation-deep-learning-methods-applications/\">Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Super Resolution**\n",
    "    * <img src=\"./assets/tut_conv_superres.PNG\" style=\"height:300px\">\n",
    "    * <a href=\"https://arxiv.org/pdf/1609.04802.pdf\">Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Style Transfer**\n",
    "    * <img src=\"./assets/tut_conv_style_trans.jpeg\" style=\"height:300px\">\n",
    "    * <a href=\"https://towardsdatascience.com/light-on-math-machine-learning-intuitive-guide-to-neural-style-transfer-ef88e46697ee\">Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Image Editing**\n",
    "    * <img src=\"./assets/tut_conv_image_edit.jpg\" style=\"height:300px\">\n",
    "    * <a href=\"http://people.csail.mit.edu/junyanz/projects/gvm/\">Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Image Generation**\n",
    "    * StyleGAN V2 - <a href=\"https://thispersondoesnotexist.com/\">thispersondoesnotexist.com</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Multi-Signals**\n",
    "    * Synthesizing Obama: Learning Lip Sync from Audio\n",
    "    * <img src=\"./assets/tut_conv_obama.png\" style=\"height:300px\">\n",
    "    * <a href=\"http://grail.cs.washington.edu/projects/AudioToObama/\">Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cute-clipart/64/000000/easter-egg.png\" style=\"height:50px;display:inline\"> Are CNNs the Holy Grail? The Problem with CNNs\n",
    "---\n",
    "#### Deep NNs are sensitive to adversarial attacks.\n",
    "* For example: consider the following image, where on the left, we have an image of a pig that is correctly classified by a state-of-the-art convolutional neural network. \n",
    "* After perturbing the image slightly (every pixel is in the range [0, 1] and changed by at most 0.005), the network now returns class “airliner” with high confidence.\n",
    "    \n",
    "<img src=\"./assets/tut_conv_pig.png\" style=\"height:200px\">\n",
    "\n",
    "<a href=\"http://gradientscience.org/intro_adversarial/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Recognition algorithms generalize poorly to new environments\n",
    "\n",
    "<img src=\"./assets/tut_conv_cow.PNG\" style=\"height:200px\">\n",
    "\n",
    "<a href=\"https://arxiv.org/pdf/1807.04975.pdf\">Recognition in Terra Incognita (Beery et al., 2018)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Neural Networks tend to exhibit undesirable biases\n",
    "<img src=\"./assets/tut_conv_fairness.PNG\" style=\"height:300px\">\n",
    "\n",
    "* The reasons why the model learns these biases are unclear. \n",
    "    * One hypothesis is that despite the balanced distribution of races in pictures labeled basketball, black persons are more represented in this class in comparison to the other classes\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/1711.11443\">ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases (Stock and Cisse, 2018)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* Convolutional Neural Networks -  <a href=\"https://www.youtube.com/watch?v=iaSUYvmCekI\"> Convolutional Neural Networks | MIT 6.S191 </a>\n",
    "    * A previous version of this lecture - <a href=\"https://www.youtube.com/watch?v=H-HVZJ7kGI0\"> Convolutional Neural Networks | MIT 6.S191 </a>\n",
    "* Deep Neural Networks with PyTorch - <a href=\"https://www.youtube.com/watch?v=_H3aw6wkCv0&t\"> Stefan Otte: Deep Neural Networks with PyTorch | PyData Berlin 2018 </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "* Some slides from CS131 and CS231n (Stanford)\n",
    "* Deep Learning with Pytorch on CIFAR10 Dataset - <a href=\"https://www.stefanfiott.com/machine-learning/cifar-10-classifier-using-cnn-in-pytorch/\">Zhenye's Blog</a>\n",
    "* CIFAR-10 Classifier Using CNN in PyTorch - <a href=\"https://www.stefanfiott.com/machine-learning/cifar-10-classifier-using-cnn-in-pytorch/\">Stefan Fiott</a>\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}