1_hands_on.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "300c5048",
   "metadata": {
    "editable": true,
    "id": "2iUXCk7tC1x5",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "# Session 1 : Training your first aircraft classifier with pytorch\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"><img alt=\"Creative Commons License\" align=\"left\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png\" /></a>&nbsp;| Florient Chouteau | <a href=\"https://supaerodatascience.github.io/deep-learning/\">https://supaerodatascience.github.io/deep-learning/</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01c6d950",
   "metadata": {
    "editable": true,
    "id": "yfn1RtChC1yD",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Intro\n",
    "\n",
    "The objectives of this session is to apply what you learned during [the previous class on Deep Learning](https://supaerodatascience.github.io/deep-learning/) on a real dataset of satellite images.\n",
    "\n",
    "Most of the vocabulary and concepts of Deep Learning and Convolutionnal Neural Network has been defined on the class linked above so you should refer to it.\n",
    "\n",
    "In this session you will:\n",
    "- Train a basic NN on a basic dataset\n",
    "- Plot ROC curve & confusion matrix to diagnose your dataset\n",
    "\n",
    "During session 2 you will be experimenting with harder datasets\n",
    "\n",
    "If you haven't done so, go to the previous notebooks to get a hands on pytorch and CNNs\n",
    "\n",
    "\n",
    "**First steps**\n",
    "- Activate the GPU runtime in colab\n",
    "- Check using `!nvidia-smi` that you detect it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2439063b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Installation script for torchinfo package\n",
    "!pip install torchinfo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c74fcbef",
   "metadata": {
    "editable": true,
    "id": "xEo4VpHqC1yF",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a5bcff0",
   "metadata": {
    "editable": true,
    "id": "FG3_sWsWC1yH",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Put your imports here\n",
    "import numpy as np\n",
    "import torchinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67df5e42",
   "metadata": {
    "editable": true,
    "id": "1nb7isjuC1yI",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Dataset\n",
    "\n",
    "Récupération et exploration du datset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0d4e7ee",
   "metadata": {
    "editable": true,
    "id": "XLml82VWC1yK",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Configuration variables\n",
    "TOY_DATASET_URL = \"https://storage.googleapis.com/fchouteau-isae-deep-learning/toy_aircraft_dataset_2023.npz\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fcb1b4a",
   "metadata": {
    "editable": true,
    "id": "Shmmb50XC1yK",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Image (reminders)\n",
    "\n",
    "A digital image is an image composed of picture elements, also known as pixels, each with finite, discrete quantities of numeric representation for its intensity or gray level that is an output from its two-dimensional functions fed as input by its spatial coordinates denoted with x, y on the x-axis and y-axis, respectively.\n",
    "\n",
    "We represent images as matrixes,\n",
    "\n",
    "Images are made of pixels, and pixels are made of combinations of primary colors (in our case Red, Green and Blue). In this context, images have chanels that are the grayscale image of the same size as a color image, made of just one of these primary colors. For instance, an image from a standard digital camera will have a red, green and blue channel. A grayscale image has just one channel.\n",
    "\n",
    "In geographic information systems, channels are often referred to as raster bands.\n",
    "\n",
    "![img](https://static.packt-cdn.com/products/9781789613964/graphics/e91171a3-f7ea-411e-a3e1-6d3892b8e1e5.png)\n",
    "\n",
    "\n",
    "For the rest of this workshop we will use the following axis conventions for images\n",
    "\n",
    "![conventions](https://storage.googleapis.com/fchouteau-isae-deep-learning/static/image_coordinates.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe53e0e",
   "metadata": {
    "editable": true,
    "id": "nPa5zHUBC1yN",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Downloading the dataset\n",
    "\n",
    "We will be using [numpy datasources](https://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html?highlight=datasources) to download the dataset. DataSources can be local files or remote files/URLs. The files may also be compressed or uncompressed. DataSource hides some of the low-level details of downloading the file, allowing you to simply pass in a valid file path (or URL) and obtain a file object.\n",
    "\n",
    "The dataset is in npz format which is a packaging format where we store several numpy arrays in key-value format\n",
    "\n",
    "Note:\n",
    "If you get an error with the code below run:\n",
    "```python\n",
    "!gsutil -m cp -r gs://isae-deep-learning/toy_aircraft_dataset.npz /tmp/storage.googleapis.com/isae-deep-learning/toy_aircraft_dataset.npz\n",
    "```\n",
    "in a cell above the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf91c4ba",
   "metadata": {
    "editable": true,
    "id": "aPxBx-2-C1yP",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "ds = np.lib.npyio.DataSource(destpath=\"/tmp/\")\n",
    "f = ds.open(TOY_DATASET_URL, \"rb\")\n",
    "\n",
    "toy_dataset = np.load(f)\n",
    "trainval_images = toy_dataset[\"train_images\"]\n",
    "trainval_labels = toy_dataset[\"train_labels\"]\n",
    "test_images = toy_dataset[\"test_images\"]\n",
    "test_labels = toy_dataset[\"test_labels\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43451185",
   "metadata": {
    "editable": true,
    "id": "dRMdfPRKC1yR",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### A bit of data exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bac3c39a",
   "metadata": {
    "editable": true,
    "id": "KLD83Y7vC1yR",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "**Q1. Labels counting**\n",
    "\n",
    "a. What is the dataset size ?\n",
    "\n",
    "b. How many images representing aircrafts ?\n",
    "\n",
    "c. How many images representing backgrounds ?\n",
    "\n",
    "d. What are the dimensions (height and width) of the images ? What are the number of channels ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "107dce85",
   "metadata": {
    "editable": true,
    "id": "5xkrtVx-C1yS",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "**Q2. Can you plot at least 8 examples of each label ? In a 4x4 grid ?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f22895",
   "metadata": {
    "editable": true,
    "id": "n_fynC7iC1yT",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Here are some examples that help you answer this question. Try them and make your own. A well-understandood dataset is the key to an efficient model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d90f9d2",
   "metadata": {
    "editable": true,
    "id": "7XcQrRWKC1yT",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d26ed56f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "l7wcKYZBC1yU",
    "outputId": "4068a524-b60a-48ec-f40d-9538e2ea425f",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "LABEL_NAMES = [\"Not an aircraft\", \"Aircraft\"]\n",
    "\n",
    "print(\"Labels counts :\")\n",
    "for l, c, label in zip(*np.unique(trainval_labels, return_counts=True), LABEL_NAMES):\n",
    "    print(f\" Label: {label} , value: {l}, count: {c}\")\n",
    "\n",
    "for l, label in enumerate(LABEL_NAMES):\n",
    "    print(\n",
    "        f\"Examples shape for label {l} : {trainval_images[trainval_labels == l, ::].shape}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cbb4059",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "ArvB0PsXC1yW",
    "outputId": "84db6bb2-22b4-4384-d197-2ddcac18d9fd",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "LABEL_NAMES = [\"Not an aircraft\", \"Aircraft\"]\n",
    "\n",
    "print(\"Labels counts :\")\n",
    "for l, c, label in zip(*np.unique(test_labels, return_counts=True), LABEL_NAMES):\n",
    "    print(f\" Label: {label} , value: {l}, count: {c}\")\n",
    "\n",
    "for l, label in enumerate(LABEL_NAMES):\n",
    "    print(f\"Examples shape for label {l} : {test_images[test_labels == l, ::].shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce69cac5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 594
    },
    "editable": true,
    "id": "ol6QpoP6C1yX",
    "outputId": "5d445c41-2ed8-4f75-ed45-5ba1e25a2c8f",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "grid_size = 4\n",
    "grid = np.zeros((grid_size * 64, grid_size * 64, 3)).astype(np.uint8)\n",
    "for i in range(grid_size):\n",
    "    for j in range(grid_size):\n",
    "        tile = np.copy(trainval_images[i * grid_size + j])\n",
    "        label = np.copy(trainval_labels[i * grid_size + j])\n",
    "        color = (0, 255, 0) if label == 1 else (255, 0, 0)\n",
    "        tile = cv2.rectangle(tile, (0, 0), (64, 64), color, thickness=2)\n",
    "        grid[i * 64 : (i + 1) * 64, j * 64 : (j + 1) * 64, :] = tile\n",
    "\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "ax.imshow(grid)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b37fa0",
   "metadata": {
    "editable": true,
    "id": "gFtNYE6EC1yY",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### A bit about train-test\n",
    "\n",
    "You just downloaded a training and a test set.\n",
    "\n",
    "- We use the training set for forward/backward\n",
    "- We use the validation set to tune hyperparameters (optimizers, early stopping)\n",
    "- We use the test set for final metrics on our tuned model\n",
    "\n",
    "![](https://github.com/SupaeroDataScience/deep-learning/blob/main/vision/osBuF.png?raw=true)\n",
    "![](https://raw.githubusercontent.com/SupaeroDataScience/deep-learning/refs/heads/main/vision/pXAfX.png)\n",
    "\n",
    "For more information as to why we use train/validation and test refer to these articles:\n",
    "\n",
    "- https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7\n",
    "- https://www.freecodecamp.org/news/what-to-do-when-your-training-and-testing-data-come-from-different-distributions-d89674c6ecd8/\n",
    "- https://kevinzakka.github.io/2016/09/26/applying-deep-learning/\n",
    "\n",
    "We will now create our validation dataset,\n",
    "\n",
    "Since we know the dataset is balanced, we can evenly sample from the dataset without taking too many risks\n",
    "\n",
    "We will do a 80/20 sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19cdaa8a",
   "metadata": {
    "editable": true,
    "id": "gHmjoZhLC1yZ",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "idxs = np.random.permutation(np.arange(trainval_images.shape[0]))\n",
    "\n",
    "train_idxs, val_idxs = idxs[: int(0.8 * len(idxs))], idxs[int(0.8 * len(idxs)) :]\n",
    "\n",
    "train_images = trainval_images[train_idxs]\n",
    "train_labels = trainval_labels[train_idxs]\n",
    "val_images = trainval_images[val_idxs]\n",
    "val_labels = trainval_labels[val_idxs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b71e7f6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "7cfe6Yu6C1yZ",
    "outputId": "9d35706f-c6b9-4d44-fde4-e7509580b865",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "train_images.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212a8386",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "What is the mean of our data ? \n",
    "Whats is the standard deviation ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdce0936",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Compute the dataset statistics in [0.,1.], we're going to use it to normalize our data\n",
    "\n",
    "mean = np.mean(train_images, axis=(0, 1, 2)) / 255.0\n",
    "std = np.std(train_images, axis=(0, 1, 2)) / 255.0\n",
    "\n",
    "mean, std"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec46f53b",
   "metadata": {
    "editable": true,
    "id": "SZ6VBCvQC1ya",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Preparing our training\n",
    "\n",
    "Remember that training a deep learning model requires:\n",
    "\n",
    "- Defining a model to train\n",
    "- Defining a loss function (cost function) to compute gradients with\n",
    "- Defining an optimizer to update parameters\n",
    "- Putting the model on the accelerator device that trains very fast (GPU, TPU)... You'll learn about GPUs later :)\n",
    "\n",
    "![](https://pbs.twimg.com/media/E_1d06cVIAcYheX?format=jpg)\n",
    "\n",
    "The training loop is \"quite basic\" : We loop over samples of the dataset (in batches) several times over :\n",
    "\n",
    "![](https://pbs.twimg.com/media/E_1d06XVcA8Dhzs?format=jpg)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5354dc9",
   "metadata": {
    "editable": true,
    "id": "10ow7xWIC1ya",
    "lines_to_next_cell": 1,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "from typing import Callable\n",
    "\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from PIL import Image\n",
    "from torch import nn, optim\n",
    "from torch.utils.data import DataLoader, Dataset\n",
    "from torchvision import datasets, transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aad7e7ea",
   "metadata": {
    "editable": true,
    "id": "AKqUcnCcC1yb",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Defining Dataset & Transforms\n",
    "\n",
    "First, we need to tell pytorch how to load our data.\n",
    "\n",
    "Have a look at : https://pytorch.org/tutorials/beginner/basics/data_tutorial.html\n",
    "\n",
    "We write our own `torch.data.Dataset` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfeca21f",
   "metadata": {
    "editable": true,
    "id": "uvFjmzHoC1yb",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "class NpArrayDataset(Dataset):\n",
    "    def __init__(\n",
    "        self,\n",
    "        images: np.ndarray,\n",
    "        labels: np.ndarray,\n",
    "        image_transforms: Callable = None,\n",
    "        label_transforms: Callable = None,\n",
    "    ):\n",
    "        self.images = images\n",
    "        self.labels = labels\n",
    "        self.image_transforms = image_transforms\n",
    "        self.label_transforms = label_transforms\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.images.shape[0]\n",
    "\n",
    "    def __getitem__(self, index: int):\n",
    "        x = self.images[index]\n",
    "        y = self.labels[index]\n",
    "\n",
    "        if self.image_transforms is not None:\n",
    "            x = self.image_transforms(x)\n",
    "        else:\n",
    "            x = torch.tensor(x)\n",
    "\n",
    "        if self.label_transforms is not None:\n",
    "            y = self.label_transforms(y)\n",
    "        else:\n",
    "            y = torch.tensor(y)\n",
    "\n",
    "        return x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9987e1f4",
   "metadata": {
    "editable": true,
    "id": "0Z4N5o8AC1yb",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Then we need to process our data (images) into \"tensors\" that torch can process, we define \"transforms\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "572fb66c",
   "metadata": {
    "editable": true,
    "id": "PahdjhR5C1yc",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# transform to convert np array in range [0,255] to torch.Tensor [0.,1.]\n",
    "# then normalize by doing x = (x - mean) / std\n",
    "image_transforms = transforms.Compose(\n",
    "    [\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean, std),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# here we don't have anything to do\n",
    "target_transforms = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59dee236",
   "metadata": {
    "editable": true,
    "id": "p9QH51F-C1yc",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Now we put everything together into something to load our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7cc525f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "CR14oNXyC1yd",
    "outputId": "55d52439-7ffc-46a6-d7fb-13c0b173e761",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# load the training data\n",
    "train_set = NpArrayDataset(\n",
    "    images=train_images,\n",
    "    labels=train_labels,\n",
    "    image_transforms=image_transforms,\n",
    "    label_transforms=target_transforms,\n",
    ")\n",
    "\n",
    "print(len(train_set))\n",
    "\n",
    "train_loader = DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "\n",
    "# load the validation data\n",
    "validation_set = NpArrayDataset(\n",
    "    images=val_images,\n",
    "    labels=val_labels,\n",
    "    image_transforms=image_transforms,\n",
    "    label_transforms=target_transforms,\n",
    ")\n",
    "\n",
    "print(len(validation_set))\n",
    "\n",
    "val_loader = DataLoader(validation_set, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42ab0bed",
   "metadata": {
    "editable": true,
    "id": "0yxUYemIC1yd",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Check that your dataset outputs correct data\n",
    "\n",
    "Always to this as a sanity check to catch bugs in your data processing pipeline\n",
    "\n",
    "Write the inverse transformation by hand to ensure it's ok\n",
    "\n",
    "![andrej](https://storage.googleapis.com/fchouteau-isae-deep-learning/static/andrej_tweet_1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00394609",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 116
    },
    "editable": true,
    "id": "Ic2sE836C1ye",
    "outputId": "db784726-e3e0-40c3-d9ff-9e0ec597d28e",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "k = np.random.randint(len(train_set))\n",
    "x, y = train_set[k]\n",
    "\n",
    "# From torch\n",
    "# Inverse transform\n",
    "x = x.numpy()\n",
    "x = x.transpose((1, 2, 0))\n",
    "x = x * std + mean\n",
    "x = x.clip(0.0, 1.0)\n",
    "x = (x * 255.0).astype(np.uint8)\n",
    "\n",
    "print(\"Inverse transform is OK ?\")\n",
    "print(\"Label {}\".format(y))\n",
    "plt.imshow(x)\n",
    "plt.show()\n",
    "\n",
    "plt.imshow(train_set.images[k])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a379432",
   "metadata": {
    "editable": true,
    "id": "4np1A43JC1yf",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f004dbd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### On which device will we train ?\n",
    "\n",
    "We will check if we have a GPU and set the \"device\" of pytorch on it so that it trains on GPU "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28d0d674",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "BmV9carLC1yf",
    "outputId": "dba28689-a129-4ae1-facc-41f9d3e55f8e",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Apple Silicon Support\n",
    "if torch.backends.mps.is_available() and torch.backends.mps.is_built():\n",
    "    DEVICE = torch.device(\"mps\")\n",
    "# NVIDIA GPU\n",
    "elif torch.cuda.is_available():\n",
    "    DEVICE = torch.device(\"cuda:0\")\n",
    "# Fallback to CPU\n",
    "else:\n",
    "    DEVICE = torch.device(\"cpu\")\n",
    "\n",
    "print(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00fcab7c",
   "metadata": {
    "editable": true,
    "id": "AJ3oVqOHC1yg",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Defining a model and computing the parameters\n",
    "\n",
    "Now we have to define a CNN to train. It's usually called a \"network\", and we define its \"architecture\".\n",
    "\n",
    "Defining a good architecture is a huge field of research (a pandora's box) that takes a lot of time, but we can define \"sane architectures\" easily:\n",
    "\n",
    "Basically, CNN architectures are a stacks of :\n",
    "- Convolution layers + non linearities\n",
    "- Pooling layer\n",
    "- A final \"activation\" layer at the end (for classification) that allows us to output probabilities\n",
    "\n",
    "![](https://cs231n.github.io/assets/cnn/convnet.jpeg)\n",
    "\n",
    "Let's define a model together:\n",
    "\n",
    "```python\n",
    "model = nn.Sequential(\n",
    "    # A block of 2 convolutions + non linearities & a pooling layers\n",
    "    # IN SHAPE (3,64,64)\n",
    "    nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3),\n",
    "    # OUT SHAPE (16,62,62)\n",
    "    nn.ReLU(),\n",
    "    # IN SHAPE (16,62,62)\n",
    "    nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3),\n",
    "    # OUT SHAPE (16,60,60)\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # OUT SHAPE (16,30,30)\n",
    "    # Another stack of these\n",
    "    nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3),\n",
    "    # OUT SHAPE (?,?,?)\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # OUT SHAPE (?,?,?)\n",
    "    # Another stack of these\n",
    "    nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3),\n",
    "    # OUT SHAPE (?,?,?)\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3),\n",
    "    # OUT SHAPE (?,?,?)\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # OUT SHAPE (?,?,?)\n",
    "    # A final classifier\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(in_features=4 * 4 * 64, out_features=256), # do you understand why 4 * 4 * 64 ?\n",
    "    nn.ReLU(),\n",
    "    nn.Dropout(p=0.25),\n",
    "    nn.Linear(in_features=256, out_features=64),\n",
    "    nn.ReLU(),\n",
    "    nn.Dropout(p=0.25),\n",
    "    nn.Linear(in_features=64, out_features=1),\n",
    "    nn.Sigmoid(),\n",
    ")\n",
    "```\n",
    "\n",
    "**Questions**\n",
    "\n",
    "Knowing that the input image size is (3,64,64), go through the model step by step,\n",
    "\n",
    "Can you fill the blanks for the shapes ?\n",
    "\n",
    "Do you understand why ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0aee9458",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "hd06b1EnC1yh",
    "outputId": "039eb3c0-b120-4288-e404-f1e70bb76a48",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Let's test this !\n",
    "\n",
    "some_model = nn.Sequential(\n",
    "    nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # Another stack of these\n",
    "    nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # Another stack of these\n",
    "    nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2),\n",
    "    # A final classifier\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(in_features=4 * 4 * 64, out_features=256),\n",
    "    nn.ReLU(),\n",
    "    nn.Dropout(p=0.25),\n",
    "    nn.Linear(in_features=256, out_features=64),\n",
    "    nn.ReLU(),\n",
    "    nn.Dropout(p=0.25),\n",
    "    nn.Linear(in_features=64, out_features=1),\n",
    "    nn.Sigmoid(),\n",
    ")\n",
    "\n",
    "# We define an input of dimensions batch_size, channels, height, width\n",
    "x = torch.rand((16, 3, 64, 64))\n",
    "\n",
    "print(x.shape)\n",
    "\n",
    "y = some_model(x)\n",
    "\n",
    "print(y.shape)\n",
    "\n",
    "# Let's visualize each shape using our summarize helper\n",
    "print(\n",
    "    torchinfo.summary(\n",
    "        some_model,\n",
    "        input_data=x,\n",
    "        col_names=[\n",
    "            \"input_size\",\n",
    "            \"output_size\",\n",
    "            \"num_params\",\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "\n",
    "# let's delete the model now, we won't need it\n",
    "del some_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306e1482",
   "metadata": {
    "editable": true,
    "id": "YPpPpXwZC1yh",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "**Let's do it yourself !**\n",
    "\n",
    "About weight init :\n",
    "- https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks/\n",
    "- https://www.pyimagesearch.com/2021/05/06/understanding-weight-initialization-for-neural-networks/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad933167",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 398
    },
    "editable": true,
    "id": "d5nx-e0VC1yh",
    "outputId": "b4f8293d-996e-4e95-bcbc-0442c991ebbe",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Let's define another model, except this time there are blanks ... it's up to you to fill them\n",
    "\n",
    "\n",
    "def _init_weights(model):\n",
    "    for m in model.modules():\n",
    "        # Initialize all convs\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "            nn.init.kaiming_normal_(m.weight, mode=\"fan_in\", nonlinearity=\"relu\")\n",
    "        if isinstance(m, nn.Linear):\n",
    "            nn.init.kaiming_normal_(m.weight, mode=\"fan_in\", nonlinearity=\"relu\")\n",
    "\n",
    "\n",
    "def model_fn():\n",
    "    model = nn.Sequential(\n",
    "        # A first convolution block\n",
    "        nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=..., out_channels=16, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # Another stack of these\n",
    "        nn.Conv2d(in_channels=..., out_channels=32, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=..., out_channels=..., kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # A final classifier\n",
    "        nn.Flatten(),\n",
    "        nn.Linear(in_features=... * ... * ..., out_features=64),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(p=0.1),\n",
    "        nn.Linear(in_features=64, out_features=1),\n",
    "        nn.Sigmoid(),\n",
    "    )\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f401c0e3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "model = model_fn()\n",
    "\n",
    "print(model)\n",
    "\n",
    "x = torch.rand(\n",
    "    (16, 3, 64, 64)\n",
    ")  # We define an input of dimensions batch_size, channels, height, width\n",
    "\n",
    "print(x.shape)\n",
    "\n",
    "y = model(x)\n",
    "\n",
    "print(y.shape)\n",
    "\n",
    "print(\n",
    "    torchinfo.summary(\n",
    "        some_model,\n",
    "        input_data=x,\n",
    "        col_names=[\n",
    "            \"input_size\",\n",
    "            \"output_size\",\n",
    "            \"num_params\",\n",
    "        ],\n",
    "    )\n",
    ")\n",
    "\n",
    "# THIS CELL SHOULD NOT GIVE AN ERROR !"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df7afaf",
   "metadata": {
    "editable": true,
    "id": "nxb-JeDnC1yk",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Hint: The answer (and there can only be one) is :\n",
    "\n",
    "<details>\n",
    "    <summary>Solution</summary> \n",
    "    \n",
    "```python\n",
    "def _init_weights(model):\n",
    "    # about weight initialization\n",
    "    # https://machinelearningmastery.com/weight-initialization-for-deep-learning-neural-networks/\n",
    "    # https://www.pyimagesearch.com/2021/05/06/understanding-weight-initialization-for-neural-networks/\n",
    "    for m in model.modules():\n",
    "        # Initialize all convs\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "            nn.init.kaiming_normal_(m.weight, mode=\"fan_in\", nonlinearity=\"relu\")\n",
    "        if isinstance(m, nn.Linear):\n",
    "            nn.init.kaiming_normal_(m.weight, mode=\"fan_in\", nonlinearity=\"relu\")\n",
    "\n",
    "\n",
    "def model_fn():\n",
    "    model = nn.Sequential(\n",
    "        # A first convolution block\n",
    "        nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # Another stack of these\n",
    "        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # A final classifier\n",
    "        nn.Flatten(),\n",
    "        nn.Linear(in_features=12 * 12 * 32, out_features=64),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(p=0.1),\n",
    "        nn.Linear(in_features=64, out_features=1),\n",
    "        nn.Sigmoid(),\n",
    "    )\n",
    "\n",
    "    _init_weights(model)\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "model = model_fn()\n",
    "\n",
    "print(model)\n",
    "\n",
    "x = torch.rand((16, 3, 64, 64))  # We define an input of dimensions batch_size, channels, height, width\n",
    "\n",
    "print(x.shape)\n",
    "\n",
    "y = model(x)\n",
    "\n",
    "print(y.shape)\n",
    "\n",
    "print(summarize(model,example_input_array=x))\n",
    "```\n",
    "\n",
    "And outputs this\n",
    "\n",
    "```\n",
    "Sequential(\n",
    "  (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (1): ReLU()\n",
    "  (2): Conv2d(16, 16, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (3): ReLU()\n",
    "  (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "  (5): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (6): ReLU()\n",
    "  (7): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (8): ReLU()\n",
    "  (9): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1))\n",
    "  (10): ReLU()\n",
    "  (11): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
    "  (12): Flatten(start_dim=1, end_dim=-1)\n",
    "  (13): Linear(in_features=4608, out_features=64, bias=True)\n",
    "  (14): ReLU()\n",
    "  (15): Dropout(p=0.1, inplace=False)\n",
    "  (16): Linear(in_features=64, out_features=1, bias=True)\n",
    "  (17): Sigmoid()\n",
    ")\n",
    "torch.Size([16, 3, 64, 64])\n",
    "torch.Size([16, 1])\n",
    "   | Name | Type      | Params | In sizes         | Out sizes       \n",
    "--------------------------------------------------------------------------\n",
    "0  | 0    | Conv2d    | 448    | [16, 3, 64, 64]  | [16, 16, 62, 62]\n",
    "1  | 1    | ReLU      | 0      | [16, 16, 62, 62] | [16, 16, 62, 62]\n",
    "2  | 2    | Conv2d    | 2.3 K  | [16, 16, 62, 62] | [16, 16, 60, 60]\n",
    "3  | 3    | ReLU      | 0      | [16, 16, 60, 60] | [16, 16, 60, 60]\n",
    "4  | 4    | MaxPool2d | 0      | [16, 16, 60, 60] | [16, 16, 30, 30]\n",
    "5  | 5    | Conv2d    | 4.6 K  | [16, 16, 30, 30] | [16, 32, 28, 28]\n",
    "6  | 6    | ReLU      | 0      | [16, 32, 28, 28] | [16, 32, 28, 28]\n",
    "7  | 7    | Conv2d    | 9.2 K  | [16, 32, 28, 28] | [16, 32, 26, 26]\n",
    "8  | 8    | ReLU      | 0      | [16, 32, 26, 26] | [16, 32, 26, 26]\n",
    "9  | 9    | Conv2d    | 9.2 K  | [16, 32, 26, 26] | [16, 32, 24, 24]\n",
    "10 | 10   | ReLU      | 0      | [16, 32, 24, 24] | [16, 32, 24, 24]\n",
    "11 | 11   | MaxPool2d | 0      | [16, 32, 24, 24] | [16, 32, 12, 12]\n",
    "12 | 12   | Flatten   | 0      | [16, 32, 12, 12] | [16, 4608]      \n",
    "13 | 13   | Linear    | 294 K  | [16, 4608]       | [16, 64]        \n",
    "14 | 14   | ReLU      | 0      | [16, 64]         | [16, 64]        \n",
    "15 | 15   | Dropout   | 0      | [16, 64]         | [16, 64]        \n",
    "16 | 16   | Linear    | 65     | [16, 64]         | [16, 1]         \n",
    "17 | 17   | Sigmoid   | 0      | [16, 1]          | [16, 1]         \n",
    "--------------------------------------------------------------------------\n",
    "320 K     Trainable params\n",
    "0         Non-trainable params\n",
    "320 K     Total params\n",
    "1.284     Total estimated model params size (MB)\n",
    "```\n",
    "\n",
    "</details>\n",
    "\n",
    "\n",
    "You should be able to understand this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbd76d92",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "DAv9FrjAC1yl",
    "outputId": "6b42440f-8806-41e3-dc97-ecb499de36bd",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# moving model to gpu if available\n",
    "model = model.to(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72ee5d15",
   "metadata": {
    "editable": true,
    "id": "LhpN-UNfC1yl",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Defining our loss and optimizer\n",
    "\n",
    "Check the definition of the binary cross entropy:\n",
    "\n",
    "https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a74f6904",
   "metadata": {
    "editable": true,
    "id": "6w1BHLnoC1ym",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "criterion = nn.BCELoss(reduction=\"mean\")\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-2, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "159d40d6",
   "metadata": {
    "editable": true,
    "id": "8d1qaMZ8C1ym",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Training with pytorch\n",
    "\n",
    "We will actually train the model, and plot training & validation metrics during training\n",
    "\n",
    "Be careful, if you train several times the same model it will continue optimizing its parameters\n",
    "\n",
    "Its advised to define a new model if you are executing the training loop several times"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d14d356",
   "metadata": {},
   "source": [
    "### Defining the Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b752d49",
   "metadata": {
    "editable": true,
    "id": "xfP8tBSMC1yn",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "def train_one_epoch(model, train_loader):\n",
    "    epoch_loss = []\n",
    "\n",
    "    for i, batch in enumerate(train_loader):\n",
    "        # get one batch\n",
    "        x, y_true = batch\n",
    "        x = x.to(DEVICE)\n",
    "        y_true = y_true.to(DEVICE)\n",
    "\n",
    "        # format the y_true so that it is compatible with the loss\n",
    "        y_true = y_true.view((-1, 1)).float()\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward\n",
    "        y_pred = model(x)\n",
    "\n",
    "        # compute loss\n",
    "        loss = criterion(y_pred, y_true)\n",
    "\n",
    "        # backward\n",
    "        loss.backward()\n",
    "\n",
    "        # update parameters\n",
    "        optimizer.step()\n",
    "\n",
    "        # save statistics\n",
    "        epoch_loss.append(loss.item())\n",
    "\n",
    "        if i % 10 == 0:\n",
    "            print(f\"Batch {i}, curr loss = {loss.item():.03f}\")\n",
    "\n",
    "    return np.asarray(epoch_loss).mean()\n",
    "\n",
    "\n",
    "def valid_one_epoch(model, valid_loader):\n",
    "    epoch_loss = []\n",
    "\n",
    "    for i, batch in enumerate(valid_loader):\n",
    "        with torch.no_grad():\n",
    "            # get one batch\n",
    "            x, y_true = batch\n",
    "            x = x.to(DEVICE)\n",
    "            y_true = y_true.to(DEVICE)\n",
    "\n",
    "            # format the y_true so that it is compatible with the loss\n",
    "            y_true = y_true.view((-1, 1)).float()\n",
    "\n",
    "            # forward\n",
    "            y_pred = model(x)\n",
    "\n",
    "            # compute loss\n",
    "            loss = criterion(y_pred, y_true)\n",
    "\n",
    "            # save statistics\n",
    "            epoch_loss.append(loss.item())\n",
    "\n",
    "    return np.asarray(epoch_loss).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d98d87c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Putting everything together to run a training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43a0c33b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "**Hyperparameters**\n",
    "\n",
    "We also define what we call hyperparameters, the \"meta-parameters\" of the training that you can modify to affect your training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4c50d06",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "EPOCHS = 10  # Set number of epochs, example 100\n",
    "LEARNING_RATE = 1e-2\n",
    "MOMENTUM = 0.9"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b549761",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Here we copy paste previous code into a function (so that you reset your training everytime) so that you are sure you have setup everything correctly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e9d4a9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "def setup_training():\n",
    "    model = model_fn()\n",
    "\n",
    "    # moving model to gpu if available\n",
    "    model = model.to(DEVICE)\n",
    "\n",
    "    print(model)\n",
    "\n",
    "    # We define an input of dimensions batch_size, channels, height, width\n",
    "    x = torch.rand((16, 3, 64, 64))\n",
    "    x = x.to(DEVICE)\n",
    "\n",
    "    print(x.shape)\n",
    "\n",
    "    y = model(x)\n",
    "\n",
    "    print(y.shape)\n",
    "\n",
    "    print(torchinfo.summary(model, input_data=x))\n",
    "\n",
    "    criterion = nn.BCELoss(reduction=\"mean\")\n",
    "    optimizer = optim.SGD(\n",
    "        model.parameters(), lr=LEARNING_RATE, momentum=MOMENTUM, weight_decay=1e-4\n",
    "    )\n",
    "\n",
    "    return model, criterion, optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bbfdecf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Why do we wrap it into a function ? Try executing your notebook in the wrong order and see what happens !\n",
    "\n",
    "You can see that if you redefine a model, then the optimizer is called on the wrong parameters !\n",
    "\n",
    "Note that we defined some hyperparameters beyond the function, be careful of what we call \"the scope\". You could also pass them as parameters of your function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9857cb44",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Let's go !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf6924a0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "editable": true,
    "id": "6fwFxOOFGQkZ",
    "outputId": "a6da7c86-8233-4257-cf0f-86d6ff13ea88",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Init the training\n",
    "model, criterion, optimizer = setup_training()\n",
    "\n",
    "# Send model to GPU\n",
    "model = model.to(DEVICE)\n",
    "\n",
    "train_losses = []\n",
    "valid_losses = []\n",
    "\n",
    "# loop over the dataset multiple times\n",
    "for epoch in range(EPOCHS):\n",
    "    model.train()\n",
    "    train_epoch_loss = train_one_epoch(model, train_loader)\n",
    "    model.eval()\n",
    "    valid_epoch_loss = valid_one_epoch(model, val_loader)\n",
    "\n",
    "    print(f\"EPOCH={epoch}, TRAIN={train_epoch_loss:.05f}, VAL={valid_epoch_loss:.05f}\")\n",
    "\n",
    "    train_losses.append(train_epoch_loss)\n",
    "    valid_losses.append(valid_epoch_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fd1a2c1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 279
    },
    "editable": true,
    "id": "sFTlE66MC1yo",
    "outputId": "3f24cd80-7722-4228-e74e-7b4e2759fcb1",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Plot training / validation loss\n",
    "plt.plot(train_losses, label=\"Training Loss\")\n",
    "plt.plot(valid_losses, label=\"Validation Loss\")\n",
    "plt.xlabel(\"No. of Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend(frameon=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c99b0e30",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Training analysis\n",
    "\n",
    "How would you analyze your training ?\n",
    "\n",
    "Is it underfitting ?\n",
    "\n",
    "Is it overfitting ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c747a0d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Model saving\n",
    "\n",
    "There are several ways to save your model :\n",
    "https://pytorch.org/tutorials/beginner/saving_loading_models.html\n",
    "\n",
    "- torch.save: Saves a serialized object to disk. This function uses Python’s pickle utility for serialization. Models, tensors, and dictionaries of all kinds of objects can be saved using this function.\n",
    "\n",
    "- torch.load: Uses pickle’s unpickling facilities to deserialize pickled object files to memory. This function also facilitates the device to load the data into (see Saving & Loading Model Across Devices).\n",
    "\n",
    "- torch.nn.Module.load_state_dict: Loads a model’s parameter dictionary using a deserialized state_dict. For more information on state_dict, see What is a state_dict?.\n",
    "\n",
    "- scripting / tracing the model: https://pytorch.org/docs/stable/jit.html\n",
    "\n",
    "The first 2 options require you to import the model definition as it uses pickle\n",
    "The third option requires you to redefine an empty model with the same architecture and load the weights, because we are only saving the \"state\" (e.g. parameters, weights, biases)\n",
    "The fourth option allow to make a \"self-contained\" model that can be used later, but comes with caveats"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15d79a0a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### State dict saving \n",
    "\n",
    "This is the recommended method because it allows to reuse the model with any code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38de2195",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# State dict saving\n",
    "with open(\"model.pt\", \"wb\") as f:\n",
    "    torch.save(model.state_dict(), f)\n",
    "\n",
    "# See below for how to reload the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4dbdab7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "To reload such a model, you have to instantiate an empty model with the same architecture then load the state dict (the weights)\n",
    "```python\n",
    "# Instantiate a new empty model\n",
    "model = model_fn()\n",
    "\n",
    "print(model)\n",
    "\n",
    "# Load state\n",
    "checkpoint_path = \"model.pt\"\n",
    "model.load_state_dict(torch.load(checkpoint_path))\n",
    "\n",
    "print(\"Model Loaded\")\n",
    "```\n",
    "\n",
    "This is very nice because you get a model that you can finetune, retrain, modify. However, this means that you have to \"port\" the model definition code to production. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8531a0cb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Model scripting\n",
    "\n",
    "But for production, in order to avoid shipping the model definition code, we like to have an \"self-contained\" binary that we can deliver to the production team (you will see such a case during our next class together for cloud computing)\n",
    "\n",
    "Here we try to \"script\" the model, meaning that we compile the graph to a static version of itself\n",
    "\n",
    "https://pytorch.org/docs/stable/jit.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "636e99e3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "import torch.jit\n",
    "\n",
    "# Put the model in eval mode\n",
    "model = model.cpu().eval()\n",
    "\n",
    "# Script the model\n",
    "scripted_model = torch.jit.script(model)\n",
    "\n",
    "# Save\n",
    "scripted_model.save(\"scripted_model.pt\")\n",
    "\n",
    "print(scripted_model)\n",
    "\n",
    "# Scripted model reloading (demo)\n",
    "scripted_model = torch.jit.load(\"scripted_model.pt\", map_location=DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f0c89f5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Download the scripted model\n",
    "\n",
    "We are going to download the scripted model to be able to re-use it elsewhere (in another notebook for example), without having to rewrite the model definition function\n",
    "\n",
    "Uncomment this on google colab to download the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21e72e4c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# from google.colab import files\n",
    "\n",
    "# files.download('scripted_model.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d71a515f",
   "metadata": {
    "editable": true,
    "id": "VqMkcDroC1yp",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "We have finished what we need to do with the model, let's delete it !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0500cfe2",
   "metadata": {
    "editable": true,
    "id": "btrb85LmC1yp",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "del model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2445d336",
   "metadata": {
    "editable": true,
    "id": "QW5XvyyZC1yq",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Testing our models and computing metrics\n",
    "\n",
    "Now that we have a trained network, it is important to measure how well it performs. We do not do that during training because theoretically we try to test on a context closer to how the final model will be used, meaning this can be another pipeline and is usually outside the training engine.\n",
    "\n",
    "You can refer to your ML course or on resources on the web to see how we can measure it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c51c3b88",
   "metadata": {
    "editable": true,
    "id": "g0TldNDQC1yq",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Loading saved model\n",
    "\n",
    "State dict method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8d405d",
   "metadata": {
    "editable": true,
    "id": "Q-kCmgWEC1yr",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Instantiate a new empty model\n",
    "model = model_fn()\n",
    "\n",
    "print(model)\n",
    "\n",
    "# Load state\n",
    "checkpoint_path = \"model.pt\"\n",
    "model.load_state_dict(\n",
    "    torch.load(\n",
    "        checkpoint_path, weights_only=True, map_location=lambda storage, loc: storage\n",
    "    )\n",
    ")\n",
    "\n",
    "print(\"Model Loaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "231867ef",
   "metadata": {
    "editable": true,
    "id": "a-rbNh7qC1yr",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Inferencing on the test dataset\n",
    "\n",
    "Now we will run predictions on the test dataset using the newly loaded model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cc13733",
   "metadata": {
    "editable": true,
    "id": "LjlrKEEOC1yr",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "test_ds = NpArrayDataset(\n",
    "    images=test_images,\n",
    "    labels=test_labels,\n",
    "    image_transforms=image_transforms,\n",
    "    label_transforms=target_transforms,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6d3164a",
   "metadata": {
    "editable": true,
    "id": "Jf3oIRA4C1yr",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "807fda4f",
   "metadata": {
    "editable": true,
    "id": "VWM757ggC1ys",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "y_true = []\n",
    "y_pred = []\n",
    "\n",
    "# Send model to correct device\n",
    "model.to(DEVICE)\n",
    "\n",
    "# Put model in evaluatio mode (very important)\n",
    "model.eval()\n",
    "\n",
    "# Disable all gradients things\n",
    "with torch.no_grad():\n",
    "    for x, y_t in tqdm.tqdm(test_ds, \"predicting\"):\n",
    "        x = x.reshape((-1,) + x.shape)\n",
    "        x = x.to(DEVICE)\n",
    "        y = model.forward(x)\n",
    "        y = y.to(\"cpu\").numpy()\n",
    "\n",
    "        y_t = int(y_t.to(\"cpu\").numpy())\n",
    "\n",
    "        y_pred.append(y)\n",
    "        y_true.append(y_t)\n",
    "y_pred = np.concatenate(y_pred, axis=0)\n",
    "y_true = np.asarray(y_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e73f45af",
   "metadata": {
    "editable": true,
    "id": "awHUQ2KxC1ys",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "print(y_pred.shape)\n",
    "\n",
    "print(y_pred[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e799fa10",
   "metadata": {
    "editable": true,
    "id": "aGqeE1UJC1ys",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "y_pred_classes = y_pred[:, 0] > 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7356cc29",
   "metadata": {
    "editable": true,
    "id": "wMuCtss8C1ys",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Confusion matrix\n",
    "Here, we are first computing the [confusion matrix]():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a55265",
   "metadata": {
    "editable": true,
    "id": "QSq5-t7dC1yt",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix\n",
    "\n",
    "print(\"Confusion matrix\")\n",
    "cm = confusion_matrix(y_true, y_pred_classes)\n",
    "\n",
    "disp = ConfusionMatrixDisplay(\n",
    "    confusion_matrix=cm, display_labels=[\"background\", \"aircraft\"]\n",
    ")\n",
    "\n",
    "disp.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41098552",
   "metadata": {
    "editable": true,
    "id": "Ao41bfOuC1yt",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### ROC curve\n",
    "\n",
    "The next metric we are computing is the [Receiver Operating Characteristic](https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html). A receiver operating characteristic curve, or ROC curve, is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The method was originally developed for operators of military radar receivers starting in 1941, which led to its name. \n",
    "\n",
    "![](https://upload.wikimedia.org/wikipedia/commons/thumb/1/13/Roc_curve.svg/512px-Roc_curve.svg.png)\n",
    "\n",
    "![](http://algolytics.com/wp-content/uploads/2018/05/roc1_en.png)\n",
    "\n",
    "It is used to choose a threshold on the output probability in case you are interesting in controling the false positive rate.\n",
    "\n",
    "**In our case : which is the positive class ? which is the negative class ?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "607d819a",
   "metadata": {
    "editable": true,
    "id": "sEj4ZBgTC1yt",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Compute ROC curve and Area Under Curver\n",
    "\n",
    "from sklearn.metrics import auc, roc_curve\n",
    "\n",
    "# We round predictions for better readability\n",
    "y_pred_probas = np.round(y_pred[:, 0], 2)\n",
    "\n",
    "fpr, tpr, thresholds = roc_curve(y_true, y_pred_probas)\n",
    "roc_auc = auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ed86a65",
   "metadata": {
    "editable": true,
    "id": "KGD6ukiMC1yu",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "lw = 2\n",
    "plt.plot(\n",
    "    fpr, tpr, color=\"darkorange\", lw=lw, label=\"ROC curve (area = %0.2f)\" % roc_auc\n",
    ")\n",
    "plt.plot([0, 1], [0, 1], color=\"navy\", lw=lw, linestyle=\"--\")\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.title(\"Receiver operating characteristic example\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55d6871a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Using the ROC curve to select an optimal threshold\n",
    "\n",
    "The ROC curve can be used to select the best decision threshold for classifying an aircraft as positive.\n",
    "\n",
    "Plot the ROC curve with thresholds assigned to points in the curve (you can round the predictions for a simpler curve)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c8a3f9e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# We round predictions every 0.05 for readability\n",
    "y_pred_probas = (y_pred[:, 0] / 0.05).astype(np.int64) * 0.05\n",
    "\n",
    "fpr, tpr, thresholds = roc_curve(y_true, y_pred_probas)\n",
    "roc_auc = auc(fpr, tpr)\n",
    "\n",
    "plt.clf()\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "plt.step(fpr, tpr, \"bo\", alpha=0.2, where=\"post\")\n",
    "plt.fill_between(fpr, tpr, alpha=0.2, color=\"b\", step=\"post\")\n",
    "\n",
    "plt.xlabel(\"False Positive Rate\")\n",
    "plt.ylabel(\"True Positive Rate\")\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.title(\"2-class ROC curve: AUC={:0.2f}\".format(roc_auc))\n",
    "plt.plot([0, 1], [0, 1], color=\"darkblue\", linestyle=\"--\")\n",
    "\n",
    "for tp, fp, t in zip(tpr, fpr, thresholds):\n",
    "    plt.annotate(\n",
    "        np.round(t, 2),\n",
    "        xy=(fp, tp),\n",
    "        xytext=(fp - 0.05, tp - 0.05),\n",
    "        arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"arc3\"),\n",
    "    )\n",
    "plt.savefig(\"roc_curve_thresholds.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ecc3668",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "Now, choose a threshold on the curve where you miss less than 10% of the aircrafts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72968875",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "selected_threshold = ...\n",
    "\n",
    "print(\"Confusion matrix\")\n",
    "\n",
    "y_pred_classes = y_pred_probas >= selected_threshold\n",
    "\n",
    "cm = confusion_matrix(y_true, y_pred_classes)\n",
    "\n",
    "disp = ConfusionMatrixDisplay(\n",
    "    confusion_matrix=cm, display_labels=[\"background\", \"aircraft\"]\n",
    ")\n",
    "\n",
    "disp.plot()\n",
    "plt.show()\n",
    "\n",
    "# How did the confusion matrix evolve ? Does it match your intuition ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8809f5ac",
   "metadata": {},
   "source": [
    "**Food for thought** : \n",
    "- How would you generalize this method if you had 10 (exclusive) classes ? Example with MNIST or [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) ?\n",
    "- Would you select one general threshold or one threshold per class ? Or maybe get the max proba then threshold it differently with each class ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de337331",
   "metadata": {
    "editable": true,
    "id": "ql5f8eLHC1yu",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Misclassified examples\n",
    "\n",
    "It is always interesting to check mis classified examples.\n",
    "\n",
    "It usually provides tips on how to improve your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aed18b1",
   "metadata": {
    "editable": true,
    "id": "Z0wvNDzmC1yv",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "misclassified_idxs = np.where(y_pred_classes != y_true)[0]\n",
    "\n",
    "print(len(misclassified_idxs))\n",
    "\n",
    "print(misclassified_idxs)\n",
    "\n",
    "misclassified_images = test_images[misclassified_idxs]\n",
    "misclassified_true_labels = test_labels[misclassified_idxs]\n",
    "misclassified_pred_labels = y_pred_classes[misclassified_idxs]\n",
    "\n",
    "grid_size = 4\n",
    "grid = np.zeros((grid_size * 64, grid_size * 64, 3)).astype(np.uint8)\n",
    "for i in range(grid_size):\n",
    "    for j in range(grid_size):\n",
    "        img = np.copy(misclassified_images[i * grid_size + j])\n",
    "        pred = np.copy(misclassified_pred_labels[i * grid_size + j])\n",
    "        color = (0, 255, 0) if pred == 1 else (255, 0, 0)\n",
    "        tile = cv2.rectangle(img, (0, 0), (64, 64), color, thickness=2)\n",
    "        grid[i * 64 : (i + 1) * 64, j * 64 : (j + 1) * 64, :] = img\n",
    "\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "ax.imshow(grid)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0a34443",
   "metadata": {
    "editable": true,
    "id": "-DDeFy4cC1yv",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## Improving our training / validation loop\n",
    "\n",
    "We will add more advanced features to our training loop for better models\n",
    "\n",
    "Copy the train / valid loop and update it accordingly"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1fb6cc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Computing accuracy during training / validation\n",
    "\n",
    "Update the `valid_one_epoch` to compute accuracy during during the validation loop, and plot its evolution during training\n",
    "\n",
    "Use the ROC curve computation where we compute the pred / true classes as inspiration\n",
    "\n",
    "Here's an example (that needs to be modified)\n",
    "```python\n",
    "\n",
    "correct_pred = 0\n",
    "total_pred = 0\n",
    "with torch.no_grad():\n",
    "    for data in valid_loader:\n",
    "        images, labels = data\n",
    "        outputs = net(images)\n",
    "        predictions = torch.round(outputs)[:,0]\n",
    "        # collect the correct predictions\n",
    "        for label, prediction in zip(labels, predictions):\n",
    "            if label == prediction:\n",
    "                correct_pred += 1\n",
    "            total_pred += 1\n",
    "            \n",
    "    # print accuracy\n",
    "    accuracy = 100 * (total_pred / total_pred)\n",
    "    print(\"Accuracy is: {:.1f} %\".format(accuracy))\n",
    "\n",
    "```                                             "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc508ebd",
   "metadata": {
    "editable": true,
    "id": "84qzXDMGC1yw",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Early stopping\n",
    "\n",
    "You've seen that it is possible to overfit it you're not careful,\n",
    "\n",
    "**Go back to your previous class and adapt the training loop to add early stopping**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ce6d921",
   "metadata": {
    "editable": true,
    "id": "NW7rLbdGC1yx",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "fd58fe29",
   "metadata": {
    "editable": true,
    "id": "4XRzekUGC1yy",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Data Augmentation\n",
    "\n",
    "\n",
    "One technique for training CNNs on images is to put your training data through data augmentation to generate similar-but-different examples to make your network more robust.\n",
    "\n",
    "You can generate \"augmented images\" on the fly or use composition to generate data\n",
    "\n",
    "- We are going to wrap our numpy arrays with `torch.utils.data.Dataset` class\n",
    "\n",
    "https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class\n",
    "\n",
    "- Here is how we use torch Compose to augment data\n",
    "\n",
    "https://pytorch.org/docs/stable/torchvision/transforms.html\n",
    "\n",
    "https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#compose-transforms\n",
    "\n",
    "Note: This step requires a bit of tinkering from numpy arrays to torch datasets, it's fine if you skip it. For the next notebook it may prove a useful way of gaining performance\n",
    "\n",
    "**Remember : We apply data augmentation only during training**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da3237e4",
   "metadata": {
    "editable": true,
    "id": "ua4UQAZWC1yy",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "import torch.functional as F\n",
    "import torch.utils\n",
    "import torchvision.transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21fcf544",
   "metadata": {
    "editable": true,
    "id": "em78uFlnC1yy",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Example (very simple) data augmentation to get your started, you can add more transforms to this list\n",
    "\n",
    "train_transform = torchvision.transforms.Compose(\n",
    "    [\n",
    "        torchvision.transforms.ToPILImage(),\n",
    "        torchvision.transforms.RandomHorizontalFlip(p=0.5),\n",
    "        torchvision.transforms.RandomVerticalFlip(p=0.5),\n",
    "        torchvision.transforms.ToTensor(),\n",
    "        torchvision.transforms.Normalize(mean, std),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b2bccb4",
   "metadata": {
    "editable": true,
    "id": "5yPlhQbzC1yy",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Example\n",
    "trainset_augmented = NpArrayDataset(\n",
    "    images=train_images,\n",
    "    labels=train_labels,\n",
    "    image_transforms=train_transform,\n",
    "    label_transforms=None,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84beb2b0",
   "metadata": {
    "editable": true,
    "id": "MeKVDcHrC1yz",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# Get image from dataset. Note: it has been converted as a torch tensor in CHW format in float32 normalized !\n",
    "img, label = trainset_augmented[0]\n",
    "img = img.numpy().transpose((1, 2, 0)) * std + mean\n",
    "img = img.clip(0.0, 1.0)\n",
    "img = (img * 255.0).astype(np.uint8)\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "\n",
    "# Compare effects of data augmentation\n",
    "img_orig = trainset_augmented.images[0]\n",
    "plt.imshow(img_orig)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97d75f01",
   "metadata": {
    "editable": true,
    "id": "ej7Jb0SNC1yz",
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# do another training and plot our metrics again. Did we change something ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21e4d111",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Best checkpoint\n",
    "\n",
    "You've seen how to save model checkpoint. However we saved the model at the end of training. What if there is an issue (like overfitting ? or our computer crashes !!!) ? \n",
    "\n",
    "How to keep a good copy of our model at any point ? \n",
    "\n",
    "The idea is that during the training, we always save the checkpoint with the lowest valid loss, then reload it at the end of training\n",
    "\n",
    "**Modify the train loop to keep the best model state dict at any point, then reload it at the end of training**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b44dc452",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8bbe1416",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "toc-hr-collapsed": true
   },
   "source": [
    "## Food for thoughts: Tooling\n",
    "\n",
    "To conclude this notebook, reflect on the following,\n",
    "\n",
    "You have launched different experiences and obtained different results,\n",
    "\n",
    "Did you feel the notebook you used was sufficient ? Which tools would you like to have in order to properly run your experiments ? (Quick google search or ask someone) Do they already exist ?\n",
    "\n",
    "### **Presentation : High level frameworks**\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/pytorch/ignite/master/assets/logo/ignite_logo_mixed.svg\" alt=\"ignite\" style=\"width: 400px;\"/>\n",
    "\n",
    "Pytorch ignite is what we call a \"high-level library\" over pytorch, its objectives is to abstract away most of the boilerplate code for training deep neural network.\n",
    "\n",
    "Usually, they make the development process easier by enabling you to focus on what's important instead of writing distributed and optimized training loops and plugging metrics / callbacks. Because we all forgot to call `.backward()` or `.zero_grad()` at least once.\n",
    "\n",
    "Here an overview of the high-level libraries available for pytorch,\n",
    "\n",
    "https://neptune.ai/blog/model-training-libraries-pytorch-ecosystem?utm_source=twitter&utm_medium=tweet&utm_campaign=blog-model-training-libraries-pytorch-ecosystem\n",
    "\n",
    "Of these, we would like to highlight three of them:\n",
    "\n",
    "- pytorch-ignite, officially sanctioned by the pytorch team (its repo lives at https://pytorch.org/ignite/), which is developped by [someone from Toulouse](https://twitter.com/vfdev_5) - yes there is a member of the pytorch team living in Toulouse, we are not THAT behind in ML/DL :wishful-thinking:\n",
    "\n",
    "- pytorch-lightning (https://www.pytorchlightning.ai/) which has recently seen its 1.0 milestone and has been developped to a company. It is more \"research oriented\" that pytorch-ignite, and with a lower abstraction level, but seems to enable more use case.\n",
    "\n",
    "- catalyst (https://github.com/catalyst-team/catalyst) \n",
    "\n",
    "- skorch (https://github.com/skorch-dev/skorch). This class was previously written in skorch. Skorch mimics the scikit-learn API and allows bridging the two libraries together. It's a bit less powerful but you write much less code than the two libraries above, and if you are very familiar with scikit-learn, it could be very useful for fast prototyping\n",
    "\n",
    "\n",
    "**Take a look at the [previous class](https://nbviewer.jupyter.org/github/SupaeroDataScience/deep-learning/blob/main/deep/PyTorch%20Ignite.ipynb), the [official examples](https://nbviewer.jupyter.org/github/pytorch/ignite/tree/master/examples/notebooks/) or the [documentation](https://pytorch.org/ignite/) if want to learn about Ignite**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a5c1cee",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## **Optional** exercises to run at home\n",
    "\n",
    "If you're done with this, you can explore a little bit more : Now that we have a nice training loop we can do hyperparameter tuning !\n",
    "\n",
    "As you can see, there are a lot of parameters we can choose:\n",
    "\n",
    "- the optimizer\n",
    "- the learning rate\n",
    "- the model architecture\n",
    " \n",
    "etc... !\n",
    "\n",
    "\n",
    "- Try to play with network hyperparameters. The dataset is small and allow fast iterations so use it to have an idea on hyperparameter sensitivity.\n",
    "    number of convolutions, other network structures, learning rates, optimizers,...\n",
    "\n",
    "- Example: Compare again SGD and ADAM\n",
    "\n",
    "- Try to use the ROC curve to select a threshold to filter only negative examples without losing any positive examples\n",
    "\n",
    "When you are done with the warmup, go to the next notebook. But remember that next datasets will be larger and you will not have the time (trainings will take longer ) to experiment on hyperparameters.\n",
    "\n",
    "**You can try more things**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "861a2172",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Optimizer Changes\n",
    "Change the optimizer from SGD to optim.Adam. Is it better ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99ea0cf9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d3a9ea8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Batch Normalization\n",
    "\n",
    "One of the most used \"layer\" beyond conv / pool / relu is \"batch normalization\",\n",
    "\n",
    "http://d2l.ai/chapter_convolutional-modern/batch-norm.html\n",
    "\n",
    "Try adding it to your network and see what happens !\n",
    "\n",
    "<details>\n",
    "\n",
    "```python\n",
    "def model_fn():\n",
    "    model = nn.Sequential(\n",
    "        # A first convolution block\n",
    "        nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3),\n",
    "        nn.BatchNorm2d(16),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=16, out_channels=16, kernel_size=3),\n",
    "        nn.BatchNorm2d(16),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # Another stack of these\n",
    "        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3),\n",
    "        nn.BatchNorm2d(32),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "        nn.BatchNorm2d(32),\n",
    "        nn.ReLU(),\n",
    "        nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3),\n",
    "        nn.BatchNorm2d(32),\n",
    "        nn.ReLU(),\n",
    "        nn.MaxPool2d(2),\n",
    "        # A final classifier\n",
    "        nn.Flatten(),\n",
    "        nn.Linear(in_features=12 * 12 * 32, out_features=64),\n",
    "        nn.BatchNorm1d(64),\n",
    "        nn.ReLU(),\n",
    "        nn.Dropout(p=0.1),\n",
    "        nn.Linear(in_features=64, out_features=1),\n",
    "        nn.Sigmoid(),\n",
    "    )\n",
    "\n",
    "    return model\n",
    "```\n",
    "    \n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ab06dd",
   "metadata": {
    "editable": true,
    "id": "xUlVYFuPC1yz",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Trying other models\n",
    "\n",
    "You have seen a class on different model structure,\n",
    "https://supaerodatascience.github.io/deep-learning/slides/2_architectures.html#/\n",
    "\n",
    "Now is the time to try and implement them. \n",
    "\n",
    "For example, try to write a VGG-11 with fewer filters by yourself... or a very small resnet using [this](https://github.com/a-martyn/resnet/blob/master/resnet.py) as inspiration\n",
    "\n",
    "You can also use models from [torchvision](https://pytorch.org/docs/stable/torchvision/models.html#classification) in your loop, or as inspiration\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "891a84bc",
   "metadata": {
    "editable": true,
    "id": "L5vTgr9OC1yz",
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "**Modify the model structure and launch another training... Is it better ?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e06122a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1de8a45",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### LR Scheduling\n",
    "\n",
    "Sometimes it's best to reduce the learning rate if you stop improving, or to reduce learning rate at the end of training\n",
    "\n",
    "Tutorial : https://www.deeplearningwizard.com/deep_learning/boosting_models_pytorch/lr_scheduling/#top-basic-learning-rate-schedules\n",
    "\n",
    "- **Modify the train loop to change the learning rate when the validation loss is stagnating**\n",
    "\n",
    "- **Modify the train loop to change the learning rate when the validation loss is stagnating**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f236094",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    }
   },
   "outputs": [],
   "source": [
    "# ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e60e449",
   "metadata": {},
   "source": [
    "### How to train neural networks ?\n",
    "\n",
    "You must have noticed that training neural networks depends on a lot of different things : hyperparameters, architectures, data, ...\n",
    "\n",
    "It's an experimental science that requires some tuning... and such tuning can yield very decreasing improvements over time. It's a kind of special hell / pandora's box.\n",
    "\n",
    "Here are some very interesting links on getting started the right way when training neural networks :\n",
    "\n",
    "http://cs231n.stanford.edu/slides/2023/lecture_7.pdf\n",
    "\n",
    "https://cs231n.stanford.edu/slides/2024/lecture_6_part_2.pdf\n",
    "\n",
    "https://karpathy.github.io/2019/04/25/recipe/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a74a3015",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_json": true,
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}