Image_Super_Resolution_using_Autoencoders.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Image Super Resolution using Autoencoders.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "gJAxdtNX1Yyi",
        "oGLw-cCn1Y0q",
        "ZXsavBJj1Y1N",
        "qMnwvfUb1Y1O",
        "wh9OSw-D1Y1Q",
        "telVTJOQ1Y1n",
        "ksg71gnK1Y13",
        "1QGkU3q21Y2E",
        "aWdvqEyz1Y2L",
        "jEYu9t021Y2Q",
        "8mBRutcQ1Y2R",
        "Ey0Cnp2A1Y2T",
        "J0r-drht1Y3V",
        "JIMWxWjV1Y3Z",
        "1NSeNnml1Y3Z"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 2",
      "language": "python",
      "name": "python2"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ilopezfr/image-superres/blob/master/Image_Super_Resolution_using_Autoencoders.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x1zxbsCB88JO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/ilopezfr/image-superres-with-autoencoders"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KKA2mwNm1YyI",
        "colab_type": "text"
      },
      "source": [
        "# Image Super Resolution Using Auto-encoders Tutorial\n",
        "---\n",
        "\n",
        "In this tutorial we'll review how to train an Auto-encoder to convert low-quality images into high-quality. We'll be using TensorFlow and Keras. \n",
        "\n",
        "## Introduction \n",
        "\n",
        "### Auto-encoder\n",
        "An Auto-encoder is a type of Neural Network that tries to learn a representation of its input data, but in a space with much smaller dimensionality. This smaller representation is able to learn important features of the input data that can be used to later reconstruct the data. \n",
        "An auto encoder is principally composed of 3 elements: an **encoder**, a **decoder** and a **loss function**.\n",
        "* Both the encoder and decoder are usually Convolutional Neural Networks.\n",
        "* The encoder tries to reduce the dimensionality of the input while the decoder tries to recover our image from this new space. \n",
        "    * First, the **encoder** takes an input and passes it through its layers, gradually reducing the receptive field of the input. At the end of the encoder, the input is reduced to a *linear feature representation*.  \n",
        "    * This linear feature representation is then fed to the **decoder** which tries to recover the image through *upsampling* it (increasing its receptive field) gradually until it reaches the end where the output has the same dimensions as the original input. \n",
        "* This architecture is ideal for preserving the dimensionality. However, the linear compression of the input is a *lossy* process, meaning it losses information in the process.\n",
        "* The **loss function** is a way of describing a meaningful difference (or distance) between the input and output. During training, our goal is to minimize such difference so that the network will eventually lean to reconstruct the best possible output.\n",
        "\n",
        "In order to teach an auto-encoder how to reconstruct an image, we need to show it pairs of low quality and high quality images. This way, then network will try to find the patterns and important encoded visual features needed to be able to reconstruct it from the low quality version.  \n",
        "\n",
        "During training, the hidden layers will capture a **dense** (compressed) representation of the input data.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D4Vz_C6I1YyL",
        "colab_type": "text"
      },
      "source": [
        "### Super-Resolution\n",
        "\n",
        "Auto-encoders have many applications in image processing, especially in the Image Transformation task. Some of these applications include: \n",
        "- Denoising\n",
        "- Super-Resolution\n",
        "- Colorization\n",
        "\n",
        "In this tuturial, we'll go over the problem of **super-resolution**, where the task is to generate a high-resolution output image from a low-resolution input.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzhGkq2q1Yyj",
        "colab_type": "text"
      },
      "source": [
        "## The encoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3K1DewX1Yyl",
        "colab_type": "text"
      },
      "source": [
        "Here we'll be designing the following encoder using TensorFlow and Keras."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VstTPiLM1Yyo",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://github.com/ilopezfr/image-superres/blob/master/img/model_encoder.png?raw=1\"  width=\"450\" />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yukv2fVA1YzN",
        "colab_type": "text"
      },
      "source": [
        "Run the complete code below, it's the entire encoder. We can see a summary of what our encoder looks like right after."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bRaxJR9U1YzP",
        "colab_type": "code",
        "outputId": "9cb770ff-abb8-40ca-af64-95659d663f42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "from keras.layers import Input, Dense, Conv2D, MaxPooling2D, Dropout, Conv2DTranspose, UpSampling2D, add\n",
        "from keras.models import Model\n",
        "from keras import regularizers\n",
        "\n",
        "# Encoder\n",
        "\n",
        "n = 256\n",
        "chan = 3\n",
        "input_img = Input(shape=(n, n, chan))\n",
        "\n",
        "l1 = Conv2D(64, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(input_img)\n",
        "l2 = Conv2D(64, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l1)\n",
        "l3 = MaxPooling2D(padding='same')(l2)\n",
        "l3 = Dropout(0.3)(l3)\n",
        "l4 = Conv2D(128, (3, 3),  padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l3)\n",
        "l5 = Conv2D(128, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l4)\n",
        "l6 = MaxPooling2D(padding='same')(l5)\n",
        "l3 = Dropout(0.5)(l3)\n",
        "l7 = Conv2D(256, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l6)\n",
        "encoder = Model(input_img, l7)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n",
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0621 02:09:04.456413 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "W0621 02:09:04.502058 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
            "\n",
            "W0621 02:09:04.512031 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
            "\n",
            "W0621 02:09:04.586672 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n",
            "\n",
            "W0621 02:09:04.592659 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
            "\n",
            "W0621 02:09:04.603779 140159259035520 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8f7kOjIW1YzW",
        "colab_type": "code",
        "outputId": "87f0c5b5-56e8-4753-959c-1e9683252dd4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 450
        }
      },
      "source": [
        "encoder.summary()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         (None, 256, 256, 3)       0         \n",
            "_________________________________________________________________\n",
            "conv2d_1 (Conv2D)            (None, 256, 256, 64)      1792      \n",
            "_________________________________________________________________\n",
            "conv2d_2 (Conv2D)            (None, 256, 256, 64)      36928     \n",
            "_________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2 (None, 128, 128, 64)      0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 128, 128, 64)      0         \n",
            "_________________________________________________________________\n",
            "conv2d_3 (Conv2D)            (None, 128, 128, 128)     73856     \n",
            "_________________________________________________________________\n",
            "conv2d_4 (Conv2D)            (None, 128, 128, 128)     147584    \n",
            "_________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2 (None, 64, 64, 128)       0         \n",
            "_________________________________________________________________\n",
            "conv2d_5 (Conv2D)            (None, 64, 64, 256)       295168    \n",
            "=================================================================\n",
            "Total params: 555,328\n",
            "Trainable params: 555,328\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ogGgm6_RWBh2",
        "colab_type": "text"
      },
      "source": [
        "Some comments: \n",
        "\n",
        "**Layers `l4 and l5`**: \n",
        "```python\n",
        "l4 = Conv2D(128, (3, 3),  padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l3)\n",
        "l5 = Conv2D(128, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l4)\n",
        "```\n",
        "- In Layer 3, in our new reduced space of 128x128, let's put more convolutions (128 of 3x3) so that we can learn more features. Because the space is smaller, we have less information. So by having more different convolutions we can try to compensate for the loss of information. \n",
        "- This gives the network much more perspective. We can imagine these convolution filters as being like a different point of view, each at the same thing (the image). An analogy is having many people (convolution filters) on a team working together and seeing same problem from different angles.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwgkECqF1Yzc",
        "colab_type": "text"
      },
      "source": [
        "## The decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "St60F-5I1Yzf",
        "colab_type": "text"
      },
      "source": [
        "Let's now build our decoder!\n",
        "\n",
        "The steps are pretty much the same as in the encoder but in **reverse order**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DQLpclKD1Yzh",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://github.com/ilopezfr/image-superres/blob/master/img/model_decoder.png?raw=1\" width=\"450\"/>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bwSrWJWZXI0q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "87b81635-634f-4c38-8f09-b173d5c19d86"
      },
      "source": [
        "# Decoder\n",
        "\n",
        "l8 = UpSampling2D()(l7)\n",
        "\n",
        "l9 = Conv2D(128, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l8)\n",
        "l10 = Conv2D(128, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l9)\n",
        "\n",
        "l11 = add([l5, l10])\n",
        "l12 = UpSampling2D()(l11)\n",
        "l3 = Dropout(0.3)(l3)\n",
        "l13 = Conv2D(64, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l12)\n",
        "l14 = Conv2D(64, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l13)\n",
        "\n",
        "l15 = add([l14, l2])\n",
        "\n",
        "# chan = 3, for RGB\n",
        "decoded = Conv2D(chan, (3, 3), padding='same', activation='relu', activity_regularizer=regularizers.l1(10e-10))(l15)\n",
        "\n",
        "# Create our network\n",
        "autoencoder = Model(input_img, decoded)\n",
        "# You'll understand later what this is\n",
        "autoencoder_hfenn = Model(input_img, decoded)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0621 02:09:08.133253 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JaeojFbI1Yzs",
        "colab_type": "text"
      },
      "source": [
        "Some comments:\n",
        "\n",
        "**Layers `l11` and  `l15`** :\n",
        "```python\n",
        "l11 = add([l5, l10])\n",
        "l15 = add([l14, l2])\n",
        "```\n",
        "- Using a **merge layer** we are performing an \"add\" operation with the layer below and a layer from the encoder: \n",
        "  - `conv2D_l4 + conv2d_l7`\n",
        "  - `conv2D_2 + conv2D_9`\n",
        "- We do this for various reasons:\n",
        "  * we want to share knowledge from the encoder to the decoder like \"Hey, I'm trying to reconstruct this part of the image, did it roughly look like that to you also?\".\n",
        "  * it helps with the **\"vanishing gradient problem\"**, in which the network looses information from going deeper. Neurons at the beginning have difficulty learning in deep networks because they are too far from the deeper networks and can't have their knowledge shared. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qzv1ncEo1Yz8",
        "colab_type": "text"
      },
      "source": [
        "## Complete network\n",
        "*Here*'s the final network :"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nE9UQBU-1Yz_",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://github.com/ilopezfr/image-superres/blob/master/img/model.png?raw=1\"  width=\"450\" />"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s1rpXscY1Y0J",
        "colab_type": "code",
        "outputId": "db5c653b-ece9-4596-d82d-dc7599cdff46",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 797
        }
      },
      "source": [
        "autoencoder.summary()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            (None, 256, 256, 3)  0                                            \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_1 (Conv2D)               (None, 256, 256, 64) 1792        input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_2 (Conv2D)               (None, 256, 256, 64) 36928       conv2d_1[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling2d_1 (MaxPooling2D)  (None, 128, 128, 64) 0           conv2d_2[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "dropout_1 (Dropout)             (None, 128, 128, 64) 0           max_pooling2d_1[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_3 (Conv2D)               (None, 128, 128, 128 73856       dropout_1[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_4 (Conv2D)               (None, 128, 128, 128 147584      conv2d_3[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "max_pooling2d_2 (MaxPooling2D)  (None, 64, 64, 128)  0           conv2d_4[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_5 (Conv2D)               (None, 64, 64, 256)  295168      max_pooling2d_2[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "up_sampling2d_1 (UpSampling2D)  (None, 128, 128, 256 0           conv2d_5[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_6 (Conv2D)               (None, 128, 128, 128 295040      up_sampling2d_1[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_7 (Conv2D)               (None, 128, 128, 128 147584      conv2d_6[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "add_1 (Add)                     (None, 128, 128, 128 0           conv2d_4[0][0]                   \n",
            "                                                                 conv2d_7[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "up_sampling2d_2 (UpSampling2D)  (None, 256, 256, 128 0           add_1[0][0]                      \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_8 (Conv2D)               (None, 256, 256, 64) 73792       up_sampling2d_2[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_9 (Conv2D)               (None, 256, 256, 64) 36928       conv2d_8[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "add_2 (Add)                     (None, 256, 256, 64) 0           conv2d_9[0][0]                   \n",
            "                                                                 conv2d_2[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "conv2d_10 (Conv2D)              (None, 256, 256, 3)  1731        add_2[0][0]                      \n",
            "==================================================================================================\n",
            "Total params: 1,110,403\n",
            "Trainable params: 1,110,403\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGJbiJ8v1Y0O",
        "colab_type": "text"
      },
      "source": [
        "### Training \n",
        "\n",
        "Let's *compile* the model to be able to train it. For now we'll simply use a **Mean Squared Error (MSE)** for the loss, we'll see later what this is and if we can go further."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SQYqnLHX1Y0P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "381e1d3e-9f70-4125-adfc-761280b5ae12"
      },
      "source": [
        "autoencoder.compile(optimizer='adadelta', loss='mean_squared_error')"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0621 02:09:32.783448 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hwHfWGBL1Y0T",
        "colab_type": "text"
      },
      "source": [
        "We have a dataset of images and we load it by batches.  the dataset doesn't really hold in memory, so we split it by batches and give it to the GPU so that it can train on a reasonable part of the dataset at each iteration.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "fkzqHc5q1Y0U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import re\n",
        "from scipy import ndimage, misc\n",
        "from skimage.transform import resize, rescale\n",
        "from matplotlib import pyplot\n",
        "import numpy as np\n",
        "def train_batches(just_load_dataset=False):\n",
        "    batches = 256 # Number of images to have at the same time in a batch\n",
        "    batch = 0 # counter images in the current batch (grows over time and then resets for each batch)\n",
        "    batch_nb = 0 # counter of current batch index\n",
        "    max_batches = -1 # If you want to train only on a limited number of images to finish the training even faster.\n",
        "    \n",
        "    ep = 4 # Number of epochs\n",
        "    images = []\n",
        "    x_train_n = []\n",
        "    x_train_down = []\n",
        "    \n",
        "    x_train_n2 = [] # Resulting high res dataset\n",
        "    x_train_down2 = [] # Resulting low res dataset\n",
        "    \n",
        "    for root, dirnames, filenames in os.walk(\"/data/cars_train\"): # generate the files names\n",
        "        for filename in filenames:\n",
        "            if re.search(\"\\.(jpg|jpeg|JPEG|png|bmp|tiff)$\", filename):\n",
        "                if batch_nb == max_batches: # If we limit the number of batches, just return earlier\n",
        "                    return x_train_n2, x_train_down2\n",
        "                filepath = os.path.join(root, filename)\n",
        "                image = pyplot.imread(filepath) # read the image file and save into an array\n",
        "                if len(image.shape) > 2:\n",
        "                    # Resize the image so that every image is the same size\n",
        "                    image_resized = resize(image, (256, 256))\n",
        "                    # Add this image to the high res dataset\n",
        "                    x_train_n.append(image_resized) \n",
        "                    # Rescale it 0.5x and 2x so that it is a low res image but still has 256x256 resolution\n",
        "                    x_train_down.append(rescale(rescale(image_resized, 0.5), 2.0)) \n",
        "                    batch += 1\n",
        "                    if batch == batches:\n",
        "                        batch_nb += 1\n",
        "                        x_train_n2 = np.array(x_train_n)\n",
        "                        x_train_down2 = np.array(x_train_down)\n",
        "                        \n",
        "                        if just_load_dataset:\n",
        "                            return x_train_n2, x_train_down2\n",
        "                        \n",
        "                        print('Training batch', batch_nb, '(', batches, ')')\n",
        "                        autoencoder.fit(x_train_down2, x_train_n2,\n",
        "                            epochs=ep,\n",
        "                            batch_size=10,\n",
        "                            shuffle=True,\n",
        "                            validation_split=0.15)\n",
        "                    \n",
        "                        x_train_n = []\n",
        "                        x_train_down = []\n",
        "                    \n",
        "                        batch = 0\n",
        "    return x_train_n2, x_train_down2\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BSxLlC51Y0Z",
        "colab_type": "text"
      },
      "source": [
        "To train the model:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nnn-4MW11Y0b",
        "colab_type": "text"
      },
      "source": [
        "```Python\n",
        "x_train_n = []\n",
        "x_train_down = []\n",
        "x_train_n, x_train_down = train_batches()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-PtY4Va1Y0e",
        "colab_type": "text"
      },
      "source": [
        "Training from scratch takes a lot of time, so we'll just load a pretrained model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0dMdX4pyZ90L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# download CARS DATA from this url: https://ai.stanford.edu/~jkrause/cars/car_dataset.html\n",
        "\n",
        "%cd /content\n",
        "!wget http://imagenet.stanford.edu/internal/car196/cars_train.tgz \n",
        "!tar -xvzf cars_train.tgz"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EYvpKEm91Y0g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train_n, x_train_down = train_batches(just_load_dataset=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYeIlN1b1Y0l",
        "colab_type": "text"
      },
      "source": [
        "And here, we load the already existing weights :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FQUbtXNi1Y0n",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "42a29f1e-c9ed-42a9-91a2-21b926ccbd12"
      },
      "source": [
        "# download the WEIGHTS from a location in Dropbox\n",
        "%cd /content\n",
        "!wget https://www.dropbox.com/s/n2s2n29ja5xytc7/weights.zip?dl=0 -O weights.zip\n",
        "!unzip weights.zip\n",
        "  \n",
        "autoencoder.load_weights(\"/content/weights/sr.img_net.mse.final_model5.no_patch.weights.best.hdf5\")\n",
        "\n",
        "\n"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W0621 02:10:24.353622 140159259035520 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
            "\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oGLw-cCn1Y0q",
        "colab_type": "text"
      },
      "source": [
        "### Display the results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Go6LJrVX1Y03",
        "colab_type": "text"
      },
      "source": [
        "Predict:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "--HPXY-j1Y04",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# We clip the output so that it doesn't produce weird colors\n",
        "sr1 = np.clip(autoencoder.predict(x_train_down), 0.0, 1.0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eZapUSRU1Y09",
        "colab_type": "text"
      },
      "source": [
        "Display and compare the results. In the following order:\n",
        "\n",
        "* The low-res input image\n",
        "* The reconstructed image\n",
        "* The original high-res image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJcQd3TM1Y0-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "image_index = 251"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gsn1b7jV1Y1B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(128, 128))\n",
        "i = 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_down[image_index])  #Input (low-res)\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(sr1[image_index])  # Output (supre-res recovered image)\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_n[image_index])  # Ground truth (high-res) \n",
        "plt.show()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X2SD2si31Y1d",
        "colab_type": "text"
      },
      "source": [
        "## Loss Metrics\n",
        "\n",
        "Below are the most common metrics to measure the similarity between a pair of images. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "telVTJOQ1Y1n",
        "colab_type": "text"
      },
      "source": [
        "### MSE"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3viqcExU1Y1q",
        "colab_type": "text"
      },
      "source": [
        "[Mean Squared Error](https://en.wikipedia.org/wiki/Root-mean-square_deviation) is a metric that indicates perfect similarity if the value is 0.\n",
        "\n",
        "The value grows beyond 0 if the average difference between pixel intensities increases.\n",
        "\n",
        "MSE has a few issues when used for similarity comparison, meaning that if you take an image and photoshop it to make it brighter and compare it with the original, the two images will be very different according to MSE as a dark image has pixel values closer to 0 and a bright image has pixel closer to 1, this means that the difference is going to be very big as MSE sees the image from a very general point of view.\n",
        "\n",
        "Anyway, for our purpose, we compare lower quality images to their higher resolution counterpart, so the brightness is retained, this means that ** MSE is still a useful metric in our case**."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D5401v-L1Y1r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def mse(orig, res):\n",
        "    return ((orig - res) ** 2).mean()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wda6KVFW1Y1w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mse(high_res, low_res)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ksg71gnK1Y13",
        "colab_type": "text"
      },
      "source": [
        "### SSIM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ko2ybLmQ1Y14",
        "colab_type": "text"
      },
      "source": [
        "[Structural similarity (SSIM)](https://en.wikipedia.org/wiki/Structural_similarity) measures the similarity between two images as would be perceived on a television or a similar media."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBQ81FP71Y15",
        "colab_type": "text"
      },
      "source": [
        "The SSIM is a value in the range $[1, -1]$ where $1$ would mean two indentical images, and lower values would show a \"perceptual\" difference.\n",
        "\n",
        "This metric compares small windows in the image rather than the whole image (like MSE), which makes it a bit more interesting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w7goFz-91Y17",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import skimage.measure\n",
        "import os\n",
        "from matplotlib import pyplot\n",
        "\n",
        "def ssim(ori, res):\n",
        "    return skimage.measure.compare_ssim(ori.astype(np.float64),\n",
        "        res.astype(np.float64),\n",
        "        gaussian_weights=True, data_range=1., win_size=1,\n",
        "        sigma=1.5, multichannel=False, use_sample_covariance=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-7FsuS9A1Y2B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ssim(high_res, low_res)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QGkU3q21Y2E",
        "colab_type": "text"
      },
      "source": [
        "### PSNR"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdEHg-OA1Y2E",
        "colab_type": "text"
      },
      "source": [
        "[Peak signal-to-noise ratio](https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio) is a metric defined using Mean Squared Error (seen above). \n",
        "\n",
        "*PSNR is most commonly used to measure the quality of reconstruction of lossy compression* - Wikipedia\n",
        "\n",
        "Low resolution and pixelization can be considered as a form of *compression* as we loose information.\n",
        "\n",
        "When there's no noise, the PSNR is infinite (because there's a division by the MSE and MSE is $0$ when both images are exactly the same).\n",
        "\n",
        "Of course, this means that we need to maximize the PSNR. The result is in decibel (dB)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LD7yncvH1Y2G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def psnr(ori, res):\n",
        "    return skimage.measure.compare_psnr(ori, res, data_range=1.)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3l7-xAxZ1Y2J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "psnr(high_res, low_res)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aWdvqEyz1Y2L",
        "colab_type": "text"
      },
      "source": [
        "### HFENN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_MwI6Zf51Y2M",
        "colab_type": "text"
      },
      "source": [
        "The HFENN (High Frequency Error Norm Normalized) metric gives  a  measure  of how high-frequency details differ between two images.\n",
        "\n",
        "This means that we can guess if an image has more or less high frequency details (which are fine details that you need to zoom in to see and that are not blurry) compared to another image.\n",
        "\n",
        "When the output value is 0 the images are identical. The greater the value, the more of a perceptual difference in both images there is."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FksgXrwa1Y2M",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "from scipy.ndimage import filters\n",
        "\n",
        "def l_o_g(img, sigma):\n",
        "    '''\n",
        "    Laplacian of Gaussian filter (channel-wise)\n",
        "    -> img: input image\n",
        "    -> sigma: gaussian_laplace sigma\n",
        "    <- filtered image\n",
        "    '''\n",
        "    while len(img.shape) < 3:\n",
        "        img = img[..., np.newaxis]\n",
        "    out = img.copy()\n",
        "    for chan in range(img.shape[2]):\n",
        "        out[..., chan] = filters.gaussian_laplace(img[..., chan], sigma)\n",
        "    return out\n",
        "\n",
        "def hfenn(orig, res):\n",
        "    '''\n",
        "    High Frequency Error Norm (Normalized) metric for comparison of original and result images\n",
        "    The metric independent to image size (in contrast to regular HFEN)\n",
        "    Inputs are expected to be float in range [0, 1] (with possible overflow)\n",
        "    -> ori: original image\n",
        "    -> res: result image\n",
        "    <- HFENN value\n",
        "    '''\n",
        "    sgima = 1.5  # From DLMRI paper\n",
        "    return np.mean((l_o_g(orig - res, sgima)) ** 2) * 1e4  # magnification"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0BNf9e7Q1Y2O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "hfenn(high_res, low_res)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jEYu9t021Y2Q",
        "colab_type": "text"
      },
      "source": [
        "## Combining Loss Functions\n",
        "\n",
        "Let's try using a loss that doesn't just tell us the pixel-wise difference in resolution, but that also if there's an improvement, for example, in high frequency details.\n",
        "\n",
        "### MSE and HFENN\n",
        "\n",
        "We can do a weight sum of both losses like:\n",
        "\n",
        "$MSE + weight * HFENN$\n",
        "\n",
        "We could choose $weight = 10$ and see what happens. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B4qFCAo61Y2U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import scipy.ndimage as nd\n",
        "import scipy.ndimage.filters as filters\n",
        "from keras import losses\n",
        "import tensorflow as tf\n",
        "\n",
        "def hfenn_loss(ori, res):\n",
        "    '''\n",
        "    HFENN-based loss\n",
        "    ori, res - batched images with 3 channels\n",
        "    See metrics.hfenn\n",
        "    '''\n",
        "    fnorm = 0.325 # norm of l_o_g operator, estimated numerically\n",
        "    sigma = 1.5 # parameter from HFEN metric\n",
        "    truncate = 4 # default parameter from filters.gaussian_laplace\n",
        "    wradius = int(truncate * sigma + 0.5)\n",
        "    eye = np.zeros((2*wradius+1, 2*wradius+1), dtype=np.float32)\n",
        "    eye[wradius, wradius] = 1.\n",
        "    ker_mat = filters.gaussian_laplace(eye, sigma)\n",
        "    with tf.name_scope('hfenn_loss'):\n",
        "        chan = 3\n",
        "        ker = tf.constant(np.tile(ker_mat[:, :, None, None], (1, 1, chan, 1)))\n",
        "        filtered = tf.nn.depthwise_conv2d(ori - res, ker, [1, 1, 1, 1], 'VALID')\n",
        "        loss = tf.reduce_mean(tf.square(filtered))\n",
        "        loss = loss / (fnorm**2)\n",
        "    return loss\n",
        "  \n",
        "\n",
        "def ae_loss(input_img, decoder):\n",
        "    mse = losses.mean_squared_error(input_img, decoder) # MSE\n",
        "    weight = 10.0 # weight\n",
        "    return mse + weight * hfenn_loss(input_img, decoder) # MSE + weight * HFENN\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "moSB0zLW1Y2g",
        "colab_type": "text"
      },
      "source": [
        "Now, just compile the model with the new loss :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xp6XzvYw1Y2h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "autoencoder.compile(optimizer='adadelta', loss=ae_loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cr6c6AE71Y2l",
        "colab_type": "text"
      },
      "source": [
        "If you wanted to train the model, that's how you'd do :"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Q_MrEDV1Y2l",
        "colab_type": "text"
      },
      "source": [
        "```Python\n",
        "x_train_n = []\n",
        "x_train_down = []\n",
        "x_train_n, x_train_down = train_batches()\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "121A8cWL1Y2m",
        "colab_type": "text"
      },
      "source": [
        "But, we'll just load the pretrained weights :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ApyDoi-O1Y2q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## TODO\n",
        "## download the weights\n",
        "\n",
        "autoencoder_hfenn.load_weights(\"/data/sr.img_net.mse_hfenn.final_model5_2.no_patch.weights.best.hdf5\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_XqH_53V1Y2t",
        "colab_type": "text"
      },
      "source": [
        "Let's see what the network can do after using our new custom loss :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9NuD7zV1Y2u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sr_hfenn = np.clip(autoencoder_hfenn.predict(x_train_down), 0.0, 1.0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKHBNKzP1Y21",
        "colab_type": "text"
      },
      "source": [
        "Display the image results:\n",
        "\n",
        "* The low resolution input image\n",
        "* A bicubic interopolated version\n",
        "* The reconstructed image with MSE\n",
        "* The reconstructed image with our custom MSE + HFENN loss\n",
        "* The original perfect image\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jETO4hJs1Y26",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "image_index = 99"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5hSKatI71Y29",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(128, 128))\n",
        "i = 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_down[image_index])\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_down[image_index], interpolation=\"bicubic\")\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(sr1[image_index])\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(sr_hfenn[image_index])  # The reconstructed image with our custom MSE + HFENN loss\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_n[image_index])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8PDB8wuL1Y3W",
        "colab_type": "text"
      },
      "source": [
        "If you look a bit closely and check out the lines and edges, you'll se that they're sharper when using MSE and HFENN compared to MSE alone."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yE28_Swy1Y3X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.figure(figsize=(128, 128))\n",
        "j = 6\n",
        "i = 1\n",
        "idx_1 = 32*j\n",
        "idx_2 = 32*(j+1)\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_down[image_index, idx_1:idx_2, idx_1:idx_2])\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_down[image_index, idx_1:idx_2, idx_1:idx_2], interpolation=\"bicubic\")\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(sr1[image_index, idx_1:idx_2, idx_1:idx_2])\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(sr_hfenn[image_index, idx_1:idx_2, idx_1:idx_2])\n",
        "i += 1\n",
        "ax = plt.subplot(10, 10, i)\n",
        "plt.imshow(x_train_n[image_index, idx_1:idx_2, idx_1:idx_2])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQ5F8U6s1Y3Y",
        "colab_type": "text"
      },
      "source": [
        "**Note** : double click on the images to make them bigger"
      ]
    }
  ]
}