C1W1_(Colab)_Inputs_to_a_pre_trained_GAN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "C1W1_8: (Colab) Inputs to a pre-trained GAN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/https-deeplearning-ai/GANs-Public/blob/master/C1W1_(Colab)_Inputs_to_a_pre_trained_GAN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bFKPt3JI7A67"
      },
      "source": [
        "# Inputs to a Pre-trained GAN\n",
        "\n",
        "### Goal\n",
        "You are going to begin playing with generators and see how they fit into machine learning and generative adversarial networks (GANs). You will get to run a pre-trained GAN, mess with the inputs, and check out the wacky outputs.\n",
        "\n",
        "In the teaser, you saw some cool outputs. How does this all work? You will dive a little deeper in this notebook and see!\n",
        "\n",
        "### Learning Objectives\n",
        "1. Gain familiarity with how GANs work (discriminator + generator).\n",
        "2. Explore the impact of noise vectors and truncation.\n",
        "3. Adapt inputs to the generator to generate different faces.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ytICXgMY3CZ"
      },
      "source": [
        "## Discriminator vs. Generator\n",
        "The first thing that you will get to explore is the difference between a discriminator and a generator. Remember that these are the two main components of a GAN!\n",
        "\n",
        "#### What is a discriminator?\n",
        "\n",
        "One of the most widely used type of ML models is the classifier, which is used to sift through items in a dataset and classify them into different categories. For example, you might be familiar with image classifiers that discriminate between images of a cat and images of a dog. For a well-trained classifier: when you give it an image of a cat, it will say \"cat\"! When you give it an image of a dog, it will say \"dog\"! You can also use other classes, like coconuts vs. starfruit.\n",
        "\n",
        "<img src=\"https://lh3.googleusercontent.com/2blMv3QiuuIb-lTccsI0SUODvmuhwKdNpU9M6eXMCKbg27vgXIB7aSJDf3csyGr8yeFVxQWDIcAg7L0VVteO-4zkENqbboi0VyqF_g=w2880-l80-sg\" alt=\"discriminative model\" width=\"400\" align=\"center\"/>\n",
        "\n",
        "> *Image Credit: [Google](https://about.google/stories/machine-learning-qa/)*\n",
        "\n",
        "In terms of probabilities, the classifier wants to find $p(y|x)$: the probability that given an image input $x$, the image class $y$ is cat, $p(y=\\text{cat}|x)$, or dog, $p(y=\\text{dog}|x)$.\n",
        "\n",
        "The discriminator is simply a classifier with two classes: real and fake. Given an input x, the discriminator will calculate the probabilities $p(y=\\text{real}|x)$ and $p(y=\\text{fake}|x)$ and classify $x$. The input $x$ can be anything that you have the generator create and is not limited to images. Your GAN can be trained on videos, text, audio, etc.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yPemox5IZC9B"
      },
      "source": [
        "#### What is a generator?\n",
        "\n",
        "Generators are designed to have a different goal from discriminators (classifiers). Imagine you're working at a tropical fruit stand and asked to sort the fruit into two categories: coconuts and starfruit. That's the job of a classifier. But what if a customer comes up to the stand, and asks: what is a starfruit? You can't just say that it's not a coconut. You would need to explain what makes something a starfruit and what doesn't, not just its differences from a coconut. That's the job of a generator: to represent different classes in general, not just distinguish them. \n",
        "\n",
        "In terms of probabilities, the generator wants to figure out $p(x|y)$: the probability that, given that you generated a starfruit $(y=\\text{starfruit})$, the resulting image $(x)$ is the one generated. The output space of possible starfruit images is huge, so that makes this challenging for the generator.\n",
        "\n",
        "This can be a much harder task than discrimination. Typically, you will need the generator to take multiple steps to improve itself for every step the discriminator takes. It's easy to tell the difference between a coconut and a starfruit when you look at a mix of them, but to know exactly all the features of all possible coconuts in the world? That's a lot, but it's really cool if you can get even close to it, because you can start generating all sorts of coconuts and starfruit when you do. I don't have a generator for these tropical fruits figured out for you to play with, but you can generate all kinds of cool things with these models, like realistic faces!\n",
        "\n",
        "In the below image, the generator is trying to find the features that represent all cats using the feedback from the discriminator.\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/tutorials/generative/images/gan2.png\" width=\"500\" display=\"block\" align=\"center\"/>\n",
        "\n",
        ">*Image Credit: TensorFlow*\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHKz2j3wZAUE"
      },
      "source": [
        "## Noise vector $z$\n",
        "\n",
        "The noise vector $z$ has the important role of making sure the images generated from the same class $y$ don't all look the same—think of it as a random seed. You generate it randomly, usually by sampling random numbers either between 0 and 1 uniformly, or from the normal distribution, which you can denote $z$ ~ $N(0, 1)$. The zero means the normal distribution has a mean of zero, and the 1 means that the normal distribution has a variance of 1. \n",
        "\n",
        "In reality, $z$ is usually larger than just 1 value to allow for more combinations of what $z$ could be. There's no special number that determines what works, but 100 is standard. Some researchers might use a power of 2, like 128 or 512, but again, nothing special about the number itself, just that it's large enough to contain a lot of possibilities. As a result, you would sample $z$ from that many different dimensions (constituting multiple normal distributions).\n",
        "\n",
        "*Fun Fact: this is also called a spherical normal and denoted $z$ ~ $N(0, I)$ where the $I$ represents the identity matrix and means the variance is 1 in all dimensions.*\n",
        "\n",
        "## Truncation trick\n",
        "\n",
        "So now that you're a bit familiar with noise vectors, here's another cool concept that people use to tune their outputs. It's called the truncation trick. I like to think of the truncation trick as a way of trading off fidelity (quality) and diversity in the samples. It works like this: when you randomly sample your noise vector $z$, you can choose to keep that random $z$ or you can sample another one. \n",
        "\n",
        "Why would you want to sample another one? \n",
        "\n",
        "Well, since I'm sampling $z$ from a normal distribution, my model will see more of those $z$ values within a standard deviation from the mean than those at the tails of the distribution—and this happens during training. This means that while the model is training, it's likely to be familiar with certain noise vectors and as a result model those areas coming from familiar noise vector regions. In these areas, my model will likely have much more realistic results, but nothing too funky, it's not taking as many risks in those regions mapped from those familiar noise vectors. This is the trade-off between fidelity (realistic, high quality images) and diversity (variety in images). \n",
        "\n",
        "<img src=\"https://build.openmodelica.org/Documentation/Modelica%203.2.3/Resources/Images/Math/Distributions/TruncatedNormal.density.png\" alt=\"truncated normal distribution\" width=\"400\"/>\n",
        "\n",
        "> *Image Credit: Modelica*\n",
        "\n",
        "\n",
        "What the truncation trick does is resamples the noise vector $z$ until it falls within some bounds of the normal distribution. In fact, it samples $z$ from a truncated normal distribution where the tails are cut off at different values (red line in graph is truncated normal, blue is original). You can tune these values and thus tune fidelity/diversity. Recall that having a lot of fidelity is not always the goal—one failure mode of that is that you get one really real image but nothing else (no diversity), and that's not very interesting or successful from a model that's supposed to model the realm of all possible human faces or that of all possible coconuts—including that of a cat pouncing after a flying coconut (but with extremely low probability)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FpCJE5hs5XvW"
      },
      "source": [
        "## Playing with code\n",
        "You're going to play with the unconditional generator from StyleGAN to generate some faces. Notice how sometimes they look super realistic, and other times, well, hope you're not too scared of the uncanny valley. As you play more with these, you might grow to love them all the same, especially when you contrast it to where they began: pure random noise, as in completely random pixel values that look like static, learning over time to become face-like.\n",
        "\n",
        "Start by running the following code block to load the model and import some libraries you will be using! (Don't worry about any errors regarding tensorflow 2.2.0 or warnings you may see)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ps2IngTZ8i4d"
      },
      "source": [
        "# Clone the official StyleGAN repository from GitHub\n",
        "!git clone https://github.com/NVlabs/stylegan.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UscKHoh16zo8"
      },
      "source": [
        "%tensorflow_version 1.x\n",
        "import os\n",
        "import pickle\n",
        "import numpy as np\n",
        "import PIL.Image\n",
        "import stylegan\n",
        "from stylegan import config\n",
        "from stylegan.dnnlib import tflib\n",
        "from tensorflow.python.util import module_wrapper\n",
        "module_wrapper._PER_MODULE_WARNING_LIMIT = 0\n",
        "\n",
        "# Initialize TensorFlow\n",
        "tflib.init_tf()\n",
        "\n",
        "# Go into that cloned directory\n",
        "path = 'stylegan/'\n",
        "if \"stylegan\" not in os.getcwd():\n",
        "    os.chdir(path)\n",
        "\n",
        "# Load pre-trained network\n",
        "# url = 'https://drive.google.com/uc?id=1MEGjdvVpUsu1jB4zrXZN7Y4kBBOzizDQ' # Downloads the pickled model file: karras2019stylegan-ffhq-1024x1024.pkl\n",
        "url = 'https://bitbucket.org/ezelikman/gans/downloads/karras2019stylegan-ffhq-1024x1024.pkl'\n",
        "with stylegan.dnnlib.util.open_url(url, cache_dir=config.cache_dir) as f:\n",
        "  print(f)\n",
        "  _G, _D, Gs = pickle.load(f)\n",
        "#   Gs.print_layers()  # Print network details"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VS0bDt8piPh9"
      },
      "source": [
        "Now you can use this separate script to play with the model without needing to reload it everytime (that would take a while).\n",
        "\n",
        "Feel free to mess with any of the code but here are some values you can try modifying:\n",
        "  *   `rnd`: The random seed that generates the noise vectors. Changing this will change the noise vectors fed to the generator; see what happens next!\n",
        "  *   `batch_size`: This is the number of images that you want generated.\n",
        "  *   `truncation`: The positive truncation value. 1 is low truncation (high diversity) and 0 is all truncation except for the mean (high quality/fidelity). A lower value increases fidelity and decreases diversity, and vice versa. These are trade-offs that you can play with.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W23SJezZ7l6p"
      },
      "source": [
        "# Set the random seed that generates the noise vectors\n",
        "rnd = np.random.RandomState(42)\n",
        "\n",
        "# Set the number of images to generate\n",
        "batch_size = 4\n",
        "\n",
        "# Set the truncation value for truncation trick sampling\n",
        "truncation = 0.7\n",
        "\n",
        "# Create a noise vector z for each sample in the batch: (batch_size, z_dim)\n",
        "z_dim = Gs.input_shape[1] # StyleGAN authors use the image dim (512) as the size of z\n",
        "print(f'Noise vector has size {z_dim}')\n",
        "noise_vectors = rnd.randn(batch_size, z_dim)\n",
        "\n",
        "# Generate image by running (sampling) the generator\n",
        "fmt = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True) # Specify the desired output format and shape\n",
        "images = Gs.run(noise_vectors,\n",
        "                None,    # No labels/conditions because it is unconditional generation (more on this in the coming lectures)\n",
        "                truncation_psi=truncation, \n",
        "                randomize_noise=False,\n",
        "                output_transform=fmt\n",
        "                )\n",
        "\n",
        "# Display images\n",
        "if batch_size > 1:\n",
        "  img = np.concatenate(images, axis=1) # Save all images in batch to a single image\n",
        "else:\n",
        "  img = images[0]\n",
        "PIL.Image.fromarray(img, 'RGB')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UssV5c31idTP"
      },
      "source": [
        "Here's another script to help familiarize you with what interpolation in GANs can look like—filling in the unknown between two objects. In this case, you are using face images where the GAN will fill in how one image can morph into another. You will generate random noise vectors for the start and end images and then create noise vectors for the the ones inbetween.\n",
        "\n",
        "Again, feel free to mess with any of the code but here are some values you should try modifying:\n",
        "  *   `rnd`: The random seed that generates the noise vectors. Changing this will change the noise vectors fed to the generator and, consequently, the images generated!\n",
        "  *   `truncation`: The truncation value between 0 and 1. 1 is no truncation (high diversity) and 0 is all truncation except for the mean (high quality/fidelity). A lower value increases fidelity and decreases diversity, and vice versa. These are trade-offs that you can play with.\n",
        "  *   `n_interpolations`: The number of images that you want generated. A 3 would mean there is one transition photo between the start and end photos. If you want to see an interpolation, this value should be greater than 2.\n",
        "\n",
        "*Fun Fact: interpolations do not have to be not nearly as smooth as the ones produced by StyleGAN. In fact, StyleGAN has a few tricks specifically to make it learn smooth interpolations!*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4YOTrHMVfkoI"
      },
      "source": [
        "# Set the random seed that generates the noise vectors\n",
        "rnd = np.random.RandomState(4)\n",
        "\n",
        "# Set the truncation value for truncation trick sampling\n",
        "truncation = 0.7\n",
        "\n",
        "# Set the number of interpolations/number of images to generate\n",
        "n_interpolation = 10\n",
        "\n",
        "# Create a noise vector z for the start and end images (batch_size = 1 since they are single image): (batch_size, z_dim)\n",
        "# And create noise for the interpolations inbetween\n",
        "z_dim = Gs.input_shape[1]\n",
        "first_noise = rnd.randn(1, z_dim)\n",
        "second_noise = rnd.randn(1, z_dim)\n",
        "percent_first_noise = np.linspace(0, 1, n_interpolation)[:, None]\n",
        "interpolation_noise = first_noise * percent_first_noise + second_noise * (1 - percent_first_noise)\n",
        "\n",
        "# Generate image by running (sampling) the generator\n",
        "fmt = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True) # Specify the desired output format and shape\n",
        "images = Gs.run(interpolation_noise,\n",
        "                None,    # No labels/conditions because it is unconditional generation!\n",
        "                truncation_psi=truncation, \n",
        "                randomize_noise=False,\n",
        "                output_transform=fmt\n",
        "                )\n",
        "\n",
        "# Display images\n",
        "if batch_size > 1:\n",
        "  img = np.concatenate(images, axis=1) # Save all images in batch to a single image\n",
        "else:\n",
        "  img = images[0]\n",
        "PIL.Image.fromarray(img, 'RGB')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}