Docx/CS224N/Tutorial/CS224N_PyTorch_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "colab": {
      "name": "CS224N PyTorch Tutorial.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6oqGiIXvrMl"
      },
      "source": [
        "# CS224N: PyTorch Tutorial (Winter '21)\n",
        "\n",
        "### Author: Dilara Soylu\n",
        "\n",
        "In this notebook, we will have a basic introduction to `PyTorch` and work on a toy NLP task. Following resources have been used in preparation of this notebook:\n",
        "* [\"Word Window Classification\" tutorial notebook]((https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1204/materials/ww_classifier.ipynb) by Matt Lamm, from Winter 2020 offering of CS224N\n",
        "* Official PyTorch Documentation on [Deep Learning with PyTorch: A 60 Minute Blitz](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) by Soumith Chintala\n",
        "* PyTorch Tutorial Notebook, [Build Basic Generative Adversarial Networks (GANs) | Coursera](https://www.coursera.org/learn/build-basic-generative-adversarial-networks-gans) by Sharon Zhou, offered on Coursera\n",
        "\n",
        "Many thanks to Angelica Sun and John Hewitt for their feedback."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4gk1UKaNvrMv"
      },
      "source": [
        "## Introduction\n",
        "[PyTorch](https://pytorch.org/) is a machine learning framework that is used in both academia and industry for various applications. PyTorch started of as a more flexible alternative to [TensorFlow](https://www.tensorflow.org/), which is another popular machine learning framework. At the time of its release, `PyTorch` appealed to the users due to its user friendly nature: as opposed to defining static graphs before performing an operation as in `TensorFlow`, `PyTorch` allowed users to define their operations as they go, which is also the approached integrated by `TensorFlow` in its following releases. Although `TensorFlow` is more widely preferred in the industry, `PyTorch` is often times the preferred machine learning framework for researchers. If you would like to learn more about the differences between the two, you can check out [this](https://blog.udacity.com/2020/05/pytorch-vs-tensorflow-what-you-need-to-know.html) blog post. \n",
        "\n",
        "Now that we have learned enough about the background of `PyTorch`, let's start by importing it into our notebook. To install `PyTorch`, you can follow the instructions here. Alternatively, you can open this notebook using `Google Colab`, which already has `PyTorch` installed in its base kernel. Once you are done with the installation process, run the following cell:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u0ukr7quvrMx"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "# Import pprint, module we use for making our print statements prettier\n",
        "import pprint\n",
        "pp = pprint.PrettyPrinter()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k10ZRdcBwDP3"
      },
      "source": [
        "We are all set to start our tutorial. Let's dive in!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OLdSN9ZXvrM0"
      },
      "source": [
        "## Tensors\n",
        "\n",
        "Tensors are the most basic building blocks in `PyTorch`.  Tensors are similar to matrices, but the have extra properties and they can represent higher dimensions. For example, an square image with 256 pixels in both sides can be represented by a `3x256x256` tensor, where the first 3 dimensions represent the color channels, red, green and blue. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c6aWvTEy7lgG"
      },
      "source": [
        "### Tensor Initialization\n",
        "There are several ways to instantiate tensors in `PyTorch`, which we will go through next. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c78_3AMEyvJd"
      },
      "source": [
        "#### **From a Python List**\n",
        "\n",
        "We can initalize a tensor from a `Python` list, which could include sublists. The dimensions and the data types will be automatically inferred by `PyTorch` when we use [`torch.tensor()`](https://pytorch.org/docs/stable/generated/torch.tensor.html). \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lsjIW9I_ztiO",
        "outputId": "b78c8e03-211d-499b-9089-262f7ac04b19"
      },
      "source": [
        "# Initialize a tensor from a Python List\n",
        "data = [\n",
        "        [0, 1], \n",
        "        [2, 3],\n",
        "        [4, 5]\n",
        "       ]\n",
        "x_python = torch.tensor(data)\n",
        "\n",
        "# Print the tensor\n",
        "x_python"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0, 1],\n",
              "        [2, 3],\n",
              "        [4, 5]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Iv6ZEoZ0RWb5"
      },
      "source": [
        "We can also call `torch.tensor()` with the optional `dtype` parameter, which will set the data type. Some useful datatypes to be familiar with are: `torch.bool`, `torch.float`, and `torch.long`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0bQF5IhsD7-n",
        "outputId": "a4729524-2b06-4545-cf1f-f83a5a7574b5"
      },
      "source": [
        "# We are using the dtype to create a tensor of particular type\n",
        "x_float = torch.tensor(data, dtype=torch.float)\n",
        "x_float"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 1.],\n",
              "        [2., 3.],\n",
              "        [4., 5.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "16VoILaaE-_j",
        "outputId": "8d236b7a-b60e-482c-c28b-df25925d7819"
      },
      "source": [
        "# We are using the dtype to create a tensor of particular type\n",
        "x_bool = torch.tensor(data, dtype=torch.bool)\n",
        "x_bool"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[False,  True],\n",
              "        [ True,  True],\n",
              "        [ True,  True]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j4HccPWFEQUB"
      },
      "source": [
        "We can also get the same tensor in our specified data type using methods such as `float()`, `long()` etc. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nh_yq0SuTS_W",
        "outputId": "f01ac372-5cb4-4be7-ae34-7c0a37b1b444"
      },
      "source": [
        "x_python.float()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 1.],\n",
              "        [2., 3.],\n",
              "        [4., 5.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kFiS1OFdTlKE"
      },
      "source": [
        "We can also use `tensor.FloatTensor`, `tensor.LongTensor`, `tensor.Tensor` classes to instantiate a tensor of particular type. `LongTensor`s are particularly important in NLP as many methods that deal with indices require the indices to be passed as a `LongTensor`, which is a 64 bit integer. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hXXWZ1H2TkNN",
        "outputId": "66564be9-f6df-4fab-c248-724635a2e272"
      },
      "source": [
        "# `torch.Tensor` defaults to float\n",
        "# Same as torch.FloatTensor(data)\n",
        "x = torch.Tensor(data) \n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 1.],\n",
              "        [2., 3.],\n",
              "        [4., 5.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QuLzDzsoytM2"
      },
      "source": [
        "#### **From a NumPy Array**\n",
        "We can also initialize a tensor from a `NumPy` array. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YtSNe8X-2Pox",
        "outputId": "e0e2222c-27db-485f-a005-9ee3f6463e72"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Initialize a tensor from a NumPy array\n",
        "ndarray = np.array(data)\n",
        "x_numpy = torch.from_numpy(ndarray)\n",
        "\n",
        "# Print the tensor\n",
        "x_numpy"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0, 1],\n",
              "        [2, 3],\n",
              "        [4, 5]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dhtcBgum3OZ3"
      },
      "source": [
        "#### **From a Tensor**\n",
        "We can also initialize a tensor from another tensor, using the following methods:\n",
        "\n",
        "* `torch.ones_like(old_tensor)`: Initializes a tensor of `1s`.\n",
        "* `torch.zeros_like(old_tensor)`: Initializes a tensor of `0s`.\n",
        "* `torch.rand_like(old_tensor)`: Initializes a tensor where all the elements are sampled from a uniform distribution between `0` and `1`.\n",
        "* `torch.randn_like(old_tensor)`: Initializes a tensor where all the elements are sampled from a normal distribution.\n",
        "\n",
        "All of these methods preserve the tensor properties of the original tensor passed in, such as the `shape` and `device`, which we will cover in a bit. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zoKVhcLh2yqe",
        "outputId": "8ed9acbe-947b-4890-db81-c83d802c0f86"
      },
      "source": [
        "# Initialize a base tensor\n",
        "x = torch.tensor([[1., 2], [3, 4]])\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [3., 4.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FncfGN6z7ELA",
        "outputId": "71db6165-ea7a-4068-da13-c0310790c4b9"
      },
      "source": [
        "# Initialize a tensor of 0s\n",
        "x_zeros = torch.zeros_like(x)\n",
        "x_zeros"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0.],\n",
              "        [0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D993dpnP6iA8",
        "outputId": "ca8a7fd3-e7d0-47f7-a7c0-1bddc5614223"
      },
      "source": [
        "# Initialize a tensor of 1s\n",
        "x_ones = torch.ones_like(x)\n",
        "x_ones"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 1.],\n",
              "        [1., 1.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VBUDeEm97IqW",
        "outputId": "455a2ce8-caf6-4252-c1be-6207e81d2649"
      },
      "source": [
        "# Initialize a tensor where each element is sampled from a uniform distribution\n",
        "# between 0 and 1\n",
        "x_rand = torch.rand_like(x)\n",
        "x_rand"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.8979, 0.7173],\n",
              "        [0.3067, 0.1246]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WYsE3lKt7IEX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2c49ce88-937e-4afe-eb1c-37a625cbc77f"
      },
      "source": [
        "# Initialize a tensor where each element is sampled from a normal distribution\n",
        "x_randn = torch.randn_like(x)\n",
        "x_randn"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-0.6749, -0.8590],\n",
              "        [ 0.6666,  1.1185]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6tqf7v38vbi"
      },
      "source": [
        "#### **By Specifying a Shape**\n",
        "We can also instantiate tensors by specifying their shapes (which we will cover in more detail in a bit). The methods we could use follow the ones in the previous section:\n",
        "* `torch.zeros()`\n",
        "* `torch.ones()`\n",
        "* `torch.rand()`\n",
        "* `torch.randn()`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dh4I4Npz-dZ4",
        "outputId": "e45a1095-0d18-4266-ceb7-a7c9de1f18e2"
      },
      "source": [
        "# Initialize a 2x3x2 tensor of 0s\n",
        "shape = (4, 2, 2)\n",
        "x_zeros = torch.zeros(shape) # x_zeros = torch.zeros(4, 3, 2) is an alternative\n",
        "x_zeros"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0., 0.],\n",
              "         [0., 0.]],\n",
              "\n",
              "        [[0., 0.],\n",
              "         [0., 0.]],\n",
              "\n",
              "        [[0., 0.],\n",
              "         [0., 0.]],\n",
              "\n",
              "        [[0., 0.],\n",
              "         [0., 0.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LEjeR24MLkN"
      },
      "source": [
        "#### **With `torch.arange()`**\n",
        "We can also create a tensor with `torch.arange(end)`, which returns a `1-D` tensor with elements ranging from `0` to `end-1`. We can use the optional `start` and `step` parameters to create tensors with different ranges.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1EjARl2aM7pA",
        "outputId": "264a9ffa-5d05-4b9d-9901-cd49cbb93264"
      },
      "source": [
        "# Create a tensor with values 0-9\n",
        "x = torch.arange(10)\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VgpkRn527zSr"
      },
      "source": [
        "### Tensor Properties\n",
        "\n",
        "Tensors have a few properties that are important for us to cover. These are namely `shape`, and the `device` properties. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JBt6e4xZT3zr"
      },
      "source": [
        "#### Data Type\n",
        "\n",
        "The `dtype` property lets us see the data type of a tensor. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rlF3k3eUT_hQ",
        "outputId": "c2416faf-77cc-46e9-a22e-a67fe14bc13d"
      },
      "source": [
        "# Initialize a 3x2 tensor, with 3 rows and 2 columns\n",
        "x = torch.ones(3, 2)\n",
        "x.dtype"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.float32"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y1vtN4Dy8FAG"
      },
      "source": [
        "#### Shape\n",
        "\n",
        "The `shape` property tells us the shape of our tensor. This can help us identify how many dimensional our tensor is as well as how many elements exist in each dimension."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "24gXLJcn7Pxs",
        "outputId": "ef11b998-9fc4-458e-8452-6d545da19694"
      },
      "source": [
        "# Initialize a 3x2 tensor, with 3 rows and 2 columns\n",
        "x = torch.Tensor([[1, 2], [3, 4], [5, 6]])\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [3., 4.],\n",
              "        [5., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vV0cE1cXAEHP",
        "outputId": "1d12cf8d-c659-4127-807e-91459ba12d3a"
      },
      "source": [
        "# Print out its shape\n",
        "# Same as x.size()\n",
        "x.shape "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([3, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MA3vnJnaAQlc",
        "outputId": "cfa251e1-be1c-4dbc-9803-bc213ca05a2a"
      },
      "source": [
        "# Print out the number of elements in a particular dimension\n",
        "# 0th dimension corresponds to the rows\n",
        "x.shape[0] "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uxXCX6y6BvhH"
      },
      "source": [
        "We can also get the size of a particular dimension with the `size()` method.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZapQmydxBVuy",
        "outputId": "24547ccf-817c-44ef-bc6a-03a0bf246a58"
      },
      "source": [
        "# Get the size of the 0th dimension\n",
        "x.size(0)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCQm7ToPOveH"
      },
      "source": [
        "We can change the shape of a tensor with the `view()` method. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J1JH3fiNO5Gu",
        "outputId": "0c270810-811b-466f-a5d2-70f240942aa3"
      },
      "source": [
        "# Example use of view()\n",
        "# x_view shares the same memory as x, so changing one changes the other\n",
        "x_view = x.view(2, 3)\n",
        "x_view"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2., 3.],\n",
              "        [4., 5., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3C3x4seqPGEI",
        "outputId": "66e48d01-5702-414c-c628-055adbdcdeef"
      },
      "source": [
        "# We can ask PyTorch to infer the size of a dimension with -1\n",
        "x_view = x.view(3, -1)\n",
        "x_view"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [3., 4.],\n",
              "        [5., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TYSCEesPITpf"
      },
      "source": [
        "We can also use `torch.reshape()` method for a similar purpose. There is a subtle difference between `reshape()` and `view()`: `view()` requires the data to be stored contiguously in the memory. You can refer to [this](https://stackoverflow.com/questions/49643225/whats-the-difference-between-reshape-and-view-in-pytorch) StackOverflow answer for more information. In simple terms, contiguous means that the way our data is laid out in the memory is the same as the way we would read elements from it. This happens because some methods, such as `transpose()` and `view()`, do not actually change how our data is stored in the memory. They just change the meta information about out tensor, so that when we use it we will see the elements in the order we expect. \n",
        "\n",
        "`reshape()` calls `view()` internally if the data is stored contiguously, if not, it returns a copy. The difference here isn't too important for basic tensors, but if you perform operations that make the underlying storage of the data non-contiguous (such as taking a transpose), you will have issues using `view()`. If you would like to match the way your tensor is stored in the memory to how it is used, you can use the `contiguous()` method.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gLGcGYE4Llom",
        "outputId": "e810557b-552f-4b0f-a37a-1924bec7fcfa"
      },
      "source": [
        "# Change the shape of x to be 3x2\n",
        "# x_reshaped could be a reference to or copy of x\n",
        "x_reshaped = torch.reshape(x, (2, 3))\n",
        "x_reshaped"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2., 3.],\n",
              "        [4., 5., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWNTZKZZQ9i6"
      },
      "source": [
        "We can use `torch.unsqueeze(x, dim)` function to add a dimension of size `1` to the provided `dim`, where `x` is the tensor. We can also use the corresponding use `torch.squeeze(x)`, which removes the dimensions of size `1`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3_IYojrJRh-m",
        "outputId": "ed315529-ea33-4cbe-9878-4581f3afbcdb"
      },
      "source": [
        "# Initialize a 5x2 tensor, with 5 rows and 2 columns\n",
        "x = torch.arange(10).reshape(5, 2)\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0, 1],\n",
              "        [2, 3],\n",
              "        [4, 5],\n",
              "        [6, 7],\n",
              "        [8, 9]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pLhg_oZ4SHh-",
        "outputId": "a32cf8a3-c1f4-4220-d948-b42fd12f8772"
      },
      "source": [
        "# Add a new dimension of size 1 at the 1st dimension\n",
        "x = x.unsqueeze(1)\n",
        "x.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([5, 1, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YoGYGbMRSo-J",
        "outputId": "9c8415c4-ebf3-40ab-9faa-b13c5fd42c62"
      },
      "source": [
        "# Squeeze the dimensions of x by getting rid of all the dimensions with 1 element\n",
        "x = x.squeeze()\n",
        "x.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([5, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQpZ4556B3lb"
      },
      "source": [
        "If we want to get the total number of elements in a tensor, we can use the `numel()` method. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P-irUWlxTB6a",
        "outputId": "c070c19a-cf0c-46cb-ec54-8254a9c83974"
      },
      "source": [
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0, 1],\n",
              "        [2, 3],\n",
              "        [4, 5],\n",
              "        [6, 7],\n",
              "        [8, 9]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "76yVMMg_CA0Q",
        "outputId": "7edfbc68-b830-459c-a5bf-fe3803a902af"
      },
      "source": [
        "# Get the number of elements in tensor.\n",
        "x.numel()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_M1U_RTpBhl2"
      },
      "source": [
        "#### **Device**\n",
        "Device property tells `PyTorch` where to store our tensor. Where a tensor is stored determines which device, `GPU` or `CPU`, would be handling the computations involving it. We can find the device of a tensor with the `device` property."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XYRGhIbnCl3b",
        "outputId": "4eac92c3-48f5-4e4e-cdf1-329bd79538b5"
      },
      "source": [
        "# Initialize an example tensor\n",
        "x = torch.Tensor([[1, 2], [3, 4]])\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [3., 4.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "byEnJyKdBgjl",
        "outputId": "04440390-a16d-4250-920e-64dade75622a"
      },
      "source": [
        "# Get the device of the tensor\n",
        "x.device"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cpu')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-vqf_-NADFX8"
      },
      "source": [
        "We can move a tensor from one device to another with the method `to(device)`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GzA6zqkXDEt1"
      },
      "source": [
        "# Check if a GPU is available, if so, move the tensor to the GPU\n",
        "if torch.cuda.is_available():\n",
        "  x.to('cuda') "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V7BMktFFAkRA"
      },
      "source": [
        "### Tensor Indexing\n",
        "In `PyTorch` we can index tensors, similar to `NumPy`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fRJN7ovWDsKV",
        "outputId": "f4bea081-d4c4-49a2-e529-fa502cf135f7"
      },
      "source": [
        "# Initialize an example tensor\n",
        "x = torch.Tensor([\n",
        "                  [[1, 2], [3, 4]],\n",
        "                  [[5, 6], [7, 8]], \n",
        "                  [[9, 10], [11, 12]] \n",
        "                 ])\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 1.,  2.],\n",
              "         [ 3.,  4.]],\n",
              "\n",
              "        [[ 5.,  6.],\n",
              "         [ 7.,  8.]],\n",
              "\n",
              "        [[ 9., 10.],\n",
              "         [11., 12.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M67ZiOF1Heyc",
        "outputId": "0821f0cc-5a93-46e6-bee4-9fd648723a27"
      },
      "source": [
        "x.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([3, 2, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "guXKE7m8AX1K",
        "outputId": "cf492c27-a67d-4fbe-b57a-54fb4f8802a4"
      },
      "source": [
        "# Access the 0th element, which is the first row\n",
        "x[0] # Equivalent to x[0, :]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [3., 4.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8m8EyVvES4-"
      },
      "source": [
        "We can also index into multiple dimensions with `:`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2Z6GFUcuEL85",
        "outputId": "f36a740c-df12-42f6-a64b-d4b3038168f4"
      },
      "source": [
        "# Get the top left element of each element in our tensor\n",
        "x[:, 0, 0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1., 5., 9.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rm8vc3nuXaEw"
      },
      "source": [
        "We can also access arbitrary elements in each dimension. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eYhcH9gaWHyW",
        "outputId": "f900ea02-4f55-4af6-c485-c5dec4baf196"
      },
      "source": [
        "# Print x again to see our tensor\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[ 1.,  2.],\n",
              "         [ 3.,  4.]],\n",
              "\n",
              "        [[ 5.,  6.],\n",
              "         [ 7.,  8.]],\n",
              "\n",
              "        [[ 9., 10.],\n",
              "         [11., 12.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R4xl6CW3RrEw",
        "outputId": "1cb93869-7764-4d22-b411-dcaadb906830"
      },
      "source": [
        "# Let's access the 0th and 1st elements, each twice\n",
        "i = torch.tensor([0, 0, 1, 1])\n",
        "x[i]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[1., 2.],\n",
              "         [3., 4.]],\n",
              "\n",
              "        [[1., 2.],\n",
              "         [3., 4.]],\n",
              "\n",
              "        [[5., 6.],\n",
              "         [7., 8.]],\n",
              "\n",
              "        [[5., 6.],\n",
              "         [7., 8.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A3QYZ8k7Wvqp",
        "outputId": "6cb46909-2800-45a0-abb3-0707c2c50d6c"
      },
      "source": [
        "# Let's access the 0th elements of the 1st and 2nd elements\n",
        "i = torch.tensor([1, 2])\n",
        "j = torch.tensor([0])\n",
        "x[i, j]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 5.,  6.],\n",
              "        [ 9., 10.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WAELXC--IHS7"
      },
      "source": [
        "We can get a `Python` scalar value from a tensor with `item()`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BM-ZujN2IGaQ",
        "outputId": "785f43c3-5670-4fa2-ed51-e72e9c0474ca"
      },
      "source": [
        "x[0, 0, 0]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(1.)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6NwxK7d_Ycgs",
        "outputId": "343abf96-2d77-49b0-9076-c0f617e7aafe"
      },
      "source": [
        "x[0, 0, 0].item()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GltnmDzeIXJM"
      },
      "source": [
        "### Operations\n",
        "PyTorch operations are very similar to those of `NumPy`. We can work with both scalars and other tensors. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W9KBzcA0G6v9",
        "outputId": "057decb7-7a56-4fcc-bc20-781cba8d0caf"
      },
      "source": [
        "# Create an example tensor\n",
        "x = torch.ones((3,2,2))\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[1., 1.],\n",
              "         [1., 1.]],\n",
              "\n",
              "        [[1., 1.],\n",
              "         [1., 1.]],\n",
              "\n",
              "        [[1., 1.],\n",
              "         [1., 1.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AUw8MAHqKuzs",
        "outputId": "fd10173a-caf5-4e45-ceca-c0b32f4bf39e"
      },
      "source": [
        "# Perform elementwise addition\n",
        "# Use - for subtraction\n",
        "x + 2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[3., 3.],\n",
              "         [3., 3.]],\n",
              "\n",
              "        [[3., 3.],\n",
              "         [3., 3.]],\n",
              "\n",
              "        [[3., 3.],\n",
              "         [3., 3.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QAfMsaz1Gw5v",
        "outputId": "d0432152-8137-46e7-902a-ec0b779d5471"
      },
      "source": [
        "# Perform elementwise multiplication\n",
        "# Use / for division\n",
        "x * 2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[2., 2.],\n",
              "         [2., 2.]],\n",
              "\n",
              "        [[2., 2.],\n",
              "         [2., 2.]],\n",
              "\n",
              "        [[2., 2.],\n",
              "         [2., 2.]]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0aq89FU7OOe7"
      },
      "source": [
        "We can apply the same operations between different tensors of compatible sizes.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BGhz62wILIfN",
        "outputId": "40f51f08-a80b-41b9-b9a1-e874c8b89e05"
      },
      "source": [
        "# Create a 4x3 tensor of 6s\n",
        "a = torch.ones((4,3)) * 6\n",
        "a"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tvDC1OzzPyLV",
        "outputId": "291f0c1f-9a64-4c23-a399-7464c6cbd39e"
      },
      "source": [
        "# Create a 1D tensor of 2s\n",
        "b = torch.ones(3) * 2\n",
        "b"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([2., 2., 2.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 126
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HUF9noMTP5NI",
        "outputId": "9cd9b5ab-c292-4987-c562-a3f8e4ec8a2f"
      },
      "source": [
        "# Divide a by b\n",
        "a / b"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[3., 3., 3.],\n",
              "        [3., 3., 3.],\n",
              "        [3., 3., 3.],\n",
              "        [3., 3., 3.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YTiVVbukXRct"
      },
      "source": [
        "We can use `tensor.matmul(other_tensor)` for matrix multiplication and `tensor.T` for transpose. Matrix multiplication can also be performed with `@`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bPoC_WcbXCw5",
        "outputId": "332f95f5-11fd-431d-c77a-e4c98dbd6660"
      },
      "source": [
        "# Alternative to a.matmul(b)\n",
        "# a @ b.T returns the same result since b is 1D tensor and the 2nd dimension\n",
        "# is inferred\n",
        "a @ b"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([36., 36., 36., 36.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e2WEoQjVYBQ8",
        "outputId": "0f97c447-5f28-4483-ac82-e1b484663b96"
      },
      "source": [
        "pp.pprint(a.shape)\n",
        "pp.pprint(a.T.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([4, 3])\n",
            "torch.Size([3, 4])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PibNpxbYYf2"
      },
      "source": [
        "We can take the mean and standard deviation along a certain dimension with the methods `mean(dim)` and `std(dim)`. That is, if we want to get the mean `3x2` matrix in a `4x3x2` matrix, we would set the `dim` to be 0. We can call these methods with no parameter to get the mean and standard deviation for the whole tensor. To use `mean` and `std` our tensor should be a floating point type. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a987teCtYg7R",
        "outputId": "0ca3799b-6dec-49ea-c1eb-d6265f390818"
      },
      "source": [
        "# Create an example tensor\n",
        "m = torch.tensor(\n",
        "    [\n",
        "     [1., 1.],\n",
        "     [2., 2.],\n",
        "     [3., 3.],\n",
        "     [4., 4.]\n",
        "    ]\n",
        ")\n",
        "\n",
        "pp.pprint(\"Mean: {}\".format(m.mean()))\n",
        "pp.pprint(\"Mean in the 0th dimension: {}\".format(m.mean(0)))\n",
        "pp.pprint(\"Mean in the 1st dimension: {}\".format(m.mean(1)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "'Mean: 2.5'\n",
            "'Mean in the 0th dimension: tensor([2.5000, 2.5000])'\n",
            "'Mean in the 1st dimension: tensor([1., 2., 3., 4.])'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rd77stQ5VQVT"
      },
      "source": [
        "We can concatenate tensors using `torch.cat`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "advfDCOPK9Gw",
        "outputId": "8efcab75-98f0-4562-a58f-fa419604956b"
      },
      "source": [
        "# Concatenate in dimension 0 and 1\n",
        "a_cat0 = torch.cat([a, a, a], dim=0)\n",
        "a_cat1 = torch.cat([a, a, a], dim=1)\n",
        "\n",
        "print(\"Initial shape: {}\".format(a.shape))\n",
        "print(\"Shape after concatenation in dimension 0: {}\".format(a_cat0.shape))\n",
        "print(\"Shape after concatenation in dimension 1: {}\".format(a_cat1.shape))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initial shape: torch.Size([4, 3])\n",
            "Shape after concatenation in dimension 0: torch.Size([12, 3])\n",
            "Shape after concatenation in dimension 1: torch.Size([4, 9])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BveswZMOjtff"
      },
      "source": [
        "Most of the operations in `PyTorch` are not in place. However, `PyTorch` offers the in place versions of operations available by adding an underscore (`_`) at the end of the method name. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ebr7nn-DaU3B",
        "outputId": "aad366a1-1bbb-4864-8be0-46ef440763b2"
      },
      "source": [
        "# Print our tensor\n",
        "a"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mP8-VtoHaKAc",
        "outputId": "94fafdfd-da16-4a94-bdd3-f9935a278ea2"
      },
      "source": [
        "# add() is not in place\n",
        "a.add(a)\n",
        "a"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.],\n",
              "        [6., 6., 6.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mY0ojINbaayp",
        "outputId": "cd396248-e6e4-4d0b-ebf9-5a7411f27882"
      },
      "source": [
        "# add_() is in place\n",
        "a.add_(a)\n",
        "a"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[48., 48., 48.],\n",
              "        [48., 48., 48.],\n",
              "        [48., 48., 48.],\n",
              "        [48., 48., 48.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Re8xiL37eAja"
      },
      "source": [
        "## Autograd\n",
        "`PyTorch` and other machine learning libraries are known for their automatic differantiation feature. That is, given that we have defined the set of operations that need to be performed, the framework itself can figure out how to compute the gradients. We can call the `backward()` method to ask `PyTorch` to calculate the gradiends, which are then stored in the `grad` attribute."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-oEvBJHWfn8H",
        "outputId": "740dc424-2b22-4d2b-ac2a-6d45e06d914b"
      },
      "source": [
        "# Create an example tensor\n",
        "# requires_grad parameter tells PyTorch to store gradients\n",
        "x = torch.tensor([2.], requires_grad=True)\n",
        "\n",
        "# Print the gradient if it is calculated\n",
        "# Currently None since x is a scalar\n",
        "pp.pprint(x.grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DTJazZXkgthP",
        "outputId": "e3d72745-0206-4c31-e11e-7b7ad833d5b5"
      },
      "source": [
        "# Calculating the gradient of y with respect to x\n",
        "y = x * x * 3 # 3x^2\n",
        "y.backward()\n",
        "pp.pprint(x.grad) # d(y)/d(x) = d(3x^2)/d(x) = 6x = 12"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([12.])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Hqc2oM3iV6a"
      },
      "source": [
        "Let's run backprop from a different tensor again to see what happens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K--Az0Xiic_z",
        "outputId": "21195fa0-8466-4a33-92df-be40e3c33745"
      },
      "source": [
        "z = x * x * 3 # 3x^2\n",
        "z.backward()\n",
        "pp.pprint(x.grad)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([48.])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HhjPkiE6i7ja"
      },
      "source": [
        "We can see that the `x.grad` is updated to be the sum of the gradients calculated so far. When we run backprop in a neural network, we sum up all the gradients for a particular neuron before making an update. This is exactly what is happening here! This is also the reason why we need to run `zero_grad()` in every training iteration (more on this later). Otherwise our gradients would keep building up from one training iteration to the other, which would cause our updates to be wrong. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pYLWqKIoaOyd"
      },
      "source": [
        "## Neural Network Module\n",
        "\n",
        "So far we have looked into the tensors, their properties and basic operations on tensors. These are especially useful to get familiar with if we are building the layers of our network from scratch. We will utilize these in Assignment 3, but moving forward, we will use predefined blocks in the `torch.nn` module of `PyTorch`. We will then put together these blocks to create complex networks. Let's start by importing this module with an alias so that we don't have to type `torch` every time we use it. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qUmrDpbhV4Tn"
      },
      "source": [
        "import torch.nn as nn"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "joGvRWjEbak0"
      },
      "source": [
        "### **Linear Layer**\n",
        "We can use `nn.Linear(H_in, H_out)` to create a a linear layer. This will take a matrix of `(N, *, H_in)` dimensions and output a matrix of `(N, *, H_out)`. The `*` denotes that there could be arbitrary number of dimensions in between. The linear layer performs the operation `Ax+b`, where `A` and `b` are initialized randomly. If we don't want the linear layer to learn the bias parameters, we can initialize our layer with `bias=False`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6XfnKI4-a5j9",
        "outputId": "867af230-5f0a-4b0b-c06c-025b1b8b0b0f"
      },
      "source": [
        "# Create the inputs\n",
        "input = torch.ones(2,3,4)\n",
        "# N* H_in -> N*H_out\n",
        "\n",
        "\n",
        "# Make a linear layers transforming N,*,H_in dimensinal inputs to N,*,H_out\n",
        "# dimensional outputs\n",
        "linear = nn.Linear(4, 2)\n",
        "nn.Linear(2,1)\n",
        "linear_output = linear(input)\n",
        "linear_output"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382]],\n",
              "\n",
              "        [[-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382]]], grad_fn=<AddBackward0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 146
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0_9XKtAFYpdI",
        "outputId": "df248f31-e5ca-4857-88c2-c0bbc61a40be"
      },
      "source": [
        "list(linear.parameters()) # Ax + b"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[-0.2491,  0.2283,  0.2765, -0.4489],\n",
              "         [ 0.3642,  0.0685, -0.3154,  0.2699]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([0.0997, 0.2510], requires_grad=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 149
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAXCCu9keUlW"
      },
      "source": [
        "### **Other Module Layers**\n",
        "There are several other preconfigured layers in the `nn` module. Some commonly used examples are `nn.Conv2d`, `nn.ConvTranspose2d`, `nn.BatchNorm1d`, `nn.BatchNorm2d`, `nn.Upsample` and `nn.MaxPool2d` among many others. We will learn more about these as we progress in the course. For now, the only important thing to remember is that we can treat each of these layers as plug and play components: we will be providing the required dimensions and `PyTorch` will take care of setting them up. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yslDOK66fYWn"
      },
      "source": [
        "### **Activation Function Layer**\n",
        "We can also use the `nn` module to apply activations functions to our tensors. Activation functions are used to add non-linearity to our network. Some examples of activations functions are `nn.ReLU()`, `nn.Sigmoid()` and `nn.LeakyReLU()`. Activation functions operate on each element seperately, so the shape of the tensors we get as an output are the same as the ones we pass in."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IrJP5CveeOON",
        "outputId": "ce8b78e3-6311-44e2-c40c-f09f81f1acd1"
      },
      "source": [
        "linear_output"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382]],\n",
              "\n",
              "        [[-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382],\n",
              "         [-0.0935,  0.6382]]], grad_fn=<AddBackward0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 150
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W9v5FjQtd4Ck",
        "outputId": "966538af-4eac-475b-dd30-9174efd7ed3c"
      },
      "source": [
        "sigmoid = nn.Sigmoid()\n",
        "output = sigmoid(linear_output)\n",
        "output"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.4766, 0.6543],\n",
              "         [0.4766, 0.6543],\n",
              "         [0.4766, 0.6543]],\n",
              "\n",
              "        [[0.4766, 0.6543],\n",
              "         [0.4766, 0.6543],\n",
              "         [0.4766, 0.6543]]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 151
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RiYTthJwhEYT"
      },
      "source": [
        "### **Putting the Layers Together**\n",
        "So far we have seen that we can create layers and pass the output of one as the input of the next. Instead of creating intermediate tensors and passing them around, we can use `nn.Sequentual`, which does exactly that. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xtJeOqLxhBLY",
        "outputId": "b2370e4a-47bc-47f2-e784-d7fcd3ce57b7"
      },
      "source": [
        "block = nn.Sequential(\n",
        "    nn.Linear(4, 2),\n",
        "    nn.Sigmoid()\n",
        ")\n",
        "\n",
        "input = torch.ones(2,3,4)\n",
        "output = block(input)\n",
        "output"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[0.3116, 0.8282],\n",
              "         [0.3116, 0.8282],\n",
              "         [0.3116, 0.8282]],\n",
              "\n",
              "        [[0.3116, 0.8282],\n",
              "         [0.3116, 0.8282],\n",
              "         [0.3116, 0.8282]]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 152
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GkJ81p3GUVPM"
      },
      "source": [
        "### Custom Modules\n",
        "\n",
        "Instead of using the predefined modules, we can also build our own by extending the `nn.Module` class. For example, we can build a the `nn.Linear` (which also extends `nn.Module`) on our own using the tensor introduced earlier! We can also build new, more complex modules, such as a custom neural network. You will be practicing these in the later assignment.\n",
        "\n",
        "To create a custom module, the first thing we have to do is to extend the `nn.Module`. We can then initialize our parameters in the `__init__` function, starting with a call to the `__init__` function of the super class. All the class attributes we define which are `nn` module objects are treated as parameters, which can be learned during the training. Tensors are not parameters, but they can be turned into parameters if they are wrapped in `nn.Parameter` class.\n",
        "\n",
        "All classes extending `nn.Module` are also expected to implement a `forward(x)` function, where `x` is a tensor. This is the function that is called when a parameter is passed to our module, such as in `model(x)`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J2P7eZiMj32_"
      },
      "source": [
        "class MultilayerPerceptron(nn.Module):\n",
        "\n",
        "  def __init__(self, input_size, hidden_size):\n",
        "    # Call to the __init__ function of the super class\n",
        "    super(MultilayerPerceptron, self).__init__()\n",
        "\n",
        "    # Bookkeeping: Saving the initialization parameters\n",
        "    self.input_size = input_size \n",
        "    self.hidden_size = hidden_size \n",
        "\n",
        "    # Defining of our model\n",
        "    # There isn't anything specific about the naming of `self.model`. It could\n",
        "    # be something arbitrary.\n",
        "    self.model = nn.Sequential(\n",
        "        nn.Linear(self.input_size, self.hidden_size),\n",
        "        nn.ReLU(),\n",
        "        nn.Linear(self.hidden_size, self.input_size),\n",
        "        nn.Sigmoid()\n",
        "    )\n",
        "    \n",
        "  def forward(self, x):\n",
        "    output = self.model(x)\n",
        "    return output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b2DrfLiBVjNT"
      },
      "source": [
        "Here is an alternative way to define the same class. You can see that we can replace `nn.Sequential` by defining the individual layers in the `__init__` method and connecting the in the `forward` method. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9-lqhsqwViIk"
      },
      "source": [
        "class MultilayerPerceptron(nn.Module):\n",
        "\n",
        "  def __init__(self, input_size, hidden_size):\n",
        "    # Call to the __init__ function of the super class\n",
        "    super(MultilayerPerceptron, self).__init__()\n",
        "\n",
        "    # Bookkeeping: Saving the initialization parameters\n",
        "    self.input_size = input_size \n",
        "    self.hidden_size = hidden_size \n",
        "\n",
        "    # Defining of our layers\n",
        "    self.linear = nn.Linear(self.input_size, self.hidden_size)\n",
        "    self.relu = nn.ReLU()\n",
        "    self.linear2 = nn.Linear(self.hidden_size, self.input_size)\n",
        "    self.sigmoid = nn.Sigmoid()\n",
        "    \n",
        "  def forward(self, x):\n",
        "    linear = self.linear(x)\n",
        "    relu = self.relu(linear)\n",
        "    linear2 = self.linear2(relu)\n",
        "    output = self.sigmoid(linear2)\n",
        "    return output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YQelcFo5bXgU"
      },
      "source": [
        "Now that we have defined our class, we can instantiate it and see what it does. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cXi0T0FZbV0y",
        "outputId": "23547003-cfef-4a6d-e3f4-5fdb7a5cdd97"
      },
      "source": [
        "# Make a sample input\n",
        "input = torch.randn(2, 5)\n",
        "\n",
        "# Create our model\n",
        "model = MultilayerPerceptron(5, 3)\n",
        "\n",
        "# Pass our input through our model\n",
        "model(input)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.6960, 0.5888, 0.6302, 0.5337, 0.6120],\n",
              "        [0.6787, 0.5964, 0.6672, 0.4974, 0.6041]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 154
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tCCbjc-Fb2-B"
      },
      "source": [
        "We can inspect the parameters of our model with `named_parameters()` and `parameters()` methods. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7d23soYIb2WZ",
        "outputId": "3bf4243d-2f85-4593-8989-680b322aca13"
      },
      "source": [
        "list(model.named_parameters())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('linear.weight', Parameter containing:\n",
              "  tensor([[-0.0094, -0.3072,  0.2230,  0.0499, -0.0917],\n",
              "          [ 0.0116, -0.2261, -0.4170, -0.1688,  0.2925],\n",
              "          [ 0.4049,  0.2189,  0.1391,  0.2115, -0.3926]], requires_grad=True)),\n",
              " ('linear.bias', Parameter containing:\n",
              "  tensor([0.1696, 0.2785, 0.3635], requires_grad=True)),\n",
              " ('linear2.weight', Parameter containing:\n",
              "  tensor([[ 0.4921,  0.5605,  0.5188],\n",
              "          [ 0.4088,  0.4430,  0.0042],\n",
              "          [-0.2919,  0.2893, -0.4794],\n",
              "          [ 0.4321, -0.1348,  0.4558],\n",
              "          [-0.4387,  0.2400,  0.3511]], requires_grad=True)),\n",
              " ('linear2.bias', Parameter containing:\n",
              "  tensor([ 0.2369, -0.0131,  0.4319,  0.1126,  0.2039], requires_grad=True))]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 155
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5JegycOdMFy"
      },
      "source": [
        "## Optimization\n",
        "We have showed how gradients are calculated with the `backward()` function. Having the gradients isn't enought for our models to learn. We also need to know how to update the parameters of our models. This is where the optomozers comes in. `torch.optim` module contains several optimizers that we can use. Some popular examples are `optim.SGD` and `optim.Adam`. When initializing optimizers, we pass our model parameters, which can be accessed with `model.parameters()`, telling the optimizers which values it will be optimizing. Optimizers also has a learning rate (`lr`) parameter, which determines how big of an update will be made in every step. Different optimizers have different hyperparameters as well."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W0F-TvV0kk-I"
      },
      "source": [
        "import torch.optim as optim"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wgak6o5dlQWF"
      },
      "source": [
        "After we have our optimization function, we can define a `loss` that we want to optimize for. We can either define the loss ourselves, or use one of the predefined loss function in `PyTorch`, such as `nn.BCELoss()`. Let's put everything together now! We will start by creating some dummy data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dGYFiaT_vXBn",
        "outputId": "48bd9b70-f225-4c20-fad7-5f7a6fa4ed9b"
      },
      "source": [
        "# Create the y data\n",
        "y = torch.ones(10, 5)\n",
        "\n",
        "# Add some noise to our goal y to generate our x\n",
        "# We want out model to predict our original data, albeit the noise\n",
        "x = y + torch.randn_like(y)\n",
        "x"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 1.6387,  0.7748,  1.7080,  0.2209,  1.9849],\n",
              "        [-0.4608,  0.8775,  1.4027,  2.0996,  1.6603],\n",
              "        [ 1.5067,  2.1198,  1.8461,  2.6047,  1.5850],\n",
              "        [ 0.4918,  0.9058,  1.6317,  0.9045,  0.4642],\n",
              "        [-0.0420,  2.1190, -0.1469,  0.5251,  1.7798],\n",
              "        [ 1.8992,  1.2926,  0.5929,  1.4380,  0.6741],\n",
              "        [ 0.2726,  1.5211,  0.5603,  3.1195,  1.8431],\n",
              "        [ 1.1206,  0.8492,  1.2665,  2.8705,  0.7252],\n",
              "        [-0.6805,  1.4661,  1.5455,  2.5870, -0.1636],\n",
              "        [ 2.4966,  2.1783,  1.8534,  1.7078, -0.0653]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 156
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BEsiOdpWvfLj"
      },
      "source": [
        "Now, we can define our model, optimizer and the loss function. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2oA2XsdsbN8p",
        "outputId": "b4cd8c6c-d2e9-4594-f238-234d61e027d9"
      },
      "source": [
        "# Instantiate the model\n",
        "model = MultilayerPerceptron(5, 3)\n",
        "\n",
        "# Define the optimizer\n",
        "adam = optim.Adam(model.parameters(), lr=1e-1)\n",
        "\n",
        "# Define loss using a predefined loss function\n",
        "loss_function = nn.BCELoss()\n",
        "\n",
        "# Calculate how our model is doing now\n",
        "y_pred = model(x)\n",
        "loss_function(y_pred, y).item()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7652291059494019"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 158
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtxU7Y8ZufSR"
      },
      "source": [
        "Let's see if we can have our model achieve a smaller loss. Now that we have everything we need, we can setup our training loop. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ogl6-Ctmuek6",
        "outputId": "b22f36a0-c702-4c89-e5d4-27c60d27c0ff"
      },
      "source": [
        "# Set the number of epoch, which determines the number of training iterations\n",
        "n_epoch = 10 \n",
        "\n",
        "for epoch in range(n_epoch):\n",
        "  # Set the gradients to 0\n",
        "  adam.zero_grad()\n",
        "\n",
        "  # Get the model predictions\n",
        "  y_pred = model(x)\n",
        "\n",
        "  # Get the loss\n",
        "  loss = loss_function(y_pred, y)\n",
        "\n",
        "  # Print stats\n",
        "  print(f\"Epoch {epoch}: traing loss: {loss}\")\n",
        "\n",
        "  # Compute the gradients\n",
        "  loss.backward()\n",
        "\n",
        "  # Take a step to optimize the weights\n",
        "  adam.step()\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: traing loss: 0.7652291059494019\n",
            "Epoch 1: traing loss: 0.5666258335113525\n",
            "Epoch 2: traing loss: 0.3737648129463196\n",
            "Epoch 3: traing loss: 0.21681034564971924\n",
            "Epoch 4: traing loss: 0.1113014817237854\n",
            "Epoch 5: traing loss: 0.05232418701052666\n",
            "Epoch 6: traing loss: 0.023548150435090065\n",
            "Epoch 7: traing loss: 0.01053957361727953\n",
            "Epoch 8: traing loss: 0.004811116959899664\n",
            "Epoch 9: traing loss: 0.002271441277116537\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZrMJ8AmqeCY-",
        "outputId": "73815ecf-9630-4273-9cbb-516258bafe10"
      },
      "source": [
        "list(model.parameters())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[ 0.0044, -0.7197, -0.7594, -0.1397, -0.0485],\n",
              "         [-0.0994, -0.1889, -0.1952, -0.5861, -0.8391],\n",
              "         [ 0.5859,  0.7440,  0.6009,  1.1375,  0.9526]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([-0.6889, -0.4001,  0.5036], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([[ 0.4578,  0.1112,  1.1189],\n",
              "         [-0.0121,  0.2187,  1.3968],\n",
              "         [ 0.3651,  0.3457,  1.0704],\n",
              "         [-0.0348,  0.2022,  1.3452],\n",
              "         [ 0.3004,  0.6618,  1.3298]], requires_grad=True),\n",
              " Parameter containing:\n",
              " tensor([0.8258, 1.2799, 0.7956, 0.2157, 0.4042], requires_grad=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 161
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nXApd82wlsF"
      },
      "source": [
        "You can see that our loss is decreasing. Let's check the predictions of our model now and see if they are close to our original `y`, which was all `1s`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gRqE7P9EtvuS",
        "outputId": "d4c76a74-0568-43a9-da28-f25340b87e7f"
      },
      "source": [
        "# See how our model performs on the training data\n",
        "y_pred = model(x)\n",
        "y_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9987, 0.9998, 0.9983, 0.9993, 0.9993],\n",
              "        [0.9993, 0.9999, 0.9990, 0.9996, 0.9997],\n",
              "        [1.0000, 1.0000, 1.0000, 1.0000, 1.0000],\n",
              "        [0.9946, 0.9988, 0.9932, 0.9959, 0.9964],\n",
              "        [0.9963, 0.9993, 0.9953, 0.9974, 0.9977],\n",
              "        [0.9987, 0.9998, 0.9983, 0.9993, 0.9993],\n",
              "        [0.9999, 1.0000, 0.9998, 1.0000, 1.0000],\n",
              "        [0.9997, 1.0000, 0.9996, 0.9999, 0.9999],\n",
              "        [0.9982, 0.9997, 0.9977, 0.9989, 0.9990],\n",
              "        [0.9997, 1.0000, 0.9996, 0.9999, 0.9999]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 162
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IJng31_Pi2R6",
        "outputId": "c69fbad9-28f2-4529-c171-1849ed811a56"
      },
      "source": [
        "# Create test data and check how our model performs on it\n",
        "x2 = y + torch.randn_like(y)\n",
        "y_pred = model(x2)\n",
        "y_pred"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.9998, 1.0000, 0.9998, 0.9999, 0.9999],\n",
              "        [0.8011, 0.8795, 0.7922, 0.7105, 0.7462],\n",
              "        [0.9993, 0.9999, 0.9990, 0.9996, 0.9997],\n",
              "        [0.9999, 1.0000, 0.9998, 1.0000, 1.0000],\n",
              "        [0.9998, 1.0000, 0.9997, 0.9999, 0.9999],\n",
              "        [0.9987, 0.9998, 0.9983, 0.9993, 0.9994],\n",
              "        [0.6955, 0.7824, 0.6890, 0.5537, 0.5997],\n",
              "        [0.9951, 0.9990, 0.9939, 0.9964, 0.9968],\n",
              "        [0.9978, 0.9996, 0.9971, 0.9986, 0.9987],\n",
              "        [0.9994, 0.9999, 0.9991, 0.9997, 0.9997]], grad_fn=<SigmoidBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 163
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8WNk6oIZw2xo"
      },
      "source": [
        "Great! Looks like our model almost perfectly learned to filter out the noise from the `x` that we passed in!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P8rUNk_1xG1v"
      },
      "source": [
        "## Demo: Word Window Classification\n",
        "\n",
        "Until this part of the notebook, we have learned the fundamentals of PyTorch and built a basic network solving a toy task. Now we will attempt to solve an example NLP task. Here are the things we will learn:\n",
        "\n",
        "1. Data: Creating a Dataset of Batched Tensors\n",
        "2. Modeling\n",
        "3. Training\n",
        "4. Prediction\n",
        "\n",
        "In this section, our goal will be to train a model that will find the words in a sentence corresponding to a `LOCATION`, which will be always of span `1` (meaning that `San Fransisco` won't be recognized as a `LOCATION`). Our task is called `Word Window Classification` for a reason. Instead of letting our model to only take a look at one word in each forward pass, we would like it to be able to consider the context of the word in question. That is, for each word, we want our model to be aware of the surrounding words. Let's dive in!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_amzuUx8BJXI"
      },
      "source": [
        "### Data\n",
        "\n",
        "The very first task of any machine learning project is to set up our training set. Usually, there will be a training corpus we will be utilizing. In NLP tasks, the corpus would generally be a `.txt` or `.csv` file where each row corresponds to a sentence or a tabular datapoint. In our toy task, we will assume that we have already read our data and the corresponding labels into a `Python` list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mDiI1PLMw10z"
      },
      "source": [
        "# Our raw data, which consists of sentences\n",
        "corpus = [\n",
        "          \"We always come to Paris\",\n",
        "          \"The professor is from Australia\",\n",
        "          \"I live in Stanford\",\n",
        "          \"He comes from Taiwan\",\n",
        "          \"The capital of Turkey is Ankara\"\n",
        "         ]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t33Uke9AE22s"
      },
      "source": [
        "#### Preprocessing\n",
        "\n",
        "To make it easier for our models to learn, we usually apply a few preprocessing steps to our data. This is especially important when dealing with text data. Here are some examples of text preprocessing:\n",
        "* **Tokenization**: Tokenizing the sentences into words.\n",
        "* **Lowercasing**: Changing all the letters to be lowercase.\n",
        "* **Noise removal:** Removing special characters (such as punctuations). \n",
        "* **Stop words removal**: Removing commonly used words.\n",
        "\n",
        "Which preprocessing steps are necessary is determined by the task at hand. For example, although it is useful to remove special characters in some tasks, for others they may be important (for example, if we are dealing with multiple languages). For our task, we will lowercase our words and tokenize. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fTGn8ANTzZXT",
        "outputId": "5ac3bc25-e5dd-4005-aa36-90836345fb15"
      },
      "source": [
        "# The preprocessing function we will use to generate our training examples\n",
        "# Our function is a simple one, we lowercase the letters\n",
        "# and then tokenize the words.\n",
        "def preprocess_sentence(sentence):\n",
        "  return sentence.lower().split()\n",
        "\n",
        "# Create our training set\n",
        "train_sentences = [sent.lower().split() for sent in corpus]\n",
        "train_sentences"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[['we', 'always', 'come', 'to', 'paris'],\n",
              " ['the', 'professor', 'is', 'from', 'australia'],\n",
              " ['i', 'live', 'in', 'stanford'],\n",
              " ['he', 'comes', 'from', 'taiwan'],\n",
              " ['the', 'capital', 'of', 'turkey', 'is', 'ankara']]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k4jzo5tp0Hza"
      },
      "source": [
        "For each training example we have, we should also have a corresponding label. Recall that the goal of our model was to determine which words correspond to a `LOCATION`. That is, we want our model to output `0` for all the words that are not `LOCATION`s and `1` for the ones that are `LOCATION`s."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3wo1kcMAHFw7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ffef4ec0-d693-4137-ba40-f9112db5ea55"
      },
      "source": [
        "# Set of locations that appear in our corpus\n",
        "locations = set([\"australia\", \"ankara\", \"paris\", \"stanford\", \"taiwan\", \"turkey\"])\n",
        "\n",
        "# Our train labels\n",
        "train_labels = [[1 if word in locations else 0 for word in sent] for sent in train_sentences]\n",
        "train_labels"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[0, 0, 0, 0, 1],\n",
              " [0, 0, 0, 0, 1],\n",
              " [0, 0, 0, 1],\n",
              " [0, 0, 0, 1],\n",
              " [0, 0, 0, 1, 0, 1]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qgVKH9M3RtPx"
      },
      "source": [
        "#### Converting Words to Embeddings\n",
        "\n",
        "Let's look at our training data a little more closely. Each datapoint we have is a sequence of words. On the other hand, we know that machine learning models work with numbers in vectors. How are we going to turn words into numbers? You may be thinking embeddings and you are right!\n",
        "\n",
        "Imagine that we have an embedding lookup table `E`, where each row corresponds to an embedding. That is, each word in our vocabulary would have a corresponding embedding row `i` in this table. Whenever we want to find an embedding for a word, we will follow these steps:\n",
        "1. Find the corresponding index `i` of the word in the embedding table: `word->index`.\n",
        "2. Index into the embedding table and get the embedding: `index->embedding`.\n",
        "\n",
        "Let's look at the first step. We should assign all the words in our vocabulary to a corresponding index. We can do it as follows:\n",
        "1. Find all the unique words in our corpus.\n",
        "2. Assign an index to each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SjTDlfPyVp5z",
        "outputId": "9f36d85f-bc74-4d45-ef79-def80b5a8f6e"
      },
      "source": [
        "# Find all the unique words in our corpus \n",
        "vocabulary = set(w for s in train_sentences for w in s)\n",
        "vocabulary"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'always',\n",
              " 'ankara',\n",
              " 'australia',\n",
              " 'capital',\n",
              " 'come',\n",
              " 'comes',\n",
              " 'from',\n",
              " 'he',\n",
              " 'i',\n",
              " 'in',\n",
              " 'is',\n",
              " 'live',\n",
              " 'of',\n",
              " 'paris',\n",
              " 'professor',\n",
              " 'stanford',\n",
              " 'taiwan',\n",
              " 'the',\n",
              " 'to',\n",
              " 'turkey',\n",
              " 'we'}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOxnKznOWXSC"
      },
      "source": [
        "`vocabulary` now contains all the words in our corpus. On the other hand, during the test time, we can see words that are not contained in our vocabulary. If we can figure out a way to represent the unknown words, our model can still reason about whether they are a `LOCATION` or not, since we are also looking at the neighboring words for each prediction. \n",
        "\n",
        "We introduce a special token, `<unk>`, to tackle the words that are out of vocabulary. We could pick another string for our unknown token if we wanted. The only requirement here is that our token should be unique: we should only be using this token for unknown words. We will also add this special token to our vocabulary. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ygxYuE1DYeR3"
      },
      "source": [
        "# Add the unknown token to our vocabulary\n",
        "vocabulary.add(\"<unk>\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bsf4haL94AFu"
      },
      "source": [
        "Earlier we mentioned that our task was called `Word Window Classification` because our model is looking at the surroundings words in addition to the given word when it needs to make a prediction. \n",
        "\n",
        "For example, let's take the sentence \"We always come to Paris\". The corresponding training label for this sentence is `0, 0, 0, 0, 1` since only Paris, the last word, is a `LOCATION`. In one pass (meaning a call to `forward()`), our model will try to generate the correct label for one word. Let's say our model is trying to generate the correct label `1` for `Paris`. If we only allow our model to see `Paris`, but nothing else, we will miss out on the important information that the word `to` often times appears with `LOCATION`s. \n",
        "\n",
        "Word windows allow our model to consider the surrounding `+N` or `-N` words of each word when making a prediction. In our earlier example for `Paris`, if we have a window size of 1, that means our model will look at the words that come immediately before and after `Paris`, which are `to`, and, well, nothing. Now, this raises another issue. `Paris` is at the end of our sentence, so there isn't another word following it. Remember that we define the input dimensions of our `PyTorch` models when we are initializing them. If we set the window size to be `1`, it means that our model will be accepting `3` words in every pass. We cannot have our model expect `2` words from time to time.\n",
        "\n",
        "The solution is to introduce a special token, such as `<pad>`, that will be added to our sentences to make sure that every word has a valid window around them. Similar to `<unk>` token, we could pick another string for our pad token if we wanted, as long as we make sure it is used for a unique purpose. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZVQsjYi6ZegI",
        "outputId": "7833a42b-0f2c-4b6a-bda1-3b06503983eb"
      },
      "source": [
        "# Add the <pad> token to our vocabulary\n",
        "vocabulary.add(\"<pad>\")\n",
        "\n",
        "# Function that pads the given sentence\n",
        "# We are introducing this function here as an example\n",
        "# We will be utilizing it later in the tutorial\n",
        "def pad_window(sentence, window_size, pad_token=\"<pad>\"):\n",
        "  window = [pad_token] * window_size\n",
        "  return window + sentence + window\n",
        "\n",
        "# Show padding example\n",
        "window_size = 2\n",
        "pad_window(train_sentences[0], window_size=window_size)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<pad>', '<pad>', 'we', 'always', 'come', 'to', 'paris', '<pad>', '<pad>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqvgWKwSNpAd"
      },
      "source": [
        "Now that our vocabularly is ready, let's assign an index to each of our words. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BNCTQnKDa4oh",
        "outputId": "54033f5b-e96d-47b0-f93c-bbc12a84bdbe"
      },
      "source": [
        "# We are just converting our vocabularly to a list to be able to index into it\n",
        "# Sorting is not necessary, we sort to show an ordered word_to_ind dictionary\n",
        "# That being said, we will see that having the index for the padding token\n",
        "# be 0 is convenient as some PyTorch functions use it as a default value\n",
        "# such as nn.utils.rnn.pad_sequence, which we will cover in a bit\n",
        "ix_to_word = sorted(list(vocabulary))\n",
        "\n",
        "# Creating a dictionary to find the index of a given word\n",
        "word_to_ix = {word: ind for ind, word in enumerate(ix_to_word)}\n",
        "word_to_ix"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'<pad>': 0,\n",
              " '<unk>': 1,\n",
              " 'always': 2,\n",
              " 'ankara': 3,\n",
              " 'australia': 4,\n",
              " 'capital': 5,\n",
              " 'come': 6,\n",
              " 'comes': 7,\n",
              " 'from': 8,\n",
              " 'he': 9,\n",
              " 'i': 10,\n",
              " 'in': 11,\n",
              " 'is': 12,\n",
              " 'live': 13,\n",
              " 'of': 14,\n",
              " 'paris': 15,\n",
              " 'professor': 16,\n",
              " 'stanford': 17,\n",
              " 'taiwan': 18,\n",
              " 'the': 19,\n",
              " 'to': 20,\n",
              " 'turkey': 21,\n",
              " 'we': 22}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 164
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "pt-0SK67hMVo",
        "outputId": "a914da1b-6e1c-4309-acd5-bd4a53b7c2b1"
      },
      "source": [
        "ix_to_word[1]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'<unk>'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 165
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ELZuteqbdWd1"
      },
      "source": [
        "Great! We are ready to convert our training sentences into a sequence of indices corresponding to each token. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZNOxip15bMfH",
        "outputId": "01bba9e6-651a-47e8-a3d7-cf5b89ca919d"
      },
      "source": [
        "# Given a sentence of tokens, return the corresponding indices\n",
        "def convert_token_to_indices(sentence, word_to_ix):\n",
        "  indices = []\n",
        "  for token in sentence:\n",
        "    # Check if the token is in our vocabularly. If it is, get it's index. \n",
        "    # If not, get the index for the unknown token.\n",
        "    if token in word_to_ix:\n",
        "      index = word_to_ix[token]\n",
        "    else:\n",
        "      index = word_to_ix[\"<unk>\"]\n",
        "    indices.append(index)\n",
        "  return indices\n",
        "\n",
        "# More compact version of the same function\n",
        "def _convert_token_to_indices(sentence, word_to_ix):\n",
        "  return [word_to_ind.get(token, word_to_ix[\"<unk>\"]) for token in sentence]\n",
        "\n",
        "# Show an example\n",
        "example_sentence = [\"we\", \"always\", \"come\", \"to\", \"kuwait\"]\n",
        "example_indices = convert_token_to_indices(example_sentence, word_to_ix)\n",
        "restored_example = [ix_to_word[ind] for ind in example_indices]\n",
        "\n",
        "print(f\"Original sentence is: {example_sentence}\")\n",
        "print(f\"Going from words to indices: {example_indices}\")\n",
        "print(f\"Going from indices to words: {restored_example}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original sentence is: ['we', 'always', 'come', 'to', 'kuwait']\n",
            "Going from words to indices: [22, 2, 6, 20, 1]\n",
            "Going from indices to words: ['we', 'always', 'come', 'to', '<unk>']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jsXw8cB1xpH"
      },
      "source": [
        "In the example above, `kuwait` shows up as `<unk>`, because it is not included in our vocabulary. Let's convert our `train_sentences` to `example_padded_indices`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JRaKQwSJH-1d",
        "outputId": "7ac54f70-e1ee-4418-a803-42c9ee873074"
      },
      "source": [
        "# Converting our sentences to indices\n",
        "example_padded_indices = [convert_token_to_indices(s, word_to_ix) for s in train_sentences]\n",
        "example_padded_indices"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[22, 2, 6, 20, 15],\n",
              " [19, 16, 12, 8, 4],\n",
              " [10, 13, 11, 17],\n",
              " [9, 7, 8, 18],\n",
              " [19, 5, 14, 21, 12, 3]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZULjHBjHsEK"
      },
      "source": [
        "Now that we have an index for each word in our vocabularly, we can create an embedding table with `nn.Embedding` class in `PyTorch`. It is called as follows `nn.Embedding(num_words, embedding_dimension)` where `num_words` is the number of words in our vocabulary and the `embedding_dimension` is the dimension of the embeddings we want to have. There is nothing fancy about `nn.Embedding`: it is just a wrapper class around a trainabe `NxE` dimensional tensor, where `N` is the number of words in our vocabulary and `E` is the number of embedding dimensions. This table is initially random, but it will change over time. As we train our network, the gradients will be backpropagated all the way to the embedding layer, and hence our word embeddings would be updated. We will initiliaze the embedding layer we will use for our model in our model, but we are showing an example here. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F4AgHzv91VXx",
        "outputId": "c718e46d-af30-459a-ed12-510926044d03"
      },
      "source": [
        "# Creating an embedding table for our words\n",
        "embedding_dim = 5\n",
        "embeds = nn.Embedding(len(vocabulary), embedding_dim)\n",
        "\n",
        "# Printing the parameters in our embedding table\n",
        "list(embeds.parameters())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[-0.5421,  0.6919,  0.8236, -1.3510,  1.4048],\n",
              "         [ 1.2983,  1.4740,  0.1002, -0.5475,  1.0871],\n",
              "         [ 1.4604, -1.4934, -0.4363, -0.3231, -1.9746],\n",
              "         [ 0.8021,  1.5121,  0.8239,  0.9865, -1.3801],\n",
              "         [ 0.3502, -0.5920,  0.9295,  0.6062, -0.6258],\n",
              "         [ 0.5038, -1.0187,  0.2860,  0.3231, -1.2828],\n",
              "         [ 1.5232, -0.5983, -0.4971, -0.5137,  1.4319],\n",
              "         [ 0.3826,  0.6501, -0.3948,  1.3998, -0.5133],\n",
              "         [-0.1728, -0.7658,  0.2873, -2.1812,  0.9506],\n",
              "         [-0.5617,  0.4552,  0.0618, -1.7503,  0.2192],\n",
              "         [-0.5405,  0.7887, -0.9843, -0.6110,  0.6391],\n",
              "         [ 0.6581, -0.7067,  1.3208,  1.3860, -1.5113],\n",
              "         [ 1.1594,  0.4977, -1.9175,  0.0916,  0.0085],\n",
              "         [ 0.3317,  1.8169,  0.0802, -0.1456, -0.7304],\n",
              "         [ 0.4997, -1.4895,  0.1237, -0.4121,  0.8909],\n",
              "         [ 0.6732,  0.4117, -0.5378,  0.6632, -2.7096],\n",
              "         [-0.4580, -0.9436, -1.6345,  0.1284, -1.6147],\n",
              "         [-0.3537,  1.9635,  1.0702, -0.1894, -0.8822],\n",
              "         [-0.4057, -1.2033, -0.7083,  0.4087, -1.1708],\n",
              "         [-0.6373,  0.5272,  1.8711, -0.5865, -0.7643],\n",
              "         [ 0.4714, -2.5822,  0.4338,  0.1537, -0.7650],\n",
              "         [-2.1828,  1.3178,  1.3833,  0.5018, -1.7209],\n",
              "         [-0.5354,  0.2153, -0.1482,  0.3903,  0.0900]], requires_grad=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wI7ZTt4OkpPp"
      },
      "source": [
        "To get the word embedding for a word in our vocabulary, all we need to do is to create a lookup tensor. The lookup tensor is just a tensor containing the index we want to look up `nn.Embedding` class expects an index tensor that is of type Long Tensor, so we should create our tensor accordingly. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nkldmcepjfh_",
        "outputId": "278c2d7e-acf3-426d-ea29-10f85cf11277"
      },
      "source": [
        "# Get the embedding for the word Paris\n",
        "index = word_to_ix[\"paris\"]\n",
        "index_tensor = torch.tensor(index, dtype=torch.long)\n",
        "paris_embed = embeds(index_tensor)\n",
        "paris_embed"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 0.6732,  0.4117, -0.5378,  0.6632, -2.7096],\n",
              "       grad_fn=<EmbeddingBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 167
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mUsdwBOxm6B4",
        "outputId": "8c4ced79-7235-43f5-ec59-cd33df1d9ca9"
      },
      "source": [
        "# We can also get multiple embeddings at once\n",
        "index_paris = word_to_ix[\"paris\"]\n",
        "index_ankara = word_to_ix[\"ankara\"]\n",
        "indices = [index_paris, index_ankara]\n",
        "indices_tensor = torch.tensor(indices, dtype=torch.long)\n",
        "embeddings = embeds(indices_tensor)\n",
        "embeddings"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 0.6732,  0.4117, -0.5378,  0.6632, -2.7096],\n",
              "        [ 0.8021,  1.5121,  0.8239,  0.9865, -1.3801]],\n",
              "       grad_fn=<EmbeddingBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2bgSW3LPltkF"
      },
      "source": [
        "Usually, we define the embedding layer as part of our model, which you will see in the later sections of our notebook. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DHCXeQOamHU1"
      },
      "source": [
        "#### Batching Sentences\n",
        "\n",
        "We have learned about batches in class. Waiting our whole training corpus to be processed before making an update is constly. On the other hand, updating the parameters after every training example causes the loss to be less stable between updates. To combat these issues, we instead update our parameters after training on a batch of data. This allows us to get a better estimate of the gradient of the global loss. In this section, we will learn how to structure our data into batches using the `torch.util.data.DataLoader` class. \n",
        "\n",
        "We will be calling the `DataLoader` class as follows: `DataLoader(data, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)`.  The `batch_size` parameter determines the number of examples per batch. In every epoch, we will be iterating over all the batches using the `DataLoader`. The order of batches is deterministic by default, but we can ask `DataLoader` to shuffle the batches by setting the `shuffle` parameter to `True`. This way we ensure that we don't encounter a bad batch multiple times.\n",
        "\n",
        "If provided, `DataLoader` passes the batches it prepares to the `collate_fn`. We can write a custom function to pass to the `collate_fn` parameter in order to print stats about our batch or perform extra processing. In our case, we will use the `collate_fn` to:\n",
        "1. Window pad our train sentences.\n",
        "2. Convert the words in the training examples to indices.\n",
        "3. Pad the training examples so that all the sentences and labels have the same length. Similarly, we also need to pad the labels. This creates an issue because when calculating the loss, we need to know the actual number of words in a given example. We will also keep track of this number in the function we pass to the `collate_fn` parameter.\n",
        "\n",
        "Because our version of the `collate_fn` function will need to access to our `word_to_ix` dictionary (so that it can turn words into indices), we will make use of the `partial` function in `Python`, which passes the parameters we give to the function we pass it. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OkvvVlo4jgFm"
      },
      "source": [
        "from torch.utils.data import DataLoader\n",
        "from functools import partial\n",
        "\n",
        "def custom_collate_fn(batch, window_size, word_to_ix):\n",
        "  # Break our batch into the training examples (x) and labels (y)\n",
        "  # We are turning our x and y into tensors because nn.utils.rnn.pad_sequence\n",
        "  # method expects tensors. This is also useful since our model will be\n",
        "  # expecting tensor inputs. \n",
        "  x, y = zip(*batch)\n",
        "\n",
        "  # Now we need to window pad our training examples. We have already defined a \n",
        "  # function to handle window padding. We are including it here again so that\n",
        "  # everything is in one place.\n",
        "  def pad_window(sentence, window_size, pad_token=\"<pad>\"):\n",
        "    window = [pad_token] * window_size\n",
        "    return window + sentence + window\n",
        "\n",
        "  # Pad the train examples.\n",
        "  x = [pad_window(s, window_size=window_size) for s in x]\n",
        "\n",
        "  # Now we need to turn words in our training examples to indices. We are\n",
        "  # copying the function defined earlier for the same reason as above.\n",
        "  def convert_tokens_to_indices(sentence, word_to_ix):\n",
        "    return [word_to_ix.get(token, word_to_ix[\"<unk>\"]) for token in sentence]\n",
        "\n",
        "  # Convert the train examples into indices.\n",
        "  x = [convert_tokens_to_indices(s, word_to_ix) for s in x]\n",
        "\n",
        "  # We will now pad the examples so that the lengths of all the example in \n",
        "  # one batch are the same, making it possible to do matrix operations. \n",
        "  # We set the batch_first parameter to True so that the returned matrix has \n",
        "  # the batch as the first dimension.\n",
        "  pad_token_ix = word_to_ix[\"<pad>\"]\n",
        "\n",
        "  # pad_sequence function expects the input to be a tensor, so we turn x into one\n",
        "  x = [torch.LongTensor(x_i) for x_i in x]\n",
        "  x_padded = nn.utils.rnn.pad_sequence(x, batch_first=True, padding_value=pad_token_ix)\n",
        "\n",
        "  # We will also pad the labels. Before doing so, we will record the number \n",
        "  # of labels so that we know how many words existed in each example. \n",
        "  lengths = [len(label) for label in y]\n",
        "  lenghts = torch.LongTensor(lengths)\n",
        "\n",
        "  y = [torch.LongTensor(y_i) for y_i in y]\n",
        "  y_padded = nn.utils.rnn.pad_sequence(y, batch_first=True, padding_value=0)\n",
        "\n",
        "  # We are now ready to return our variables. The order we return our variables\n",
        "  # here will match the order we read them in our training loop.\n",
        "  return x_padded, y_padded, lenghts  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Q0gXea-bCsz"
      },
      "source": [
        "This function seems long, but it really doesn't have to be. Check out the alternative version below where we remove the extra function declarations and comments. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dZfcmAJXbLcq"
      },
      "source": [
        "def _custom_collate_fn(batch, window_size, word_to_ix):\n",
        "  # Prepare the datapoints\n",
        "  x, y = zip(*batch)  \n",
        "  x = [pad_window(s, window_size=window_size) for s in x]\n",
        "  x = [convert_tokens_to_indices(s, word_to_ix) for s in x]\n",
        "\n",
        "  # Pad x so that all the examples in the batch have the same size\n",
        "  pad_token_ix = word_to_ix[\"<pad>\"]\n",
        "  x = [torch.LongTensor(x_i) for x_i in x]\n",
        "  x_padded = nn.utils.rnn.pad_sequence(x, batch_first=True, padding_value=pad_token_ix)\n",
        "\n",
        "  # Pad y and record the length\n",
        "  lengths = [len(label) for label in y]\n",
        "  lenghts = torch.LongTensor(lengths)\n",
        "  y = [torch.LongTensor(y_i) for y_i in y]\n",
        "  y_padded = nn.utils.rnn.pad_sequence(y, batch_first=True, padding_value=0)\n",
        "\n",
        "  return x_padded, y_padded, lenghts  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dS1WuQO0Khxx"
      },
      "source": [
        "Now, we can see the `DataLoader` in action. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RfB0JKL2vZ6p",
        "outputId": "1d689d9e-05fb-4f45-9fb8-9363af241131"
      },
      "source": [
        "# Parameters to be passed to the DataLoader\n",
        "data = list(zip(train_sentences, train_labels))\n",
        "batch_size = 2\n",
        "shuffle = True\n",
        "window_size = 2\n",
        "collate_fn = partial(custom_collate_fn, window_size=window_size, word_to_ix=word_to_ix)\n",
        "\n",
        "# Instantiate the DataLoader\n",
        "loader = DataLoader(data, batch_size=batch_size, shuffle=shuffle, collate_fn=collate_fn)\n",
        "\n",
        "# Go through one loop\n",
        "counter = 0\n",
        "for batched_x, batched_y, batched_lengths in loader:\n",
        "  print(f\"Iteration {counter}\")\n",
        "  print(\"Batched Input:\")\n",
        "  print(batched_x)\n",
        "  print(\"Batched Labels:\")\n",
        "  print(batched_y)\n",
        "  print(\"Batched Lengths:\")\n",
        "  print(batched_lengths)\n",
        "  print(\"\")\n",
        "  counter += 1"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteration 0\n",
            "Batched Input:\n",
            "tensor([[ 0,  0, 22,  2,  6, 20, 15,  0,  0],\n",
            "        [ 0,  0, 19, 16, 12,  8,  4,  0,  0]])\n",
            "Batched Labels:\n",
            "tensor([[0, 0, 0, 0, 1],\n",
            "        [0, 0, 0, 0, 1]])\n",
            "Batched Lengths:\n",
            "tensor([5, 5])\n",
            "\n",
            "Iteration 1\n",
            "Batched Input:\n",
            "tensor([[ 0,  0, 19,  5, 14, 21, 12,  3,  0,  0],\n",
            "        [ 0,  0, 10, 13, 11, 17,  0,  0,  0,  0]])\n",
            "Batched Labels:\n",
            "tensor([[0, 0, 0, 1, 0, 1],\n",
            "        [0, 0, 0, 1, 0, 0]])\n",
            "Batched Lengths:\n",
            "tensor([6, 4])\n",
            "\n",
            "Iteration 2\n",
            "Batched Input:\n",
            "tensor([[ 0,  0,  9,  7,  8, 18,  0,  0]])\n",
            "Batched Labels:\n",
            "tensor([[0, 0, 0, 1]])\n",
            "Batched Lengths:\n",
            "tensor([4])\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "93QOZSsMTFNF"
      },
      "source": [
        "The batched input tensors you see above will be passed into our model. On the other hand, we started off saying that our model will be a window classifier. The way our input tensors are currently formatted, we have all the words in a sentence in one datapoint. When we pass this input to our model, it needs to create the windows for each word, make a prediction as to whether the center word is a `LOCATION` or not for each window, put the predictions together and return. \n",
        "\n",
        "We could avoid this problem if we formatted our data by breaking it into windows beforehand. In this example, we will instead how our model take care of the formatting. \n",
        "\n",
        "Given that our `window_size` is `N` we want our model to make a prediction on every `2N+1` tokens. That is, if we have an input with `9` tokens, and a `window_size` of `2`, we want our model to return `5` predictions. This makes sense because before we padded it with `2` tokens on each side, our input also had `5` tokens in it! \n",
        "\n",
        "We can create these windows by using for loops, but there is a faster `PyTorch` alternative, which is the `unfold(dimension, size, step)` method. We can create the windows we need using this method as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RMZu-pxLVxHQ",
        "outputId": "bf7467d5-632d-4c0a-8bf2-91a24e834f2d"
      },
      "source": [
        "# Print the original tensor\n",
        "print(f\"Original Tensor: \")\n",
        "print(batched_x)\n",
        "print(\"\")\n",
        "\n",
        "# Create the 2 * 2 + 1 chunks\n",
        "chunk = batched_x.unfold(1, window_size*2 + 1, 1)\n",
        "print(f\"Windows: \")\n",
        "print(chunk)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original Tensor: \n",
            "tensor([[ 0,  0,  9,  7,  8, 18,  0,  0]])\n",
            "\n",
            "Windows: \n",
            "tensor([[[ 0,  0,  9,  7,  8],\n",
            "         [ 0,  9,  7,  8, 18],\n",
            "         [ 9,  7,  8, 18,  0],\n",
            "         [ 7,  8, 18,  0,  0]]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XlDbOpeoSKxd"
      },
      "source": [
        "### Model\n",
        "\n",
        "Now that we have prepared our data, we are ready to build our model. We have learned how to write custom `nn.Module` classes. We will do the same here and put everything we have learned so far together. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JLTU4h76NLYm"
      },
      "source": [
        "class WordWindowClassifier(nn.Module):\n",
        "\n",
        "  def __init__(self, hyperparameters, vocab_size, pad_ix=0):\n",
        "    super(WordWindowClassifier, self).__init__()\n",
        "    \n",
        "    \"\"\" Instance variables \"\"\"\n",
        "    self.window_size = hyperparameters[\"window_size\"]\n",
        "    self.embed_dim = hyperparameters[\"embed_dim\"]\n",
        "    self.hidden_dim = hyperparameters[\"hidden_dim\"]\n",
        "    self.freeze_embeddings = hyperparameters[\"freeze_embeddings\"]\n",
        "\n",
        "    \"\"\" Embedding Layer \n",
        "    Takes in a tensor containing embedding indices, and returns the \n",
        "    corresponding embeddings. The output is of dim \n",
        "    (number_of_indices * embedding_dim).\n",
        "\n",
        "    If freeze_embeddings is True, set the embedding layer parameters to be\n",
        "    non-trainable. This is useful if we only want the parameters other than the\n",
        "    embeddings parameters to change. \n",
        "\n",
        "    \"\"\"\n",
        "    self.embeds = nn.Embedding(vocab_size, self.embed_dim, padding_idx=pad_ix)\n",
        "    if self.freeze_embeddings:\n",
        "      self.embed_layer.weight.requires_grad = False\n",
        "\n",
        "    \"\"\" Hidden Layer\n",
        "    \"\"\"\n",
        "    full_window_size = 2 * window_size + 1\n",
        "    self.hidden_layer = nn.Sequential(\n",
        "      nn.Linear(full_window_size * self.embed_dim, self.hidden_dim), \n",
        "      nn.Tanh()\n",
        "    )\n",
        "\n",
        "    \"\"\" Output Layer\n",
        "    \"\"\"\n",
        "    self.output_layer = nn.Linear(self.hidden_dim, 1)\n",
        "\n",
        "    \"\"\" Probabilities \n",
        "    \"\"\"\n",
        "    self.probabilities = nn.Sigmoid()\n",
        "\n",
        "  def forward(self, inputs):\n",
        "    \"\"\"\n",
        "    Let B:= batch_size\n",
        "        L:= window-padded sentence length\n",
        "        D:= self.embed_dim\n",
        "        S:= self.window_size\n",
        "        H:= self.hidden_dim\n",
        "        \n",
        "    inputs: a (B, L) tensor of token indices\n",
        "    \"\"\"\n",
        "    B, L = inputs.size()\n",
        "\n",
        "    \"\"\"\n",
        "    Reshaping.\n",
        "    Takes in a (B, L) LongTensor\n",
        "    Outputs a (B, L~, S) LongTensor\n",
        "    \"\"\"\n",
        "    # Fist, get our word windows for each word in our input.\n",
        "    token_windows = inputs.unfold(1, 2 * self.window_size + 1, 1)\n",
        "    _, adjusted_length, _ = token_windows.size()\n",
        "\n",
        "    # Good idea to do internal tensor-size sanity checks, at the least in comments!\n",
        "    assert token_windows.size() == (B, adjusted_length, 2 * self.window_size + 1)\n",
        "\n",
        "    \"\"\"\n",
        "    Embedding.\n",
        "    Takes in a torch.LongTensor of size (B, L~, S) \n",
        "    Outputs a (B, L~, S, D) FloatTensor.\n",
        "    \"\"\"\n",
        "    embedded_windows = self.embeds(token_windows)\n",
        "\n",
        "    \"\"\"\n",
        "    Reshaping.\n",
        "    Takes in a (B, L~, S, D) FloatTensor.\n",
        "    Resizes it into a (B, L~, S*D) FloatTensor.\n",
        "    -1 argument \"infers\" what the last dimension should be based on leftover axes.\n",
        "    \"\"\"\n",
        "    embedded_windows = embedded_windows.view(B, adjusted_length, -1)\n",
        "\n",
        "    \"\"\"\n",
        "    Layer 1.\n",
        "    Takes in a (B, L~, S*D) FloatTensor.\n",
        "    Resizes it into a (B, L~, H) FloatTensor\n",
        "    \"\"\"\n",
        "    layer_1 = self.hidden_layer(embedded_windows)\n",
        "\n",
        "    \"\"\"\n",
        "    Layer 2\n",
        "    Takes in a (B, L~, H) FloatTensor.\n",
        "    Resizes it into a (B, L~, 1) FloatTensor.\n",
        "    \"\"\"\n",
        "    output = self.output_layer(layer_1)\n",
        "\n",
        "    \"\"\"\n",
        "    Softmax.\n",
        "    Takes in a (B, L~, 1) FloatTensor of unnormalized class scores.\n",
        "    Outputs a (B, L~, 1) FloatTensor of (log-)normalized class scores.\n",
        "    \"\"\"\n",
        "    output = self.probabilities(output)\n",
        "    output = output.view(B, -1)\n",
        "\n",
        "    return output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Avy1fnyAvEcd"
      },
      "source": [
        "### Training\n",
        "\n",
        "We are now ready to put everything together. Let's start with preparing our data and intializing our model. We can then intialize our optimizer and define our loss function. This time, instead of using one of the predefined loss function as we did before, we will define our own loss function. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bInu1VqjHsfj"
      },
      "source": [
        "# Prepare the data\n",
        "data = list(zip(train_sentences, train_labels))\n",
        "batch_size = 2\n",
        "shuffle = True\n",
        "window_size = 2\n",
        "collate_fn = partial(custom_collate_fn, window_size=window_size, word_to_ix=word_to_ix)\n",
        "\n",
        "# Instantiate a DataLoader\n",
        "loader = DataLoader(data, batch_size=batch_size, shuffle=shuffle, collate_fn=collate_fn)\n",
        "\n",
        "# Initialize a model\n",
        "# It is useful to put all the model hyperparameters in a dictionary\n",
        "model_hyperparameters = {\n",
        "    \"batch_size\": 4,\n",
        "    \"window_size\": 2,\n",
        "    \"embed_dim\": 25,\n",
        "    \"hidden_dim\": 25,\n",
        "    \"freeze_embeddings\": False,\n",
        "}\n",
        "\n",
        "vocab_size = len(word_to_ix)\n",
        "model = WordWindowClassifier(model_hyperparameters, vocab_size)\n",
        "\n",
        "# Define an optimizer\n",
        "learning_rate = 0.01\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
        "\n",
        "# Define a loss function, which computes to binary cross entropy loss\n",
        "def loss_function(batch_outputs, batch_labels, batch_lengths):   \n",
        "    # Calculate the loss for the whole batch\n",
        "    bceloss = nn.BCELoss()\n",
        "    loss = bceloss(batch_outputs, batch_labels.float())\n",
        "\n",
        "    # Rescale the loss. Remember that we have used lengths to store the \n",
        "    # number of words in each training example\n",
        "    loss = loss / batch_lengths.sum().float()\n",
        "\n",
        "    return loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHxpxDkFHfQE"
      },
      "source": [
        "Unlike our earlier example, this time instead of passing all of our training data to the model at once in each epoch, we will be utilizing batches. Hence, in each training epoch iteration, we also iterate over the batches."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QL9IDgIOvHca"
      },
      "source": [
        "# Function that will be called in every epoch\n",
        "def train_epoch(loss_function, optimizer, model, loader):\n",
        "  \n",
        "  # Keep track of the total loss for the batch\n",
        "  total_loss = 0\n",
        "  for batch_inputs, batch_labels, batch_lengths in loader:\n",
        "    # Clear the gradients\n",
        "    optimizer.zero_grad()\n",
        "    # Run a forward pass\n",
        "    outputs = model.forward(batch_inputs)\n",
        "    # Compute the batch loss\n",
        "    loss = loss_function(outputs, batch_labels, batch_lengths)\n",
        "    # Calculate the gradients\n",
        "    loss.backward()\n",
        "    # Update the parameteres\n",
        "    optimizer.step()\n",
        "    total_loss += loss.item()\n",
        "\n",
        "  return total_loss\n",
        "\n",
        "\n",
        "# Function containing our main training loop\n",
        "def train(loss_function, optimizer, model, loader, num_epochs=10000):\n",
        "\n",
        "  # Iterate through each epoch and call our train_epoch function\n",
        "  for epoch in range(num_epochs):\n",
        "    epoch_loss = train_epoch(loss_function, optimizer, model, loader)\n",
        "    if epoch % 100 == 0: print(epoch_loss)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cjf75cnzJ4n6"
      },
      "source": [
        "Let's start training!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kav8kwVBJ6XW",
        "outputId": "0bafb3dc-6806-47bc-c5dc-915195e1d05b"
      },
      "source": [
        "num_epochs = 1000\n",
        "train(loss_function, optimizer, model, loader, num_epochs=num_epochs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.3274914249777794\n",
            "0.24941639229655266\n",
            "0.1968013420701027\n",
            "0.1381114460527897\n",
            "0.11672545038163662\n",
            "0.09148690290749073\n",
            "0.07141915801912546\n",
            "0.05857925023883581\n",
            "0.04900792893022299\n",
            "0.04107789508998394\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T-k7Pav4LdQJ"
      },
      "source": [
        "### Prediction\n",
        "\n",
        "Let's see how well our model is at making predictions. We can start by creating our test data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-v5X69a2Lkbm"
      },
      "source": [
        "# Create test sentences\n",
        "test_corpus = [\"She comes from Paris\"]\n",
        "test_sentences = [s.lower().split() for s in test_corpus]\n",
        "test_labels = [[0, 0, 0, 1]]\n",
        "\n",
        "# Create a test loader\n",
        "test_data = list(zip(test_sentences, test_labels))\n",
        "batch_size = 1\n",
        "shuffle = False\n",
        "window_size = 2\n",
        "collate_fn = partial(custom_collate_fn, window_size=2, word_to_ix=word_to_ix)\n",
        "test_loader = torch.utils.data.DataLoader(test_data, \n",
        "                                           batch_size=1, \n",
        "                                           shuffle=False, \n",
        "                                           collate_fn=collate_fn)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlBa8xaNMZgv"
      },
      "source": [
        "Let's loop over our test examples to see how well we are doing. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dGYn8CAoMTjX",
        "outputId": "8d8fd126-ae7e-48e2-fa3e-8eb1a14f99c7"
      },
      "source": [
        "for test_instance, labels, _ in test_loader:\n",
        "  outputs = model.forward(test_instance)\n",
        "  print(labels)\n",
        "  print(outputs)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[0, 0, 0, 1]])\n",
            "tensor([[0.0339, 0.1031, 0.0500, 0.9770]], grad_fn=<ViewBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}