LSTM_multiclass_text_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "LSTM_multiclass_text_classification.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nolll77/LSTM_multiclass_text_classification/blob/master/LSTM_multiclass_text_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yBWolfWZbAt7",
        "colab_type": "text"
      },
      "source": [
        "# LSTM in Pytorch\n",
        "\n",
        "![Predicting item ratings based on customer reviews](https://drive.google.com/uc?id=1fi3BRAmlN7ljSnZLz2TuZmz9qZstZPxi)\n",
        "\n",
        "\n",
        "Human language is filled with ambiguity, many-a-times the same phrase can have multiple interpretations based on the context and can even appear confusing to humans. Such challenges make natural language processing an interesting but hard problem to solve. However, we’ve seen a lot of advancement in NLP in the past couple of years and it’s quite fascinating to explore the various techniques being used. This article aims to cover one such technique in deep learning using Pytorch: Long Short Term Memory (LSTM) models.\n",
        "\n",
        "If you’re new to NLP or need an in-depth read on preprocessing and word embeddings, you can check out the following [article](https://towardsdatascience.com/getting-started-with-natural-language-processing-nlp-2c482420cc05)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1CphM93sb_HA",
        "colab_type": "text"
      },
      "source": [
        "## Gentle Intro to RNNs and LSTMs :\n",
        "\n",
        "What sets language models apart from conventional neural networks is their dependency on context. Conventional feed-forward networks assume inputs to be independent of one another. For NLP, we need a mechanism to be able to use sequential information from previous inputs to determine the current output. Recurrent Neural Networks (RNNs) tackle this problem by having loops, allowing information to persist through the network.\n",
        "\n",
        "![An unrolled Recurrent Neural Network](https://drive.google.com/uc?id=1vFo5VFVHYGM4PTrKNVH3--iXJ-bNOD7u)\n",
        "\n",
        "However, conventional RNNs have the issue of exploding and vanishing gradients and are not good at processing long sequences because they suffer from short term memory.\n",
        "\n",
        "Long Short Term Memory networks (LSTM) are a special kind of RNN, which are capable of learning long-term dependencies. They do so by maintaining an internal memory state called the “cell state” and have regulators called “gates” to control the flow of information inside each LSTM unit. \n",
        "\n",
        "[Here](https://towardsdatascience.com/getting-started-with-natural-language-processing-nlp-2c482420cc05)’s an excellent source explaining the specifics of LSTMs.\n",
        "\n",
        "\n",
        "![Structure of an LSTM cell. (source : Varsamopoulos, Savvas & Bertels, Koen & Almudever, Carmen. (2018). Designing neural network based decoders for surface codes.)\n",
        "](https://drive.google.com/uc?id=1bXZ4F4OporY_NeanJ1m0sZMVMSBf8WJL)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WtQQgmoCbAt9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#library imports\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re\n",
        "import spacy\n",
        "from collections import Counter\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "import torch.nn.functional as F\n",
        "import string\n",
        "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence\n",
        "from sklearn.metrics import mean_squared_error"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CEy_MbT0bAuG",
        "colab_type": "text"
      },
      "source": [
        "## Basic LSTM in Pytorch with random numbers\n",
        "\n",
        "Before we jump into the main problem, let’s take a look at the basic structure of an LSTM in Pytorch, using a random input. This is a useful step to perform before getting into complex inputs because it helps us learn how to debug the model better, check if dimensions add up and ensure that our model is working as expected.\n",
        "\n",
        "Even though we’re going to be dealing with text, since our model can only work with numbers, we convert the input into a sequence of numbers where each number represents a particular word (more on this in the next section)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dKbssHHNbAuH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#input\n",
        "x = torch.tensor([[1,2, 12,34, 56,78, 90,80],\n",
        "                 [12,45, 99,67, 6,23, 77,82],\n",
        "                 [3,24, 6,99, 12,56, 21,22]])"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOBa-b3mbAuM",
        "colab_type": "text"
      },
      "source": [
        "#### using two different models\n",
        "\n",
        "We first pass the input (3×8) through an [embedding](https://en.wikipedia.org/wiki/Word_embedding) layer, because word embeddings are better at capturing context and are spatially more efficient than one-hot vector representations.\n",
        "\n",
        "In Pytorch, we can use the nn.Embedding module to create this layer, which takes the vocabulary size and desired word-vector length as input. You can optionally provide a padding index, to indicate the index of the padding element in the embedding matrix.\n",
        "\n",
        "In the following example, our vocabulary consists of 100 words, so our input to the embedding layer can only be from 0–100, and it returns us a 100×7 embedding matrix, with the 0th index representing our padding element."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jpj39mVxbAuN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model1 = nn.Embedding(100, 7, padding_idx=0)\n",
        "model2 = nn.LSTM(input_size=7, hidden_size=3, num_layers=1, batch_first=True)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zJVix9ZvbAuS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "out1 = model1(x)\n",
        "out2 = model2(out1)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1I3uAZbwbAuW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 526
        },
        "outputId": "6475147e-1c50-4987-fdd6-5418aabe722e"
      },
      "source": [
        "print(out1.shape)\n",
        "print(out1)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([3, 8, 7])\n",
            "tensor([[[-1.0969,  0.3833, -0.0982, -0.0817, -0.1877, -0.5058,  0.9958],\n",
            "         [ 2.1139,  0.5521, -0.9124,  0.0139, -0.1538, -0.3513, -2.5576],\n",
            "         [-1.2045, -1.2763,  0.2394, -2.0269, -0.2634, -1.7427, -0.3318],\n",
            "         [-1.5396, -0.7897,  1.3016,  0.3635, -0.7474,  0.6954,  0.8867],\n",
            "         [ 1.6468,  0.2826, -0.2182,  1.1096,  0.4718,  0.5952,  0.5073],\n",
            "         [-0.1784,  2.5837, -0.0215,  1.0938, -0.7102, -1.2993,  0.5261],\n",
            "         [ 1.3587, -1.9472, -0.3211,  0.2783, -1.2317,  0.6844, -1.1047],\n",
            "         [ 0.3260,  1.5871,  0.8937,  0.1045,  1.1027, -0.0351, -0.8258]],\n",
            "\n",
            "        [[-1.2045, -1.2763,  0.2394, -2.0269, -0.2634, -1.7427, -0.3318],\n",
            "         [-0.1222,  0.1632, -0.5621,  0.7660,  0.9702, -0.5147, -0.0697],\n",
            "         [ 0.0515,  0.1854, -1.1795,  1.9484, -0.2502, -0.5535, -0.2956],\n",
            "         [ 0.4359,  2.7316,  0.6009, -0.6831,  0.5884, -0.1225, -1.1036],\n",
            "         [ 0.6661,  0.0205, -0.6748, -0.0934, -1.6146,  2.3368, -0.4889],\n",
            "         [ 0.3242,  0.2338, -1.8680,  0.4918,  0.0916,  0.1128, -0.9728],\n",
            "         [ 0.2173, -0.1342,  0.0167, -0.1934,  0.7749,  0.6617,  1.4193],\n",
            "         [-1.1680, -0.3930, -0.0509, -2.2198, -0.0467, -0.4564, -1.1406]],\n",
            "\n",
            "        [[-0.4381,  0.0178,  0.6589,  0.1879,  1.1929,  0.7261,  0.1031],\n",
            "         [ 0.6284, -0.6504,  0.4278, -0.2298,  0.9270, -1.0197, -0.3137],\n",
            "         [ 0.6661,  0.0205, -0.6748, -0.0934, -1.6146,  2.3368, -0.4889],\n",
            "         [ 0.0515,  0.1854, -1.1795,  1.9484, -0.2502, -0.5535, -0.2956],\n",
            "         [-1.2045, -1.2763,  0.2394, -2.0269, -0.2634, -1.7427, -0.3318],\n",
            "         [ 1.6468,  0.2826, -0.2182,  1.1096,  0.4718,  0.5952,  0.5073],\n",
            "         [ 0.7373,  0.0663,  0.5630, -1.9483, -1.6785, -1.2919,  0.4477],\n",
            "         [-1.0118,  0.8055, -0.6790, -1.5518, -0.4946,  0.7419,  0.2158]]],\n",
            "       grad_fn=<EmbeddingBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvKtDsCzmAsJ",
        "colab_type": "text"
      },
      "source": [
        "##### We pass the embedding layer’s output into an LSTM layer (created using nn.LSTM), which takes as input the word-vector length, length of the hidden state vector and number of layers. Additionally, if the first element in our input’s shape has the batch size, we can specify batch_first = True\n",
        "\n",
        "The LSTM layer outputs three things:\n",
        "\n",
        "The consolidated output — of all hidden states in the sequence\n",
        "\n",
        "*   The consolidated output — of all hidden states in the sequence\n",
        "*   Hidden state of the last LSTM unit — the final output\n",
        "*   Cell state\n",
        "\n",
        "Hidden state of the last LSTM unit — the final output\n",
        "Cell state\n",
        "\n",
        "\n",
        "We can verify that after passing through all layers, our output has the expected dimensions:\n",
        "\n",
        "3×8 -> embedding -> 3x8x7 -> LSTM (with hidden size=3)-> 3×3"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X1KXmI48bAub",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "e10cc16e-b45d-4f3e-9dd7-410285f74d5c"
      },
      "source": [
        "out, (ht, ct) = model2(out1)\n",
        "print(ht)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[[-0.0455, -0.0326, -0.1335],\n",
            "         [-0.1509,  0.2473, -0.0080],\n",
            "         [-0.2018,  0.2954,  0.0275]]], grad_fn=<StackBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y0lAnVsYbAui",
        "colab_type": "text"
      },
      "source": [
        "#### using nn.sequential"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3G1Z8IMzbAuk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model3 = nn.Sequential(nn.Embedding(100, 7, padding_idx=0),\n",
        "                        nn.LSTM(input_size=7, hidden_size=3, num_layers=1, batch_first=True))"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FlPUA67QbAup",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 490
        },
        "outputId": "db0e783b-3481-4ea7-9951-7ed755eb877e"
      },
      "source": [
        "out, (ht, ct) = model3(x)\n",
        "print(out)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[[-0.0903,  0.3432, -0.0756],\n",
            "         [-0.0602,  0.6366, -0.0986],\n",
            "         [-0.0996,  0.3600, -0.0648],\n",
            "         [-0.0509,  0.2888, -0.0256],\n",
            "         [-0.0213, -0.2174, -0.0161],\n",
            "         [ 0.0167,  0.1775,  0.0022],\n",
            "         [-0.2140,  0.4243, -0.0795],\n",
            "         [-0.0702,  0.3409, -0.1100]],\n",
            "\n",
            "        [[-0.0644,  0.0896, -0.0424],\n",
            "         [-0.1845,  0.4240, -0.0694],\n",
            "         [-0.3029,  0.6818, -0.0105],\n",
            "         [-0.2619,  0.3583, -0.0103],\n",
            "         [-0.1965,  0.3919, -0.0990],\n",
            "         [-0.0777,  0.1169, -0.1686],\n",
            "         [ 0.0638, -0.1781, -0.0332],\n",
            "         [ 0.0261,  0.0061, -0.0348]],\n",
            "\n",
            "        [[ 0.0533,  0.2313,  0.2094],\n",
            "         [-0.0817,  0.1106, -0.0394],\n",
            "         [-0.1622,  0.3708, -0.0891],\n",
            "         [-0.2874,  0.6930, -0.0095],\n",
            "         [-0.1387,  0.3535, -0.0616],\n",
            "         [-0.0428, -0.2579, -0.0285],\n",
            "         [ 0.0901, -0.4416,  0.0039],\n",
            "         [ 0.0175,  0.1434,  0.0339]]], grad_fn=<TransposeBackward0>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7YZG146NbAut",
        "colab_type": "text"
      },
      "source": [
        "## Multiclass Text Classification - Predicting ratings from review comments\n",
        "\n",
        "Let’s now look at an application of LSTMs.\n",
        "\n",
        "Problem Statement: Given an item’s review comment, predict the rating ( takes integer values from 1 to 5, 1 being worst and 5 being best)\n",
        "\n",
        "We are going to predict item ratings based on customer reviews bsed on this dataset from Kaggle :\n",
        "\n",
        "[Here](https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews) or [here](https://towardsdatascience.com/getting-started-with-natural-language-processing-nlp-2c482420cc05)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0xM2-bJOtwiE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "b625c122-1ebb-49d6-ef7e-32fd9199bd59"
      },
      "source": [
        "# Auth from my Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wW4jvLDhbAuu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 223
        },
        "outputId": "2e02faf7-bea3-4c28-b511-5eb9689d4593"
      },
      "source": [
        "#loading the data\n",
        "path = \"/content/drive/My Drive/Dataset/Jovian/Womens Clothing E-Commerce Reviews.csv\"\n",
        "reviews = pd.read_csv(path)\n",
        "print(reviews.shape)\n",
        "reviews.head()"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(23486, 11)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>Clothing ID</th>\n",
              "      <th>Age</th>\n",
              "      <th>Title</th>\n",
              "      <th>Review Text</th>\n",
              "      <th>Rating</th>\n",
              "      <th>Recommended IND</th>\n",
              "      <th>Positive Feedback Count</th>\n",
              "      <th>Division Name</th>\n",
              "      <th>Department Name</th>\n",
              "      <th>Class Name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>767</td>\n",
              "      <td>33</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Absolutely wonderful - silky and sexy and comf...</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>Initmates</td>\n",
              "      <td>Intimate</td>\n",
              "      <td>Intimates</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1080</td>\n",
              "      <td>34</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Love this dress!  it's sooo pretty.  i happene...</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>General</td>\n",
              "      <td>Dresses</td>\n",
              "      <td>Dresses</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2</td>\n",
              "      <td>1077</td>\n",
              "      <td>60</td>\n",
              "      <td>Some major design flaws</td>\n",
              "      <td>I had such high hopes for this dress and reall...</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>General</td>\n",
              "      <td>Dresses</td>\n",
              "      <td>Dresses</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3</td>\n",
              "      <td>1049</td>\n",
              "      <td>50</td>\n",
              "      <td>My favorite buy!</td>\n",
              "      <td>I love, love, love this jumpsuit. it's fun, fl...</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>General Petite</td>\n",
              "      <td>Bottoms</td>\n",
              "      <td>Pants</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4</td>\n",
              "      <td>847</td>\n",
              "      <td>47</td>\n",
              "      <td>Flattering shirt</td>\n",
              "      <td>This shirt is very flattering to all due to th...</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>6</td>\n",
              "      <td>General</td>\n",
              "      <td>Tops</td>\n",
              "      <td>Blouses</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0  Clothing ID  Age  ...   Division Name Department Name  Class Name\n",
              "0           0          767   33  ...       Initmates        Intimate   Intimates\n",
              "1           1         1080   34  ...         General         Dresses     Dresses\n",
              "2           2         1077   60  ...         General         Dresses     Dresses\n",
              "3           3         1049   50  ...  General Petite         Bottoms       Pants\n",
              "4           4          847   47  ...         General            Tops     Blouses\n",
              "\n",
              "[5 rows x 11 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OSPJbOibAuy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reviews['Title'] = reviews['Title'].fillna('')\n",
        "reviews['Review Text'] = reviews['Review Text'].fillna('')\n",
        "reviews['review'] = reviews['Title'] + ' ' + reviews['Review Text']"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fCnmB2fNbAu2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "outputId": "b011ebb4-bf59-4c0a-afef-d4199b5856fa"
      },
      "source": [
        "#keeping only relevant columns and calculating sentence lengths\n",
        "reviews = reviews[['review', 'Rating']]\n",
        "reviews.columns = ['review', 'rating']\n",
        "reviews['review_length'] = reviews['review'].apply(lambda x: len(x.split()))\n",
        "reviews.head()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>rating</th>\n",
              "      <th>review_length</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Absolutely wonderful - silky and sexy and com...</td>\n",
              "      <td>4</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Love this dress!  it's sooo pretty.  i happen...</td>\n",
              "      <td>5</td>\n",
              "      <td>62</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Some major design flaws I had such high hopes ...</td>\n",
              "      <td>3</td>\n",
              "      <td>102</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>My favorite buy! I love, love, love this jumps...</td>\n",
              "      <td>5</td>\n",
              "      <td>25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Flattering shirt This shirt is very flattering...</td>\n",
              "      <td>5</td>\n",
              "      <td>38</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              review  rating  review_length\n",
              "0   Absolutely wonderful - silky and sexy and com...       4              8\n",
              "1   Love this dress!  it's sooo pretty.  i happen...       5             62\n",
              "2  Some major design flaws I had such high hopes ...       3            102\n",
              "3  My favorite buy! I love, love, love this jumps...       5             25\n",
              "4  Flattering shirt This shirt is very flattering...       5             38"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4IL1Td2ebAu7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#changing ratings to 0-numbering\n",
        "zero_numbering = {1:0, 2:1, 3:2, 4:3, 5:4}\n",
        "reviews['rating'] = reviews['rating'].apply(lambda x: zero_numbering[x])"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XMS5zjKXbAu_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "63c242aa-5210-43b3-8fe6-914d61544b3f"
      },
      "source": [
        "#mean sentence length\n",
        "np.mean(reviews['review_length'])"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "60.832921740611425"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-IBXBdrbns20",
        "colab_type": "text"
      },
      "source": [
        "#### Metric\n",
        "We usually take accuracy as our metric for most classification problems, however, ratings are ordered. If the actual value is 5 but the model predicts a 4, it is not considered as bad as predicting a 1. Hence, instead of going with accuracy, we choose RMSE — root mean squared error as our North Star metric. Also, rating prediction is a pretty hard problem, even for humans, so a prediction of being off by just 1 point or lesser is considered pretty good.\n",
        "\n",
        "Preprocessing\n",
        "As mentioned earlier, we need to convert our text into a numerical form that can be fed to our model as input. I’ve used spacy for tokenization after removing punctuation, special characters, and lower casing the text:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Tg8KerVbAvE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#tokenization\n",
        "tok = spacy.load('en')\n",
        "def tokenize (text):\n",
        "    text = re.sub(r\"[^\\x00-\\x7F]+\", \" \", text)\n",
        "    regex = re.compile('[' + re.escape(string.punctuation) + '0-9\\\\r\\\\t\\\\n]') # remove punctuation and numbers\n",
        "    nopunct = regex.sub(\" \", text.lower())\n",
        "    return [token.text for token in tok.tokenizer(nopunct)]"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vBL1zic6n43k",
        "colab_type": "text"
      },
      "source": [
        "#### We count the number of occurrences of each token in our corpus and get rid of the ones that don’t occur too frequently :"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t8_FUXF_bAvI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#count number of occurences of each word\n",
        "counts = Counter()\n",
        "for index, row in reviews.iterrows():\n",
        "    counts.update(tokenize(row['review']))"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-5HKRiUoUDF",
        "colab_type": "text"
      },
      "source": [
        "#### We lost about 6000 words! This is expected because our corpus is quite small, less than 25k reviews, the chance of having repeated words is quite small.\n",
        "\n",
        "We then create a vocabulary to index mapping and encode our review text using this mapping. I’ve chosen the maximum length of any review to be 70 words because the average length of reviews was around 60."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W_Pgx_QLbAvO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "ad6fdabb-a6b9-4b80-92c4-3f10a2eb0a6d"
      },
      "source": [
        "#deleting infrequent words\n",
        "print(\"num_words before:\",len(counts.keys()))\n",
        "for word in list(counts):\n",
        "    if counts[word] < 2:\n",
        "        del counts[word]\n",
        "print(\"num_words after:\",len(counts.keys()))"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num_words before: 14138\n",
            "num_words after: 8263\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-4U-5IrTbAvS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#creating vocabulary\n",
        "vocab2index = {\"\":0, \"UNK\":1}\n",
        "words = [\"\", \"UNK\"]\n",
        "for word in counts:\n",
        "    vocab2index[word] = len(words)\n",
        "    words.append(word)"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BZWG9qjUbAvW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def encode_sentence(text, vocab2index, N=70):\n",
        "    tokenized = tokenize(text)\n",
        "    encoded = np.zeros(N, dtype=int)\n",
        "    enc1 = np.array([vocab2index.get(word, vocab2index[\"UNK\"]) for word in tokenized])\n",
        "    length = min(N, len(enc1))\n",
        "    encoded[:length] = enc1[:length]\n",
        "    return encoded, length"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H-J-gX7mbAva",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "outputId": "f8d61746-d4df-4781-9337-1c43207bfb7c"
      },
      "source": [
        "reviews['encoded'] = reviews['review'].apply(lambda x: np.array(encode_sentence(x,vocab2index )))\n",
        "reviews.head()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>review</th>\n",
              "      <th>rating</th>\n",
              "      <th>review_length</th>\n",
              "      <th>encoded</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Absolutely wonderful - silky and sexy and com...</td>\n",
              "      <td>3</td>\n",
              "      <td>8</td>\n",
              "      <td>[[2, 3, 4, 5, 6, 7, 8, 7, 9, 0, 0, 0, 0, 0, 0,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Love this dress!  it's sooo pretty.  i happen...</td>\n",
              "      <td>4</td>\n",
              "      <td>62</td>\n",
              "      <td>[[2, 10, 11, 12, 5, 13, 14, 15, 16, 5, 17, 18,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Some major design flaws I had such high hopes ...</td>\n",
              "      <td>2</td>\n",
              "      <td>102</td>\n",
              "      <td>[[54, 55, 56, 57, 17, 58, 59, 60, 61, 62, 11, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>My favorite buy! I love, love, love this jumps...</td>\n",
              "      <td>4</td>\n",
              "      <td>25</td>\n",
              "      <td>[[68, 109, 110, 2, 17, 10, 2, 10, 2, 10, 11, 1...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Flattering shirt This shirt is very flattering...</td>\n",
              "      <td>4</td>\n",
              "      <td>38</td>\n",
              "      <td>[[122, 123, 11, 123, 52, 92, 122, 19, 124, 125...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                              review  ...                                            encoded\n",
              "0   Absolutely wonderful - silky and sexy and com...  ...  [[2, 3, 4, 5, 6, 7, 8, 7, 9, 0, 0, 0, 0, 0, 0,...\n",
              "1   Love this dress!  it's sooo pretty.  i happen...  ...  [[2, 10, 11, 12, 5, 13, 14, 15, 16, 5, 17, 18,...\n",
              "2  Some major design flaws I had such high hopes ...  ...  [[54, 55, 56, 57, 17, 58, 59, 60, 61, 62, 11, ...\n",
              "3  My favorite buy! I love, love, love this jumps...  ...  [[68, 109, 110, 2, 17, 10, 2, 10, 2, 10, 11, 1...\n",
              "4  Flattering shirt This shirt is very flattering...  ...  [[122, 123, 11, 123, 52, 92, 122, 19, 124, 125...\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F5RFBIvtbAvf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4474dbcc-acd1-42fc-b168-f1dc83bcc6b7"
      },
      "source": [
        "#check how balanced the dataset is\n",
        "Counter(reviews['rating'])"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Counter({0: 842, 1: 1565, 2: 2871, 3: 5077, 4: 13131})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_jxGCANHbAvk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = list(reviews['encoded'])\n",
        "y = list(reviews['rating'])\n",
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2)"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEMEl0mxbAvn",
        "colab_type": "text"
      },
      "source": [
        "#### Pytorch Dataset\n",
        "\n",
        "The dataset is quite straightforward because we’ve already stored our encodings in the input dataframe. We also output the length of the input sequence in each case, because we can have LSTMs that take variable-length sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8MiEsz5QbAvo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ReviewsDataset(Dataset):\n",
        "    def __init__(self, X, Y):\n",
        "        self.X = X\n",
        "        self.y = Y\n",
        "        \n",
        "    def __len__(self):\n",
        "        return len(self.y)\n",
        "    \n",
        "    def __getitem__(self, idx):\n",
        "        return torch.from_numpy(self.X[idx][0].astype(np.int32)), self.y[idx], self.X[idx][1]"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5n3Xq81CbAvs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_ds = ReviewsDataset(X_train, y_train)\n",
        "valid_ds = ReviewsDataset(X_valid, y_valid)"
      ],
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rrh3t_sxogqo",
        "colab_type": "text"
      },
      "source": [
        "#### Pytorch training loop\n",
        "The training loop is pretty standard. I’ve used Adam optimizer and cross-entropy loss.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TUqRC-wQbAvx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_model(model, epochs=10, lr=0.001):\n",
        "    parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
        "    optimizer = torch.optim.Adam(parameters, lr=lr)\n",
        "    for i in range(epochs):\n",
        "        model.train()\n",
        "        sum_loss = 0.0\n",
        "        total = 0\n",
        "        for x, y, l in train_dl:\n",
        "            x = x.long()\n",
        "            y = y.long()\n",
        "            y_pred = model(x, l)\n",
        "            optimizer.zero_grad()\n",
        "            loss = F.cross_entropy(y_pred, y)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            sum_loss += loss.item()*y.shape[0]\n",
        "            total += y.shape[0]\n",
        "        val_loss, val_acc, val_rmse = validation_metrics(model, val_dl)\n",
        "        if i % 5 == 1:\n",
        "            print(\"train loss %.3f, val loss %.3f, val accuracy %.3f, and val rmse %.3f\" % (sum_loss/total, val_loss, val_acc, val_rmse))\n",
        "\n",
        "def validation_metrics (model, valid_dl):\n",
        "    model.eval()\n",
        "    correct = 0\n",
        "    total = 0\n",
        "    sum_loss = 0.0\n",
        "    sum_rmse = 0.0\n",
        "    for x, y, l in valid_dl:\n",
        "        x = x.long()\n",
        "        y = y.long()\n",
        "        y_hat = model(x, l)\n",
        "        loss = F.cross_entropy(y_hat, y)\n",
        "        pred = torch.max(y_hat, 1)[1]\n",
        "        correct += (pred == y).float().sum()\n",
        "        total += y.shape[0]\n",
        "        sum_loss += loss.item()*y.shape[0]\n",
        "        sum_rmse += np.sqrt(mean_squared_error(pred, y.unsqueeze(-1)))*y.shape[0]\n",
        "    return sum_loss/total, correct/total, sum_rmse/total"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOkgXYMybAv1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 5000\n",
        "vocab_size = len(words)\n",
        "train_dl = DataLoader(train_ds, batch_size=batch_size, shuffle=True)\n",
        "val_dl = DataLoader(valid_ds, batch_size=batch_size)"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4zGZpP9low2K",
        "colab_type": "text"
      },
      "source": [
        "## LSTM Model\n",
        "I’ve used 3 variations for the model:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4oL7C1BdbAv5",
        "colab_type": "text"
      },
      "source": [
        "### 1- LSTM with fixed length input\n",
        "\n",
        "This pretty much has the same structure as the basic LSTM we saw earlier, with the addition of a dropout layer to prevent overfitting. Since we have a classification problem, we have a final linear layer with 5 outputs. This implementation actually works the best among the classification LSTMs, with an accuracy of about 64% and a root-mean-squared-error of only 0.817"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OgV19Ze6bAv5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTM_fixed_len(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim) :\n",
        "        super().__init__()\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, 5)\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "        \n",
        "    def forward(self, x, l):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        lstm_out, (ht, ct) = self.lstm(x)\n",
        "        return self.linear(ht[-1])"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_e4E2gtybAv9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_fixed =  LSTM_fixed_len(vocab_size, 50, 50)"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C_kB5b0DbAwA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "48e64bc3-48ea-4f0b-c368-fa81cfb48a97"
      },
      "source": [
        "train_model(model_fixed, epochs=30, lr=0.01)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 1.277, val loss 1.216, val accuracy 0.560, and val rmse 1.360\n",
            "train loss 1.191, val loss 1.200, val accuracy 0.560, and val rmse 1.357\n",
            "train loss 1.146, val loss 1.176, val accuracy 0.554, and val rmse 1.315\n",
            "train loss 1.080, val loss 1.088, val accuracy 0.562, and val rmse 1.298\n",
            "train loss 0.974, val loss 1.048, val accuracy 0.562, and val rmse 1.005\n",
            "train loss 0.925, val loss 1.030, val accuracy 0.588, and val rmse 0.982\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wnzKSAFnbAwG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "d8c1d3ab-f99c-4d87-def3-ba9898953764"
      },
      "source": [
        "train_model(model_fixed, epochs=30, lr=0.01)"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 0.927, val loss 1.075, val accuracy 0.581, and val rmse 1.160\n",
            "train loss 0.830, val loss 1.013, val accuracy 0.595, and val rmse 1.002\n",
            "train loss 0.808, val loss 1.009, val accuracy 0.608, and val rmse 0.910\n",
            "train loss 0.757, val loss 0.987, val accuracy 0.610, and val rmse 0.885\n",
            "train loss 0.692, val loss 0.972, val accuracy 0.617, and val rmse 0.865\n",
            "train loss 0.643, val loss 0.992, val accuracy 0.621, and val rmse 0.854\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hANFIkaUbAwK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "57d8a352-ccc4-48b4-a4df-baf2d1a3f77a"
      },
      "source": [
        "train_model(model_fixed, epochs=30, lr=0.01)"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 0.643, val loss 0.977, val accuracy 0.622, and val rmse 0.838\n",
            "train loss 0.584, val loss 1.033, val accuracy 0.625, and val rmse 0.846\n",
            "train loss 0.562, val loss 1.024, val accuracy 0.631, and val rmse 0.813\n",
            "train loss 0.518, val loss 1.060, val accuracy 0.627, and val rmse 0.820\n",
            "train loss 0.486, val loss 1.083, val accuracy 0.633, and val rmse 0.821\n",
            "train loss 0.464, val loss 1.126, val accuracy 0.628, and val rmse 0.834\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPNWqpOdbAwO",
        "colab_type": "text"
      },
      "source": [
        "### 2- LSTM with variable length input\n",
        "\n",
        "We can modify our model a bit to make it accept variable-length inputs. This ends up increasing the training time though, because of the pack_padded_sequence function call which returns a padded batch of variable-length sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C9IPC4JybAwO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTM_variable_input(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim) :\n",
        "        super().__init__()\n",
        "        self.hidden_dim = hidden_dim\n",
        "        self.dropout = nn.Dropout(0.3)\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, 5)\n",
        "        \n",
        "    def forward(self, x, s):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        x_pack = pack_padded_sequence(x, s, batch_first=True, enforce_sorted=False)\n",
        "        out_pack, (ht, ct) = self.lstm(x_pack)\n",
        "        out = self.linear(ht[-1])\n",
        "        return out"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AO0b0NWybAwS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = LSTM_variable_input(vocab_size, 50, 50)"
      ],
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nW1AamJebAwW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "5b47909b-9306-47c6-bb65-b84a7437859e"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.1)"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 1.285, val loss 1.248, val accuracy 0.534, and val rmse 1.340\n",
            "train loss 1.098, val loss 1.133, val accuracy 0.566, and val rmse 1.253\n",
            "train loss 0.917, val loss 0.962, val accuracy 0.600, and val rmse 0.943\n",
            "train loss 0.821, val loss 0.916, val accuracy 0.617, and val rmse 0.843\n",
            "train loss 0.767, val loss 0.915, val accuracy 0.632, and val rmse 0.862\n",
            "train loss 0.748, val loss 0.928, val accuracy 0.616, and val rmse 0.829\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zriXqOGSbAwZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "941a700c-974d-4eb2-d308-99c8812083cb"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 0.771, val loss 0.962, val accuracy 0.619, and val rmse 0.838\n",
            "train loss 0.705, val loss 0.939, val accuracy 0.629, and val rmse 0.827\n",
            "train loss 0.676, val loss 0.949, val accuracy 0.619, and val rmse 0.830\n",
            "train loss 0.671, val loss 0.955, val accuracy 0.622, and val rmse 0.838\n",
            "train loss 0.657, val loss 0.959, val accuracy 0.620, and val rmse 0.828\n",
            "train loss 0.649, val loss 0.958, val accuracy 0.622, and val rmse 0.833\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IdP-iFiVbAwd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "6fff471a-6c82-4689-c55f-53d088a8f22d"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 0.647, val loss 1.034, val accuracy 0.619, and val rmse 0.862\n",
            "train loss 0.614, val loss 0.996, val accuracy 0.618, and val rmse 0.828\n",
            "train loss 0.610, val loss 0.998, val accuracy 0.615, and val rmse 0.830\n",
            "train loss 0.596, val loss 1.009, val accuracy 0.622, and val rmse 0.836\n",
            "train loss 0.598, val loss 1.018, val accuracy 0.607, and val rmse 0.868\n",
            "train loss 0.592, val loss 1.025, val accuracy 0.611, and val rmse 0.849\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAUUf31ibAwg",
        "colab_type": "text"
      },
      "source": [
        "### 3- LSTM with pretrained Glove word embeddings\n",
        "\n",
        "Instead of training our own word embeddings, we can use pre-trained Glove word vectors that have been trained on a massive corpus and probably have better context captured. For our problem, however, this doesn’t seem to help much.\n",
        "\n",
        "Download weights [here](https://nlp.stanford.edu/projects/glove/)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t6KKZh1MbAwh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def load_glove_vectors(glove_file=\"/content/drive/My Drive/Dataset/Jovian/Glove6B/glove.6B.50d.txt\"):\n",
        "    \"\"\"Load the glove word vectors\"\"\"\n",
        "    word_vectors = {}\n",
        "    with open(glove_file) as f:\n",
        "        for line in f:\n",
        "            split = line.split()\n",
        "            word_vectors[split[0]] = np.array([float(x) for x in split[1:]])\n",
        "    return word_vectors"
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P-YXQNq0bAwl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_emb_matrix(pretrained, word_counts, emb_size = 50):\n",
        "    \"\"\" Creates embedding matrix from word vectors\"\"\"\n",
        "    vocab_size = len(word_counts) + 2\n",
        "    vocab_to_idx = {}\n",
        "    vocab = [\"\", \"UNK\"]\n",
        "    W = np.zeros((vocab_size, emb_size), dtype=\"float32\")\n",
        "    W[0] = np.zeros(emb_size, dtype='float32') # adding a vector for padding\n",
        "    W[1] = np.random.uniform(-0.25, 0.25, emb_size) # adding a vector for unknown words \n",
        "    vocab_to_idx[\"UNK\"] = 1\n",
        "    i = 2\n",
        "    for word in word_counts:\n",
        "        if word in word_vecs:\n",
        "            W[i] = word_vecs[word]\n",
        "        else:\n",
        "            W[i] = np.random.uniform(-0.25,0.25, emb_size)\n",
        "        vocab_to_idx[word] = i\n",
        "        vocab.append(word)\n",
        "        i += 1   \n",
        "    return W, np.array(vocab), vocab_to_idx"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T1LHR0aPbAwq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "word_vecs = load_glove_vectors()\n",
        "pretrained_weights, vocab, vocab2index = get_emb_matrix(word_vecs, counts)"
      ],
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2lDpYxxwbAwt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTM_glove_vecs(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, glove_weights) :\n",
        "        super().__init__()\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.embeddings.weight.data.copy_(torch.from_numpy(glove_weights))\n",
        "        self.embeddings.weight.requires_grad = False ## freeze embeddings\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, 5)\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "        \n",
        "    def forward(self, x, l):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        lstm_out, (ht, ct) = self.lstm(x)\n",
        "        return self.linear(ht[-1])"
      ],
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BNbPjItubAwy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = LSTM_glove_vecs(vocab_size, 50, 50, pretrained_weights)"
      ],
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MII275yGbAw-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "0d7e453b-3bf0-44b4-e189-5e07f4174ced"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.1)"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 1.272, val loss 1.245, val accuracy 0.560, and val rmse 1.360\n",
            "train loss 1.211, val loss 1.208, val accuracy 0.560, and val rmse 1.360\n",
            "train loss 1.206, val loss 1.204, val accuracy 0.560, and val rmse 1.360\n",
            "train loss 1.203, val loss 1.206, val accuracy 0.560, and val rmse 1.360\n",
            "train loss 1.201, val loss 1.207, val accuracy 0.559, and val rmse 1.359\n",
            "train loss 1.197, val loss 1.206, val accuracy 0.560, and val rmse 1.357\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KFhC_KhYbAxB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "dfc0031a-7e5e-4e11-9267-6e0fddfa0642"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 1.219, val loss 1.223, val accuracy 0.559, and val rmse 1.360\n",
            "train loss 1.194, val loss 1.210, val accuracy 0.558, and val rmse 1.358\n",
            "train loss 1.189, val loss 1.209, val accuracy 0.558, and val rmse 1.357\n",
            "train loss 1.173, val loss 1.192, val accuracy 0.554, and val rmse 1.355\n",
            "train loss 1.144, val loss 1.157, val accuracy 0.555, and val rmse 1.350\n",
            "train loss 1.119, val loss 1.140, val accuracy 0.561, and val rmse 1.334\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "asAZ_c7sbAxH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "31c08d02-c542-4171-fdfd-a789e14501ec"
      },
      "source": [
        "train_model(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train loss 1.132, val loss 1.113, val accuracy 0.560, and val rmse 1.328\n",
            "train loss 1.089, val loss 1.109, val accuracy 0.559, and val rmse 1.096\n",
            "train loss 1.059, val loss 1.078, val accuracy 0.576, and val rmse 1.173\n",
            "train loss 1.029, val loss 1.045, val accuracy 0.579, and val rmse 1.135\n",
            "train loss 0.999, val loss 1.013, val accuracy 0.589, and val rmse 1.014\n",
            "train loss 0.989, val loss 1.006, val accuracy 0.588, and val rmse 0.963\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVK_YzpnbAxL",
        "colab_type": "text"
      },
      "source": [
        "## Predicting ratings using regression instead of classification\n",
        "\n",
        "Since ratings have an order, and a prediction of 3.6 might be better than rounding off to 4 in many cases, it is helpful to explore this as a regression problem. Not surprisingly, this approach gives us the lowest error of just 0.799 because we don’t have just integer predictions anymore.\n",
        "\n",
        "The only change to our model is that instead of the final layer having 5 outputs, we have just one. The training loop changes a bit too, we use MSE loss and we don’t need to take the argmax anymore to get the final prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j6S0JCIBbAxL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_model_regr(model, epochs=10, lr=0.001):\n",
        "    parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
        "    optimizer = torch.optim.Adam(parameters, lr=lr)\n",
        "    for i in range(epochs):\n",
        "        model.train()\n",
        "        sum_loss = 0.0\n",
        "        total = 0\n",
        "        for x, y, l in train_dl:\n",
        "            x = x.long()\n",
        "            y = y.float()\n",
        "            y_pred = model(x, l)\n",
        "            optimizer.zero_grad()\n",
        "            loss = F.mse_loss(y_pred, y.unsqueeze(-1))\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            sum_loss += loss.item()*y.shape[0]\n",
        "            total += y.shape[0]\n",
        "        val_loss = validation_metrics_regr(model, val_dl)\n",
        "        if i % 5 == 1:\n",
        "            print(\"train mse %.3f val rmse %.3f\" % (sum_loss/total, val_loss))\n",
        "\n",
        "def validation_metrics_regr (model, valid_dl):\n",
        "    model.eval()\n",
        "    correct = 0\n",
        "    total = 0\n",
        "    sum_loss = 0.0\n",
        "    for x, y, l in valid_dl:\n",
        "        x = x.long()\n",
        "        y = y.float()\n",
        "        y_hat = model(x, l)\n",
        "        loss = np.sqrt(F.mse_loss(y_hat, y.unsqueeze(-1)).item())\n",
        "        total += y.shape[0]\n",
        "        sum_loss += loss.item()*y.shape[0]\n",
        "    return sum_loss/total"
      ],
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8baMAjuCbAxQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LSTM_regr(torch.nn.Module) :\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim) :\n",
        "        super().__init__()\n",
        "        self.embeddings = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)\n",
        "        self.lstm = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
        "        self.linear = nn.Linear(hidden_dim, 1)\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "        \n",
        "    def forward(self, x, l):\n",
        "        x = self.embeddings(x)\n",
        "        x = self.dropout(x)\n",
        "        lstm_out, (ht, ct) = self.lstm(x)\n",
        "        return self.linear(ht[-1])"
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yZn861O0bAxT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model =  LSTM_regr(vocab_size, 50, 50)"
      ],
      "execution_count": 69,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VKGwRHPxbAxX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "fbc2c9ff-2372-41e1-d227-24bbba2c7625"
      },
      "source": [
        "train_model_regr(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train mse 1.590 val rmse 1.196\n",
            "train mse 1.215 val rmse 1.115\n",
            "train mse 1.143 val rmse 1.123\n",
            "train mse 1.103 val rmse 1.127\n",
            "train mse 1.080 val rmse 1.129\n",
            "train mse 1.056 val rmse 1.130\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zayWnyUwbAxb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 126
        },
        "outputId": "a40937f8-9499-4b4c-b480-e67190eed41e"
      },
      "source": [
        "train_model_regr(model, epochs=30, lr=0.05)"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train mse 1.481 val rmse 1.151\n",
            "train mse 1.023 val rmse 1.077\n",
            "train mse 0.622 val rmse 0.855\n",
            "train mse 0.466 val rmse 0.795\n",
            "train mse 0.400 val rmse 0.782\n",
            "train mse 0.376 val rmse 0.789\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KD11KcL1pmDv",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion:\n",
        "LSTM appears to be theoretically involved, but its Pytorch implementation is pretty straightforward. Also, while looking at any problem, it is very important to choose the right metric, in our case if we’d gone for accuracy, the model seems to be doing a very bad job, but the RMSE shows that it is off by less than 1 rating point, which is comparable to human performance!\n",
        "\n",
        "References:\n",
        "\n",
        "https://www.usfca.edu/data-institute/certificates/deep-learning-part-one\n",
        "\n",
        "https://colah.github.io/posts/2015-08-Understanding-LSTMs/\n",
        "\n",
        "http://web.stanford.edu/class/cs224n/\n",
        "\n",
        "https://www.jovian.ml/blog/multiclass-text-classification-using-lstm-in-pytorch"
      ]
    }
  ]
}