07_introduction_to_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/umar052001/Deep-Learning-with-Tensorflow/blob/master/07_introduction_to_NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction to NLP fundamentals in Tensorflow\n",
        "\n",
        "NLP has the goal of deriving information out of natural language (could be sequences text and speech)\n",
        "\n",
        "Another common term for NLP problems is sequence to sequence problems (seq2seq)."
      ],
      "metadata": {
        "id": "c5gLJMjaKL3d"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Get helper functions"
      ],
      "metadata": {
        "id": "l10QhBhLKWp5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://raw.githubusercontent.com/umar052001/Deep-Learning-with-Tensorflow/master/helperFunctions.py\n",
        "\n",
        "from helperFunctions import unzip_data, create_tensorboard_callback, plot_loss_curves, compare_historys"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VvyChvatKWmh",
        "outputId": "cb961f24-b8f9-4a47-8c1f-3168fffab8b9"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-09-15 22:22:42--  https://raw.githubusercontent.com/umar052001/Deep-Learning-with-Tensorflow/master/helperFunctions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10512 (10K) [text/plain]\n",
            "Saving to: ‘helperFunctions.py’\n",
            "\n",
            "\rhelperFunctions.py    0%[                    ]       0  --.-KB/s               \rhelperFunctions.py  100%[===================>]  10.27K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-09-15 22:22:42 (81.3 MB/s) - ‘helperFunctions.py’ saved [10512/10512]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Get a text dataset\n",
        "\n",
        "The dataset we're going to use is Kaggle's introduction to NLP dataset (text samples of tweets labled as diaster or not diaster)"
      ],
      "metadata": {
        "id": "hGcaFZRlKWjq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
        "unzip_data(\"nlp_getting_started.zip\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LNNqYPl8KWgs",
        "outputId": "30ad63da-c1e4-464b-9da4-b56598b472e3"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-09-15 22:22:45--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.164.176, 172.253.115.128, 172.217.0.48, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.164.176|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip’\n",
            "\n",
            "\rnlp_getting_started   0%[                    ]       0  --.-KB/s               \rnlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.004s  \n",
            "\n",
            "2022-09-15 22:22:45 (158 MB/s) - ‘nlp_getting_started.zip’ saved [607343/607343]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Visualizing text dataset"
      ],
      "metadata": {
        "id": "SzAJQxzsKWeQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "train_df =  pd.read_csv(\"train.csv\")\n",
        "test_df = pd.read_csv(\"test.csv\")"
      ],
      "metadata": {
        "id": "6WCgQol1KWbV"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "PggJght1KWYv",
        "outputId": "f7c4fd27-26e0-45f9-84cd-589ea60884b3"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   id keyword location                                               text  \\\n",
              "0   1     NaN      NaN  Our Deeds are the Reason of this #earthquake M...   \n",
              "1   4     NaN      NaN             Forest fire near La Ronge Sask. Canada   \n",
              "2   5     NaN      NaN  All residents asked to 'shelter in place' are ...   \n",
              "3   6     NaN      NaN  13,000 people receive #wildfires evacuation or...   \n",
              "4   7     NaN      NaN  Just got sent this photo from Ruby #Alaska as ...   \n",
              "\n",
              "   target  \n",
              "0       1  \n",
              "1       1  \n",
              "2       1  \n",
              "3       1  \n",
              "4       1  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-3b428b5c-8080-45a7-bee4-00938262e799\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-3b428b5c-8080-45a7-bee4-00938262e799')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-3b428b5c-8080-45a7-bee4-00938262e799 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-3b428b5c-8080-45a7-bee4-00938262e799');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_df_shuffled = train_df.sample(frac=1, random_state=42)\n",
        "train_df_shuffled.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "95avT_hEKWWS",
        "outputId": "b20d6d5c-1c87-4c10-d306-c94cc799aa77"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        id      keyword               location  \\\n",
              "2644  3796  destruction                    NaN   \n",
              "2227  3185       deluge                    NaN   \n",
              "5448  7769       police                     UK   \n",
              "132    191   aftershock                    NaN   \n",
              "6845  9810       trauma  Montgomery County, MD   \n",
              "\n",
              "                                                   text  target  \n",
              "2644  So you have a new weapon that can cause un-ima...       1  \n",
              "2227  The f$&amp;@ing things I do for #GISHWHES Just...       0  \n",
              "5448  DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...       1  \n",
              "132   Aftershock back to school kick off was great. ...       0  \n",
              "6845  in response to trauma Children of Addicts deve...       0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8aa0eb40-d8a5-4111-8a69-ae215aef84a5\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2644</th>\n",
              "      <td>3796</td>\n",
              "      <td>destruction</td>\n",
              "      <td>NaN</td>\n",
              "      <td>So you have a new weapon that can cause un-ima...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2227</th>\n",
              "      <td>3185</td>\n",
              "      <td>deluge</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The f$&amp;amp;@ing things I do for #GISHWHES Just...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5448</th>\n",
              "      <td>7769</td>\n",
              "      <td>police</td>\n",
              "      <td>UK</td>\n",
              "      <td>DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>191</td>\n",
              "      <td>aftershock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Aftershock back to school kick off was great. ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6845</th>\n",
              "      <td>9810</td>\n",
              "      <td>trauma</td>\n",
              "      <td>Montgomery County, MD</td>\n",
              "      <td>in response to trauma Children of Addicts deve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8aa0eb40-d8a5-4111-8a69-ae215aef84a5')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8aa0eb40-d8a5-4111-8a69-ae215aef84a5 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8aa0eb40-d8a5-4111-8a69-ae215aef84a5');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "test_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "TpX6xipcKWTW",
        "outputId": "33e3dc44-269c-4afd-ca3d-c3cb900cd167"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1328f32d-8823-4ff3-8103-4e15232b999e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1328f32d-8823-4ff3-8103-4e15232b999e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-1328f32d-8823-4ff3-8103-4e15232b999e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-1328f32d-8823-4ff3-8103-4e15232b999e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_df.target.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "piKOlf6WKWQj",
        "outputId": "bbcc46d0-aa48-4282-a115-c7441c6fea9f"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pretty balanced binary classification "
      ],
      "metadata": {
        "id": "udoosU3BKWN1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# How many samples?\n",
        "len(train_df), len(test_df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpg_xao5KWLY",
        "outputId": "bed936bd-3d41-40d8-aa8a-e8b222662669"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(7613, 3263)"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's visualize random samples\n",
        "import random\n",
        "random_index = random.randint(0, len(train_df)-5)\n",
        "for row in train_df_shuffled[[\"text\",\"target\"]][random_index:random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target: {target}\", \"(real diaster)\" if target > 0 else \"(not real diaster)\")\n",
        "  print(f\"Text: \\n{text}\\n\")\n",
        "  print(\"---\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zLYVAXE2KWJG",
        "outputId": "f03384b7-b2f6-4a7c-fab0-c8b192835c95"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 1 (real diaster)\n",
            "Text: \n",
            "FedEx no longer will ship potential bioterror pathogens - FedEx Corp. (NYSE: FDX) will no longer deliver packages ... http://t.co/2kdq56xTWs\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real diaster)\n",
            "Text: \n",
            "ÛÏRichmond Coaches were devastated to hear of the death of their second driver Mr Chance who was sittingÛ_: Jam... http://t.co/dIalTa6t69\n",
            "\n",
            "---\n",
            "\n",
            "Target: 1 (real diaster)\n",
            "Text: \n",
            "600 passengers abandoned at LRT station during Tuesday's hailstorm http://t.co/vgF41IuPkn #yyc #yycstorm #abstorm\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real diaster)\n",
            "Text: \n",
            "#DU19 who gon get in this rap battle with me\n",
            "\n",
            "---\n",
            "\n",
            "Target: 0 (not real diaster)\n",
            "Text: \n",
            "CAMILA'S DOING A FOLLOW SPREE TONIGHT IM SCREAMING OF HAPPINESS\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Split data into training and validation sets"
      ],
      "metadata": {
        "id": "cZo9t15MKWGp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split"
      ],
      "metadata": {
        "id": "c7mQA2GSKWEL"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_sentences, val_sentences,  train_labels, val_labels = train_test_split(train_df_shuffled[\"text\"].to_numpy(),\n",
        "                                                                             train_df_shuffled[\"target\"].to_numpy(),\n",
        "                                                                             test_size=0.1,\n",
        "                                                                             random_state=42)"
      ],
      "metadata": {
        "id": "LhW6jzMAKWBu"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "len(train_sentences), len(val_sentences),  len(train_labels), len(val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "44IOpD9wKV_N",
        "outputId": "63c49a35-1f26-4320-e998-3d7d9985cbd9"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6851, 762, 6851, 762)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check the first 10 samples\n",
        "train_sentences[:10], train_labels[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s9Wt6C88KV9M",
        "outputId": "bbee0359-dad4-413e-bf2e-da8e4fb70a53"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "        'Imagine getting flattened by Kurt Zouma',\n",
              "        '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "        \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "        'Somehow find you and I collide http://t.co/Ee8RpOahPk',\n",
              "        '@EvaHanderek @MarleyKnysh great times until the bus driver held us hostage in the mall parking lot lmfao',\n",
              "        'destroy the free fandom honestly',\n",
              "        'Weapons stolen from National Guard Armory in New Albany still missing #Gunsense http://t.co/lKNU8902JE',\n",
              "        '@wfaaweather Pete when will the heat wave pass? Is it really going to be mid month? Frisco Boy Scouts have a canoe trip in Okla.',\n",
              "        'Patient-reported outcomes in long-term survivors of metastatic colorectal cancer - British Journal of Surgery http://t.co/5Yl4DC1Tqt'],\n",
              "       dtype=object), array([0, 0, 1, 0, 0, 1, 1, 0, 1, 1]))"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Converting text into numbers\n",
        "\n",
        "There are few ways to do this, namely:\n",
        "* Tokenization\n",
        "*  Embedding"
      ],
      "metadata": {
        "id": "XPkvmLowKV6i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Text vectorization(tokenization)"
      ],
      "metadata": {
        "id": "5whBnO-QKV4I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_sentences[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JR3TToA2KV13",
        "outputId": "3a28fd97-0433-41f8-c850-405f80461462"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "       'Imagine getting flattened by Kurt Zouma',\n",
              "       '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "       \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "       'Somehow find you and I collide http://t.co/Ee8RpOahPk'],\n",
              "      dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "text_vectorizor = TextVectorization(max_tokens=None, # defines how many words in vocabulary (automatically adds <oov>)\n",
        "                                    standardize=\"lower_and_strip_punctuation\",\n",
        "                                    split=\"whitespace\",\n",
        "                                    ngrams=None, # create groups of n words\n",
        "                                    output_mode=\"int\", # how to map tokens to numbers\n",
        "                                    output_sequence_length=None, # how long do you want your sequence\n",
        "                                    pad_to_max_tokens=False)"
      ],
      "metadata": {
        "id": "Rm2yfEiMKVzi"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the average number of tokens\n",
        "round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ogh0E3I0KVxK",
        "outputId": "1f276d18-40b7-4bf7-eb2f-94f00ffe189a"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_vocab_length = 10000 # max number of words in our vocabulary\n",
        "max_length = 15 # max length our vars will be\n",
        "text_vectorizor = TextVectorization(max_tokens=max_vocab_length,\n",
        "                                    output_mode=\"int\",\n",
        "                                    output_sequence_length=max_length)"
      ],
      "metadata": {
        "id": "gpcr2dgeKVuf"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# fit text vectorizer to the training text\n",
        "text_vectorizor.adapt(train_sentences)"
      ],
      "metadata": {
        "id": "kKObfc6QKVrs"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a sample sentence and tokenize it\n",
        "sample_sentence = \"There's a flood in my street\"\n",
        "text_vectorizor([sample_sentence])"
      ],
      "metadata": {
        "id": "cY6GtpaYKVpS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7b002da8-86ba-4317-b577-a094c2dae241"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[264,   3, 232,   4,  13, 698,   0,   0,   0,   0,   0,   0,   0,\n",
              "          0,   0]])>"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Choose random sentence from training dataset and tokenize it\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text : \\n{random_sentence}\")\n",
        "print(f\"\\n\\nVectorized Version : {text_vectorizor([random_sentence])}\")"
      ],
      "metadata": {
        "id": "almISClDKVm6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd9afd3c-8452-4422-cb9b-fb3699f43c9f"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text : \n",
            "Finnish Nuclear Plant to Move Ahead After Financing Secured-&gt; http://t.co/uHkXMXaB9l\n",
            "\n",
            "\n",
            "Vectorized Version : [[2456  105 2800    5  769 1155   43 5641 8701    1    0    0    0    0\n",
            "     0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get unique words in vocabulary\n",
        "words_in_vocab = text_vectorizor.get_vocabulary()\n",
        "top_5_words = words_in_vocab[:5]\n",
        "bottom_5_words = words_in_vocab[-5:]\n",
        "print(f\"Number of words in vocab: {len(words_in_vocab)}\")\n",
        "print(f\"5 most common words words: {top_5_words}\")\n",
        "print(f\"5 least common words words: {bottom_5_words}\")"
      ],
      "metadata": {
        "id": "ofIrw1mzKVkY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "58f6ac88-1ae0-4419-d958-8d983018a5c4"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of words in vocab: 10000\n",
            "5 most common words words: ['', '[UNK]', 'the', 'a', 'in']\n",
            "5 least common words words: ['pages', 'paeds', 'pads', 'padres', 'paddytomlinson1']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating an embedding using Embedding Layer\n",
        "\n",
        "Embedding is rich neumerical representation which can be learned during training.\n",
        "\n",
        "Params we care more about are:\n",
        "* `input_dim` = the size of our vocabulary\n",
        "* `output_dim` = the size of our embedding vector, for example, a value of 100 would mean each token gets represented by a vector 100 long.\n",
        "* `input_length` = length of sequences being passed to the embedding layer"
      ],
      "metadata": {
        "id": "gCvDXYnzcdpR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "embedding = layers.Embedding(input_dim=max_vocab_length, #set input shape\n",
        "                             output_dim=128, # output shape\n",
        "                             input_length=max_length)\n",
        "embedding"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "v9PYO25acdl8",
        "outputId": "0ec2593a-2661-4f4d-8df3-20c55d09cf05"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.layers.embeddings.Embedding at 0x7f9bb05db810>"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "random_sentence = random.choice(train_sentences)\n",
        "\n",
        "print(f\"Original text : \\n{random_sentence}\")\n",
        "print(f\"\\n\\nEmbededed Version : {text_vectorizor([random_sentence])}\")\n",
        "\n",
        "sample_embed = embedding(text_vectorizor([random_sentence]))\n",
        "sample_embed"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hGmAq94Ncdiz",
        "outputId": "b1cc0300-9c75-48e5-c1bd-6cbeaaf2ebb3"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text : \n",
            "@King_Naruto_ As long as I see Madara bleeding I'm good ??\n",
            "\n",
            "\n",
            "Embededed Version : [[  1  26 480  26   8  99   1 587  32 136   0   0   0   0   0]]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              "array([[[ 0.04683903,  0.01486573, -0.02780171, ..., -0.0079063 ,\n",
              "         -0.00039761,  0.01829282],\n",
              "        [ 0.04488498, -0.02935326, -0.04652852, ...,  0.03380029,\n",
              "         -0.04191084,  0.03300612],\n",
              "        [-0.02270782, -0.02571425,  0.02391774, ..., -0.02828163,\n",
              "          0.0467344 ,  0.03818215],\n",
              "        ...,\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614],\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614],\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614]]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check out single token's embeding\n",
        "sample_embed[0][0], sample_embed[0],[0], random_sentence[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nGx-jEDucdgL",
        "outputId": "c37316e4-b08e-4c53-8af5-18a40e7343c9"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(128,), dtype=float32, numpy=\n",
              " array([ 0.04683903,  0.01486573, -0.02780171, -0.04809263, -0.02039313,\n",
              "         0.0146823 , -0.03714484, -0.02970212,  0.02704105, -0.02105966,\n",
              "        -0.02007029,  0.03466674, -0.01439226,  0.00977935,  0.01270122,\n",
              "         0.00495011, -0.02477835, -0.04981147, -0.01685236, -0.01151722,\n",
              "        -0.00014649, -0.03305632, -0.01183579,  0.04062596,  0.03516364,\n",
              "        -0.03775354,  0.00790917,  0.03267522,  0.0475976 ,  0.04066389,\n",
              "         0.03434283, -0.02115428,  0.00119604, -0.00301099, -0.04582844,\n",
              "         0.03415031, -0.03281531, -0.0435425 ,  0.03948966,  0.02683032,\n",
              "        -0.01792346,  0.03272321, -0.03027393, -0.04318608, -0.02771096,\n",
              "         0.03527823, -0.01396351, -0.01461094,  0.00439065,  0.03701483,\n",
              "        -0.01840919,  0.04468466,  0.01678491,  0.03630351,  0.01724771,\n",
              "        -0.0167951 , -0.00484623, -0.02024409, -0.0214394 ,  0.02833028,\n",
              "        -0.01249665,  0.0161353 ,  0.0003186 ,  0.01682338,  0.04549502,\n",
              "         0.03462574,  0.04082406, -0.01568005, -0.00948702, -0.0398042 ,\n",
              "         0.01250989,  0.0479355 , -0.02283531, -0.02170458, -0.0310452 ,\n",
              "         0.01692245,  0.01254071,  0.03018693, -0.03493471, -0.02362235,\n",
              "         0.04791889, -0.02645922,  0.01623521,  0.0231486 , -0.02446629,\n",
              "        -0.03692323, -0.01310451,  0.01403974, -0.02472149,  0.00255431,\n",
              "        -0.01959183,  0.0437793 ,  0.00688825,  0.04301891, -0.02738134,\n",
              "        -0.02210674,  0.03946013, -0.03748605, -0.03712817,  0.04131385,\n",
              "        -0.02702525, -0.00253954,  0.01155084,  0.00324725,  0.04726348,\n",
              "        -0.0401041 , -0.03059827, -0.04720951, -0.00074751,  0.03046102,\n",
              "        -0.01632509,  0.00573846, -0.01555086,  0.03454349,  0.03700637,\n",
              "        -0.00289785,  0.00299016,  0.03843105,  0.02141228, -0.03571709,\n",
              "        -0.0444975 , -0.03347621,  0.01380973,  0.0462019 ,  0.00903865,\n",
              "        -0.0079063 , -0.00039761,  0.01829282], dtype=float32)>,\n",
              " <tf.Tensor: shape=(15, 128), dtype=float32, numpy=\n",
              " array([[ 0.04683903,  0.01486573, -0.02780171, ..., -0.0079063 ,\n",
              "         -0.00039761,  0.01829282],\n",
              "        [ 0.04488498, -0.02935326, -0.04652852, ...,  0.03380029,\n",
              "         -0.04191084,  0.03300612],\n",
              "        [-0.02270782, -0.02571425,  0.02391774, ..., -0.02828163,\n",
              "          0.0467344 ,  0.03818215],\n",
              "        ...,\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614],\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614],\n",
              "        [ 0.02767852, -0.01007007,  0.03506594, ..., -0.02855086,\n",
              "         -0.01879266,  0.03650614]], dtype=float32)>,\n",
              " [0],\n",
              " '@')"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Modeling a text dataset (running a series of experiments)\n",
        "\n",
        "We'll start from baseline and move on from there:\n",
        "* Model 0: Naive Bayes (baseline)\n",
        "* Model 1: Feed-Forward neural network (dense model)\n",
        "* Model 2: LSTM model (RNN)\n",
        "* Model 3: GRU model (RNN)\n",
        "* Model 4: Bidirectional-LSTM model (RNN)\n",
        "* Model 5: 1D CNN\n",
        "* Model 6: Tensorflow Hub pretrained feature extractor (using transfer learning for NLP)\n",
        "* Model 7: Same as model 6 with 10% of training data"
      ],
      "metadata": {
        "id": "ujP4J0t_cdc-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model 0 : Getting a baseline\n",
        "\n",
        "We'll use Sklearn's multinomial Naive Bayes using TF-IDF formula to vonvert our words to numbers.\n",
        "> Its common practice to use non-DL algorithms as a baseline because of there speed and later using DL to see if we can improve upon them."
      ],
      "metadata": {
        "id": "vJTjrlgScdZ_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "model_0 = Pipeline([\n",
        "                    (\"tfidf\", TfidfVectorizer()), # convert words to numbers using tfidf\n",
        "                    (\"clf\", MultinomialNB()) # Model the text\n",
        "])\n",
        "# Fit the pipeline to training data\n",
        "model_0.fit(train_sentences, train_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EVW1yExxcdW3",
        "outputId": "8650959e-0a7b-4b48-d5c7-90de358d5a51"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('tfidf', TfidfVectorizer()), ('clf', MultinomialNB())])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate our baseline model\n",
        "baseline_score = model_0.score(val_sentences, val_labels)\n",
        "print(f\"Baseline model achieves an accuracy of {baseline_score * 100:.2f}%\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GevpMCNWcdTu",
        "outputId": "c9e3e16b-ccaa-436d-f283-f3b0ecb93da3"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline model achieves an accuracy of 79.27%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds[:20]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BaqNdC0EcdRc",
        "outputId": "76462cf6-a1fc-4887-e0d5-858460ba86bb"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating an evaluation function for our model experiments"
      ],
      "metadata": {
        "id": "146y8Q1ocdOg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def calculate_results(y_true, y_preds):\n",
        "  \"\"\"\n",
        "  Calculates model accuracy, precision, recall and f1 score of a binary classification model\n",
        "  Args:\n",
        "    y_true: Ground truth labels\n",
        "    y_preds: Prediction labels\n",
        "  Returns:\n",
        "    A dictionary of accuracy score, precision recall and f1-score\n",
        "  \"\"\"\n",
        "  # Calculate model accuracy \n",
        "  model_accuracy = accuracy_score(y_true, y_preds) * 100\n",
        "  # Calculate model precision, recall and f1-score using \"weigted\" average\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_preds, average=\"weighted\")\n",
        "  # Result dictionary\n",
        "  model_results = {\"accuracy\":model_accuracy,\n",
        "                   \"precision\":model_precision,\n",
        "                   \"recall\": model_recall,\n",
        "                   \"f1\": model_f1}\n",
        "  return model_results"
      ],
      "metadata": {
        "id": "9_lqErRqKVib"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get baseline results\n",
        "baseline_results = calculate_results(val_labels,baseline_preds)\n",
        "baseline_results"
      ],
      "metadata": {
        "id": "Dn8ref6GKVgN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a9dff0b6-e0d5-4de4-a554-1cc8dfe8121d"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model 1: A simple dense model"
      ],
      "metadata": {
        "id": "B7GmraeQuaNT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensorboard callback (need to create a new one for each model)\n",
        "from helperFunctions import create_tensorboard_callback\n",
        "\n",
        "# Create a dir to save logs\n",
        "SAVE_DIR = \"model_logs\""
      ],
      "metadata": {
        "id": "ocMhZeoouZxg"
      },
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Build model with Functional Api\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizor(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.GlobalAveragePooling1D()(x) # condense the feature vector for each token to one vector\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x) # sigmoid for binary\n",
        "model_1 = tf.keras.Model(inputs, outputs, name=\"model_1_dense\")"
      ],
      "metadata": {
        "id": "obYbvIYVuYqb"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_1.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aeuXgRGWuYnT",
        "outputId": "b2d06352-3193-42c7-f8a4-005b0dff0031"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (TextV  (None, 15)               0         \n",
            " ectorization)                                                   \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " global_average_pooling1d (G  (None, 128)              0         \n",
            " lobalAveragePooling1D)                                          \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"adam\",\n",
        "                metrics=[\"accuracy\"])"
      ],
      "metadata": {
        "id": "MJgvhdNduYh5"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_1_history = model_1.fit(x=train_sentences,\n",
        "                              y=train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR,\n",
        "                                                                     experiment_name=\"model_1_dense\")])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "26w4Pn0_uYee",
        "outputId": "5cfc0d75-683f-4fbe-f119-c6979a99ff0b"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_1_dense/20220915-222250\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 8s 9ms/step - loss: 0.6104 - accuracy: 0.6859 - val_loss: 0.5340 - val_accuracy: 0.7598\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.4403 - accuracy: 0.8189 - val_loss: 0.4678 - val_accuracy: 0.7808\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.3460 - accuracy: 0.8615 - val_loss: 0.4599 - val_accuracy: 0.7953\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.2844 - accuracy: 0.8904 - val_loss: 0.4636 - val_accuracy: 0.7927\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.2367 - accuracy: 0.9101 - val_loss: 0.4825 - val_accuracy: 0.7848\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bhc-JxVnuYam",
        "outputId": "163eaf90-5948-4204-ffe6-6f4d6ddf3e56"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 3ms/step - loss: 0.4825 - accuracy: 0.7848\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4825327694416046, 0.7847769260406494]"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1_pred_probs = model_1.predict(val_sentences)\n",
        "model_1_pred_probs.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gKWn9RufuYYN",
        "outputId": "2e6137d3-9611-46e9-9fbd-ed2706c5d12d"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(762, 1)"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1_pred_probs[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PAj1RXwmuYVn",
        "outputId": "a02d66af-ee9b-40f9-efa0-357eb05e005e"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.32648388], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_probs))\n",
        "model_1_preds[:20]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ba8CkgzNuYTY",
        "outputId": "c82e66f1-a30f-409a-cb96-43aae55a594e"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(20,), dtype=float32, numpy=\n",
              "array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0.,\n",
              "       0., 0., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1_results = calculate_results(val_labels, model_1_preds)\n",
        "model_1_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NGrhQ61OuX8v",
        "outputId": "180621d8-f45c-4386-c070-08b1a6cfffe2"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.4776902887139,\n",
              " 'precision': 0.789165199286798,\n",
              " 'recall': 0.7847769028871391,\n",
              " 'f1': 0.7818959205825942}"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HLsbF8qBuX5e",
        "outputId": "11e480da-6c11-4bc3-8540-199cd4cd9ad7"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It looks like our basline model is outperforming our deep learning model"
      ],
      "metadata": {
        "id": "wzipaFpKuX0M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Visualizing learned embeddings"
      ],
      "metadata": {
        "id": "fUCCKQGEuXwy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the vocabulary from the text vectorization layer\n",
        "words_in_vocab = text_vectorizor.get_vocabulary()\n",
        "len(words_in_vocab)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YpP8EQtWuXup",
        "outputId": "e662b016-238f-4c5b-d610-b9b8a156fede"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10000"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_1.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RKCa7Ev-uXsT",
        "outputId": "e328561b-432b-43f4-97e2-39b13065ec97"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_1 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (TextV  (None, 15)               0         \n",
            " ectorization)                                                   \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " global_average_pooling1d (G  (None, 128)              0         \n",
            " lobalAveragePooling1D)                                          \n",
            "                                                                 \n",
            " dense (Dense)               (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the weight metrics of embedding layer\n",
        "embed_weights = model_1.get_layer(\"embedding\").get_weights()[0]\n",
        "print(embed_weights.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BkHTEJHsuXpu",
        "outputId": "b6848746-c0a9-42bb-8eb4-20265957b26b"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 128)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we've got embedding matrix our model has learned to represent our tokens let's visualize it."
      ],
      "metadata": {
        "id": "lYnWOqj_uXnM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating embedding files\n",
        "import io\n",
        "out_v = io.open('vectors.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('metadata.tsv', 'w', encoding='utf-8')\n",
        "\n",
        "for index, word in enumerate(words_in_vocab):\n",
        "  if index == 0:\n",
        "    continue  # skip 0, it's padding.\n",
        "  vec = embed_weights[index]\n",
        "  out_v.write('\\t'.join([str(x) for x in vec]) + \"\\n\")\n",
        "  out_m.write(word + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ],
      "metadata": {
        "id": "INxc6tjluXkQ"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Download files from colab to projector\n",
        "try:\n",
        "  from google.colab import files\n",
        "  files.download('vectors.tsv')\n",
        "  files.download('metadata.tsv')\n",
        "except Exception:\n",
        "  pass"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "zxrtLAZjuXic",
        "outputId": "06e76365-834c-406e-ac20-630e73248372"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_896bb724-9b9d-4766-8b78-8af9e06ce4d7\", \"vectors.tsv\", 15385009)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_7e50ab69-6ded-4714-b38a-2d9df8a083cd\", \"metadata.tsv\", 80388)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Recurrent Neural Network\n",
        "\n",
        "RNN's are used for sequence data.\n",
        "The premise of RNN is to use the representation of a previous input to aid the representation of a later input."
      ],
      "metadata": {
        "id": "fGtFtPP8uXeD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model 2: LSTM\n",
        "\n",
        "LSTM = one of the most popular LSTM cells\n",
        "\n",
        "Our structure for RNN's typically looks like this:\n",
        "```\n",
        "Input (text) -> Tokenize -> Embedding -> Layers(RNN/dense) -> Output(label probability)\n",
        "```"
      ],
      "metadata": {
        "id": "HyiXtl9PuXZ2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create LSTM model\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizor(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.LSTM(units=64, return_sequences=True)\n",
        "x = layers.LSTM(64)(x)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_2 = tf.keras.Model(inputs, outputs, name=\"model_2_LSTM\")"
      ],
      "metadata": {
        "id": "05CLd-cAuXTH"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Compile the model\n",
        "model_2.compile(loss=\"binary_crossentropy\",\n",
        "              optimizer=tf.keras.optimizers.Adam(),\n",
        "              metrics=[\"accuracy\"])"
      ],
      "metadata": {
        "id": "sOIK6aDCuXQv"
      },
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# fit the model\n",
        "history_model_2_LSTM = model_2.fit(train_sentences,\n",
        "                                   train_labels,\n",
        "                                   epochs=5,\n",
        "                                   validation_data=(val_sentences, val_labels),\n",
        "                                   callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                          \"model_2_LSTM\")])"
      ],
      "metadata": {
        "id": "hgbCrJn9uXOG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa5fe86d-eeaa-465f-eaec-608506a30c63"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_2_LSTM/20220915-222313\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 6s 8ms/step - loss: 0.2243 - accuracy: 0.9222 - val_loss: 0.5774 - val_accuracy: 0.7835\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 1s 7ms/step - loss: 0.1578 - accuracy: 0.9416 - val_loss: 0.5800 - val_accuracy: 0.7861\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.1265 - accuracy: 0.9526 - val_loss: 0.7928 - val_accuracy: 0.7808\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 1s 7ms/step - loss: 0.1062 - accuracy: 0.9594 - val_loss: 0.8337 - val_accuracy: 0.7822\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 1s 7ms/step - loss: 0.0865 - accuracy: 0.9661 - val_loss: 1.1276 - val_accuracy: 0.7756\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_2.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "id": "9doKOIfeuXLv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df6508b5-ceff-4a76-d5c3-d963093b97e6"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 3ms/step - loss: 1.1276 - accuracy: 0.7756\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1.1276111602783203, 0.7755905389785767]"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_2_pred_probs = model_2.predict(val_sentences)\n",
        "model_2_pred_probs[:10]"
      ],
      "metadata": {
        "id": "x8lxqVuLKVdw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb907967-1746-48aa-9653-83d7369190cb"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[6.4824376e-04],\n",
              "       [6.6403204e-01],\n",
              "       [9.9980170e-01],\n",
              "       [1.0188251e-03],\n",
              "       [1.7482950e-04],\n",
              "       [9.9863678e-01],\n",
              "       [8.9947730e-01],\n",
              "       [9.9986112e-01],\n",
              "       [9.9973589e-01],\n",
              "       [3.8505879e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))\n",
        "model_2_preds[:10]"
      ],
      "metadata": {
        "id": "U6P76mMVKVbP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2fbb127e-c06d-44df-cea7-fc3486e0826b"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_2_results = calculate_results(val_labels,\n",
        "                                    model_2_preds)"
      ],
      "metadata": {
        "id": "sOPKeEns1Gjl"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_2_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WadkAuPE1Wna",
        "outputId": "1021584b-378e-4317-e6fb-08ae91899368"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.55905511811024,\n",
              " 'precision': 0.7815353725945549,\n",
              " 'recall': 0.7755905511811023,\n",
              " 'f1': 0.7718737954958428}"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "id": "yyzjJed11Y-J",
        "outputId": "a271cbf0-c804-47fa-cc58-1b12306727d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model 3: GRU\n",
        "\n",
        "Has similars features to an LSTM cell but has less params."
      ],
      "metadata": {
        "id": "bbNAhKKU1cVV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizor(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.GRU(64)(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_3 = tf.keras.Model(inputs, outputs, name=\"model_3_GRU\")"
      ],
      "metadata": {
        "id": "cAvsjnxuHMlV"
      },
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_3.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "POAzZorYHMh1",
        "outputId": "d55216db-742e-43ec-91ee-b1636cbdb38f"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_3_GRU\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_3 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (TextV  (None, 15)               0         \n",
            " ectorization)                                                   \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " gru (GRU)                   (None, 64)                37248     \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,317,313\n",
            "Trainable params: 1,317,313\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"adam\",\n",
        "                metrics=[\"accuracy\"])"
      ],
      "metadata": {
        "id": "d81Rxv-IHMfS"
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "history_model_3 = model_3.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                     \"model_3_GRU\")])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d8Yp1ad7HMcn",
        "outputId": "3648270d-66df-4abb-a5f4-b371431d6a03"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_3_GRU/20220915-222326\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 3s 8ms/step - loss: 0.1540 - accuracy: 0.9375 - val_loss: 0.8225 - val_accuracy: 0.7717\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.0825 - accuracy: 0.9701 - val_loss: 0.7263 - val_accuracy: 0.7782\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.0762 - accuracy: 0.9699 - val_loss: 0.9795 - val_accuracy: 0.7598\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.0630 - accuracy: 0.9753 - val_loss: 1.0195 - val_accuracy: 0.7730\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 1s 6ms/step - loss: 0.0496 - accuracy: 0.9787 - val_loss: 1.2009 - val_accuracy: 0.7743\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_pred_probs = model_3.predict(val_sentences)\n",
        "model_3_pred_probs.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XJsMD2SyHMZU",
        "outputId": "3fb34ef6-5eb5-4d71-d490-6ff26572f1f9"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(762, 1)"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_pred_probs[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hTtfYsSLHMTz",
        "outputId": "f8e43d1b-8614-4c10-fbe7-f8b4cb922ab7"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[6.36108685e-04],\n",
              "       [9.14308250e-01],\n",
              "       [9.99909997e-01],\n",
              "       [1.00187391e-01],\n",
              "       [1.10839654e-04],\n",
              "       [9.99806106e-01],\n",
              "       [9.89575624e-01],\n",
              "       [9.99960423e-01],\n",
              "       [9.99907017e-01],\n",
              "       [8.14427555e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))"
      ],
      "metadata": {
        "id": "-fY2R33SHMQ0"
      },
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_preds[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R5jQDQI3HMN2",
        "outputId": "17510cd9-1cd3-4c8a-d431-bec76a2cbd1e"
      },
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_results = calculate_results(y_true=val_labels,\n",
        "                                    y_preds=model_3_preds)\n",
        "model_3_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7j_UfpdTHMLG",
        "outputId": "64e7890e-d503-4b05-b80d-71501c366283"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.42782152230971,\n",
              " 'precision': 0.7752857985262857,\n",
              " 'recall': 0.7742782152230971,\n",
              " 'f1': 0.7725974162749719}"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QQQNCCVcHMII",
        "outputId": "adbf2754-fc04-4014-9a1b-a3d3dec043be"
      },
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model 4: Bidirectional RNN\n",
        "\n",
        "Normal RNN goes from left to right however bidirectional RNN goes left to right as well as left to right."
      ],
      "metadata": {
        "id": "VfFAu9CRHME_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizor(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x)\n",
        "x = layers.Bidirectional(layers.LSTM(64))(x)\n",
        "outputs =  layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_4 = tf.keras.Model(inputs, outputs, name=\"model_4_bidirectional\")"
      ],
      "metadata": {
        "id": "nSWCIn3ZHMCL"
      },
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_4.summary()"
      ],
      "metadata": {
        "id": "BsICUy15HL_P",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61df2684-62b2-48b8-8fc8-9c686d5850c1"
      },
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_4_bidirectional\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_4 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (TextV  (None, 15)               0         \n",
            " ectorization)                                                   \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " bidirectional (Bidirectiona  (None, 128)              98816     \n",
            " l)                                                              \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 1)                 129       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,378,945\n",
            "Trainable params: 1,378,945\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_4.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"adam\",\n",
        "                metrics=[\"accuracy\"])"
      ],
      "metadata": {
        "id": "9-2XgBMlHL8I"
      },
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "history_4 = model_4.fit(train_sentences,\n",
        "                        train_labels,\n",
        "                        epochs=5,\n",
        "                        validation_data=(val_sentences,val_labels),\n",
        "                        callbacks=[create_tensorboard_callback(SAVE_DIR, \"model_4_bidirectional\")])"
      ],
      "metadata": {
        "id": "-SpK7naVHL5U",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a820a49d-7fe3-447f-a1f3-31e541fbf306"
      },
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_4_bidirectional/20220915-222336\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 6s 12ms/step - loss: 0.1090 - accuracy: 0.9672 - val_loss: 0.9014 - val_accuracy: 0.7730\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0536 - accuracy: 0.9766 - val_loss: 1.1459 - val_accuracy: 0.7651\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0446 - accuracy: 0.9793 - val_loss: 1.2301 - val_accuracy: 0.7717\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0459 - accuracy: 0.9809 - val_loss: 1.1616 - val_accuracy: 0.7703\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0456 - accuracy: 0.9787 - val_loss: 1.3460 - val_accuracy: 0.7730\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions\n",
        "model_4_pred_probs = model_4.predict(val_sentences)\n",
        "model_4_pred_probs"
      ],
      "metadata": {
        "id": "kCxeJMmNHL2p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6453ea07-f34c-4dc9-b4d5-f47eaa60411d"
      },
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[7.34928937e-04],\n",
              "       [5.62680781e-01],\n",
              "       [9.99959588e-01],\n",
              "       [1.26831740e-01],\n",
              "       [4.19456883e-05],\n",
              "       [9.98944461e-01],\n",
              "       [9.39269364e-01],\n",
              "       [9.99990344e-01],\n",
              "       [9.99978900e-01],\n",
              "       [9.97304082e-01],\n",
              "       [9.82857382e-05],\n",
              "       [9.94861662e-01],\n",
              "       [1.60009993e-04],\n",
              "       [2.72851557e-01],\n",
              "       [5.77841893e-05],\n",
              "       [3.67007754e-03],\n",
              "       [7.54111970e-05],\n",
              "       [7.20649696e-05],\n",
              "       [1.07771335e-02],\n",
              "       [9.99734938e-01],\n",
              "       [9.99460995e-01],\n",
              "       [8.26300602e-05],\n",
              "       [9.99905348e-01],\n",
              "       [2.87173985e-04],\n",
              "       [9.99753535e-01],\n",
              "       [9.99986172e-01],\n",
              "       [1.43785102e-04],\n",
              "       [4.79823124e-04],\n",
              "       [1.73009656e-04],\n",
              "       [3.09776694e-01],\n",
              "       [9.99182761e-01],\n",
              "       [3.76936223e-04],\n",
              "       [1.07100559e-03],\n",
              "       [4.87984333e-04],\n",
              "       [5.06865978e-01],\n",
              "       [2.42250651e-01],\n",
              "       [9.99969363e-01],\n",
              "       [1.53558090e-01],\n",
              "       [6.40805140e-02],\n",
              "       [9.99945879e-01],\n",
              "       [9.38466012e-01],\n",
              "       [6.99507000e-05],\n",
              "       [2.24461015e-02],\n",
              "       [5.51617668e-05],\n",
              "       [8.70629668e-01],\n",
              "       [9.99921441e-01],\n",
              "       [9.98862505e-01],\n",
              "       [9.39635813e-01],\n",
              "       [4.67227132e-04],\n",
              "       [8.82180810e-01],\n",
              "       [1.57048984e-04],\n",
              "       [4.25054282e-01],\n",
              "       [6.22723214e-02],\n",
              "       [2.98079615e-03],\n",
              "       [4.18777347e-01],\n",
              "       [1.87445676e-03],\n",
              "       [4.01992525e-04],\n",
              "       [9.99893308e-01],\n",
              "       [1.56453811e-04],\n",
              "       [1.20164441e-04],\n",
              "       [5.45650767e-03],\n",
              "       [9.99930143e-01],\n",
              "       [9.99713123e-01],\n",
              "       [3.48630856e-04],\n",
              "       [9.99978065e-01],\n",
              "       [9.99991655e-01],\n",
              "       [9.94565248e-01],\n",
              "       [3.67126554e-01],\n",
              "       [9.96679902e-01],\n",
              "       [1.48498237e-01],\n",
              "       [1.55043870e-03],\n",
              "       [1.27521321e-01],\n",
              "       [9.99964237e-01],\n",
              "       [4.06250404e-03],\n",
              "       [9.97434199e-01],\n",
              "       [9.99183714e-01],\n",
              "       [4.37796954e-03],\n",
              "       [9.99557793e-01],\n",
              "       [1.30010480e-02],\n",
              "       [3.92727554e-02],\n",
              "       [2.90128752e-04],\n",
              "       [7.64932856e-02],\n",
              "       [9.99988198e-01],\n",
              "       [1.95459492e-04],\n",
              "       [1.86619844e-04],\n",
              "       [2.61728768e-04],\n",
              "       [8.00300550e-05],\n",
              "       [1.54568101e-04],\n",
              "       [2.79411767e-03],\n",
              "       [9.99939203e-01],\n",
              "       [9.99974251e-01],\n",
              "       [1.46858321e-04],\n",
              "       [7.95856774e-01],\n",
              "       [1.35584240e-04],\n",
              "       [9.99972820e-01],\n",
              "       [5.18306673e-01],\n",
              "       [3.01265478e-01],\n",
              "       [9.99829292e-01],\n",
              "       [9.99623418e-01],\n",
              "       [9.84991014e-01],\n",
              "       [9.99990106e-01],\n",
              "       [4.26942046e-04],\n",
              "       [2.93674711e-05],\n",
              "       [9.98182058e-01],\n",
              "       [9.99346793e-01],\n",
              "       [1.12304173e-03],\n",
              "       [9.99339879e-01],\n",
              "       [9.99834180e-01],\n",
              "       [9.76501979e-05],\n",
              "       [9.98969793e-01],\n",
              "       [9.97686744e-01],\n",
              "       [2.30985228e-04],\n",
              "       [9.90746379e-01],\n",
              "       [2.84669601e-04],\n",
              "       [5.76142862e-04],\n",
              "       [9.26438719e-04],\n",
              "       [5.47546804e-01],\n",
              "       [9.99856114e-01],\n",
              "       [6.08954486e-03],\n",
              "       [9.23979533e-05],\n",
              "       [9.99992132e-01],\n",
              "       [5.23148447e-05],\n",
              "       [7.33529814e-05],\n",
              "       [9.49008703e-01],\n",
              "       [6.09778159e-04],\n",
              "       [3.01299529e-04],\n",
              "       [9.98829186e-01],\n",
              "       [3.93561713e-05],\n",
              "       [2.11826584e-04],\n",
              "       [9.99526620e-01],\n",
              "       [1.80585019e-04],\n",
              "       [9.99992132e-01],\n",
              "       [9.99990940e-01],\n",
              "       [9.99986172e-01],\n",
              "       [9.99970198e-01],\n",
              "       [2.13831867e-04],\n",
              "       [9.99947071e-01],\n",
              "       [1.88785464e-01],\n",
              "       [1.00048969e-03],\n",
              "       [1.67458624e-04],\n",
              "       [9.99980330e-01],\n",
              "       [9.91904736e-01],\n",
              "       [2.72851557e-01],\n",
              "       [9.99664664e-01],\n",
              "       [1.09315701e-01],\n",
              "       [1.07845245e-03],\n",
              "       [4.23446181e-05],\n",
              "       [3.06001493e-05],\n",
              "       [2.18061842e-02],\n",
              "       [9.99864697e-01],\n",
              "       [2.21836468e-04],\n",
              "       [1.52292964e-03],\n",
              "       [6.74223003e-04],\n",
              "       [1.82630742e-04],\n",
              "       [1.13427603e-04],\n",
              "       [9.99917269e-01],\n",
              "       [9.99365032e-01],\n",
              "       [8.68087634e-04],\n",
              "       [9.99794543e-01],\n",
              "       [8.06766184e-05],\n",
              "       [9.99287784e-01],\n",
              "       [3.16201680e-04],\n",
              "       [7.75856385e-03],\n",
              "       [9.99449193e-01],\n",
              "       [3.33820772e-03],\n",
              "       [7.09061205e-05],\n",
              "       [9.99985218e-01],\n",
              "       [6.95979863e-04],\n",
              "       [9.99973893e-01],\n",
              "       [8.07209730e-01],\n",
              "       [9.99948025e-01],\n",
              "       [9.97300208e-01],\n",
              "       [9.99575794e-01],\n",
              "       [1.00206751e-04],\n",
              "       [9.99982476e-01],\n",
              "       [1.31115041e-04],\n",
              "       [4.82064299e-03],\n",
              "       [6.98631746e-04],\n",
              "       [5.84801793e-01],\n",
              "       [9.99979973e-01],\n",
              "       [1.38903561e-04],\n",
              "       [9.86323237e-01],\n",
              "       [9.99472678e-01],\n",
              "       [9.99988556e-01],\n",
              "       [9.99897599e-01],\n",
              "       [5.51147095e-05],\n",
              "       [1.53063738e-04],\n",
              "       [9.99993801e-01],\n",
              "       [8.31103680e-05],\n",
              "       [6.16952384e-05],\n",
              "       [5.69466176e-03],\n",
              "       [9.98775303e-01],\n",
              "       [8.06524695e-05],\n",
              "       [9.89590917e-05],\n",
              "       [5.40179390e-05],\n",
              "       [2.03246862e-04],\n",
              "       [1.53168861e-04],\n",
              "       [9.98590258e-04],\n",
              "       [9.99677181e-01],\n",
              "       [1.07888971e-03],\n",
              "       [2.39201658e-03],\n",
              "       [9.99975681e-01],\n",
              "       [9.99881387e-01],\n",
              "       [1.28660426e-01],\n",
              "       [1.47934770e-04],\n",
              "       [9.99986291e-01],\n",
              "       [9.99834418e-01],\n",
              "       [9.99851823e-01],\n",
              "       [9.94242609e-01],\n",
              "       [9.97787237e-01],\n",
              "       [5.85834794e-02],\n",
              "       [9.99941945e-01],\n",
              "       [9.33584597e-05],\n",
              "       [2.20890483e-03],\n",
              "       [3.00262291e-05],\n",
              "       [4.19175376e-05],\n",
              "       [9.99935269e-01],\n",
              "       [9.99320507e-01],\n",
              "       [9.91794407e-01],\n",
              "       [3.26621264e-01],\n",
              "       [8.04485902e-02],\n",
              "       [2.76520685e-03],\n",
              "       [8.83158587e-04],\n",
              "       [5.79830026e-04],\n",
              "       [9.99962807e-01],\n",
              "       [9.57638025e-02],\n",
              "       [6.91608060e-04],\n",
              "       [9.99985218e-01],\n",
              "       [9.98901367e-01],\n",
              "       [8.59252870e-01],\n",
              "       [4.73460750e-05],\n",
              "       [4.73186956e-04],\n",
              "       [9.99664187e-01],\n",
              "       [1.36749536e-01],\n",
              "       [2.18260422e-01],\n",
              "       [4.71106381e-04],\n",
              "       [2.02494055e-01],\n",
              "       [1.27294883e-01],\n",
              "       [3.66640423e-04],\n",
              "       [1.49314248e-04],\n",
              "       [9.99461114e-01],\n",
              "       [1.05823847e-02],\n",
              "       [9.99987006e-01],\n",
              "       [9.99941945e-01],\n",
              "       [2.13404957e-04],\n",
              "       [4.82718897e-05],\n",
              "       [9.99786794e-01],\n",
              "       [1.11034198e-04],\n",
              "       [5.70378266e-04],\n",
              "       [1.11041710e-01],\n",
              "       [1.02228769e-04],\n",
              "       [9.99783933e-01],\n",
              "       [2.94370930e-05],\n",
              "       [1.12311507e-03],\n",
              "       [9.99940276e-01],\n",
              "       [6.01668563e-03],\n",
              "       [9.99932766e-01],\n",
              "       [9.99993801e-01],\n",
              "       [1.05094118e-02],\n",
              "       [7.76784218e-05],\n",
              "       [1.72675747e-04],\n",
              "       [1.16006137e-04],\n",
              "       [2.30500355e-05],\n",
              "       [9.99971628e-01],\n",
              "       [9.99971390e-01],\n",
              "       [1.02227658e-01],\n",
              "       [9.99914885e-01],\n",
              "       [1.09014210e-04],\n",
              "       [5.54246595e-04],\n",
              "       [1.03088365e-04],\n",
              "       [1.48579973e-04],\n",
              "       [2.41875096e-04],\n",
              "       [9.99951839e-01],\n",
              "       [6.00720115e-04],\n",
              "       [4.18908021e-05],\n",
              "       [9.99949574e-01],\n",
              "       [9.36892320e-05],\n",
              "       [2.57379405e-04],\n",
              "       [9.99817312e-01],\n",
              "       [1.15644318e-04],\n",
              "       [2.12087226e-03],\n",
              "       [4.51767482e-05],\n",
              "       [9.99960423e-01],\n",
              "       [9.93274927e-01],\n",
              "       [4.68286335e-01],\n",
              "       [1.61576699e-02],\n",
              "       [9.99757111e-01],\n",
              "       [5.48433326e-03],\n",
              "       [9.94431674e-01],\n",
              "       [4.13970426e-02],\n",
              "       [9.99716580e-01],\n",
              "       [9.89264846e-01],\n",
              "       [9.97365296e-01],\n",
              "       [4.22468364e-01],\n",
              "       [9.79892444e-03],\n",
              "       [3.20141733e-01],\n",
              "       [7.89697398e-04],\n",
              "       [1.20854797e-03],\n",
              "       [1.46404200e-04],\n",
              "       [1.42579488e-02],\n",
              "       [4.56895577e-05],\n",
              "       [1.02438280e-04],\n",
              "       [5.10502383e-02],\n",
              "       [9.99949932e-01],\n",
              "       [1.39691474e-04],\n",
              "       [7.21166190e-03],\n",
              "       [6.46603480e-03],\n",
              "       [8.65510479e-02],\n",
              "       [1.78099841e-01],\n",
              "       [1.44652018e-04],\n",
              "       [8.60405999e-05],\n",
              "       [9.99860287e-01],\n",
              "       [2.28757504e-02],\n",
              "       [4.12512660e-01],\n",
              "       [9.99992728e-01],\n",
              "       [7.25687350e-05],\n",
              "       [9.85392511e-01],\n",
              "       [3.58064249e-02],\n",
              "       [4.48154006e-03],\n",
              "       [3.01516373e-02],\n",
              "       [1.42257981e-04],\n",
              "       [3.07023828e-03],\n",
              "       [9.99691248e-01],\n",
              "       [3.93973947e-01],\n",
              "       [9.99969244e-01],\n",
              "       [3.31993580e-01],\n",
              "       [4.88342157e-05],\n",
              "       [9.99974608e-01],\n",
              "       [1.63624078e-04],\n",
              "       [9.99987006e-01],\n",
              "       [5.69735188e-04],\n",
              "       [1.74002285e-04],\n",
              "       [9.99948144e-01],\n",
              "       [1.90550796e-04],\n",
              "       [5.09037091e-05],\n",
              "       [9.99864817e-01],\n",
              "       [6.45459077e-05],\n",
              "       [2.14226951e-04],\n",
              "       [5.41940928e-01],\n",
              "       [9.99640107e-01],\n",
              "       [1.30397864e-04],\n",
              "       [1.76115951e-03],\n",
              "       [9.99963522e-01],\n",
              "       [9.99176800e-01],\n",
              "       [9.98722017e-01],\n",
              "       [1.84121549e-01],\n",
              "       [9.97856081e-01],\n",
              "       [9.94470596e-01],\n",
              "       [5.32695442e-04],\n",
              "       [8.95745543e-05],\n",
              "       [1.15876214e-03],\n",
              "       [2.50831177e-03],\n",
              "       [5.77268511e-05],\n",
              "       [1.65677562e-01],\n",
              "       [2.22114995e-02],\n",
              "       [7.35680442e-05],\n",
              "       [9.99926805e-01],\n",
              "       [9.99986172e-01],\n",
              "       [9.99987364e-01],\n",
              "       [2.37636850e-04],\n",
              "       [2.09484473e-01],\n",
              "       [5.04225045e-02],\n",
              "       [1.68347061e-02],\n",
              "       [9.94758904e-01],\n",
              "       [4.05954719e-01],\n",
              "       [1.02191436e-04],\n",
              "       [1.80318326e-04],\n",
              "       [1.16786192e-04],\n",
              "       [9.99698639e-01],\n",
              "       [1.60572759e-04],\n",
              "       [1.66053025e-04],\n",
              "       [7.86714591e-05],\n",
              "       [1.05254883e-04],\n",
              "       [2.28258267e-01],\n",
              "       [9.94074166e-01],\n",
              "       [2.27112509e-02],\n",
              "       [1.05550178e-04],\n",
              "       [4.10815235e-04],\n",
              "       [1.78250062e-04],\n",
              "       [9.99989748e-01],\n",
              "       [9.99832273e-01],\n",
              "       [9.88099515e-01],\n",
              "       [8.41695547e-01],\n",
              "       [3.62865321e-05],\n",
              "       [9.98353958e-01],\n",
              "       [9.99975562e-01],\n",
              "       [5.83093107e-01],\n",
              "       [3.07266377e-02],\n",
              "       [9.99706924e-01],\n",
              "       [5.77598158e-03],\n",
              "       [9.99825537e-01],\n",
              "       [3.18802804e-01],\n",
              "       [1.48026564e-03],\n",
              "       [9.56411511e-02],\n",
              "       [5.61455777e-03],\n",
              "       [9.99980211e-01],\n",
              "       [6.94645271e-02],\n",
              "       [3.10907380e-05],\n",
              "       [1.39043492e-04],\n",
              "       [4.05954719e-01],\n",
              "       [9.99984741e-01],\n",
              "       [4.29102729e-05],\n",
              "       [1.53897479e-01],\n",
              "       [9.99974132e-01],\n",
              "       [3.64927691e-05],\n",
              "       [9.99992132e-01],\n",
              "       [6.19036873e-05],\n",
              "       [2.96410057e-04],\n",
              "       [6.72732931e-05],\n",
              "       [9.99170303e-01],\n",
              "       [9.99718368e-01],\n",
              "       [3.10435542e-04],\n",
              "       [1.53259156e-04],\n",
              "       [9.98522222e-01],\n",
              "       [9.99977946e-01],\n",
              "       [5.40464759e-01],\n",
              "       [1.14394323e-04],\n",
              "       [1.82235625e-03],\n",
              "       [9.70117390e-01],\n",
              "       [8.90629235e-05],\n",
              "       [9.99981403e-01],\n",
              "       [9.71463859e-01],\n",
              "       [9.99984622e-01],\n",
              "       [9.99973416e-01],\n",
              "       [2.27855192e-03],\n",
              "       [3.00074428e-01],\n",
              "       [1.44537858e-04],\n",
              "       [9.99895453e-01],\n",
              "       [5.31629801e-01],\n",
              "       [9.96741354e-01],\n",
              "       [3.40971281e-04],\n",
              "       [7.06400990e-01],\n",
              "       [2.06874873e-04],\n",
              "       [5.73979960e-05],\n",
              "       [1.62890717e-01],\n",
              "       [3.38576152e-03],\n",
              "       [9.98435199e-01],\n",
              "       [5.80938300e-04],\n",
              "       [9.99992371e-01],\n",
              "       [9.99966621e-01],\n",
              "       [3.00054264e-04],\n",
              "       [5.62680781e-01],\n",
              "       [5.82976732e-04],\n",
              "       [2.11969615e-04],\n",
              "       [3.11880201e-01],\n",
              "       [9.99619126e-01],\n",
              "       [1.35358470e-03],\n",
              "       [3.40531068e-03],\n",
              "       [4.17103074e-05],\n",
              "       [2.35100817e-02],\n",
              "       [8.23468872e-05],\n",
              "       [9.99789536e-01],\n",
              "       [9.99657869e-01],\n",
              "       [9.99976397e-01],\n",
              "       [9.91319716e-01],\n",
              "       [9.96589541e-01],\n",
              "       [5.01277624e-03],\n",
              "       [6.08813389e-05],\n",
              "       [9.94310856e-01],\n",
              "       [9.99800980e-01],\n",
              "       [9.99981523e-01],\n",
              "       [1.52541252e-04],\n",
              "       [3.18357907e-03],\n",
              "       [2.14254949e-04],\n",
              "       [9.99982834e-01],\n",
              "       [9.99986768e-01],\n",
              "       [3.47473804e-04],\n",
              "       [2.21823156e-01],\n",
              "       [9.99983311e-01],\n",
              "       [3.48226604e-04],\n",
              "       [6.50167407e-04],\n",
              "       [9.99901891e-01],\n",
              "       [2.06130993e-04],\n",
              "       [1.28258165e-04],\n",
              "       [9.99415517e-01],\n",
              "       [2.47467220e-01],\n",
              "       [9.87375855e-01],\n",
              "       [9.99979377e-01],\n",
              "       [1.06349944e-04],\n",
              "       [1.23955688e-04],\n",
              "       [5.43170208e-05],\n",
              "       [8.39544809e-05],\n",
              "       [1.02778773e-04],\n",
              "       [9.99974012e-01],\n",
              "       [1.23405989e-04],\n",
              "       [8.37676167e-01],\n",
              "       [9.98433292e-01],\n",
              "       [3.05978153e-02],\n",
              "       [7.89110735e-03],\n",
              "       [6.11013675e-05],\n",
              "       [3.01223714e-04],\n",
              "       [9.99988794e-01],\n",
              "       [9.99912262e-01],\n",
              "       [7.91299623e-04],\n",
              "       [1.56985960e-04],\n",
              "       [1.19848824e-04],\n",
              "       [1.26441533e-04],\n",
              "       [9.99574482e-01],\n",
              "       [8.39485583e-05],\n",
              "       [9.99171495e-01],\n",
              "       [9.99296188e-01],\n",
              "       [8.14830005e-01],\n",
              "       [9.95875061e-01],\n",
              "       [1.53343072e-02],\n",
              "       [7.70620629e-03],\n",
              "       [1.93065658e-04],\n",
              "       [2.32212603e-01],\n",
              "       [6.24422789e-01],\n",
              "       [7.98450530e-01],\n",
              "       [2.18878444e-02],\n",
              "       [2.19418405e-04],\n",
              "       [9.36156503e-05],\n",
              "       [2.28418037e-04],\n",
              "       [3.40768485e-04],\n",
              "       [1.24926995e-02],\n",
              "       [1.15811877e-01],\n",
              "       [9.99989033e-01],\n",
              "       [9.99888659e-01],\n",
              "       [5.18306673e-01],\n",
              "       [9.99807060e-01],\n",
              "       [4.37264377e-03],\n",
              "       [1.56646667e-04],\n",
              "       [9.99894857e-01],\n",
              "       [5.64719143e-04],\n",
              "       [4.43398254e-04],\n",
              "       [3.42691988e-01],\n",
              "       [9.96721327e-01],\n",
              "       [3.81281134e-05],\n",
              "       [9.92686510e-01],\n",
              "       [9.93472338e-01],\n",
              "       [9.97849226e-01],\n",
              "       [9.99631763e-01],\n",
              "       [4.49996296e-04],\n",
              "       [5.33775473e-03],\n",
              "       [9.76548016e-01],\n",
              "       [7.92479259e-05],\n",
              "       [2.42441008e-03],\n",
              "       [3.57468039e-01],\n",
              "       [9.99103367e-01],\n",
              "       [9.52571094e-01],\n",
              "       [3.45655310e-04],\n",
              "       [1.67212100e-04],\n",
              "       [9.97178197e-01],\n",
              "       [1.23966020e-03],\n",
              "       [2.41628484e-04],\n",
              "       [3.25312372e-04],\n",
              "       [3.41818452e-01],\n",
              "       [9.99991536e-01],\n",
              "       [9.99624968e-01],\n",
              "       [3.17124948e-02],\n",
              "       [9.99987125e-01],\n",
              "       [9.99967575e-01],\n",
              "       [5.64267539e-05],\n",
              "       [9.99962568e-01],\n",
              "       [4.30685747e-03],\n",
              "       [9.95910525e-01],\n",
              "       [9.80700627e-02],\n",
              "       [3.13088269e-04],\n",
              "       [1.57902177e-04],\n",
              "       [2.09251069e-04],\n",
              "       [1.93584966e-03],\n",
              "       [1.01834034e-04],\n",
              "       [1.79363057e-01],\n",
              "       [2.40663765e-04],\n",
              "       [9.99973536e-01],\n",
              "       [1.71475374e-04],\n",
              "       [9.95403409e-01],\n",
              "       [9.96770740e-01],\n",
              "       [1.21003739e-03],\n",
              "       [1.68694402e-04],\n",
              "       [9.99991059e-01],\n",
              "       [1.71860040e-04],\n",
              "       [9.99782026e-01],\n",
              "       [3.67159605e-01],\n",
              "       [1.09748042e-03],\n",
              "       [6.93251908e-01],\n",
              "       [5.13920968e-04],\n",
              "       [4.36134636e-03],\n",
              "       [9.99967575e-01],\n",
              "       [4.00437530e-05],\n",
              "       [1.43537298e-04],\n",
              "       [1.52525166e-02],\n",
              "       [9.99984145e-01],\n",
              "       [1.39959902e-03],\n",
              "       [1.76745132e-04],\n",
              "       [9.97870088e-01],\n",
              "       [8.93966353e-05],\n",
              "       [1.09156608e-04],\n",
              "       [2.59600143e-04],\n",
              "       [6.12326205e-01],\n",
              "       [1.94672568e-04],\n",
              "       [4.09980476e-01],\n",
              "       [2.59980828e-01],\n",
              "       [2.42016418e-03],\n",
              "       [6.47402630e-05],\n",
              "       [2.24090647e-02],\n",
              "       [7.25593927e-05],\n",
              "       [9.99930859e-01],\n",
              "       [9.98709798e-01],\n",
              "       [6.73840893e-03],\n",
              "       [6.59037905e-05],\n",
              "       [3.34209122e-04],\n",
              "       [9.99982476e-01],\n",
              "       [9.95250821e-01],\n",
              "       [9.99983072e-01],\n",
              "       [2.86030176e-04],\n",
              "       [9.99540806e-01],\n",
              "       [1.65316189e-04],\n",
              "       [9.99420047e-01],\n",
              "       [9.75427806e-01],\n",
              "       [6.37223493e-05],\n",
              "       [9.99982715e-01],\n",
              "       [4.67311998e-04],\n",
              "       [9.99392152e-01],\n",
              "       [9.99978304e-01],\n",
              "       [5.46665804e-04],\n",
              "       [1.05257088e-04],\n",
              "       [9.96409118e-01],\n",
              "       [6.44018510e-05],\n",
              "       [4.50841635e-02],\n",
              "       [9.99986291e-01],\n",
              "       [3.46933037e-01],\n",
              "       [9.99471009e-01],\n",
              "       [2.37016063e-02],\n",
              "       [9.99902964e-01],\n",
              "       [9.97346282e-01],\n",
              "       [2.88896821e-03],\n",
              "       [9.33251649e-05],\n",
              "       [9.96527374e-01],\n",
              "       [7.61766787e-05],\n",
              "       [2.49953553e-01],\n",
              "       [9.99958396e-01],\n",
              "       [9.99675155e-01],\n",
              "       [9.99989986e-01],\n",
              "       [9.99771655e-01],\n",
              "       [7.94070303e-01],\n",
              "       [9.98389602e-01],\n",
              "       [8.16332817e-04],\n",
              "       [9.87051487e-01],\n",
              "       [9.97335017e-01],\n",
              "       [9.99847531e-01],\n",
              "       [1.00949575e-04],\n",
              "       [9.99542356e-01],\n",
              "       [9.99865055e-01],\n",
              "       [1.23021530e-03],\n",
              "       [1.16905803e-03],\n",
              "       [2.18878444e-02],\n",
              "       [2.93655467e-04],\n",
              "       [9.97955680e-01],\n",
              "       [9.93731916e-01],\n",
              "       [9.99974728e-01],\n",
              "       [2.58266780e-04],\n",
              "       [3.65874112e-05],\n",
              "       [6.35443648e-05],\n",
              "       [1.09238498e-01],\n",
              "       [1.68735941e-03],\n",
              "       [1.30410353e-03],\n",
              "       [9.92750287e-01],\n",
              "       [3.23733257e-05],\n",
              "       [3.67018282e-01],\n",
              "       [7.99131468e-02],\n",
              "       [7.03315309e-04],\n",
              "       [9.99926686e-01],\n",
              "       [6.70025067e-04],\n",
              "       [9.98268366e-01],\n",
              "       [8.55021528e-04],\n",
              "       [3.60958802e-05],\n",
              "       [1.68768252e-04],\n",
              "       [9.99988675e-01],\n",
              "       [9.94118929e-01],\n",
              "       [5.30723446e-05],\n",
              "       [2.75643796e-01],\n",
              "       [9.93111253e-01],\n",
              "       [3.06013499e-05],\n",
              "       [9.99935031e-01],\n",
              "       [1.77553549e-04],\n",
              "       [9.97421741e-01],\n",
              "       [9.36711252e-01],\n",
              "       [8.98253493e-05],\n",
              "       [3.82225782e-01],\n",
              "       [1.39770478e-01],\n",
              "       [1.86806393e-03],\n",
              "       [9.99936223e-01],\n",
              "       [2.78529339e-03],\n",
              "       [2.85091639e-01],\n",
              "       [9.99587238e-01],\n",
              "       [3.81684124e-01],\n",
              "       [4.02349412e-01],\n",
              "       [3.81281134e-05],\n",
              "       [1.87543631e-01],\n",
              "       [8.13946724e-01],\n",
              "       [9.99992132e-01],\n",
              "       [9.68333036e-02],\n",
              "       [9.64417995e-04],\n",
              "       [9.99960423e-01],\n",
              "       [2.98004448e-01],\n",
              "       [9.99668956e-01],\n",
              "       [2.98004448e-01],\n",
              "       [9.99989748e-01],\n",
              "       [3.95954812e-05],\n",
              "       [4.51656245e-03],\n",
              "       [5.97235048e-05],\n",
              "       [9.99991775e-01],\n",
              "       [7.60479306e-04],\n",
              "       [2.86160823e-04],\n",
              "       [7.77198366e-05],\n",
              "       [1.59170770e-04],\n",
              "       [2.35268817e-04],\n",
              "       [1.22761761e-04],\n",
              "       [1.39398267e-03],\n",
              "       [1.30388536e-04],\n",
              "       [1.67518840e-04],\n",
              "       [9.99918699e-01],\n",
              "       [6.79751567e-04],\n",
              "       [6.88096043e-04],\n",
              "       [1.26697152e-04],\n",
              "       [2.91463566e-05],\n",
              "       [4.14940082e-02],\n",
              "       [9.99540567e-01],\n",
              "       [7.56190391e-04],\n",
              "       [1.78674772e-01],\n",
              "       [3.80400405e-03],\n",
              "       [9.99078989e-01],\n",
              "       [9.99928236e-01],\n",
              "       [8.83642424e-05],\n",
              "       [5.88491275e-05],\n",
              "       [1.07745946e-04],\n",
              "       [3.54571421e-05],\n",
              "       [9.99962926e-01],\n",
              "       [2.91463566e-05],\n",
              "       [7.04608683e-04],\n",
              "       [9.98037159e-01],\n",
              "       [9.99981761e-01],\n",
              "       [9.99987245e-01],\n",
              "       [9.99985099e-01],\n",
              "       [9.99989986e-01],\n",
              "       [1.15334638e-04],\n",
              "       [2.53249542e-04],\n",
              "       [9.13494639e-03],\n",
              "       [9.98489261e-01],\n",
              "       [9.99981999e-01],\n",
              "       [6.42326474e-01],\n",
              "       [3.21326545e-04],\n",
              "       [9.99669194e-01],\n",
              "       [4.83289182e-01],\n",
              "       [5.52088168e-05],\n",
              "       [4.63479955e-05],\n",
              "       [9.14497825e-04],\n",
              "       [1.22535729e-03],\n",
              "       [6.18575359e-05],\n",
              "       [4.24425525e-04],\n",
              "       [5.52204132e-01],\n",
              "       [9.99986053e-01],\n",
              "       [1.15084280e-04],\n",
              "       [9.99978781e-01],\n",
              "       [9.97750342e-01],\n",
              "       [6.02944847e-03],\n",
              "       [1.02157658e-03],\n",
              "       [6.81285858e-01],\n",
              "       [4.87167865e-01],\n",
              "       [3.91090522e-04],\n",
              "       [3.92769798e-05]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))"
      ],
      "metadata": {
        "id": "p7xnmKRHHLzX"
      },
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_4_preds[:10]"
      ],
      "metadata": {
        "id": "OB1ukzOcHLwr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6c4c68de-1335-4dcf-dc5f-b98b0dbca5c7"
      },
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_4_results = calculate_results(val_labels, model_4_preds)"
      ],
      "metadata": {
        "id": "Qufop4PBHLtx"
      },
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_4_results"
      ],
      "metadata": {
        "id": "sKCc0m1VHLqx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31d1844b-13ca-4f80-bb36-74e4f921e9f5"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.29658792650919,\n",
              " 'precision': 0.7745161732477092,\n",
              " 'recall': 0.7729658792650919,\n",
              " 'f1': 0.7709704727215051}"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_3_results"
      ],
      "metadata": {
        "id": "Jwa5jZ0AHLnj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e7e2d3af-57ce-4b40-f4f6-c11d01a3a652"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.42782152230971,\n",
              " 'precision': 0.7752857985262857,\n",
              " 'recall': 0.7742782152230971,\n",
              " 'f1': 0.7725974162749719}"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "id": "co83l2quHLkP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "377fbf78-2eba-42fd-9f41-fd174e524aaf"
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model 5: CNN 1D\n",
        "\n",
        "```\n",
        "Inputs(text) -> Tokenization -> Embedding -> Layers (typically Conv1D, Pooling) -> Output (class probabilities)\n",
        "```"
      ],
      "metadata": {
        "id": "BGoOsulAHLhI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# test our embedding layer, Conv1D and max poooling layer\n",
        "embedding_test = embedding(text_vectorizor([\"This is a test sentence\"]))\n",
        "conv_1d = layers.Conv1D(filters=32,\n",
        "                        kernel_size=5,\n",
        "                        activation=\"relu\",\n",
        "                        padding=\"valid\")\n",
        "conv_1d_output = conv_1d(embedding_test)\n",
        "max_pool = layers.GlobalMaxPooling1D()\n",
        "max_pool_output = max_pool(conv_1d_output)\n",
        "\n",
        "embedding_test.shape, conv_1d_output.shape, max_pool_output.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BppXTGypbu85",
        "outputId": "2c5d1e5f-640d-4073-cd7a-ba6fe5a5b890"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([1, 15, 128]), TensorShape([1, 11, 32]), TensorShape([1, 32]))"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "conv_1d_output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JRjPLzg8fEXU",
        "outputId": "5384b925-aef8-4eb2-a53f-b038cd81feec"
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 11, 32), dtype=float32, numpy=\n",
              "array([[[0.        , 0.        , 0.        , 0.        , 0.        ,\n",
              "         0.        , 0.05671427, 0.        , 0.        , 0.05657839,\n",
              "         0.        , 0.04272874, 0.        , 0.        , 0.01265009,\n",
              "         0.01530685, 0.        , 0.        , 0.00670021, 0.        ,\n",
              "         0.03828759, 0.        , 0.07022645, 0.0526075 , 0.02860191,\n",
              "         0.        , 0.03691458, 0.        , 0.03173704, 0.        ,\n",
              "         0.        , 0.00806698],\n",
              "        [0.06563078, 0.        , 0.00265307, 0.01894198, 0.        ,\n",
              "         0.        , 0.02862652, 0.03003649, 0.        , 0.05449942,\n",
              "         0.02766292, 0.04798076, 0.03144095, 0.        , 0.01788286,\n",
              "         0.00604304, 0.        , 0.00323401, 0.04930263, 0.        ,\n",
              "         0.05348131, 0.00108098, 0.04296707, 0.        , 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.04937521,\n",
              "         0.        , 0.        ],\n",
              "        [0.        , 0.        , 0.02684141, 0.08681433, 0.        ,\n",
              "         0.        , 0.01700733, 0.01304037, 0.        , 0.02182348,\n",
              "         0.        , 0.01482597, 0.        , 0.        , 0.        ,\n",
              "         0.        , 0.        , 0.03891912, 0.        , 0.        ,\n",
              "         0.04511645, 0.03138616, 0.04943492, 0.02949298, 0.04275861,\n",
              "         0.        , 0.03989956, 0.        , 0.        , 0.        ,\n",
              "         0.        , 0.        ],\n",
              "        [0.05010298, 0.        , 0.0191797 , 0.03209282, 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.06212537, 0.03173548,\n",
              "         0.04531386, 0.00380419, 0.00378663, 0.        , 0.02919849,\n",
              "         0.03540036, 0.        , 0.        , 0.0552802 , 0.01992498,\n",
              "         0.04312649, 0.01949832, 0.04370002, 0.0246014 , 0.        ,\n",
              "         0.        , 0.        , 0.0018532 , 0.01520715, 0.03696201,\n",
              "         0.        , 0.        ],\n",
              "        [0.04487315, 0.        , 0.01187366, 0.06516935, 0.        ,\n",
              "         0.        , 0.02897119, 0.02189978, 0.03892457, 0.        ,\n",
              "         0.01422691, 0.        , 0.026847  , 0.        , 0.        ,\n",
              "         0.03312023, 0.        , 0.        , 0.02016871, 0.        ,\n",
              "         0.        , 0.06879664, 0.01766114, 0.02371897, 0.        ,\n",
              "         0.04901989, 0.01474225, 0.        , 0.        , 0.        ,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079417, 0.03272855, 0.        ,\n",
              "         0.        , 0.01947135, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312936, 0.        , 0.        , 0.00458979, 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411275,\n",
              "         0.        , 0.03301793, 0.02732301, 0.05910173, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079418, 0.03272856, 0.        ,\n",
              "         0.        , 0.01947136, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312936, 0.        , 0.        , 0.0045898 , 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411276,\n",
              "         0.        , 0.03301793, 0.02732302, 0.05910174, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079417, 0.03272855, 0.        ,\n",
              "         0.        , 0.01947135, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312935, 0.        , 0.        , 0.00458979, 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411277,\n",
              "         0.        , 0.03301793, 0.027323  , 0.05910173, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079417, 0.03272855, 0.        ,\n",
              "         0.        , 0.01947135, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312936, 0.        , 0.        , 0.00458979, 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411276,\n",
              "         0.        , 0.03301793, 0.02732301, 0.05910174, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079418, 0.03272855, 0.        ,\n",
              "         0.        , 0.01947135, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312936, 0.        , 0.        , 0.0045898 , 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411276,\n",
              "         0.        , 0.03301793, 0.02732301, 0.05910174, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ],\n",
              "        [0.02304003, 0.        , 0.01079418, 0.03272856, 0.        ,\n",
              "         0.        , 0.01947135, 0.        , 0.04206143, 0.        ,\n",
              "         0.02312935, 0.        , 0.        , 0.00458979, 0.        ,\n",
              "         0.        , 0.        , 0.        , 0.        , 0.01411276,\n",
              "         0.        , 0.03301793, 0.02732301, 0.05910175, 0.        ,\n",
              "         0.        , 0.02436728, 0.        , 0.        , 0.00381872,\n",
              "         0.        , 0.        ]]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "max_pool_output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DTZXjIpgfEh9",
        "outputId": "ceab711a-569a-44cd-aa5c-5f79723d5f8f"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 32), dtype=float32, numpy=\n",
              "array([[0.06563078, 0.        , 0.02684141, 0.08681433, 0.        ,\n",
              "        0.        , 0.05671427, 0.03003649, 0.06212537, 0.05657839,\n",
              "        0.04531386, 0.04798076, 0.03144095, 0.0045898 , 0.02919849,\n",
              "        0.03540036, 0.        , 0.03891912, 0.0552802 , 0.01992498,\n",
              "        0.05348131, 0.06879664, 0.07022645, 0.05910175, 0.04275861,\n",
              "        0.04901989, 0.03989956, 0.0018532 , 0.03173704, 0.04937521,\n",
              "        0.        , 0.00806698]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "inputs = layers.Input(shape = (1,), dtype=\"string\")\n",
        "x = text_vectorizor(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.Conv1D(filters=64,\n",
        "                  activation=\"relu\",\n",
        "                  padding=\"valid\",\n",
        "                  kernel_size=5)(x)\n",
        "x = layers.GlobalMaxPool1D()(x)\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_5 = tf.keras.Model(inputs, outputs, name=\"model_5_conv1d\")\n",
        "\n",
        "# Compile\n",
        "model_5.compile(loss=\"binary_crossentropy\",\n",
        "              optimizer=\"adam\",\n",
        "              metrics=[\"accuracy\"])\n",
        "#Fit\n",
        "history_5 = model_5.fit(train_sentences,\n",
        "                        train_labels,\n",
        "                        validation_data=(val_sentences,val_labels),\n",
        "                        epochs=5,\n",
        "                        callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                               \"model_5_conv_1d\")])"
      ],
      "metadata": {
        "id": "HZhcG03CHLeh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "033dc502-554d-4401-94d1-fdddc5c4cc30"
      },
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_5_conv_1d/20220915-222357\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 3s 6ms/step - loss: 0.1280 - accuracy: 0.9559 - val_loss: 0.8643 - val_accuracy: 0.7743\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 7ms/step - loss: 0.0736 - accuracy: 0.9731 - val_loss: 1.0470 - val_accuracy: 0.7677\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0615 - accuracy: 0.9755 - val_loss: 1.0980 - val_accuracy: 0.7677\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0530 - accuracy: 0.9790 - val_loss: 1.1900 - val_accuracy: 0.7651\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0513 - accuracy: 0.9777 - val_loss: 1.1735 - val_accuracy: 0.7598\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_5.summary()"
      ],
      "metadata": {
        "id": "u41i85l3HLbn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7fccc163-40ce-44bf-cb82-97f31c40c954"
      },
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_5_conv1d\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " input_5 (InputLayer)        [(None, 1)]               0         \n",
            "                                                                 \n",
            " text_vectorization_1 (TextV  (None, 15)               0         \n",
            " ectorization)                                                   \n",
            "                                                                 \n",
            " embedding (Embedding)       (None, 15, 128)           1280000   \n",
            "                                                                 \n",
            " conv1d_1 (Conv1D)           (None, 11, 64)            41024     \n",
            "                                                                 \n",
            " global_max_pooling1d_1 (Glo  (None, 64)               0         \n",
            " balMaxPooling1D)                                                \n",
            "                                                                 \n",
            " dense_4 (Dense)             (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,321,089\n",
            "Trainable params: 1,321,089\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make Predictions\n",
        "model_5_pred_probs = model_5.predict(val_sentences)\n",
        "model_5_pred_probs.shape"
      ],
      "metadata": {
        "id": "l6E4My33HLYy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a8c7df2e-dfc4-44f3-ae04-de3ed9d28ba5"
      },
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(762, 1)"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_5_preds = tf.squeeze(tf.round(model_5_pred_probs))"
      ],
      "metadata": {
        "id": "_fnS7fGgHLV5"
      },
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_5_preds[:10]"
      ],
      "metadata": {
        "id": "7-keDmbXHLS7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc95da17-238d-48d1-8fae-ffb38b485787"
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_5_results = calculate_results(val_labels, model_5_preds)"
      ],
      "metadata": {
        "id": "j2gPshaDHLQf"
      },
      "execution_count": 84,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_5_results"
      ],
      "metadata": {
        "id": "qAQQPXGXHLNW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bcf80976-47a5-4939-b05d-5fee2ae6759c"
      },
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 75.98425196850394,\n",
              " 'precision': 0.7594533682067887,\n",
              " 'recall': 0.7598425196850394,\n",
              " 'f1': 0.759488581844846}"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "id": "V00yWZBLHLKa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e092f249-3a40-45f0-c318-709285a98996"
      },
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model 6 : Tensorflow Hub pretrained Sentence Encoder"
      ],
      "metadata": {
        "id": "Pa-gY2BEHLHy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_hub as hub\n",
        "embed = hub.load(\"https://tfhub.dev/google/universal-sentence-encoder/4\")\n",
        "embed_samples = embed([sample_sentence,\n",
        "                       \"When you call the USE on a sentence, it turns it into numbers\"])\n",
        "print(embed_samples[0][:50])"
      ],
      "metadata": {
        "id": "EydjuzFDHLEz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3782d436-4d88-40d4-e833-23730249f7c8"
      },
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[-0.01157028  0.0248591   0.02878048 -0.012715    0.03971538  0.0882776\n",
            "  0.02680984  0.05589836 -0.0106873  -0.00597291  0.00639323 -0.01819518\n",
            "  0.00030813  0.09105888  0.05874644 -0.03180628  0.01512474 -0.05162929\n",
            "  0.00991367 -0.06865347 -0.04209306  0.02678981  0.03011006  0.00321069\n",
            " -0.00337973 -0.04787357  0.0226672  -0.00985925 -0.04063613 -0.01292092\n",
            " -0.04666384  0.05630299 -0.03949255  0.00517686  0.02495829 -0.0701444\n",
            "  0.02871508  0.04947684 -0.00633979 -0.08960192  0.02807118 -0.00808364\n",
            " -0.01360602  0.0599865  -0.10361787 -0.05195374  0.00232954 -0.02332531\n",
            " -0.03758105  0.03327728], shape=(50,), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "embed_samples.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pj6pjTqQpADc",
        "outputId": "404797b4-3c13-48f7-9a82-f7df3282ef4d"
      },
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([2, 512])"
            ]
          },
          "metadata": {},
          "execution_count": 88
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a keras layer using a USE pretrained layer from tensorflow hub\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=False,\n",
        "                                        name=\"USE\")"
      ],
      "metadata": {
        "id": "iEywUTcwHLB3"
      },
      "execution_count": 89,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create model using sequential api\n",
        "model_6 = tf.keras.Sequential([\n",
        "    sentence_encoder_layer,\n",
        "    layers.Dense(64, activation=\"relu\"),\n",
        "    layers.Dense(1, activation=\"sigmoid\",name=\"output_layer\")\n",
        "], name=\"model_6_USE\")\n",
        "\n",
        "# Compile\n",
        "model_6.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=\"adam\",\n",
        "                metrics=[\"accuracy\"])"
      ],
      "metadata": {
        "id": "QxCPB6vNHK8C"
      },
      "execution_count": 90,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_6.summary()"
      ],
      "metadata": {
        "id": "KF68A6-PHK4p",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f63dcffe-a360-4eae-9dcb-97cf58af8222"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " USE (KerasLayer)            (None, 512)               256797824 \n",
            "                                                                 \n",
            " dense_5 (Dense)             (None, 64)                32832     \n",
            "                                                                 \n",
            " output_layer (Dense)        (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train a classifier on top of USE pretrained embeddings\n",
        "model_6_history = model_6.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences,val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR,\"model_6_USE\")])"
      ],
      "metadata": {
        "id": "ISaxj2GfHK1t",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ce88e5ff-073a-419f-95c8-de493392ba85"
      },
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_6_USE/20220915-222430\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 6s 17ms/step - loss: 0.5029 - accuracy: 0.7889 - val_loss: 0.4494 - val_accuracy: 0.8031\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.4151 - accuracy: 0.8177 - val_loss: 0.4386 - val_accuracy: 0.8189\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 15ms/step - loss: 0.4016 - accuracy: 0.8228 - val_loss: 0.4318 - val_accuracy: 0.8202\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 15ms/step - loss: 0.3941 - accuracy: 0.8254 - val_loss: 0.4311 - val_accuracy: 0.8097\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 4s 18ms/step - loss: 0.3863 - accuracy: 0.8272 - val_loss: 0.4273 - val_accuracy: 0.8136\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions\n",
        "model_6_pred_probs = model_6.predict(val_sentences)\n",
        "model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))\n",
        "model_6_preds[:10]"
      ],
      "metadata": {
        "id": "qpPSny92HKyy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "986ab868-2519-4ba7-b024-97ee5b6b39d7"
      },
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_results = calculate_results(val_labels,model_6_preds)"
      ],
      "metadata": {
        "id": "Tv0_zcmfHKv4"
      },
      "execution_count": 94,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_results"
      ],
      "metadata": {
        "id": "CH_ss6EJHKs-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3136c0cd-0e93-4c47-d18f-bb541dea838b"
      },
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.36482939632546,\n",
              " 'precision': 0.8141694427619625,\n",
              " 'recall': 0.8136482939632546,\n",
              " 'f1': 0.8127905275621398}"
            ]
          },
          "metadata": {},
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "id": "18xXclWAHKp4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "249991d7-e9e1-4f4f-b15b-4bc935847fb1"
      },
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ohh yeah baby! Baseline model is no more in lead"
      ],
      "metadata": {
        "id": "Rf7LHq-_HKm-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model 7: TF Hub Pretrained USE but with 10% of training data\n",
        "\n",
        "Transfer learning really comeinto play where we don't have much data."
      ],
      "metadata": {
        "id": "QIYV2rKcHKj2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# NOTE: Creating data splits like below could create data leakage problem as we seen in model 7.\n",
        "\n",
        "# Create subsets of 10% of training data\n",
        "# train_10_percent = train_df_shuffled[[\"text\",\"target\"]].sample(frac=0.1, random_state=42)\n",
        "# train_sentences_10_percent = train_10_percent[\"text\"].to_list()\n",
        "# train_labels_10_percent = train_10_percent[\"target\"].to_list()\n",
        "# len(train_sentences_10_percent),len(train_labels_10_percent)"
      ],
      "metadata": {
        "id": "jHx3sMSDHKgY"
      },
      "execution_count": 97,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Making better split\n",
        "train_10_percent_split = int(0.1 * len(train_sentences))\n",
        "train_sentences_10_percent = train_sentences[:train_10_percent_split]\n",
        "train_labels_10_percent = train_labels[:train_10_percent_split]"
      ],
      "metadata": {
        "id": "KA9WbMO_4irO"
      },
      "execution_count": 98,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "pd.Series(np.array(train_labels_10_percent)).value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KJ9vLCOi5sWB",
        "outputId": "c4ed7ce8-db99-4d64-e5d2-5e9b5a52121d"
      },
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    406\n",
              "1    279\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating model same as model 6\n",
        "model_7 = tf.keras.models.clone_model(model_6,)\n",
        "\n",
        "# Compile model\n",
        "model_7.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "# Get summary\n",
        "model_7.summary()"
      ],
      "metadata": {
        "id": "J3CR5M4mHKdF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "489abcbf-805b-486b-f008-0a0821e94eac"
      },
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " USE (KerasLayer)            (None, 512)               256797824 \n",
            "                                                                 \n",
            " dense_5 (Dense)             (None, 64)                32832     \n",
            "                                                                 \n",
            " output_layer (Dense)        (None, 1)                 65        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the model to 10% data\n",
        "model_7_history = model_7.fit(train_sentences_10_percent,\n",
        "                              train_labels_10_percent,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences,val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR,\n",
        "                                                                     \"model_7_10_percent\")])"
      ],
      "metadata": {
        "id": "jnncSvk4HKaE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3721a815-6e45-4afa-aa6b-acef4bcd2414"
      },
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/model_7_10_percent/20220915-222457\n",
            "Epoch 1/5\n",
            "22/22 [==============================] - 3s 43ms/step - loss: 0.6702 - accuracy: 0.6642 - val_loss: 0.6430 - val_accuracy: 0.7467\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 1s 26ms/step - loss: 0.5938 - accuracy: 0.7927 - val_loss: 0.5811 - val_accuracy: 0.7769\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 1s 25ms/step - loss: 0.5150 - accuracy: 0.8175 - val_loss: 0.5275 - val_accuracy: 0.7848\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 0s 22ms/step - loss: 0.4553 - accuracy: 0.8263 - val_loss: 0.4973 - val_accuracy: 0.7795\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 0s 21ms/step - loss: 0.4173 - accuracy: 0.8438 - val_loss: 0.4856 - val_accuracy: 0.7874\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions\n",
        "model_7_pred_probs = model_7.predict(val_sentences)\n",
        "model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))\n",
        "model_7_preds[:10]"
      ],
      "metadata": {
        "id": "aO6MoCmVHKVc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "210578d1-eec2-4da2-a3c9-504951953e8e"
      },
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_7_results = calculate_results(val_labels,\n",
        "                                    model_7_preds)\n",
        "model_7_results"
      ],
      "metadata": {
        "id": "Z0bgVHhuHKR6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "26e49335-b83b-4767-d52a-e7ea079f574d"
      },
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.74015748031496,\n",
              " 'precision': 0.7897494631352898,\n",
              " 'recall': 0.7874015748031497,\n",
              " 'f1': 0.7853521734440729}"
            ]
          },
          "metadata": {},
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_results"
      ],
      "metadata": {
        "id": "_rfKEtVqHKOX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c203d5a0-d42d-40f5-a634-327b343ec51e"
      },
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706,\n",
              " 'f1': 0.7862189758049549}"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Compare the performance of each of our model"
      ],
      "metadata": {
        "id": "ErZGAGx62_D6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Combine model results into a data frame\n",
        "all_model_results = pd.DataFrame({\"baseline\": baseline_results,\n",
        "                                 \"simple_dense\": model_1_results,\n",
        "                                 \"LSTM\": model_2_results,\n",
        "                                  \"GRU\": model_3_results,\n",
        "                                  \"Bidrectional_LSTM\": model_4_results,\n",
        "                                  \"conv1d\": model_5_results,\n",
        "                                  \"tf_hub_use_encoder\": model_6_results,\n",
        "                                  \"tf_hub_use_encoder_10%\": model_7_results})\n",
        "all_model_results = all_model_results.transpose()"
      ],
      "metadata": {
        "id": "xXPXUdOxQJIb"
      },
      "execution_count": 111,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "all_model_results[\"accuracy\"] = all_model_results[\"accuracy\"]/100\n",
        "all_model_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "QsR5hsBMQJGU",
        "outputId": "b7e58b1e-a2d1-480d-cf43-8141bb6ef156"
      },
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        accuracy  precision    recall        f1\n",
              "baseline                0.792651   0.811139  0.792651  0.786219\n",
              "simple_dense            0.784777   0.789165  0.784777  0.781896\n",
              "LSTM                    0.775591   0.781535  0.775591  0.771874\n",
              "GRU                     0.774278   0.775286  0.774278  0.772597\n",
              "Bidrectional_LSTM       0.772966   0.774516  0.772966  0.770970\n",
              "conv1d                  0.759843   0.759453  0.759843  0.759489\n",
              "tf_hub_use_encoder      0.813648   0.814169  0.813648  0.812791\n",
              "tf_hub_use_encoder_10%  0.787402   0.789749  0.787402  0.785352"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-eb936355-2806-4f06-bc54-7705f40c286b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>simple_dense</th>\n",
              "      <td>0.784777</td>\n",
              "      <td>0.789165</td>\n",
              "      <td>0.784777</td>\n",
              "      <td>0.781896</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LSTM</th>\n",
              "      <td>0.775591</td>\n",
              "      <td>0.781535</td>\n",
              "      <td>0.775591</td>\n",
              "      <td>0.771874</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>GRU</th>\n",
              "      <td>0.774278</td>\n",
              "      <td>0.775286</td>\n",
              "      <td>0.774278</td>\n",
              "      <td>0.772597</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Bidrectional_LSTM</th>\n",
              "      <td>0.772966</td>\n",
              "      <td>0.774516</td>\n",
              "      <td>0.772966</td>\n",
              "      <td>0.770970</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conv1d</th>\n",
              "      <td>0.759843</td>\n",
              "      <td>0.759453</td>\n",
              "      <td>0.759843</td>\n",
              "      <td>0.759489</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_use_encoder</th>\n",
              "      <td>0.813648</td>\n",
              "      <td>0.814169</td>\n",
              "      <td>0.813648</td>\n",
              "      <td>0.812791</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_use_encoder_10%</th>\n",
              "      <td>0.787402</td>\n",
              "      <td>0.789749</td>\n",
              "      <td>0.787402</td>\n",
              "      <td>0.785352</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-eb936355-2806-4f06-bc54-7705f40c286b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-eb936355-2806-4f06-bc54-7705f40c286b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-eb936355-2806-4f06-bc54-7705f40c286b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 112
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot and compare all models results\n",
        "all_model_results.plot(kind=\"bar\", figsize=(10,7)).legend(bbox_to_anchor=(1.0,1.0))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 562
        },
        "id": "HWDfEG30QJD5",
        "outputId": "2b3aa681-394e-4c01-fc2d-0a6dcd905219"
      },
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.legend.Legend at 0x7f996f49abd0>"
            ]
          },
          "metadata": {},
          "execution_count": 113
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAIQCAYAAACi4/d6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5xVdb3/8febm4hcUhyviCByGxFFkTQvVF6OlKKpKZhpnoz0pJaXo1QnK7odLe0X5engXUsz81jiJc1KpbJSREFAQFRCyQshgoIIA5/fH2uNboaBGXQz6zus1/PxmAd7fddi7w/7Mcx+z/reHBECAAAAUtKm6AIAAACAhgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkp11RL7zttttGr169inp5AACAZnv88cf/FRE1RddRJoWF1F69emny5MlFvTwAAECz2f5H0TWUDd39AAAASA4hFQAAAMkhpAIAACA5hY1JBQAAaM0ef/zx7dq1a3eNpEHixt/GWiNpel1d3Rn77rvvq41dQEgFAAB4D9q1a3fNDjvsMLCmpmZxmzZtouh6WpM1a9Z44cKFtS+//PI1kkY2dg2pHwAA4L0ZVFNTs5SAuvHatGkTNTU1S5TdhW78mhasBwAAYHPShoD63uXv3XqzKCEVAAAAyWFMKgAAQBX0GnvPvtV8vnn//fHHq/l8rQ13UgEAALBBq1atavHXJKQCAAC0YocddlifPfbYY+Duu+++xw9+8INtJen222/vWltbO7B///61BxxwQD9JWrJkSZsTTjihV79+/Wr79etXe8MNN3xAkjp16jSk/rmuv/76rY8//vheknT88cf3Ovnkk3sOHjx4wFlnndXjwQcf7LT33nsPGDhwYO2QIUMGTJ06dQtJqqur05gxY3r07dt3j379+tV+5zvf2W7ixIldDjvssD71z/vrX/+66+GHH95HG4HufgAAgFbs5ptvnrf99tuvfvPNNz1kyJDak0466fWzzz6710MPPTRrwIABK1955ZW2kjR27Ngdu3btunrOnDkzJWnhwoVtm3rul156qcOUKVNmtWvXTq+99lqbxx57bFb79u31m9/8pstFF13U4/7773/28ssvr5k/f36HmTNnzmjfvr1eeeWVtjU1Nau/+MUv9vznP//Zbqeddqq77rrrup9++un/2ph/FyEVAACgFbv00ku3v+eeez4gSS+//HL78ePH1wwbNuyNAQMGrJSk7bfffrUkTZo0qeutt976XP3fq6mpWd3Ucx933HGL27XL4uJrr73W9qSTTuo9b968jrZj1apVlqQ//vGPXc8888yF7du3V+XrnXjiiYuuvvrqbb7whS8smjJlSuc77rjj+Y35dxFSAQAAWqm77767y8MPP9xl8uTJs7p06bJm2LBh/YcMGbJ89uzZHZv7HLbfefzWW2+58lznzp3X1D+++OKLdx4+fPgbDzzwwLOzZ8/u8NGPfrT/hp73rLPOWvTxj398944dO8bRRx+9uD7ENhdjUgEAAFqp119/vW23bt1Wd+nSZc0TTzzRcerUqVutWLGizaOPPtpl1qxZHSSpvrt/+PDhS3/4wx9uV/9367v7u3fvvmrKlCkdV69erTvvvHPr9b3W0qVL2/bo0WOlJE2YMGHb+vZDDz106YQJE7atn1xV/3q9evVatf3226+6/PLLdxwzZsxGdfVL3EkFAACoiiKWjDr++OOXXHXVVTW77bbbHrvtttuKvfbaa9l2221XN378+Hmf+MQndl+zZo26d+++6pFHHnnme9/73kunn356z759++7Rpk2b+MpXvvLP00477fVvfvObC4455pjdt9lmm7q99tpr+bJlyxq9iXnxxRe/fMYZZ/S+9NJLdzr88MNfr28/77zzFs6ZM2eLAQMG7NGuXbs47bTTFn7lK19ZKEmjRo1adOWVV7bbZ599Vmzsv80RxWyUMHTo0Jg8eXIhrw0AALAxbD8eEUMr26ZOnTpvr7322ug7hGVy6qmn9hwyZMjy8847r9H3aerUqdvutddevRo7x51UAADeh15j72nymnkdT97g+T1792zyOW77Xl2T1wyc9XST1wAtZY899hi45ZZbrpkwYcIL7+Xvb/4h9RvdmnHNkk1fBwAAQInMmDHjff3WxMQpAAAAJKdZIdX2kbZn255re2wj53vaftD2E7an2f5Y9UsFAABAWTQZUm23lXSlpBGSaiWNtl3b4LL/knRbRAyRNErS/1S7UAAAAJRHc+6kDpM0NyKei4iVkm6VdEyDa0JS1/xxN0n/rF6JAAAAKJvmTJzaWVLlrKwXJX2wwTXfkPQ72+dI2krSYY09ke0xksZIUs+eTc9kBAAAaDW+0W3f6j7fkhZfd1WSJk2a1Om6667rfsMNNzQ6K3/evHntzzzzzF3uu+++5xo7Xy3Vmjg1WtINEdFD0sck/cz2Os8dEVdFxNCIGFpTU1OllwYAAMD61NU1vXxZpUMOOWT5+gKqlO0ktakDqtS8kLpA0i4Vxz3ytkqflXSbJEXEXyV1lLStAAAAsMnMnj27Q+/evfcYOXJk7912222PI488crc33nijzc4777znWWedtXNtbe3A6667bus77rij69577z2gtrZ24IgRI3ZbsmRJG0l6+OGHOw0ZMmRA//79a/fcc8+BixcvbnP33Xd3+chHPrK7JN1zzz2dBwwYUDtgwIDagQMH1i5evLjN7NmzO/Tt23cPSVq+fLlPOOGEXv369asdOHBg7V133dVFksaPH9/9iCOO6HPwwQf33XXXXQedeeaZPTb239ac7v7HJPW13VtZOB0lqeGqxPMlHSrpBtsDlYXUhRtbzHvR1CLK8zo2/Rx73rhnk9c8ddpTzS0JAACgxcybN6/jhAkT5h1xxBHLPvnJT/b6/ve/XyNJ3bt3r5s5c+bTL730Urujjz66z6RJk+Z07dp1zVe/+tUdvvWtb23/7W9/++VPfepTfW6++eZnhw8fvvy1115r07lz5zWVz3355ZfvMH78+H8cccQRy5YsWdKmU6dOa1599dV3zl966aXb2dacOXNmPvHEEx0/9rGP9X322WenS9LMmTM7TZ06deaWW265Zvfddx904YUXvrL77ruvau6/q8mQGhF1ts+WdL+ktpKui4gZtsdJmhwREyVdIOlq2+cpm0T1mShqv1UAAJA+Ntupmh122GHlEUccsUySPv3pTy8aP378dpJ06qmnLpakhx56aKtnn32247BhwwZI0qpVq7zvvvu+OW3atI7bbbfdquHDhy+XpG222WZNw+fef//937zwwgt3OfHEE18bPXr04j59+qx1zSOPPNL5nHPOeVWShgwZsmKnnXZa+dRTT3WUpIMOOmhp9+7dV0vS7rvvvuLZZ5/doqohVZIi4l5J9zZou6Ti8UxJBzb3RZEAfjgAADYhejpbju1Gj7t06bJGkiJCBx100NK77rrr+crrHn300S2beu7vfve7Lx977LFL7rzzzm4HH3zwgHvuueeZTp06rRNmG9OhQ4d3bli2bds2Vq1a5Q1d39Dmvy1qlTw9YOAGz7NfMgAALa+pz2dp8/+Mfumllzr8/ve/3+qwww5bdvPNN2/zoQ996M2ZM2d2qj//4Q9/eNkFF1zQc/r06VsMGjTo7aVLl7aZN29e+8GDB6949dVX2z/88MOdhg8fvnzx4sXrdPfPmDFji2HDhr01bNiwtx5//PFO06dP7zhs2LDl9ecPPPDAN3/+859vM3LkyDemTZu2xUsvvdRh8ODBK/7+97930vtESN0MNfXbq1Sd32D57RUAgAoFLRnVq1evFT/+8Y+3GzNmTKe+ffuuuPDCCxdec80129Wf32mnneomTJgwb9SoUbutXLnSkvT1r399weDBg9+++eabnz333HN7rlixok3Hjh3XTJo0aU7lc1922WXbPfLII11tR//+/d864YQTlsyfP799/fmLLrro1VNPPXXXfv361bZt21YTJkyYt+WWW1ZlyCchFe/ZZvnba1PDIBgCAQBITLt27XTnnXeu1ZW/YMGCte4kjRw58o2RI0eu86E8fPjw5VOnTp1V2XbUUUe9cdRRR70hSTfeeOM6S1H1799/5TPPPDNDkjp16hS33377vIbXnHvuuYskLao/fvDBB+du3L+qeuukAgAAAFVDSAUAAGilKu9qbm7o7kdpVGOs7uY407TpGbgNl0Ve1569m97muLW9L01ihQwA2KQIqUCVbZZjdaugtb0vLJ8DAMUipAJAgVjeDgAax5hUAAAAJIc7qQAAAFWw54177lvN53vqtKcKWXd1/Pjx3SdPnrzVTTfdNP/888/fqXPnzqvHjRv3SkvXwZ1UAACAzcCaNWu0evXqosuoGkIqAABAKzV79uwOvXr1GvSJT3yiV79+/fa46KKLdhw0aNDAfv361Z533nk71V/3k5/8pHu/fv1q+/fvX3vsscf2lqRbbrml2+DBgwcMHDiw9kMf+lC/F154Iake9qSKAQCkq3nLuL3/Jctu+15dk8/BhDLgXfPnz9/i2muvfX7JkiWv/epXv9p62rRpT0eEDjvssN1/+9vfdq6pqan7wQ9+sONf//rXWTvuuGPdK6+80laSDj/88DdHjRo1q02bNrriiiu2HTdu3A5XX331i0X/e+oRUgEAAFqxHXfcceWhhx66bMyYMT0mTZrUtba2tlaSli9f3mbWrFkdp0yZ0uboo49evOOOO9ZJ0vbbb79akp5//vkOxx57bI+FCxe2X7lyZZtddtnl7SL/HQ3R3Q8AANCKderUaY0kRYS+9KUvvTRr1qyZs2bNmjl//vzp55133r/W9/fOPvvsnv/xH//x6pw5c2b+5Cc/+cfbb7+dVC5MqhgAAAC8NyNGjFj6s5/9bNslS5a0kaTnn3++/YIFC9r927/929K77rpr65dffrmtJNV397/xxhtte/bsuUqSbrjhhu7FVd44uvsBAACqoKglo+odd9xxS2fMmNFxv/32GyBld1hvvvnm54cOHbriggsueOnggw8e0KZNmxg0aNDy//u//5v31a9+9Z+jR4/u061bt7qDDjrojfnz529RZP0NEVIBAABaqf79+6985plnZtQff+1rX3v1a1/72qsNrzvnnHMWnXPOOYsq20455ZTXTznllNcbXnvuuecukrRIkq644op/boKym4XufgAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOS1ABAABUwdMDBu5bzecbOOvpJtdd/fa3v73dddddV9O3b98Vr7zySvuZM2d2Gjt27IJx48a9Us1aikBIBQAAaKWuvfbamt///vdzOnbsGHPnzu1w++23b110TdVCdz8AAEArdPLJJ/d88cUXtxgxYkTfa665Zpvhw4cvb9++fRRdV7VwJxUAAKAVuuWWW+Y//PDD3R5++OE5O+64Y13R9VQbd1IBAACQHEIqAAAAkkNIBQAAQHIYkwoAAFAFzVkyalOZP39+u/3226922bJlbW3HhAkTtn/66aenb7PNNmuKqun9IqQCAAC0UgsWLHiq/vErr7wyrchaqo3ufgAAACSHkAoAAIDkNCuk2j7S9mzbc22PbeT8D20/mX/Nsf169UsFAABIypo1a9a46CJaq/y9W++Y2SZDqu22kq6UNEJSraTRtmsrr4mI8yJi74jYW9KPJd3xvqoGAABI3/SFCxd2I6huvDVr1njhwoXdJE1f3zXNmTg1TNLciHhOkmzfKukYSTPXc/1oSV/fyFoBAABalbq6ujNefvnla15++eVBYgjlxlojaXpdXd0Z67ugOSF1Z0kvVBy/KOmDjV1oe1dJvSX9cT3nx0gaI0k9e/ZsxksDAACkad99931V0sii69hcVTv1j5J0e0SsbuxkRFwVEUMjYmhNTU2VXxoAAACbi+aE1AWSdqk47pG3NWaUpF+836IAAABQbs0JqY9J6mu7t+0OyoLoxIYX2R4gaWtJf61uiQAAACibJkNqRNRJOlvS/ZKelnRbRMywPc525TiMUZJujYjYNKUCAACgLJq1LWpE3Cvp3gZtlzQ4/kb1ygIAAECZsVwCAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMlpVki1faTt2bbn2h67nmtOtD3T9gzbt1S3TAAAAJRJu6YusN1W0pWSDpf0oqTHbE+MiJkV1/SV9GVJB0bEYtvbbaqCAQAAsPlrzp3UYZLmRsRzEbFS0q2SjmlwzeckXRkRiyUpIl6tbpkAAAAok+aE1J0lvVBx/GLeVqmfpH62/2L7b7aPbOyJbI+xPdn25IULF763igEAALDZq9bEqXaS+kr6sKTRkq62/YGGF0XEVRExNCKG1tTUVOmlAQAAsLlpTkhdIGmXiuMeeVulFyVNjIhVEfG8pDnKQisAAACw0ZoTUh+T1Nd2b9sdJI2SNLHBNb9RdhdVtrdV1v3/XBXrBAAAQIk0GVIjok7S2ZLul/S0pNsiYobtcbZH5pfdL2mR7ZmSHpT0nxGxaFMVDQAAgM1bk0tQSVJE3Cvp3gZtl1Q8Dknn518AAADA+8KOUwAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5zQqpto+0Pdv2XNtjGzn/GdsLbT+Zf51R/VIBAABQFu2ausB2W0lXSjpc0ouSHrM9MSJmNrj0lxFx9iaoEQAAACXTnDupwyTNjYjnImKlpFslHbNpywIAAECZNSek7izphYrjF/O2ho63Pc327bZ3aeyJbI+xPdn25IULF76HcgEAAFAG1Zo4dZekXhExWNIDkm5s7KKIuCoihkbE0Jqamiq9NAAAADY3zQmpCyRV3hntkbe9IyIWRcTb+eE1kvatTnkAAAAoo+aE1Mck9bXd23YHSaMkTay8wPaOFYcjJT1dvRIBAABQNk3O7o+IOttnS7pfUltJ10XEDNvjJE2OiImSzrU9UlKdpNckfWYT1gwAAIDNXJMhVZIi4l5J9zZou6Ti8Zclfbm6pQEAAKCs2HEKAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQ0K6TaPtL2bNtzbY/dwHXH2w7bQ6tXIgAAAMqmyZBqu62kKyWNkFQrabTt2kau6yLpi5L+Xu0iAQAAUC7NuZM6TNLciHguIlZKulXSMY1c9y1Jl0paUcX6AAAAUELNCak7S3qh4vjFvO0dtveRtEtE3LOhJ7I9xvZk25MXLly40cUCAACgHN73xCnbbSRdIemCpq6NiKsiYmhEDK2pqXm/Lw0AAIDNVHNC6gJJu1Qc98jb6nWRNEjSQ7bnSdpf0kQmTwEAAOC9ak5IfUxSX9u9bXeQNErSxPqTEbEkIraNiF4R0UvS3ySNjIjJm6RiAAAAbPaaDKkRUSfpbEn3S3pa0m0RMcP2ONsjN3WBAAAAKJ92zbkoIu6VdG+DtkvWc+2H339ZAAAAKDN2nAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJaVZItX2k7dm259oe28j5M20/ZftJ23+2XVv9UgEAAFAWTYZU220lXSlphKRaSaMbCaG3RMSeEbG3pMskXVH1SgEAAFAazbmTOkzS3Ih4LiJWSrpV0jGVF0TE0orDrSRF9UoEAABA2bRrxjU7S3qh4vhFSR9seJHtL0g6X1IHSR9t7Ilsj5E0RpJ69uy5sbUCAACgJKo2cSoiroyIPpIulvRf67nmqogYGhFDa2pqqvXSAAAA2Mw0J6QukLRLxXGPvG19bpV07PspCgAAAOXWnJD6mKS+tnvb7iBplKSJlRfY7ltx+HFJz1SvRAAAAJRNk2NSI6LO9tmS7pfUVtJ1ETHD9jhJkyNioqSzbR8maZWkxZJO25RFAwAAYPPWnIlTioh7Jd3boO2SisdfrHJdAAAAKDF2nAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOc0KqbaPtD3b9lzbYxs5f77tmban2f6D7V2rXyoAAADKosmQarutpCsljZBUK2m07doGlz0haWhEDJZ0u6TLql0oAAAAyqM5d1KHSZobEc9FxEpJt0o6pvKCiHgwIpbnh3+T1KO6ZQIAAKBMmhNSd5b0QsXxi3nb+nxW0m8bO2F7jO3JticvXLiw+VUCAACgVKo6ccr2KZKGSvp+Y+cj4qqIGBoRQ2tqaqr50gAAANiMtGvGNQsk7VJx3CNvW4vtwyR9VdLwiHi7OuUBAACgjJpzJ/UxSX1t97bdQdIoSRMrL7A9RNIESSMj4tXqlwkAAIAyaTKkRkSdpLMl3S/paUm3RcQM2+Nsj8wv+76kzpJ+ZftJ2xPX83QAAABAk5rT3a+IuFfSvQ3aLql4fFiV6wIAAECJseMUAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASE6zQqrtI23Ptj3X9thGzh9ie4rtOtsnVL9MAAAAlEmTIdV2W0lXShohqVbSaNu1DS6bL+kzkm6pdoEAAAAon3bNuGaYpLkR8Zwk2b5V0jGSZtZfEBHz8nNrNkGNAAAAKJnmdPfvLOmFiuMX87aNZnuM7cm2Jy9cuPC9PAUAAABKoEUnTkXEVRExNCKG1tTUtORLAwAAoBVpTkhdIGmXiuMeeRsAAACwSTQnpD4mqa/t3rY7SBolaeKmLQsAAABl1mRIjYg6SWdLul/S05Jui4gZtsfZHilJtvez/aKkT0qaYHvGpiwaAAAAm7fmzO5XRNwr6d4GbZdUPH5M2TAAAAAA4H1jxykAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAktOskGr7SNuzbc+1PbaR81vY/mV+/u+2e1W7UAAAAJRHkyHVdltJV0oaIalW0mjbtQ0u+6ykxRGxu6QfSrq02oUCAACgPJpzJ3WYpLkR8VxErJR0q6RjGlxzjKQb88e3SzrUtqtXJgAAAMrEEbHhC+wTJB0ZEWfkx5+W9MGIOLvimun5NS/mx8/m1/yrwXONkTQmP+wvaXa1/iHv07aS/tXkVeXD+7Iu3pPG8b40jvelcbwv6+I9aVxK78uuEVFTdBFl0q4lXywirpJ0VUu+ZnPYnhwRQ4uuIzW8L+viPWkc70vjeF8ax/uyLt6TxvG+lFtzuvsXSNql4rhH3tboNbbbSeomaVE1CgQAAED5NCekPiapr+3etjtIGiVpYoNrJko6LX98gqQ/RlPjCAAAAID1aLK7P8qRA9cAAB1DSURBVCLqbJ8t6X5JbSVdFxEzbI+TNDkiJkq6VtLPbM+V9JqyINuaJDcEIRG8L+viPWkc70vjeF8ax/uyLt6TxvG+lFiTE6cAAACAlsaOUwAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJKdFF/NPje2DJPWNiOtt10jqHBHPF11XkWx3knSBpJ4R8TnbfSX1j4i7Cy6txdk+f0PnI+KKlqolJba3adAUkl4v+7JztvfZ0PmImNJStaTC9o+VfX80KiLObcFykmK7raTfR8RHiq4lZbYPldRJ0n0RsaroetCyShtSbX9d0lBl27NeL6m9pJ9LOrDIuhJwvaTHJR2QHy+Q9CtJpQupkn4g6UlJv5X0tiQXW04yHlcWPCrfj862p0o6IyLmFVJV8SZLmq53t3CsfH9C0kdbvKLiTc7/PFBSraRf5seflDSzkIoSERGrba+x3S0ilhRdT4psXy5piaQ1ks6S9LFiK0JLK21IlfQJSUMkTZGkiPin7S7FlpSEPhFxku3RkhQRy22XNZwNkTRa0seVBbNfSPpD2e8YRkTvxtptHyfpfyUd2bIVJeN8ZZuZvCXpVkm/jog3iy2pWBFxoyTZPkvSQRFRlx//r6Q/FVlbIt6U9JTtByQtq28s6x3mPJR+KyJez5t6Sjoxf/xUMVWhSGUek7oyDxshSba3KrieVKy0vaXefV/6KLuLWDoRMTUixkbE3so2rDhG0kzbIwsuLUkRcYek7YquoygR8f8i4iBJ5yjbJvoPtm+zvXfBpaVga0ldK447521ld4ekr0mapOwX4fqvsrpD0q22z82HQ9wk6UFJf5V0daGVoRBlvpN6m+0Jkj5g+3OS/l38J5Ckr0u6T9Iutm9W1k33mUIrKlg+XnmIpD0lvSjp1WIrSpPtzir3L76SpIh4zvadkraU9GlJ/ZQNGymz/5b0hO0HlQ2DOETSNwqtKAERcWN+U6BnRMwuup6iRcRfJB1p+xRlu1yOj4gPF1sVilTqHadsHy7pCGU/NO+PiAcKLikJtrtL2l/Z+/K3iPhXE39ls2T735V1NXWUdLuk2yKi9AF1PRPKtpY0UtJPIqKUv+zZ3k3ZltDHSHpBWZf/PRHxVqGFJcL2DpI+mB/+PSJeLrKeFNg+WtnY9w4R0Tu/6z4uIkrZW2O7naR/k7RK0iOSzpO0n6SvRcTUImtDMUodUrEu2wdKejIiluW/ze4j6UcR8Y+CS2txttcomwhT/29f6z9LiT9Ivt6gKSQtkjQpIko7biz/fpkm6U5JS7Xu90spV4NYH9sDImJW0XUUyfbjyibUPRQRQ/K26RExqNjKimH7bmVd+50k9YiI02zvJGmcpIiIzxVaIFpcabv780kelyobQ+f8KyKi6wb/4ubvp5L2sr2Xsokg1yobFzS80KqKwdIwjYiIb67vnO2eETG/JetJSOX70rmwKlqP3ymbGFNmqyJiSYO5qWuKKiYBu0bEUbY7SPqblE1qlnQGY7vLqbQhVdJlko6OiKeLLiQxdRERto+RdGVEXGv7s0UXVZDTI+IzRReRItsHSNpZ2d3TV20PljRW0sHKJg2V0aKI+EnRRaTE9vj1nZL0gZasJVEzbJ8sqW2+JvW5yrq5y2qC7b/mj9fqeYiIso/rLqXSdvfb/ktElH1N1HXYfljZxKnTlU1ueFXS1IjYs9DCCmB7SkRscIH2MrL9fUlHKZsMtLuyCQ5nSPqepAkRsaLA8grD98u6bL+hbHOQxlYIuTwitm3hkpKSb57yVVXMjVC2BFMp/w8BDZU5pP5I0g6SfqOKH6D5MjqllU9uOFnSYxHxJ9s9JX04Im4quLQWZ3uWsnVSG10ntow7CEmS7ZmS9omIFba3VjZJaFCJF/GXREhtjO0/SvqviFjn7qDt59e35i7KKZ849VlJxyrrqZGyDWXulHQtO06VT5lD6vWNNEdE/HuLF4Mk5XeBHlPjITUioow7CK0Txmw/UT/po8xs10la3tgplXS8e76F7oqIaOx9KS3bd2nD28WWdVLmLyS9LulGZcv9SVIPSadJ2iYiTiqqNhSjtCEVjWNC2bsIX42z/bqyxcfrHZIf13+vlPUDlu+X9ch/rtwTEaXcGKQh2/UTUY9T1qP38/x4tKRXIuK8QgormO05EdFvY89h81W6iVO2L4qIy2z/WI38JlvW7egqMKEMTTmmwfHlevf/Ulm30MWGHS3ph7YnSfqlpPvqt0gto4h4WMq2AY2IoRWn7rI9uaCyUvCa7U9K+r+IWCNJtttI+qSkxYVWhkKULqRKqg9fZf5BsCGvEFDfcXHlge32kgZJWlDyRf0/oGwNwyslyfajkmqUBdWLN/QXN3O/KrqAVEXE6fn/nxHK7hZeafuBiDij4NKKtpXt3SLiOUmy3VtSmbfoHqWsJ+9/bNeH0g8o2xp1VGFVoTB092MtTCh7l+3/lfTjiJhhu5uyRaZXS9pG0oUR8YtCCyyI7b9IGhURL+THT0o6VNmH6/URcWiR9RUl3175oYh4xtnCl9dJOl7SPEmfKetEu0p5UD1S+eohzO73kZKukvScsl6IXSWNiYjfFVpYAvKdDxURi4quBcUp3Z1UBqw3qauyyR9HVLSFpNKFVEkHR8SZ+ePTJc2JiGPzFRB+K6mUIVXZFo4vVBz/Of8gWWS7zHeBvijphvzxaEmDJfWWNETSj5StIVtKtkdIOknShyU9JOkaZVsOl1pE3Jevjzogb5rFuN1Mw3Bq+3C2Li+f0oVUZfskYz0i4vSia0jIyorHhyvvzo2IlxvsEFM2W1ceRMTZFYc1LVxLSuoqlsg5StJN+Qft721fVmBdKThV2VjUzxPC3pXfWf68ssmHkvSQ7QkstdSoa8UOZaVTupBaP2BdkmxvKalnRMwusKSk2O6nbGvU7SNiUL6T0MiI+HbBpRXhddtHKVun70Bl6/fVr+W3ZZGFFezvtj8XEVdXNtr+vKRHC6opBWts76hsgsehkr5Tca7M3y+KiNFF15Con0pqL+l/8uNP522lHKtre+L6Tknq3pK1IA2lC6n1bB+t7K5qB0m9832Bx9Hdr6sl/aekCZIUEdNs3yKpjCH185LGKxuj+6WIeDlvP1TSPYVVVbzzJP0m386xfpzlvpK2ULYId1ldomxCZltJEyNihvTOckPPFVlY0Vjabr32i4i9Ko7/aHtqYdUU72BJp0h6s0G7JQ1r+XJQtNKGVEnfUPZN/5CU7Qucz6wsu04R8WiD7uxSLhUTEXOUTfJo2H6/7YEFlJSEfGWDD9n+qKQ98uZ7IuKPBZZVuIi42/aukrpEROVyOZMlfbegslLB0naNW227T0Q8K0m2d1M2ObOs/iZpeWWPZz3b9HiWUJlD6qqIWNIgjLHUgfQv232Uvxe2T5D0UrElJel8Sf+v6CKKlIfSUgfThvK1Pxc3aFuW76RT5vF0LG3XuP+U9KDtytn9pZ0XEBEjNnDukPWdw+arzCF1Rt5d2TafXXmupHX2ly6hLyhbEmWA7QWSnlfW/YK1lXrmFDZa2b9fJtv+pVjabi0R8Yf886d/3jSbiWVNs/3XiDig6Dqw6bUpuoACnaOsq/JtZUsJLZX0pUIrSkBEPBcRhymbpT0gIg6KiHkFl5Ui7rpjY5T9+6Vyabuj86+jCq0oAba/IGnLiJgWEdMkdbL9H0XX1Qp0LLoAtAwW85dku62krSJiadG1FMX2+Rs6HxFXtFQtqbD9hhoPF1b2wVLmngg0sIE1mC3poxFR5jVk0QjbT0bE3g3anoiIIUXV1BrYnhIR+xRdBza90n7I5jPWz1Q2SP0xSV1t/ygivl9sZYXpkv/ZX9J+kuqXAjlaJV1WKCK6NH0V8I4NrcFc6vWZbfeQ9GNlS7lJ0p8kfTEiXiyuqiS0te3I7xblN0w6FFwTkIzS3kmt/w3W9qck7SNprKTHI2JwwaUVyvYkSR+PiDfy4y7KZm4zaB2oAtv/FxHHF11HS7L9gKRbJP0sbzpF0qci4vDiqiqe7e8rmyw1IW/6vKQXIuKC4qpKH3eby6PMY1Lb57t9HKtsTcNVYtyYJG2vtXdaWpm3AaiO3YouoAA1EXF9RNTlXzeo3LuT1btY0oOSzsq//iDpokIrKpjttrYfbOKyT7dIMShcabv7lf3mOk/SVEmT8vUNSzsmtcJNkh61/ev8+Fi9ux85gPevjL8ML7J9irJJqpI0WtKiDVxfChGxRtkOUz8tupZURMRq22tsd4uIJeu5ZnpL14VilLa7vzG22+XrHJaa7X2U7fwhSZMi4omKc1s3WKgcwEYo46SP/CbAjyUdoCykPyLpnIh4odDCCmb7QGUby+yq7KZR/U5cZbzb/g7bd0oaIukBScvq2yPi3MKKQiFKHVJtf1zZMlTvLGcREeOKqyh9ZfyABaqpjOPpbN+obGvhxfnxNpJ+EBH/XmxlxbI9S9k2w4+rYqepiCj1XWbbpzXWHhE3tnQtKFZpu/tt/6+kTpI+IukaSSeopLPYN1LZFyUH3q+Liy6gAIMre2Ai4jXbpQrq67EkIn5bdBGpiYgbbW8pqWdEsB1qiZU2pEr6UEQMtj0tIr5p+3JJ/LBoWnlvvQMbYPsprX+d1KhfOSQifteihaWhTeVQofxOapk/f+o9mM/wv0Nr78Q1pbiSimf7aGXLtnWQ1Nv23pLGRcTIYitDSyvzD4m38j+X295J2SD+HQusB0DrVvodlDbgckl/tf2r/PiTkr5TYD2p+GD+59CKtpD00QJqSck3JA2T9JAkRcSTtks9TresyhxS77b9AUmXKRsPJGXd/tgwuvuBRkTEP4quIVURcZPtyXo3fB0XETOLrCkFEfGRomtI1KqIWGKv9XGzpqhiUJzSTpzKx7ucpWwWeyjbAeWnEbGi0MISYPsgSX0j4nrbNZI6R8Tz+bltIuK1YisE0mV7f2Uz2Qcq665sK2lZRHQttDAkx/b2kr4raaeIGGG7VtIBEXFtwaUVyva1ytaMHSvpeEnnSmofEWcWWhhaXJkX879R2cz+8co+UGqVrRFaara/rmxix5fzpvaSfl5/noAKNOknytYBfUbSlpLOkHRloRUhVTdIul/STvnxHElfKqyadJyj7PP5bWVr6y4V70splflO6syIqG2qrWxsP6lsfbop9cvk5JPLSr1dLNBctidHxNDK/zdlXHYKTbP9WETsV/n9Ub9ld9G1ASko85jUKbb3j4i/SZLtD0qaXHBNKVgZEWE7JMn2VkUXBLQyy213kPSk7cskvaRy91ph/ZbZ7q58VYh8qEijuyyVge27tIEVZJjdXz6lC6kVy8S0l/SI7fn58a6SZhVZWyJusz1B0gdsf07Sv0u6uuCagNbk08pC6dnKFmrfRdm4OqCh8yVNlNTH9l8k1Shbs7usfpD/eZykHfTuULPRkl4ppCIUqnTd/fn2fOvFDF3J9uGSjlA2k//+iHig4JIAYLNku52k/sp+3s6OiFUV5w4v48/f+iEzTbVh81e6kAoAm1Ij+7FLksq+Hzs2Xlm3obb9tKSPR8Rz+XFvSfdGxMBiK0NLK113Pxpn+w1teLccls8BmudaNbIfO/AelHVd6vMkPWT7OWXvwa6SxhRbEopASIUkKSK6FF0DsJlgP3ZUSym7OiPiPtt9JQ3Im2ZFxNsb+jvYPBFSsQ7b+0g6SNkPyD9HxBMFlwS0JuzHDrwPtttL+rykQ/Kmh2xPqByvi3JgTCrWYvsSZftq35E3HSvpVxHx7eKqAloP2w820hwRUfb92LGRbN8REccVXUdLs32NshV4bsybPi1pdUScUVxVKAIhFWuxPVvSXvXbw+bbxz4ZEf2LrQwANi+2O0m6QFLPiPhc3sXdPyLuLri0QtmeGhF7NdWGzR8LTKOhf0rqWHG8haQFBdUCtDq2u9m+wvbk/Oty292KrgtJul7ZkJAD8uMFkui1klbb7lN/YHs3MQmxlBiTioaWSJph+wFlY1IPl/So7fGSFBHnFlkc0ApcJ2m6pBPz408rCyOl67ZFk/pExEm2R0tSRCy3XdYZ/ZX+U9nY7srZ/acXWxKKQEhFQ7/Ov+o9VFAdQGvVJyIqd5j6pu0nC6sGKVuZD6mq3xa1jyom25VVRPyhfuhD3jSb2f3lREjFWiLixqavArABb9k+KCL+LL2zuP9bBdeENH1d0n2SdrF9s6QDJX2m0IoSYPsLkm6OiGn58da2PxsR/1NwaWhhTJzCWmwfJelbene3HBbzBzaC7b2VzUrupuz/z2uSPhMRUwstDEmy3V3S/sq+V/4WEf8quKTC2X4yIvZu0PZERAwpqiYUg5CKtdieq2zs3FPBNwfwntnuKkkRsbToWpCm/C77kxGxzPYpkvaR9KOI+EfBpRXK9lOSBtd/BtluK2laROxRbGVoaXT3o6EXJE0noAIbx/YpEfFz2+c3aJckRcQVhRSGlP1U0l6295J0vrItdW+SNLzQqop3n6Rf2p6QH38+b0PJEFLR0EWS7rX9sNbeLYcPWGDDtsr/bGyLYX7pQ2PqIiJsHyPpyoi41vZniy4qARcrC6Zn5ccPSLqmuHJQFLr7sRbbv5P0pqSnJK2pb4+IbxZWFNCK2D4wIv7SVBuQ3wy4T9nySodIelXS1IjYs9DCgEQQUrEW29MjYlDRdQCtle0pEbFPU22A7R0knSzpsYj4k+2ekj4cETcVXFqh8rG639C6E3h3K7IutDxCKtZi+zJJv4+I3xVdC9Ca2D5A0ockfUnSDytOdZX0CbZ0BJrH9ixJ50l6XBU7TUXEosKKQiEYk4qGzpJ0oe23Ja0SS1ABzdVBUmdlP1crx6UulXRCIRUhabbf0LvjlTtIai/pzYgo+za6SyLit0UXgeJxJxUAqsj2rmVfQggbL98O9RhJ+0fE2KLrKZLt/5bUVtIdWnsC75TCikIhCKmQJNkeEBGzbDc6bo4fDkDz2H5A0icj4vX8eGtJt0bEvxVbGVoDFq2XbD/YSHNExEdbvBgUiu5+1Dtf0hhJl1e0Vf4Gww8HoHm2rQ+okhQRi21vV2RBSJPt4yoO20gaKmlFQeUkIyI+UnQNSAMhFZKkiBiTP/yppPsiYqntrynbAeVbxVUGtDprbPeMiPlS1v0v1klF446ueFwnaZ6yLv9Ss729pO9K2ikiRtiulXRARFxbcGloYXT3Yy22p0XEYNsHKQunP5B0SUR8sODSgFbB9pGSrpL0sLKJhwdLGhMR9xdaGFod21+OiO8VXUdLs/1bSddL+mpE7GW7naQnWD+2fNoUXQCSU7/cx8clXR0R9yibdQqgGSLiPmU9EL+UdKukfQmoeI8+WXQBBdk2Im5TvqFMRNSpYikqlAchFQ0tyPdLPknZ9qhbiO8ToNnyWdpHStonIu6W1Mn2sILLQuvkogsoyDLb3ZUPk7G9v6QlxZaEItDdj7XY7qTsA/apiHjG9o6S9mRxf6B5bP9U2R2gj0bEwHx2/+8iYr+CS0MrU9adyvJVZn4saZCk6ZJqJJ0QEdMKLQwtjpAKAFVUHywqlxKyPZUdp7CxyrwcVT4Otb+yu8mzI2JVxbnDI+KBwopDi6EbFwCqa5Xttnq3q7JG+dg6YCP9qugCihIRdRExIyKmVwbU3KWFFIUWR0gFgOoaL+nXkraz/R1Jf1a2nA6wFtu72b7L9r9sv2r7Ttu71Z+PCL5vGlfWsbqlwzqpAFAltttIel7SRZIOVfZhemxEPF1oYUjVLZKulPSJ/HiUpF9IYsm/DWOcYkkwJhUAqqjM4wixcerXpW7QxvjlJpR1QlkZ0d0PANX1B9vH50tRAeuwvY3tbST91vZY271s72r7Ikn3Fl1fKzCv6ALQMriTCgBVZPsNSVsp2+ZyhbIu/4iIroUWhmTYfl5Zl3Vjv8hEROzWSHtp5EshXiCpZ0R8znZfSf3zdYdRIoRUAACQDNu/lPS4pFMjYlAeWh+JiL0LLg0tjIlTAFAF+QLk6xURU1qqFrQOtk9trD0ibmrpWhLTJyJOsj1akiJiOcNnyomQCgDVcXn+Z0dJQyVNVdadO1jSZEkHFFQX0lW5C1lHZStCTJFU9pC60vaWenet4T6S3i62JBSBkAoAVRARH5Ek23dI2icinsqPB0n6RoGlIVERcU7lse0PSLq1oHJS8nVJ90naxfbNkg6U9JlCK0IhGJMKAFVke0ZE7NFUG9CQ7faSpkdE/6JrKZrt7pL2V9Yb8beI+FfBJaEAhFQAqCLbv5C0TNLP86ZPSeocEaOLqwopsn2X3l2Yvo2kWkm3RcTY4qoqnu0DJT0ZEctsnyJpH0k/ioh/FFwaWhghFQCqyHZHSWdJOiRvmiTppxGxoriqkCLbwysO6yT9IyJeLKqeVNieJmkvZeO5r5d0raQTI2L4Bv8iNjuEVAAAkIz6HaVsXyJpQURcyy5T5cTEKQCoAtu3RcSJtp9SI3uLN9z+ErB9nKRLJW2nbOwlGz9k3rD9ZUmnSDrEdhtJ7QuuCQXgTioAVIHtHSPiJdu7Nnae8XRoyPZcSUdHxNNF15IS2ztIOlnSYxHxJ9s9JX2Y9WPLh5AKAJuI7W0lLQp+0KIRtv8SEQcWXQeQKkIqAFSB7f0l/bek1yR9S9LPJG2rbNb2qRFxX4HlISF5N78kDZe0g6TfqGKx+oi4o4i6UmH7Db07ZKaDsq7+NyOiW3FVoQiMSQWA6viJpK9I6ibpj5JGRMTfbA+Q9Atli5MDknR0xePlko6oOA5JpQ6pEdGl/nG+HeoxytZMRclwJxUAqsD2kxGxd/746YgYWHHuiYgYUlx1aI1sfzkivld0HSng/9D/b++OXa+swjiAf59qMNCfGg7SlkvQoBA1O7S4tAjSFtRmu6T/QOJ/4BY0lCRE4eLk2KaQDg0FFbQUofSTiEB6Gu4VLvJDKew97/X9fOBy33vO8uVOD+c857zLZCUV4On4e+P5z0fmrAbwX5xJsrgidaMdIlm1y7yRxD3DC6RIBXg6TlTVblbXCL24fs76975xsdhiNTrAIJvtEA+S/JjVlj8Lo0gFeAq6+/nRGXjmLHIFvrvfe9y8NojleG50AABgT0tdSX2SM6MDMA1FKgBMqKourb+fVGxdnSDONlK8L4TT/QAwofWrc48nuel99P9eVd3yvy2DnlQAmNb1JPeS7N84YJesVgi7u3fGxNoaVlIXwnY/AEyou89196EkN7p7Z+NzIMnl0fm2gDaIhVCkAsAYR/YYOzV5ipmpqmNVda2qfquqX6vqq6o69nC+uz8amY/pKFIBYEJVdXbdl/pqVd3e+PyQ5M7ofDPwaZLPkxxN8nJWK6efDU3EEA5OAcCEqupgksNZvU3q/MbU/e6+OybVfFTV7e4+/sjYN919YlQmxlCkAgDDVdVL68cPszpYdiWrFxq8k+Rwd18YlY0xFKkAwHDrdofO3qf3u7uP7THOM0yRCgDA7LgnFQCYjap6d6/x7v5k6iyMpUgFAObkzY3nfUneSnIriSJ1YWz3AwCzVVWHklzp7sXfIbs07kkFAObsjySvjA7B9Gz3AwCzUVXXsjrln6wW017L6nJ/FsZ2PwAwG1V1cuPngyQ/dffPo/IwjiIVAIDZ0ZMKAMxGVZ2uqu+q6veq2q2q+1W1OzoX07OSCgDMRlV9n+Tt7v52dBbGspIKAMzJLwpUEiupAMAMVNXp9ePJJEeTfJnkr4fz3f3FiFyMo0gFAIarqo8fM93d/f5kYZgFRSoAsDWq6kJ3Xxydg/+fnlQAYJucGR2AaShSAYBtUqMDMA1FKgCwTfQpLoQiFQDYJlZSF0KRCgAMV1WX1t9P6jm9OkEcZsDpfgBguKq6k+R4kpvd/froPIz3wugAAABJrie5l2R/Ve1ujFdW96TujInFKLb7AYDhuvtcdx9KcqO7dzY+B5JcHp2P6SlSAYA5ObLH2KnJUzCc7X4AYLiqOpvkgyTHqur2xtSBJF+PScVIDk4BAMNV1cEkh5NcTHJ+Y+p+d98dk4qRFKkAAMyOnlQAAGZHkQoAwOwoUgEAmB1FKgAAs/MPBSxptoz/OxgAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(\"f1\", ascending=False)[\"f1\"].plot(kind=\"bar\", figsize=(10,7));"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 545
        },
        "id": "VW0NEGfBQJB3",
        "outputId": "1a8f94cd-d623-45ff-87ac-2321f3890509"
      },
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAIQCAYAAACljesqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7xndV3v8dfbQfLCTWPKlDuHUFJQGlHD1DQ6kArmJaHUvGKe0BKPhadCo9NFSztaZJBEmCmix3JUBE0xy7QYkIuA2BzUgC6OqKh4wZHP+WOtLT82e8/eM9/NrLVZr+fjsR/zWxdmv/k99p793mt91/ebqkKSJEnb5i5DB5AkSVrNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNdhjqE+++++61zz77DPXpJUmSlu2iiy76YlWtXejYYGVqn332YcOGDUN9ekmSpGVL8vnFjnmbT5IkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqYFlSpIkqcEOQwdotc9J7xs6AgCf+/3HDx1BkiQNwCtTkiRJDZZVppIcmeTqJBuTnLTA8b2SXJDkk0kuS/LTKx9VkiRpfJYsU0nWAKcCRwEHAcclOWjeab8BnFNVDwGOBf50pYNKkiSN0XKuTB0GbKyqa6rqZuBs4Jh55xSwS/96V+DfVy6iJEnSeC2nTN0PuHZm+7p+36xXAc9Ich1wLvDihf6iJMcn2ZBkw6ZNm7YhriRJ0ris1AD044C/rKo9gJ8G/irJ7f7uqjq9qtZV1bq1a9eu0KeWJEkaznLK1PXAnjPbe/T7Zj0POAegqj4O3A3YfSUCSpIkjdlyytSFwAFJ9k2yI90A8/Xzzvk34HEASR5AV6a8jydJku70lixTVbUZOAE4H7iK7qm9K5KckuTo/rSXAS9IcinwNuDZVVV3VGhJkqSxWNYM6FV1Lt3A8tl9J8+8vhI4fGWjSZIkjd+qX05GtzeWJXbAZXYkSXd+lilNhiVTknRHsExJE2fJlKQ2LnQsSZLUwDIlSZLUwDIlSZLUwDFTkrSAsYwlcxyZNH6WKUnSsoylYMK4Sqbvi7zNJ0mS1MAyJUmS1MAyJUmS1MAyJUmS1MAB6JIkacVNaWC+V6YkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaWKYkSZIaLKtMJTkyydVJNiY5aYHjf5Tkkv7jM0m+svJRJUmSxmeHpU5IsgY4FTgCuA64MMn6qrpy7pyqeunM+S8GHnIHZJUkSRqd5VyZOgzYWFXXVNXNwNnAMVs4/zjgbSsRTpIkaeyWU6buB1w7s31dv+92kuwN7At8eJHjxyfZkGTDpk2btjarJEnS6Kz0APRjgXdW1XcXOlhVp1fVuqpat3bt2hX+1JIkSdvfcsrU9cCeM9t79PsWcize4pMkSROynDJ1IXBAkn2T7EhXmNbPPynJ/YF7AR9f2YiSJEnjtWSZqqrNwAnA+cBVwDlVdUWSU5IcPXPqscDZVVV3TFRJkqTxWXJqBICqOhc4d96+k+dtv2rlYkmSJK0OzoAuSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUYFllKsmRSa5OsjHJSYuc87NJrkxyRZK3rmxMSZKkcdphqROSrAFOBY4ArgMuTLK+qq6cOecA4BXA4VX15SQ/cEcFliRJGpPlXJk6DNhYVddU1c3A2cAx8855AXBqVX0ZoKq+sLIxJUmSxmk5Zep+wLUz29f1+2b9MPDDST6W5BNJjlzoL0pyfJINSTZs2rRp2xJLkiSNyEoNQN8BOAB4DHAc8OdJdpt/UlWdXlXrqmrd2rVrV+hTS5IkDWc5Zep6YM+Z7T36fbOuA9ZX1Xeq6rPAZ+jKlSRJ0p3acsrUhcABSfZNsiNwLLB+3jl/S3dViiS70932u2YFc0qSJI3SkmWqqjYDJwDnA1cB51TVFUlOSXJ0f9r5wA1JrgQuAF5eVTfcUaElSZLGYsmpEQCq6lzg3Hn7Tp55XcCJ/YckSdJkOAO6JElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSA8uUJElSg2WVqSRHJrk6ycYkJy1w/NlJNiW5pP94/spHlSRJGp8dljohyRrgVOAI4DrgwiTrq+rKeae+vapOuAMySpIkjdZyrkwdBmysqmuq6mbgbOCYOzaWJEnS6rCcMnU/4NqZ7ev6ffM9JcllSd6ZZM+F/qIkxyfZkGTDpk2btiGuJEnSuKzUAPT3APtU1cHAB4GzFjqpqk6vqnVVtW7t2rUr9KklSZKGs5wydT0we6Vpj37f91TVDVX17X7zTcCPrkw8SZKkcVtOmboQOCDJvkl2BI4F1s+ekOSHZjaPBq5auYiSJEnjteTTfFW1OckJwPnAGuAvquqKJKcAG6pqPfCSJEcDm4EvAc++AzNLkiSNxpJlCqCqzgXOnbfv5JnXrwBesbLRJEmSxs8Z0CVJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhpYpiRJkhosq0wlOTLJ1Uk2JjlpC+c9JUklWbdyESVJksZryTKVZA1wKnAUcBBwXJKDFjhvZ+CXgX9e6ZCSJEljtZwrU4cBG6vqmqq6GTgbOGaB834beDXwrRXMJ0mSNGrLKVP3A66d2b6u3/c9SQ4F9qyq961gNkmSpNFrHoCe5C7A64CXLePc45NsSLJh06ZNrZ9akiRpcMspU9cDe85s79Hvm7Mz8EDgI0k+BzwcWL/QIPSqOr2q1lXVurVr1257akmSpJFYTpm6EDggyb5JdgSOBdbPHayqG6tq96rap6r2AT4BHF1VG+6QxJIkSSOyZJmqqs3ACcD5wFXAOVV1RZJTkhx9RweUJEkasx2Wc1JVnQucO2/fyYuc+5j2WJIkSauDM6BLkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1WFaZSnJkkquTbExy0gLHfzHJ5UkuSfKPSQ5a+aiSJEnjs2SZSrIGOBU4CjgIOG6BsvTWqnpQVT0YeA3wuhVPKkmSNELLuTJ1GLCxqq6pqpuBs4FjZk+oqq/ObN4TqJWLKEmSNF47LOOc+wHXzmxfBzxs/klJfgk4EdgReOxCf1GS44HjAfbaa6+tzSpJkjQ6KzYAvapOrar9gV8DfmORc06vqnVVtW7t2rUr9aklSZIGs5wydT2w58z2Hv2+xZwNPKkllCRJ0mqxnDJ1IXBAkn2T7AgcC6yfPSHJATObjwf+deUiSpIkjdeSY6aqanOSE4DzgTXAX1TVFUlOATZU1XrghCQ/CXwH+DLwC3dkaEmSpLFYzgB0qupc4Nx5+06eef3LK5xLkiRpVXAGdEmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAaWKUmSpAbLKlNJjkxydZKNSU5a4PiJSa5MclmSDyXZe+WjSpIkjc+SZSrJGuBU4CjgIOC4JAfNO+2TwLqqOhh4J/CalQ4qSZI0Rsu5MnUYsLGqrqmqm4GzgWNmT6iqC6rqG/3mJ4A9VjamJEnSOC2nTN0PuHZm+7p+32KeB7x/oQNJjk+yIcmGTZs2LT+lJEnSSK3oAPQkzwDWAX+w0PGqOr2q1lXVurVr167kp5YkSRrEDss453pgz5ntPfp9t5HkJ4FfBx5dVd9emXiSJEnjtpwrUxcCByTZN8mOwLHA+tkTkjwEOA04uqq+sPIxJUmSxmnJMlVVm4ETgPOBq4BzquqKJKckObo/7Q+AnYB3JLkkyfpF/jpJkqQ7leXc5qOqzgXOnbfv5JnXP7nCuSRJklYFZ0CXJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqYJmSJElqsKwyleTIJFcn2ZjkpAWOPyrJxUk2J3nqyseUJEkapyXLVJI1wKnAUcBBwHFJDpp32r8BzwbeutIBJUmSxmyHZZxzGLCxqq4BSHI2cAxw5dwJVfW5/tgtd0BGSZKk0VrObb77AdfObF/X79tqSY5PsiHJhk2bNm3LXyFJkjQq23UAelWdXlXrqmrd2rVrt+enliRJukMsp0xdD+w5s71Hv0+SJGnyllOmLgQOSLJvkh2BY4H1d2wsSZKk1WHJMlVVm4ETgPOBq4BzquqKJKckORogyUOTXAc8DTgtyRV3ZGhJkqSxWM7TfFTVucC58/adPPP6Qrrbf5IkSZPiDOiSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNllWmkhyZ5OokG5OctMDx70vy9v74PyfZZ6WDSpIkjdGSZSrJGuBU4CjgIOC4JAfNO+15wJer6r8BfwS8eqWDSpIkjdFyrkwdBmysqmuq6mbgbOCYeeccA5zVv34n8LgkWbmYkiRJ45Sq2vIJyVOBI6vq+f32M4GHVdUJM+d8qj/nun77//XnfHHe33U8cHy/eSBw9Ur9jzTaHfjikmdNj+/L7fmeLMz3ZWG+Lwvzfbk935OFjel92buq1i50YIftmaKqTgdO356fczmSbKiqdUPnGBvfl9vzPVmY78vCfF8W5vtye74nC1st78tybvNdD+w5s71Hv2/Bc5LsAOwK3LASASVJksZsOWXqQuCAJPsm2RE4Flg/75z1wC/0r58KfLiWun8oSZJ0J7Dkbb6q2pzkBOB8YA3wF1V1RZJTgA1VtR44A/irJBuBL9EVrtVkdLceR8L35fZ8Txbm+7Iw35eF+b7cnu/JwlbF+7LkAHRJkiQtzhnQJUmSGlimJEmSGlimJEmSGlimJEmSGmzXSTvHol9v8O+q6ieGzjJGSR4JHFBVZyZZC+xUVZ8dOtcYJHkccA/gvKr6ztB5hpTkHsDLgL2q6gVJDgAOrKr3Dhxtu0ty73m7CviKU8RAkhO3dLyqXre9soxJkkO3dLyqLt5eWcYiyR/Tfe8sqKpesh3jbJVJlqmq+m6SW5LsWlU3Dp1nTJK8ElhHt9zPmcBdgbcAhw+ZawySvBa4EbgFeBHw08MmGtyZwEXAI/rt64F3AJMrU3TvQwGza5LulORS4PlV9blBUo3DHwKXAO8Hvs1t36Mp2wB8iluXSpl9Xwp47HZPNLwN/Z+HAwcBb++3nwZcOUiiZZpkmep9Hbg8yQeBm+Z2jrn5bic/AzwEuBigqv49yc7DRhpGX55+u6q+0u/aC/jZ/vXlw6Qalf2r6ulJjgOoqm9MdYHzqtp3of1Jngz8GXDk9k00Kg8BjgMeT1c63wZ8yKt2nEg3yfU3gbOBv6mqrw8baVhVdRZAkhcBj6yqzf32nwH/MGS2pUx5zNS7gN8EPkr3DT73MXU39//IFUCSew6cZ0jvAs5O8pL+1vCbgQuAjwN/Pmiycbg5yd259Wtlf7orD+pV1buAHxg6x5Cq6tKqOqmqHkw3wfMxwJVJjh442qCq6v9U1SOBF9Mtx/ahJOckefDA0cbgXsAuM9s79ftGa7JXpqrqrP4HwV5VdfXQeUbknCSnAbsleQHwXCZaHKrqY8CRSZ5BtwLAG6rqMcOmGpVXAucBeyb5a7pL888eNNHIJNmJaf/S+j39+MuHAA8CrgO+MGyicaiqa5K8G7g78Ezgh+lui07Z7wOfTHIB3e3PRwGvGjTREiY7A3qSJ9Ldy9+xqvbtfxs4paom/dsSQJIjgJ+i+yI+v6o+OHCkQfSLdv934DvAPwEvBR4K/GZVXTpktrFI8v3Aw+m+Vj5RVV9c4j+5U1pkkPW9gKOBP6mqSf5CApDkuXS3x+8GvBM4p6omX6SS7Ee39NoxwLV0t/reV1XfHDTYSCS5D/CwfvOfq+o/h8yzlCmXqYvoBvh9pKoe0u/7VFU9cNhkGosk76W7pXcPYI+q+oUk9wVOAaqqXjBowIElORy4pKpu6q/eHQq8vqo+P3C07a5/cGNWATcAH62qSY+vS3IL3UDrua+L2/zQmeovsP37chnwbuCr3P59meRTjotJcv+q+vTQORYz2dt8wHeq6sZ542VvGSrMWPQDZl9NN84j/UdV1S5b/A/vnPauqick2RH4BHQD8oHnO64BgDcChyQ5hG4w7Rl048oePWiqAVTVby12LMleVfVv2zPPyDgFzcJmv2Z2GizF6vEBuoeARmnKZeqKJD8HrOnnx3kJ3a2cqXsN8MSqumroICNwWpKP969v81tiVU19TAPA5qqqJMcAp1bVGUmeN3SooSR5BHA/uqtRX0hyMHAS8ON0A4yn6jlV9eyhQ4zQDVX1J0OHGJMkb1jsELDb9syytaZ8m+8ewK8zMzaI7jH4bw0abGBJPlZVk59TSktL8vd0A9CfQzdA9AvApVX1oEGDDSDJHwBPoBs4/N/o/j15PvB7wGlT/nclycVVtcUJKqfI9+X2knyNbiLghZ4Kfm1V7b6dIy3bZMuUFpbk9cB9gL9l5gu6f8R7UvoB6M8DnkR3xQG6iSnfDZzhDOi5D/BzwIVV9Q9J9gIeU1VvHjjadpfkSuDQqvpWknvRDSh+4MQn6wQgyafp5placA6yKc70DZaphST5MPAbVXW7u0RJPrvYfG5jMLkyleQ9bHm6+kkOhpyT5MwFdldVPXe7hxlYkrcBXwHOonuUG2AP4BeAe1fV04fKpnGZ/4MxySfnHmyZuv5qw4UsXKaqqqY40zdJNgPfWOgQEx2n2i/L9K2qWuh9GbUplqm5wbFPprsC85Z++zjgv6rqpYME0+gk+UxV/fDWHpsKH1a4VZKv0E0APOdR/fbcezLZX9IslgvzfVlc/2/L+6pq1UwCPLkB6FX199AtFVJV62YOvSfJhkX+szu9JL9aVa9ZbKHJiS6z86UkTwP+b1XdApDkLnTrRH150GTj4MMKtzpm3vZrufX7aJJL7EgNngj8UZKP0q3Pd97c0jJjNbkyNeOeSfarqmsAkuwLTHnplLkfiJMtlAs4lu7Ky58mmStPu9EtKXPsYKnG478sUt+zG91cZKcCJPkXYC1dofq1IYONwG3+/5PcFXggcP3EJ+98x9ABxqqqntN/nRxFd9fo1CQfrKrnDxxtUZO7zTcnyZHA6cA1dL857g0cX1UfGDSYRqmf6ZuqumHoLGPhwwq3SvIx4NiqurbfvgR4HN0vaGdW1eOGzDekfpHaP66qK5LsSjcR7neBewP/s6reNmjAgfTLdX2kqv61XyD8L4CnAJ8Dnj3Vgfmz+kJ1JP0Tw2N+mm+yV6aq6rx+fqn797s+vZruz640B+Zv2fwSleSIqS6zM2MXugG0PzWzr+gWiJ6aHeeKVO8f+6+ZGya+WDjAj1fVL/avnwN8pqqe1D8N+n5gkmUK+GXgL/vXxwEHA/vSrV/4err5ySYpyVHA04HHAB8B3kS3JNFoTbZM9Y33hXQDRQE+kuS0CT/u/odDB1hlzmDEs/FuD1X1nKEzjMhtVrSvqhNmNtdu5yxjc/PM6yPob29V1X/OW4FiajbP/Lx5AvDmvoD/XZLXDJhrDJ5FN1bqhavlIsdkyxTdUhh3Bf60335mv2+092TvSHMD8wGS3B3Yq6quHjDS4JKsX+wQ8P3bM8sYJflhuu+ZH6yqB/Yzfh9dVf974GhD+OckL5i/oHGSFwL/MlCmsfhKkifQzdF2ON3cbXPzuN19yGADuyXJD9E9zPI44Hdmjk35faGqjhs6w9aacpl6aFUdMrP94SSXDpZmJJI8ke4q1Y7Avv0adKdM9DbfjwPPAL4+b3+Aw7Z/nNH5c+DlwGkAVXVZkrcCUyxTLwX+tl+iam6sy48C30c36euUvRB4A934ul+pqv/s9z8OeN9gqYZ3Mt0DP2uA9VV1BXxv+p5rhgw2tNU47cqUy9R3k+xfVf8PIMl+dIMip+5VdEXhI9CtQdc/6ThFnwC+MXvVbk6SSV+1692jqv5l3q2aUT++fEfpn0r7sSSPBX6k3/2+qvrwgLFGoao+QzeIeP7+85M8YIBIo1BV702yN7BzVc1OtbIB+N2BYo3Fqpt2Zcpl6uXABUlmn+ZzDAh8p6punPcDcpKPfFbVUVs49qjFjk3IF5PsT//1keSpwH8MG2lYfXmafIHaCicC/2foEEPp50768rx9N/WrL0x5TOaqm3ZlsmWqqj7UP813YL/r6tUy0O0OdkV/q2JN//68BLjdOkm6VZKPV9Ujhs4xgF+im17k/kmuBz5Ld1tUWq5Jj0Dfgqm/LxuSvJ1VNO3KXYYOMJQkvwTcvaouq6rLgHsk+R9D5xqBF9Pdpvg23SPLXwV+ZdBE43e3oQMMoaquqaqfpHta7f5V9UgX9tVWmuRV72WY+vsyO+3KE/uPJwyaaAlTnrTzkqp68Lx9rpU0I8ka4J5V9dWhs4zZ1FZ/T3Lilo5X1eu2VxaNX7/Q8UI/aEL3C+0k75BsYW6/AI+tqqnPT7aqTPKLuLcmSapvk31x2HHgTIPrn8b6RbrB+BcCuyR5fVX9wbDJNCI7938eCDwUmJtC4ok4DYDmqaqdlz5rkrY0t9+k5/1Lsgfwx3RTaQD8A/DLVXXdcKm2bMpXpv6AbtD5af2uFwLXVtXLhks1vLkrdkl+HjgUOAm4qKoOHjjaaE31ima/COnjq+pr/fbOdE+wOThfWiFJ/m9VPWXoHNtTkg8CbwX+qt/1DODnq+qI4VJt2WTHTNEtvnkB8KL+40PArw6aaBzu2s8O/yS6uU++w4Tv3ydZk+SCJU575nYJMz4/yG1nt7653ydp5ew3dIABrK2qM6tqc//xl4x8JYHJ3uarqlvoZm9+49BZRuY0uoU2LwU+2s+DMtkxU1X13SS3JNm1qm5c5JxPbe9cI/Fm4F+S/E2//SRuXWtM0sqY4i+zNyR5Breu23gcMOpF5qd8m+9wugkq96YrlXMzrE7xt4AtSrJDPx/KJCV5N93iox8EbprbX1UvGSzUSCQ5lFsXZP1oVX1y5ti95k1GKGkrTe0BF4D+l/g/Bh5BVyb/CXjxvMXER2XKZerTdEtAXMTMzOf9QpOTluTxdNMjfO+R/6o6ZbhEw0ryCwvtr6qztneW1WSKPwSklTbFMZlJzqJbeujL/fa9gT+squcOm2xxk73NB9xYVe8fOsTYJPkz4B7ATwBvAp7KxJ/QqqqzXPx5m0x94kFpJfza0AEGcPDsVe2q+lKSURfKKZepC/on+t7FbWdYvXjx/2QSfqyqDk5yWVX9VpLXApMunS7+vM2medlbWoYkl7P4PFM19wR1VX1guwYbh7vMDhPor0yNuq+MOtwd7GH9n+tm9hXw2AGyjMk3+z+/keS+dIP+fmjAPGPwKm6/+LNj6yS1GGyn0AoAAA7aSURBVPWM3gN7LfDxJO/ot58G/M6AeZY02TJVVT8xdIaRem+S3ehW7b6o3/emAfOMwUKLP98yVJhVxNt80iKq6vNDZxirqnpzkg3cenHjyVV15ZCZljLlAeg/CPwucN+qOirJQcAjquqMgaMNqh8b9CK6J7SKbubZN1bVtwYNNqAkZ9DNQ3YS8BS6xZ/vWlW/OGiwEUjySOCAqjozyVpgp6r6bH/s3lX1pWETSuOW5OF0T649gG4owRrgpqraZdBg2ipTnrTzL4Hzgfv225/BBX0BzqJ7ku8NdN/gB9HNJzRlLv68gCSvpBsc+4p+112Bt8wdt0hJy/IndPMo/Stwd+D5wKmDJtJWm/KVqQur6qGzj50utPjx1CS5sqoOWmqflOQSuvm3Lp75HrrMpYek5UuyoarWzX7vTHE6hNVusmOmgJuSfD/90xT9pdYFZ7iemIuTPLyqPgGQ5GHAhoEzDWILq7oD4NN83FxVlWTue8hV7qWt940kOwKXJHkN8B9M+67RqjTlMnUi3Wr3+yf5GN26P08dNtJwZh7TvSvwT0n+rd/eG/j0kNkGNLdy+5OB+3DrLazjgP8aJNG4nJPkNGC3JC8Angv8+cCZpNXmmXTl6QS6iaT3pBubqVVksrf5oFsmBTiQ7qmjq/tFfeeOHVFVHxws3HbWT9+/qCk/eTJ3GX6pfVOU5Ajgp+i+h86f0veMJM2ZdJnaEpfC0JwkVwGPr6pr+u19gXOr6gHDJpO02i2wTiwArhO7ukz5Nt9SnCNHc14KfCTJNXRfF3sDxw8baThJvsaWZ272kW5p+c5ggXVitbpYphbnJTsBUFXnJTkAuH+/69NV9e0t/Td3ZlW189AZpDsR14m9E7BMSUtIclfghcCj+l0fSXLa7Bi7qUpyKPBIul8+/rGqPjlwJGm1cZ3YOwHHTC0iybuq6slD59DwkryJ7inHs/pdzwS+W1XPHy7V8JKcTLdm1rv6XU8C3lFV/3u4VNLqkuSCBXZXVU19ndhVZbJlKsk9gJcBe1XVC/rbOAdW1XsHjqaRSXJpVR2y1L6pSXI1cMjcUkP9UkSXVNWBwyaTpO1ryhODnUl3SfUR/fb1gL9RayHfTbL/3EaS/XCgKMC/A3eb2f4+uu8jScuUZNckr0uyof94bZJdh86lrTPlMVP7V9XTkxwHUFXfSOITfFrIy+nGNcw+zfecYSONwo3AFUk+SDdm6gjgX5K8AaCqXjJkOGmV+AvgU8DP9tvPpPtl32Emq8iUy9TN/W2JuaUw9mdm8J80p6o+NHcbuN919ZSf5pvxN/3HnI8MlENazfavqtkZz3+rX/dSq8iUy9QrgfOAPZP8NXA48OxBE2mUkvwS8NdVdVm/fa8kz6uqPx042qCq6qylz5K0hG8meWRV/SN8bxLPbw6cSVtpsgPQAfqFjh9Od+vmE1X1xYEjaYSSXFJVD563b/Kruid5AvDb3Dpzs5N2SlspyYPpnhTele576EvAs6vq0kGDaatMtkz17f+SqropyTOAQ4HXT3kNOi2sXwT64Oq/WZKsAS6rqh8ZNtmwkmykG9dxeU31HxJphSTZBaCqvjp0Fm29Kd/meyNwSJJDgBPppvR/M/DoQVNpjM4D3p7ktH77hf2+qbsW+JRFStp6SZ5RVW9JcuK8/QBU1esGCaZtMuUytbmqKskxwKlVdUaS5w0dSqP0a3QF6kX99geBNw0XZzR+FTg3yd9z25mb/SEgLe2e/Z8LLc/kLyirzJRv8/093dWF59AtE/IF4NKqetCgwaRVIskHgK8DlwO3zO2vqt8aLJS0yiQ5vKo+ttQ+jduUy9R9gJ8DLqyqf0iyF/CYqnrzwNE0Mv34uldx+4HW+w2Za2hJPlVVDxw6h7SaJbm4qg5dap/GbbJlSlquJJ8GXgpcxMzM51V1w2ChRiDJa4C/q6oPDJ1FWm2SPAL4MeBXgD+aObQL8DNTX65qtZnsmKkkX+PW+9I70i1k+/Wqchp/zXdjVb1/6BAj9CLgfyb5NvAdnBpB2ho7AjvR/RyeHTf1VeCpgyTSNvPKFNAvI3MM8PCqOmnoPBqXJL8PrAHexW0HWl88WChJdwpJ9nZKntXPMjXDiRi1kCQXLLC7quqx2z3MCCS5f1V9OsmCYzosmdLy9WtbPq2qvtJv3ws4u6r++7DJtDWmfJtvdhHJuwDrgG8NFEcjVlU/MXSGkTkROB547cy+2d/KJlkypW20+1yRAqiqLyf5gSEDaetNtkwBT5x5vRn4HN2tPuk2kvwg8LvAfavqqCQHAY+oqjMGjjaIqjq+f/lG4Lyq+mqS36RbReC3h0smrUq3JNmrqv4Nutt+OM/UquNtvkUkeUVV/d7QOTS8JO8HzgR+vaoOSbID8Mmpz0mW5LKqOjjJI+lK1B8CJ1fVwwaOJq0aSY4ETgf+nu4hjh8Hjq+q8wcNpq1yl6EDjNjThg6g0di9qs6hn5iyqjYzM0XChM29B48H/ryq3kf3hJKkZaqq8+iu6r4dOBv4UYvU6mOZWlyGDqDRuCnJ99Nfek/ycODGYSONwvX9eoVPp1tW5vvw3xRpq/RPkx8JHFpV7wXukeSwgWNpK3mbbxHOQKs5/VNrfww8EPgUsBZ4alVdNmiwgSW5B90Pgcur6l+T/BDwICfxlJYvyRvprno/tqoe0D/N94GqeujA0bQVLFOLcJoEzerHSR1Id8Xy6qr6zsyxI6rqg4OFk7Rqzf3iPvszJ8mlzoC+unhJfnHvGDqAxqOqNlfVFVX1qdki1Xv1IKEk3Rl8J8kabh1GsJaZhcO1Oky2TCXZL8l7knwxyReSvDvJ9xaurarfHTKfVhXH10naVm8A/gb4gSS/A/wj3VQsWkWmPM/UW4FTgZ/pt48F3gb4WLe2lvfKJW21JHcBPgv8KvA4ul/MnlRVVw0aTFttsmOm5ubImbfP+9Taaj6sIGlbOT73zmFyt/mS3DvJvYH3JzkpyT5J9k7yq8C5Q+fTqvS5oQNIWrU+lOQp/RQJWqUmd2UqyWfpbsss9IVbVbXfAvs1Yf0UAC8D9qqqFyQ5ADiwnxNGkrZZkq8B96Rb1uxbdD+bqqp2GTSYtsrkypS0tZK8HbgIeFZVPbAvV/9UVQ8eOJokaQQmOwA9ybMW2l9Vb97eWTR6+1fV05McB1BV3/CSvKQW/WTAi6qqi7dXFrWbbJkCZmeXvRvdkxQXA5YpzXdzkrtz6zww+wPfHjaSpFXutf2fdwPWAZfS3eI7GNgAPGKgXNoGky1TVfXi2e0ku9EtMinN90rgPGDPJH8NHA48e9BEkla1qvoJgCTvoluX7/J++4HAqwaMpm3gmKlekrsCn6qqA4fOovHpFzp+ON1vjp+oqi8OHEnSnUCSK6rqR5bap3GbbJlK8h5unWzxLsBBwDlVddJwqTRGSQ4HLqmqm5I8AzgUeH1VfX7gaJJWuSRvA24C3tLv+nlgp6o6brhU2lpTLlOPntncDHy+qq4bKo/GK8llwCF0YxnOBM4AfraqHr3F/1CSlpDkbsCLgEf1uz4KvLGqvjVcKm2tyZYpablmVnU/Gbi+qs5w1nNJ0pzJDkBP8mTg1cAP0I2DcaI0LeZrSV4BPAN4VL+e1l0HziRpFUtyTlX9bJLLWWB9z/nLnWncJntlKslG4IkuKKmlJLkP8HPAhVX1D0n2Ah7jnGSStlWSH6qq/0iy90LHHZO5uky5TH2sqg4fOockSQBJdgduqKn+YF7FJlem+tt7AI8G7gP8LTMTMFbVu4bIpfHq186a+0bZke4W39eratfhUklazZI8HPh94EvAbwN/BexO93T5s6rqvAHjaStNcczUE2defwP4qZntAixTuo2q2nnudb+MzDF0c05J0rb6E+B/AbsCHwaOqqpPJLk/8Da6iYK1SkzuytRyJXlFVf3e0Dk0Tkk+WVUPGTqHpNUpySVzi6UnuaqqHjBzzH9fVpkpXplarqcBlinN3hqG7hL8OsA5YCS1uGXm9TfnHfMqxypjmVpchg6g0Zi9NbwZ+BzdrT5J2laHJPkq3c+au/ev6bfvNlwsbQvL1OL8zUAAVNVztnTcW8KStlZVrRk6g1bOXYYOMGJemdJyPW3oAJKk4UyuTCV5df/nUj8A37Ed4ujOweItSRM2uaf5+qn7DwYucm01rQTX6ZOkaZvimKnzgC8DO80M+APX5tO288qUJE3Y5G7zVdXLq2o34MNVtcvMx87Anw2dT6uSt4QlacImV6Zm7L7AviO3ewqNXpL9krwnyReTfCHJu5PsN3e8qn53yHySpGFNrkwleVE/burAJJfNfHwWuHzofBqltwLn0K3leF+6K1FvGzSRJGk0pjgAfVfgXnSzm580c+hrVfWlYVJpzJJcVlUHz9t3aVUdMlQmSdJ4TK5MScuV5N79y1+je2jhbLrJXJ8O3KuqXjFUNknSeFimpEX0t36LhZ/Wq6rab4H9kqSJsUxJkiQ1mOI8U9JWSfKshfZX1Zu3dxZJ0vhYpqSlPXTm9d2AxwEXA5YpSZK3+aStlWQ34Oyqcl4ySdL05pmSVsBNwL5Dh5AkjYO3+aQlJHkP3VN90P0CchDdJJ6SJHmbT1pKkkfPbG4GPl9V1w2VR5I0LpYpSZKkBo6ZkpaQ5MlJ/jXJjUm+muRrSb46dC5J0jh4ZUpaQpKNwBOr6qqhs0iSxscrU9LS/ssiJUlajFempEUkeXL/8tHAfYC/Bb49d7yq3jVELknSuFimpEUkOXMLh6uqnrvdwkiSRssyJTVK8oqq+r2hc0iShuGYKand04YOIEkajmVKapehA0iShmOZktp5r1ySJswyJbXzypQkTZhlSlpEklf3fy41Juod2yGOJGmkfJpPWkSSy4GDgYuq6tCh80iSxmmHoQNII3Ye8GVgp3lr8YVunqldhoklSRoTb/NJi6iql1fVbsCHq2qXmY+dgT8bOp8kaRwsU9LSdl9g35HbPYUkaZS8zSctIsmLgP8B7JfksplDOwP/NEwqSdLYOABdWkSSXYF7Ab8HnDRz6GtV9aVhUkmSxsYyJUmS1MAxU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ3+Pyz34nqaSGoEAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Uploading training logs to tensorboard"
      ],
      "metadata": {
        "id": "CMhS-BCeQI_1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!tensorboard dev upload --logdir ./model_logs/ \\\n",
        "  --name \"NLP modeling experiments\" \\\n",
        "  --description \"comparing multiple different types of architechtures on Kaggle's tweet classification dataset\" \\\n",
        "  --one_shot"
      ],
      "metadata": {
        "id": "gil6A-LxQI9A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "All the experiments are here : https://tensorboard.dev/experiment/3cGdHZkJS16wwagztKqwZQ/"
      ],
      "metadata": {
        "id": "ofJGmKncQI69"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Finding the most wrong examples"
      ],
      "metadata": {
        "id": "7mCu3OWiQI4m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download pretrained model from course\n",
        "!wget https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
        "!unzip 08_model_6_USE_feature_extractor.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yHj_umNKYo9O",
        "outputId": "5886781c-93ee-494b-d5af-69a5dcaff7bb"
      },
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-09-15 23:03:22--  https://storage.googleapis.com/ztm_tf_course/08_model_6_USE_feature_extractor.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.2.112, 172.253.115.128, 172.253.122.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.2.112|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 960779165 (916M) [application/zip]\n",
            "Saving to: ‘08_model_6_USE_feature_extractor.zip’\n",
            "\n",
            "08_model_6_USE_feat 100%[===================>] 916.27M  56.0MB/s    in 22s     \n",
            "\n",
            "2022-09-15 23:03:44 (42.3 MB/s) - ‘08_model_6_USE_feature_extractor.zip’ saved [960779165/960779165]\n",
            "\n",
            "Archive:  08_model_6_USE_feature_extractor.zip\n",
            "   creating: 08_model_6_USE_feature_extractor/\n",
            "   creating: 08_model_6_USE_feature_extractor/assets/\n",
            "   creating: 08_model_6_USE_feature_extractor/variables/\n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.data-00000-of-00001  \n",
            "  inflating: 08_model_6_USE_feature_extractor/variables/variables.index  \n",
            "  inflating: 08_model_6_USE_feature_extractor/saved_model.pb  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_pretrained = tf.keras.models.load_model(\"08_model_6_USE_feature_extractor\")\n",
        "model_6_pretrained.evaluate(val_sentences, val_labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2NY0263nZW7W",
        "outputId": "476a92fe-ed62-4310-b9bd-25342676269b"
      },
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named \"keras_metadata.pb\" in the SavedModel directory.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 10ms/step - loss: 0.4272 - accuracy: 0.8163\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.42723122239112854, 0.8162729740142822]"
            ]
          },
          "metadata": {},
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions\n",
        "model_6_pretrained_pred_probs = model_6_pretrained.predict(val_sentences)\n",
        "model_6_pretrained_preds = tf.squeeze(tf.round(model_6_pretrained_pred_probs))\n",
        "model_6_pretrained_preds[:10]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "caf1XOnTZrwm",
        "outputId": "bcec8101-e5e6-4ac1-e421-e14d49b05cae"
      },
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create DataFrame with validation sentences and best performing model's predictions labels + probabilities\n",
        "val_df = pd.DataFrame({\"text\":val_sentences,\n",
        "                       \"target\":val_labels,\n",
        "                       \"pred\": model_6_pretrained_preds,\n",
        "                       \"pred_prob\": tf.squeeze(model_6_pretrained_pred_probs)})\n",
        "val_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "xVgY1nHmQI23",
        "outputId": "de836e2f-d1af-4a2a-f720-658a7ba992bb"
      },
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                text  target  pred  pred_prob\n",
              "0  DFR EP016 Monthly Meltdown - On Dnbheaven 2015...       0   0.0   0.159757\n",
              "1  FedEx no longer to transport bioterror germs i...       0   1.0   0.747162\n",
              "2  Gunmen kill four in El Salvador bus attack: Su...       1   1.0   0.988749\n",
              "3  @camilacabello97 Internally and externally scr...       1   0.0   0.196229\n",
              "4  Radiation emergency #preparedness starts with ...       1   1.0   0.707808"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4ed09dad-80d6-4155-a204-3fddc721647a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>DFR EP016 Monthly Meltdown - On Dnbheaven 2015...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.159757</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>FedEx no longer to transport bioterror germs i...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.747162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Gunmen kill four in El Salvador bus attack: Su...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.988749</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@camilacabello97 Internally and externally scr...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.196229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Radiation emergency #preparedness starts with ...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.707808</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4ed09dad-80d6-4155-a204-3fddc721647a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4ed09dad-80d6-4155-a204-3fddc721647a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4ed09dad-80d6-4155-a204-3fddc721647a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the wrong predictions and sort by probs\n",
        "most_wrong = val_df[val_df[\"target\"] != val_df[\"pred\"]].sort_values(\"pred_prob\", ascending=False)\n",
        "most_wrong.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "XNHIWxYwQI0P",
        "outputId": "8583148c-1f45-4741-aa21-d6e4b7accf14"
      },
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  text  target  pred  \\\n",
              "31   ? High Skies - Burning Buildings ? http://t.co...       0   1.0   \n",
              "759  FedEx will no longer transport bioterror patho...       0   1.0   \n",
              "628  @noah_anyname That's where the concentration c...       0   1.0   \n",
              "209  Ashes 2015: AustraliaÛªs collapse at Trent Br...       0   1.0   \n",
              "251  @AshGhebranious civil rights continued in the ...       0   1.0   \n",
              "\n",
              "     pred_prob  \n",
              "31    0.910196  \n",
              "759   0.876982  \n",
              "628   0.852300  \n",
              "209   0.835454  \n",
              "251   0.827213  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ecb9ae94-9c7c-4a8a-854c-a49c4a8cc848\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>? High Skies - Burning Buildings ? http://t.co...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.910196</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>759</th>\n",
              "      <td>FedEx will no longer transport bioterror patho...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.876982</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>@noah_anyname That's where the concentration c...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.852300</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>209</th>\n",
              "      <td>Ashes 2015: AustraliaÛªs collapse at Trent Br...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.835454</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251</th>\n",
              "      <td>@AshGhebranious civil rights continued in the ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.827213</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ecb9ae94-9c7c-4a8a-854c-a49c4a8cc848')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ecb9ae94-9c7c-4a8a-854c-a49c4a8cc848 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ecb9ae94-9c7c-4a8a-854c-a49c4a8cc848');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "most_wrong.tail()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "eaYrl9j_QIyV",
        "outputId": "1c1c6b3e-ea89-448a-8b29-afb5bfa29a4e"
      },
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  text  target  pred  \\\n",
              "411  @SoonerMagic_ I mean I'm a fan but I don't nee...       1   0.0   \n",
              "233                    I get to smoke my shit in peace       1   0.0   \n",
              "38   Why are you deluged with low self-image? Take ...       1   0.0   \n",
              "244  Reddit Will Now QuarantineÛ_ http://t.co/pkUA...       1   0.0   \n",
              "23   Ron &amp; Fez - Dave's High School Crush https...       1   0.0   \n",
              "\n",
              "     pred_prob  \n",
              "411   0.043918  \n",
              "233   0.042087  \n",
              "38    0.038998  \n",
              "244   0.038949  \n",
              "23    0.037186  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b7b9482f-719e-4c3e-8bec-5991c6c6bdc4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>411</th>\n",
              "      <td>@SoonerMagic_ I mean I'm a fan but I don't nee...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.043918</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>233</th>\n",
              "      <td>I get to smoke my shit in peace</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.042087</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>Why are you deluged with low self-image? Take ...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038998</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>244</th>\n",
              "      <td>Reddit Will Now QuarantineÛ_ http://t.co/pkUA...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.038949</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>Ron &amp;amp; Fez - Dave's High School Crush https...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.037186</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b7b9482f-719e-4c3e-8bec-5991c6c6bdc4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b7b9482f-719e-4c3e-8bec-5991c6c6bdc4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b7b9482f-719e-4c3e-8bec-5991c6c6bdc4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Making and Visualizing Predictions with test dataset"
      ],
      "metadata": {
        "id": "tl1arl-DQIup"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "test_sentences = test_df[\"text\"].to_list()\n",
        "test_samples = random.sample(test_sentences, 10)\n",
        "for test_sample in test_samples:\n",
        "  pred_prob = tf.squeeze(model_6_pretrained.predict([test_sample]))\n",
        "  pred = tf.round(pred_prob)\n",
        "  print(f\"Pred: {int(pred)}, Prob: {pred_prob}\")\n",
        "  print(f\"\\nText: \\n{test_sample}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d1x4q-E-QIsf",
        "outputId": "ab9d95b5-b886-4f53-db25-ab17f58ca043"
      },
      "execution_count": 128,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 0, Prob: 0.031238485127687454\n",
            "\n",
            "Text: \n",
            "@jassy_curtis wait I'm fckng obsessed with your song sirens\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.4562392234802246\n",
            "\n",
            "Text: \n",
            "Demolition underway on @OU_Football stadium project. Stay updated throughout at http://t.co/ju09z2y7g0 http://t.co/UP0hOCgGrj\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9596754908561707\n",
            "\n",
            "Text: \n",
            "Experts in France begin examining airplane debris found on Reunion Island: French air accident experts on Wedn...  http://t.co/KuBsM16OuD\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.08991207927465439\n",
            "\n",
            "Text: \n",
            "@jackienatalydlt I do.... I only get the iced annihilation??\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.6158299446105957\n",
            "\n",
            "Text: \n",
            "kayaking about killed us so mom and grandma came to the rescue.\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.2476567029953003\n",
            "\n",
            "Text: \n",
            "@johngreen The catastrophic history of you and me\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9771777987480164\n",
            "\n",
            "Text: \n",
            "Per Malaysia's PM 'this piece of debris ... is conclusively a part from the wing of #MH370.' More right now http://t.co/Bd3ik4oz2r\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.3282272219657898\n",
            "\n",
            "Text: \n",
            "A Sinkhole caved in a Brooklyn Street today on 5th Ave &amp; 64th Street. http://t.co/eUKhLKgGdO\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.6379731297492981\n",
            "\n",
            "Text: \n",
            "HQhed mhtw4fnet\n",
            "\n",
            "A sad reminder of nuclear devastation - Washington Post\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9345405697822571\n",
            "\n",
            "Text: \n",
            "Mourning notices for stabbing arson victims stir Û÷politics of griefÛª in Israel: Posters for Shira Banki and A... http://t.co/WbCtkGGTY9\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Custom example\n",
        "pred_prob = tf.squeeze(model_6_pretrained.predict([\"\"\"Strong #earthquake of 7.6 magnitude strikes #Papua_New_Guinea. #tsunami alerted. Another 6.1 magnitude strikes #Indonesia. Pray for the safety. #EarthquakePH #Disaster #DisasterRelief \"\"\"]))\n",
        "pred = tf.round(pred_prob)\n",
        "print(f\"Pred: {int(pred)}, Prob: {pred_prob}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TWCmWHooQIqc",
        "outputId": "561072f4-6007-488b-9faf-0783bf2edc19"
      },
      "execution_count": 131,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 1, Prob: 0.9912919402122498\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The speed score tradeoff"
      ],
      "metadata": {
        "id": "n79UjCiwQIoX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's make a function to measure the time of prediction\n",
        "import time\n",
        "def pred_timer(model, samples):\n",
        "  \"\"\"\n",
        "  Time how long a model takes to make prediction on a sample.\n",
        "  Args:\n",
        "    model: model by which you'll make prediction\n",
        "    sample: samples list\n",
        "  Returns:\n",
        "    total_time: total time to make prediction\n",
        "    time_per_pred: time one prediction gets\n",
        "  \"\"\"\n",
        "  start_time = time.perf_counter()\n",
        "  model.predict(samples)\n",
        "  end_time = time.perf_counter()\n",
        "  total_time = end_time - start_time\n",
        "  time_per_pred = total_time/len(samples)\n",
        "  return total_time, time_per_pred"
      ],
      "metadata": {
        "id": "vgB_oiGeQImO"
      },
      "execution_count": 134,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_total_pred_time, model_6_per_pred_time = pred_timer(model_6_pretrained, val_sentences)\n",
        "model_6_total_pred_time, model_6_per_pred_time"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hAsaRA8XQIkI",
        "outputId": "1a92db39-bff0-4b99-b52e-16c5cf89f449"
      },
      "execution_count": 135,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.24438149199977488, 0.00032071061942227675)"
            ]
          },
          "metadata": {},
          "execution_count": 135
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_total_pred_time, baseline_per_pred_time = pred_timer(model_0, val_sentences)\n",
        "baseline_total_pred_time, baseline_per_pred_time"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EY1vIn-0QIh3",
        "outputId": "9b81e2ef-c431-482f-f7bc-ce691998eed6"
      },
      "execution_count": 136,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.021298624999872118, 2.795095144340173e-05)"
            ]
          },
          "metadata": {},
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_6_pretrained_results = calculate_results(val_labels,model_6_pretrained_preds)\n",
        "model_6_pretrained_results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xtjF0MDCnUOo",
        "outputId": "5dafcc5b-cc2c-4be0-d5a6-96f0d35b43cb"
      },
      "execution_count": 137,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.62729658792651,\n",
              " 'precision': 0.818446310697231,\n",
              " 'recall': 0.8162729658792651,\n",
              " 'f1': 0.8148082644367335}"
            ]
          },
          "metadata": {},
          "execution_count": 137
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure(figsize=(10,7))\n",
        "plt.scatter(baseline_per_pred_time, baseline_results[\"f1\"], label=\"baseline\")\n",
        "plt.scatter(model_6_per_pred_time, model_6_results[\"f1\"], label=\"model_6\")\n",
        "plt.legend()\n",
        "plt.title(\"F1 score vs Time per prediction\")\n",
        "plt.xlabel(\"Time per pred\")\n",
        "plt.ylabel(\"F1 score\");"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "id": "67a-pBPVQIfM",
        "outputId": "57d88cef-7dc5-4b36-e3dc-f8a8fd7fd827"
      },
      "execution_count": 140,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAG5CAYAAAA3e7gZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7xVdZ3v8ddHRLEfSgN0S0HByVRUBpqj5o8a05ualdIdp3DQxl9Zmtl0HW96+yHjzbKxe+vq2HSt1CY1c6gxMx3JX5ll6SEQRaXIUEBLRMEB8Qf4uX+sdXRzPD82wmaf7+H1fDz2w7W/a63v+q7v2R3efb/re3ZkJpIkSRr4Nmt3AyRJktQcg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukjZZEbF9RKyIiCHtbstAFxELIuK/1tv/MyK+9SrrmRsRB2zQxkmbEIObNAjU/6iuqkNI12vbet/FETEvIl6MiGPb3NSNKiLe0dAfKyMiG/sIIDNfl5lr2t3WkmTmFzPzxP6Oi4jLIuIL3c7dLTNva1njpEHO4CYNHu+vQ0jX69G6/B7gFOA3bWwbABGx+ca8Xmb+vKs/gN3q4uENffTIxmzPhhCV9frdvbF/DpI2HIObNMhl5kWZeTPwbH/HRsRhEXF/RPxnRCyOiH9o2HdERMyOiKcj4vcRcWhdvm1EXBsRT0bE/Ij4SMM50yJiekRcHhFPA8dGxDYR8e2IeKy+xhd6mqqs610VEX/WUDYpIp6IiKER8ZaI+FlELK/Lvr+ufRMRY+tRuM3r97fV7fllPSr344gYERFX1Pd9d0SMbTh/l4j4aX3v8yLig31c67aI+FJE3FXX9aNu9/b2+rrLIuKexunE+txzI+IXwDPAjj3UvyAizqp/fk9FxKURMazed0BELIqIT0fEH4FLI2KziDiz/lkujYiru7XnmIh4uN73mW7XmhYRlze837+h7Qsj4tiIOAmYCvyPrr5saGfXlOuWEfG1iHi0fn0tIrbs1ubTI+Lx+vNyXDM/V2kwM7hJavRt4KOZ+Xpgd+AWgIjYC/hX4AxgOPBOYEF9zlXAImBb4EjgixFxYEOdRwDT6/OuAC4DVgNvASYBBwOvmHarRwzvBP66ofhvgemZ+QLwv4AZwBuA0cCF63PjDaYAxwDbAX9et+FS4M+AB4CzASLitcBPgSuBN9bnfT0ixvdR94eB44E3U/XBBXVd2wE/Ab5QX+cfgB9ExKiGc48BTgJeDzzcS/1TgUPqdr8V+GzDvjfVde9Q1/MJYDLwV1Q/u6eAi+r2jAf+pb7mtsAIqj5+hYjYAbiBqv9HAROB2Zl5MdXP+5/q0c3393D6Z4C31+f8BbBXD23ehupncQJwUUS8oZd7lzYJBjdp8LimHvFYFhHXvMo6XgDGR8TWmflUZnZNr54AXJKZP83MFzNzcWY+GBFjgP2AT2fms5k5G/gWVUDpcmdmXpOZLwJbA4cBf5+ZKzPzceCrVKGnJ1cCR0E1RVgfd2VDW3cAtq2vfcervOfuLs3M32fmcqpA8vvMvCkzVwP/RhU2Ad4HLMjMSzNzdWbOAn4A/E0fdX83M+/LzJXA54AP1qONRwPXZ+b1df/+FOik6qsul2Xm3PpaL/RS/z9n5sLMfBI4l7rvai8CZ2fmc5m5CvgY8JnMXJSZzwHTgCPr0ccjgesy8/Z63+fq83vyt8BNmfm9zHwhM5fWn4NmTAXOyczHM3MJ8I9UYbHLC/X+FzLzemAFsHOTdUuDksFNGjwmZ+bw+jX5Vdbx11Rh4eF6GnKfunwM8Psejt8WeDIz/7Oh7GGqEZIuCxu2dwCGAo91hUzg/1GNWPXkB8A+EfFmqlG+F4Gf1/v+BxDAXVGtVDy+2Zvsx58atlf18P519fYOwN4NYXkZVRB5Ux91N/bFw1R9MbKu62+61bU/1chcT+c2W/+2De+XZGbjdPkOwL83XO8BYA3wX+rzXqqrDppLe7lmb5+NZmzL2qOH3du8tA7MXZ7h5f6XNkk+oCrpJZl5N3BERAwFTgWupvqHeSHV9Ft3jwJ/FhGvbwhv2wOLG6tt2F4IPAeM7PYPcm/teSoiZgAfAnYFrsrMrPf9EfgIVM9YATdFxO2ZOb/pG14/C4GfZea71+GcMQ3b21ONKD1R1/XdzPxIj2dVso99vdX/aMP77ucvBI7PzF90ryQiHqPq7673r6GaLu3JQqopzp701+ZHqQLk3F7aLKkbR9ykQS4itqgfUg9gaEQMix5WJdbHTY2IbeqpuKd5eXrs28BxEXFQ/VD7dhGxS2YuBH4JfKmudwLVtOrl3esHyMzHqJ5L+98RsXVd159HxF/1cQtXUk29HsnL06RExN9ERNdzV09RhYTepvNa4TrgrfVD/EPr154RsWsf5xwdEePrIHQO1fN6a6j66/0RcUhEDKn78oCG+2vWxyNidL3I4DNAXws2vgGcWz+jRkSMiogj6n3TgffViw62qNva278XVwD/NSI+GBGbR7WYY2K970/0sJCiwfeAz9bXHgl8nl4+O5IqBjdp8JtBNcW3L3Bxvf3OXo49BlgQ1QrQj1FN/ZGZdwHHUT2Pthz4GdVICVTPUY2lGin5d6rnqG7qoz0fBrYA7qcKXNNZe0qwu2uBnYA/ZuY9DeV7Ar+O6u+xXQt8MjMf6qOeDaoeYTyY6rm7R4E/Al8GtuzjtO9SLc74IzAMOK2uayHVIo7/CSyhGsU6g3X/HX0l1c/7Iarpyy/0cez/peq3GRHxn8CvgL3r9swFPl7X9xjVz2lRT5XUf1LlMOB04ElgNtVCA6gC//g+nrv8AtWzfHOAe6n+ZE1fbZY2eVHPOkiSWigibgMuz8xX9Y0DTdS/ADixn9AsqXCOuEmSJBXC4CZJklQIp0olSZIK4YibJElSITaJv+M2cuTIHDt2bLubIUmS1K+ZM2c+kZmjetq3SQS3sWPH0tnZ2e5mSJIk9Ssievs+YqdKJUmSSmFwkyRJKoTBTZIkqRCbxDNuPXnhhRdYtGgRzz77bLubUoxhw4YxevRohg4d2u6mSJK0Sdpkg9uiRYt4/etfz9ixY4mIdjdnwMtMli5dyqJFixg3bly7myNJ0iZpk50qffbZZxkxYoShrUkRwYgRIxyhlCSpjTbZ4AYY2taR/SVJUntt0sFNkiSpJAa3NlqwYAG77757S+q+7bbbeN/73gfAtddey3nnndeS60iSpI1nk12csCk5/PDDOfzww9vdDEmStJ4ccWvSNbMWs995tzDuzJ+w33m3cM2sxRuk3tWrVzN16lR23XVXjjzySJ555hnOOecc9txzT3bffXdOOukkMhOACy64gPHjxzNhwgSmTJkCwMqVKzn++OPZa6+9mDRpEj/60Y9ecY3LLruMU089FYBjjz2W0047jX333Zcdd9yR6dOnv3Tc+eefz5577smECRM4++yzN8j9SZKkDcfg1oRrZi3mrB/ey+Jlq0hg8bJVnPXDezdIeJs3bx6nnHIKDzzwAFtvvTVf//rXOfXUU7n77ru57777WLVqFddddx0A5513HrNmzWLOnDl84xvfAODcc8/lwAMP5K677uLWW2/ljDPOYOXKlX1e87HHHuOOO+7guuuu48wzzwRgxowZ/O53v+Ouu+5i9uzZzJw5k9tvv32970+SJG04BrcmnH/jPFa9sGatslUvrOH8G+etd91jxoxhv/32A+Doo4/mjjvu4NZbb2Xvvfdmjz324JZbbmHu3LkATJgwgalTp3L55Zez+ebVLPeMGTM477zzmDhxIgcccADPPvssjzzySJ/XnDx5Mpttthnjx4/nT3/600v1zJgxg0mTJvG2t72NBx98kN/97nfrfX+SJGnD8Rm3Jjy6bNU6la+L7n9iIyI45ZRT6OzsZMyYMUybNu2lv532k5/8hNtvv50f//jHnHvuudx7771kJj/4wQ/Yeeed16qnK5D1ZMstt3xpu2saNjM566yz+OhHP7re9yRJ0qAy52q4+RxYvgi2GQ0HfR4mfLAtTXHErQnbDt9qncrXxSOPPMKdd94JwJVXXsn+++8PwMiRI1mxYsVLz6C9+OKLLFy4kHe96118+ctfZvny5axYsYJDDjmECy+88KUANmvWrFfVjkMOOYRLLrmEFStWALB48WIef/zx9b09SZLKNudq+PFpsHwhkNV/f3xaVd4Gjrg14YxDduasH9671nTpVkOHcMYhO/dxVnN23nlnLrroIo4//njGjx/PySefzFNPPcXuu+/Om970Jvbcc08A1qxZw9FHH83y5cvJTE477TSGDx/O5z73Of7+7/+eCRMm8OKLLzJu3LiXnolbFwcffDAPPPAA++yzDwCve93ruPzyy3njG9+43vcoSVKxbj4HXug2w/bCqqq8DaNu0TVSM5h1dHRkZ2fnWmUPPPAAu+66a9N1XDNrMeffOI9Hl61i2+FbccYhOzN50nYbuqkD3rr2myRJRZs2HOgpKwVMW9aSS0bEzMzs6GmfI25Nmjxpu00yqEmStEnbZnQ9TdpDeRv4jJskSVJvDvo8DO32TPvQraryNjC4SZIk9WbCB+H9F8A2Y4Co/vv+C9q2qtSpUkmSpL5M+GDbglp3jrhJkiQVwuAmSZJUCIObJElSIQxug8TYsWN54okn1uuYZcuWceSRR7LLLruw6667vvSNDpIkaWBwcUKzBtD3lLXKJz/5SQ499FCmT5/O888/zzPPPNPuJkmSpAaOuDWjRd9TtmDBAnbZZReOPfZY3vrWtzJ16lRuuukm9ttvP3baaSfuuusunnzySSZPnsyECRN4+9vfzpw5cwBYunQpBx98MLvtthsnnngijd+Acfnll7PXXnsxceJEPvrRj7JmzZremvCS5cuXc/vtt3PCCScAsMUWWzB8+PD1uj9JkrRhGdya0df3lK2n+fPnc/rpp/Pggw/y4IMPcuWVV3LHHXfwla98hS9+8YucffbZTJo0iTlz5vDFL36RD3/4wwD84z/+I/vvvz9z587lAx/4AI888ghQfSXV97//fX7xi18we/ZshgwZwhVXXNFvO/7whz8watQojjvuOCZNmsSJJ57IypUr1/v+JEnShmNwa8byRetWvg7GjRvHHnvswWabbcZuu+3GQQcdRESwxx57sGDBAu644w6OOeYYAA488ECWLl3K008/ze23387RRx8NwHvf+17e8IY3AHDzzTczc+ZM9txzTyZOnMjNN9/MQw891G87Vq9ezW9+8xtOPvlkZs2axWtf+1rOO++89b4/SZK04fiMWzNa+D1lW2655Uvbm2222UvvN9tsM1avXs3QoUPXqb7M5O/+7u/40pe+tE7njR49mtGjR7P33nsDcOSRRxrcJEkaYBxxa0Ybv6fsHe94x0tTnbfddhsjR45k66235p3vfCdXXnklADfccANPPfVU1dSDDmL69Ok8/vjjADz55JM8/PDD/V7nTW96E2PGjGHevHlANXI3fvz4VtySJEl6lRxxa0bX6tE2rCqdNm0axx9/PBMmTOA1r3kN3/nOdwA4++yzOeqoo9htt93Yd9992X777QEYP348X/jCFzj44IN58cUXGTp0KBdddBE77LBDv9e68MILmTp1Ks8//zw77rgjl156aUvvTZIkrZtoXI04WHV0dGRnZ+daZQ888AC77rprm1pULvtNkqTWioiZmdnR0z6nSiVJkgrhVOkmZunSpRx00EGvKL/55psZMWJEG1okSZKatUkHt8wkItrdjI1qxIgRzJ49+1WduylMq0uSNJBtslOlw4YNY+nSpYaRJmUmS5cuZdiwYe1uiiRJm6xNdsRt9OjRLFq0iCVLlrS7KcUYNmwYo0ev/9+ukyRJr84mG9yGDh3KuHHj2t0MSZKkpm2yU6WSJEmlMbhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBWipcEtIg6NiHkRMT8izuxh//YRcWtEzIqIORFxWF0+oi5fERH/3O2cv4yIe+s6L4iIaOU9SJIkDRQtC24RMQS4CHgPMB44KiLGdzvss8DVmTkJmAJ8vS5/Fvgc8A89VP0vwEeAnerXoRu+9ZIkSQNPK0fc9gLmZ+ZDmfk8cBVwRLdjEti63t4GeBQgM1dm5h1UAe4lEfFmYOvM/FVmJvCvwOQW3oMkSdKA0crgth2wsOH9orqs0TTg6IhYBFwPfKKJOhf1UycAEXFSRHRGROeSJUvWpd2SJEkDUrsXJxwFXJaZo4HDgO9GxAZpU2ZenJkdmdkxatSoDVGlJElSW7UyuC0GxjS8H12XNToBuBogM+8EhgEj+6lzdD91SpIkDUqtDG53AztFxLiI2IJq8cG13Y55BDgIICJ2pQpuvc5rZuZjwNMR8fZ6NemHgR+1ovGSJEkDzeatqjgzV0fEqcCNwBDgksycGxHnAJ2ZeS1wOvDNiPgU1UKFY+tFB0TEAqqFC1tExGTg4My8HzgFuAzYCrihfkmSJA16UeekQa2joyM7Ozvb3QxJkqR+RcTMzOzoaV+7FydIkiSpSQY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSpES4NbRBwaEfMiYn5EnNnD/u0j4taImBURcyLisIZ9Z9XnzYuIQxrKF0TEvRExOyI6W9l+SZKkgWTzVlUcEUOAi4B3A4uAuyPi2sy8v+GwzwJXZ+a/RMR44HpgbL09BdgN2Ba4KSLemplr6vPelZlPtKrtkiRJA1ErR9z2AuZn5kOZ+TxwFXBEt2MS2Lre3gZ4tN4+ArgqM5/LzD8A8+v6JEmSNlmtDG7bAQsb3i+qyxpNA46OiEVUo22faOLcBGZExMyIOKm3i0fESRHRGRGdS5YsefV3IUmSNEC0e3HCUcBlmTkaOAz4bkT016b9M/NtwHuAj0fEO3s6KDMvzsyOzOwYNWrUhm21JElSG7QyuC0GxjS8H12XNToBuBogM+8EhgEj+zo3M7v++zjw7ziFKkmSNhGtDG53AztFxLiI2IJqscG13Y55BDgIICJ2pQpuS+rjpkTElhExDtgJuCsiXhsRr6+Pfy1wMHBfC+9BkiRpwGjZqtLMXB0RpwI3AkOASzJzbkScA3Rm5rXA6cA3I+JTVM+uHZuZCcyNiKuB+4HVwMczc01E/Bfg3yOiq+1XZuZ/tOoeJEmSBpKoctLg1tHRkZ2d/sk3SZI08EXEzMzs6GlfuxcnSJIkqUkGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQjQV3CJi/4g4rt4eFRHjWtssSZIkdddvcIuIs4FPA2fVRUOBy1vZKEmSJL1SMyNuHwAOB1YCZOajwOtb2ShJkiS9UjPB7fnMTCABIuK1rW2SJEmSetJMcLs6Iv4fMDwiPgLcBHyztc2SJElSd5v3tTMiAvg+sAvwNLAz8PnM/OlGaJskSZIa9BncMjMj4vrM3AMwrEmSJLVRM1Olv4mIPVveEkmSJPWpzxG32t7A1Ih4mGplaVANxk1oacskSZK0lmaC2yEtb4UkSZL61e9UaWY+DAwH3l+/htdl/YqIQyNiXkTMj4gze9i/fUTcGhGzImJORBzWsO+s+rx5EXFIs3VKkiQNVs18c8IngSuAN9avyyPiE02cNwS4CHgPMB44KiLGdzvss8DVmTkJmAJ8vT53fP1+N+BQ4OsRMaTJOiVJkgalZqZKTwD2zsyVABHxZeBO4MJ+ztsLmJ+ZD9XnXQUcAdzfcEwCW9fb2wCP1ttHAFdl5nPAHyJifl0fTdQpSZI0KDWzqjSANQ3v19Rl/dkOWNjwflFd1mgacHRELAKuB7pG8no7t5k6q0ZHnBQRnRHRuWTJkiaaK0mSNLA1E9wuBX4dEdMiYhrwK+DbG+j6RwGXZeZo4DDguxHRTJv6lZkXZ2ZHZnaMGjVqQ1QpSZLUVv1OlWbm/4mI24D966LjMnNWE3UvBsY0vB9dlzU6geoZNjLzzogYBozs59z+6pQkSRqUmlmc8Hbgd5l5QWZeAPw+IvZuou67gZ0iYlxEbEG12ODabsc8AhxUX2dXYBiwpD5uSkRsGRHjgJ2Au5qsU5IkaVBqZnHCvwBva3i/ooeyV8jM1RFxKnAjMAS4JDPnRsQ5QGdmXgucDnwzIj5FtVDh2MxMYG5EXE216GA18PHMXAPQU53N364kSVK5ospJfRwQMTszJ3Yrm1PSNyd0dHRkZ2dnu5shSZLUr4iYmZkdPe1rZiHAQxFxWkQMrV+fBB7asE2UJElSf5oJbh8D9qVaBLCI6rtLT2ployRJkvRKzawqfZxqEYAkSZLaqJlVpf8UEVvX06Q3R8SSiDh6YzROkiRJL2tmqvTgzHwaeB+wAHgLcEYrGyVJkqRXaia4dU2nvhf4t8xc3sL2SJIkqRfN/B236yLiQWAVcHJEjAKebW2zJEmS1F2/I26ZeSbVqtKOzHwBeAY4otUNkyRJ0tqaGXEjM59s2F4JrGxZiyRJktSjZp5xkyRJ0gBgcJMkSSrEqwpuEbHLhm6IJEmS+vZqR9xmbNBWSJIkqV+9Lk6IiAt62wUMb01zJEmS1Ju+VpUeB5wOPNfDvqNa0xxJkiT1pq/gdjdwX2b+svuOiJjWshZJkiSpR30FtyPp5RsSMnNca5ojSZKk3vS1OOF1mfnMRmuJJEmS+tRXcLumayMifrAR2iJJkqQ+9BXcomF7x1Y3RJIkSX3rK7hlL9uSJElqg74WJ/xFRDxNNfK2Vb1N/T4zc+uWt06SJEkv6TW4ZeaQjdkQSZIk9c0vmZckSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCGNwkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCmFwkyRJKkRLg1tEHBoR8yJifkSc2cP+r0bE7Pr124hY1rDvyxFxX/36UEP5ZRHxh4bzJrbyHiRJkgaKzVtVcUQMAS4C3g0sAu6OiGsz8/6uYzLzUw3HfwKYVG+/F3gbMBHYErgtIm7IzKfrw8/IzOmtarskSdJA1MoRt72A+Zn5UGY+D1wFHNHH8UcB36u3xwO3Z+bqzFwJzAEObWFbJUmSBrxWBrftgIUN7xfVZa8QETsA44Bb6qJ7gEMj4jURMRJ4FzCm4ZRzI2JOPdW6ZS91nhQRnRHRuWTJkvW9F0mSpLYbKIsTpgDTM3MNQGbOAK4Hfkk1CncnsKY+9ixgF2BP4M+AT/dUYWZenJkdmdkxatSoFjdfkiSp9VoZ3Baz9ijZ6LqsJ1N4eZoUgMw8NzMnZua7gQB+W5c/lpXngEuppmQlSZIGvVYGt7uBnSJiXERsQRXOru1+UETsAryBalStq2xIRIyotycAE4AZ9fs31/8NYDJwXwvvQZIkacBo2arSzFwdEacCNwJDgEsyc25EnAN0ZmZXiJsCXJWZ2XD6UODnVTbjaeDozFxd77siIkZRjcLNBj7WqnuQJEkaSGLtvDQ4dXR0ZGdnZ7ubIUmS1K+ImJmZHT3tGyiLEyRJktQPg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFaKlwS0iDo2IeRExPyLO7GH/VyNidv36bUQsa9j35Yi4r359qKF8XET8uq7z+xGxRSvvQZIkaaBoWXCLiCHARcB7gPHAURExvvGYzPxUZk7MzInAhcAP63PfC7wNmAjsDfxDRGxdn/Zl4KuZ+RbgKeCEVt2DJEnSQNLKEbe9gPmZ+VBmPg9cBRzRx/FHAd+rt8cDt2fm6j9g4xIAAAwKSURBVMxcCcwBDo2IAA4EptfHfQeY3JLWS5IkDTCtDG7bAQsb3i+qy14hInYAxgG31EX3UAW110TESOBdwBhgBLAsM1c3UedJEdEZEZ1LlixZ75uRJElqt4GyOGEKMD0z1wBk5gzgeuCXVKNwdwJr1qXCzLw4Mzsys2PUqFEbur2SJEkbXSuD22KqUbIuo+uynkzh5WlSADLz3Pr5t3cDAfwWWAoMj4jNm6hTkiRpUGllcLsb2KleBboFVTi7tvtBEbEL8AaqUbWusiERMaLengBMAGZkZgK3AkfWh/4d8KMW3oMkSdKAsXn/h7w6mbk6Ik4FbgSGAJdk5tyIOAfozMyuEDcFuKoOZV2GAj+v1iLwNHB0w3NtnwauiogvALOAb7fqHiRJkgaSWDsvDU4dHR3Z2dnZ7mZIkiT1KyJmZmZHT/sGyuIESZIk9cPgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBXC4CZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObJElSIQxukiRJhTC4SZIkFcLgJkmSVAiDmyRJUiEMbpIkSYUwuEmSJBVi83Y3oHTXzFrM+TfO49Flq9h2+FacccjOTJ60XbubJUmSBiGD23q4ZtZizvrhvax6YQ0Ai5et4qwf3gtgeJMkSRucU6Xr4fwb570U2rqsemEN5984r00tkiRJg5nBbT08umzVOpVLkiStD4Pbeth2+FbrVC5JkrQ+DG7r4YxDdmaroUPWKttq6BDOOGTnNrVIkiQNZi5OWA9dCxBcVSpJkjYGg9t6mjxpO4OaJEnaKJwqlSRJKoTBTZIkqRAGN0mSpEIY3CRJkgphcJMkSSqEwU2SJKkQBjdJkqRCtDS4RcShETEvIuZHxJk97P9qRMyuX7+NiGUN+/4pIuZGxAMRcUFERF1+W11n13lvbOU9SJIkDRQt+wO8ETEEuAh4N7AIuDsirs3M+7uOycxPNRz/CWBSvb0vsB8wod59B/BXwG31+6mZ2dmqtkuSJA1ErRxx2wuYn5kPZebzwFXAEX0cfxTwvXo7gWHAFsCWwFDgTy1sqyRJ0oDXyuC2HbCw4f2iuuwVImIHYBxwC0Bm3gncCjxWv27MzAcaTrm0nib9XNcUag91nhQRnRHRuWTJkvW/G0mSpDYbKIsTpgDTM3MNQES8BdgVGE0V9g6MiHfUx07NzD2Ad9SvY3qqMDMvzsyOzOwYNWpUy29AkiSp1Vr5JfOLgTEN70fXZT2ZAny84f0HgF9l5gqAiLgB2Af4eWYuBsjM/4yIK6mmZP+1r4bMnDnziYh4+FXdRflGAk+0uxGDlH3bOvZta9ivrWPfts6m2Lc79LajlcHtbmCniBhHFdimAH/b/aCI2AV4A3BnQ/EjwEci4ktAUC1M+FpEbA4Mz8wnImIo8D7gpv4akpmb7JBbRHRmZke72zEY2betY9+2hv3aOvZt69i3a2vZVGlmrgZOBW4EHgCuzsy5EXFORBzecOgU4KrMzIay6cDvgXuBe4B7MvPHVAsVboyIOcBsqkD4zVbdgyRJ0kDSyhE3MvN64PpuZZ/v9n5aD+etAT7aQ/lK4C83bCslSZLKMFAWJ6h1Lm53AwYx+7Z17NvWsF9bx75tHfu2Qaw9QylJkqSByhE3SZKkQhjcJEmSCmFwG6Ai4tCImBcR8yPizB72bxkR36/3/zoixjbsO6sunxcRh/RXZ0SMq+uYX9e5RV1+bEQsqb+lYnZEnNjau269jdyvp9ZlGREjG8ojIi6o982JiLe17o43ngHStwdExPKGz+xai6FKtJH79Yq6/L6IuKT+s0t+Zlvbt4PuMwsbvW+/HRH31J/N6RHxuv6uUbTM9DXAXsAQqj+HsiPV97XeA4zvdswpwDfq7SnA9+vt8fXxW1J9jdjv6/p6rRO4GphSb38DOLnePhb453b3R8H9OgkYCywARjZc4zDgBqq/Ufh24Nft7ptB1LcHANe1uz8K7tfD6s9lUH139MkN5X5mW9O3g+oz26a+3bqh3v8DnNnXNUp/OeI2MO0FzM/MhzLzeeAq4IhuxxwBfKfeng4cFBFRl1+Vmc9l5h+A+XV9PdZZn3NgXQd1nZNbeG/ttNH6FSAzZ2Xmgh7acQTwr1n5FTA8It68Qe904xsofTvYbOx+vb7+XCZwF9U33nRdw89sa/p2MNrYffs0VCPDwFZA9nONohncBqbtgIUN7xfVZT0ek9UfO14OjOjj3N7KRwDL6jp6utZfNww/N36FWYk2Zr+ubztKM1D6FmCfetrkhojYbV1uYgBqS7/W03jHAP+xDu0ozUDpWxhcn1loQ99GxKXAH4FdgAv7uUbRDG7qy4+BsZk5AfgpL/8/F2mg+g2wQ2b+BdUv72va3J5SfR24PTN/3u6GDELd+9bP7AaQmccB21J9U9OH2tycljK4DUyLgcbRrdF1WY/HRPUdrtsAS/s4t7fypVTTHpt3Kyczl2bmc3X5tyj/Wys2Zr+ubztKMyD6NjOfzswV9fb1wNDGxQsF2uj9GhFnA6OA/76O7SjNgOjbQfiZhTb9PsjqW5euAv66n2uUrd0P2fl65Yvqq8geonows+shzN26HfNx1n7o8up6ezfWfrDzIaqHOnutE/g31l6ccEq9/eaG630A+FW7+6akfm2ocwFrP0D/XtZ+0PuudvfNIOrbN/HyHxbfC3ik632Jrzb8LjgR+CWwVbdr+JltXd8Oqs/sxu7b+jP5lvrcAL4CfKWva5T+ansDfPXyg6lWIP2WahXNZ+qyc4DD6+1hVIFrPtWDrjs2nPuZ+rx5wHv6qrMu37GuY35d55Z1+ZeAufX/QG4Fdml3vxTWr6dRPYexGngU+FZdHsBF9fH3Ah3t7pdB1LenNnxmfwXs2+5+KaxfV9dls+vX5/3MtrxvB91ndmP2LdXM4S/qz+V9wBXUq0z7ukbJL7/ySpIkqRA+4yZJklQIg5skSVIhDG6SJEmFMLhJkiQVwuAmSZJUCIObpKJExIiImF2//hgRi+vtFRHx9Xa3b2OKiLERcV+72yFp49m8/0MkaeDIzKXARICImAasyMyvtLVRvYiIzfPl7wFu+XmSBj9H3CQNChFxQERcV29Pi4jvRMTPI+LhiPhvEfFPEXFvRPxH/UXfRMRfRsTPImJmRNwYEW/uod7LIuIbEdEZEb+NiPfV5UMi4vyIuDsi5kTERxva8fOIuBa4v4f6VkTEVyNibkTcHBGj6vLbIuJrEdEJfLK3ttXl90TEPVR/GV7SJsTgJmmw+nPgQOBw4HLg1szcA1gFvLcObxcCR2bmXwKXAOf2UtdYqq8jei/wjYgYBpwALM/MPYE9gY9ExLj6+LcBn8zMt/ZQ12uBzszcDfgZcHbDvi0yswO4oI+2XQp8IqsvJZe0iXGqVNJgdUNmvhAR91J91+F/1OX3UgWxnYHdgZ9GBPUxj/VS19WZ+SLwu4h4CNgFOBiYEBFH1sdsA+wEPE/1XZ5/6KWuF4Hv19uXAz9s2NdV3mPbImI4MDwzb6+P+y7wnr46QdLgYnCTNFg9B5CZL0bEC/ny9/u9SPW7L4C5mblPE3V1/27ArM//RGbe2LgjIg4AVq5DOxvr7jqvx7bVwU3SJsypUkmbqnnAqIjYByAihkbEbr0c+zcRsVlE/DmwY33ujcDJDc/LvTUiXtvEdTcDukbp/ha4o9m2ZeYyYFlE7F8fN7WJ60kaRBxxk7RJyszn62nOCyJiG6rfh18D5vZw+CPAXcDWwMcy89mI+BbVlOtvoprPXAJMbuLSK4G9IuKzwOPAh9axbccBl0REAjPW5Z4llS9enj2QJHUXEZcB12Xm9A1U34rMfN2GqEvSpsepUkmSpEI44iZJklQIR9wkSZIKYXCTJEkqhMFNkiSpEAY3SZKkQhjcJEmSCvH/Adyxx+IhvMDRAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0jPHg-b0QHA0"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}