practical_sessions/3-Text_Preprocessing/Practical_Session-Text_Preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "30f9e621",
   "metadata": {},
   "source": [
    "# Text Preprocessing\n",
    "## CSSM530: Automated Text Processing for Social Sciences\n",
    "### Table of Contents:\n",
    "\n",
    "1. [Text Preprocessing](#text_preproc)\n",
    "    - [Removing punctuation](#remove_punc)\n",
    "    - [Removing URLs](#remove_url)\n",
    "    - [Lower Casing](#lower)\n",
    "    - [Tokenization](#tokenize)\n",
    "    - [Removing Stop Words](#remove_stop)\n",
    "    - [Removing Emoji](#remove_emoji)\n",
    "    - [Stemming & Lemmatization](#stem_lemma)\n",
    "2. [Converting Text to Numbers (Text Vectorization)](#vectorization)\n",
    "    - [Bag-of-Words](#bow)\n",
    "    - [Term Frequency–Inverse Document Frequency (TF-IDF)](#tfidf)\n",
    "    - [Building a Simple Classifier](#classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b885f8",
   "metadata": {},
   "source": [
    "## 0- Discussion\n",
    "\n",
    "- Preprocessing $\\neq$ removing\n",
    "    - In most cases, you may want to keep or replace stop words, URLs, puncuations, emojis. For example,\n",
    "        - URLs can be replaced with standard tokens like \\<URL\\>\n",
    "        - Instead of removing hashtags or out-of-vocabulary words, you may use sequence analysis algorithms to find the most likely combination of words: #whataday &#8594; what a day\n",
    "- Preprocessing is highly context-dependent\n",
    "    - The methods you are going to be used will change dramatically depending on your use case!\n",
    "    - Therefore, you have to decide what features to remove, replace or keep depending on your research."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bff186a",
   "metadata": {},
   "source": [
    "# 1-  Text Preprocessing <a class=\"anchor\" id=\"text_preproc\"></a>\n",
    "\n",
    "- For a comprehensive overview Text Preprocessing for Social Media text, please see the following tutorial:\n",
    "    - [Preprocessing Social Media Text](https://bit.ly/nlpcss201-preproc) by [Steve Wilson](https://steverw.com/) tutorial presented at [NLP+CSS 201 tutorial series](https://nlp-css-201-tutorials.github.io/nlp-css-201-tutorials/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "10f166aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>user_type</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>expanded_urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>nicolebyer</td>\n",
       "      <td>comedy</td>\n",
       "      <td>1501332398092414978</td>\n",
       "      <td>2022-03-08 23:01:51+00:00</td>\n",
       "      <td>@ashleyn1cole 🥰💜🥰💜</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>nicolebyer</td>\n",
       "      <td>comedy</td>\n",
       "      <td>1501241039440400385</td>\n",
       "      <td>2022-03-08 16:58:50+00:00</td>\n",
       "      <td>😭😭😭😭😭😭 https://t.co/X7LngGNLlt</td>\n",
       "      <td>https://twitter.com/laurenlapkus/status/150124...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>nicolebyer</td>\n",
       "      <td>comedy</td>\n",
       "      <td>1500957090990362625</td>\n",
       "      <td>2022-03-07 22:10:31+00:00</td>\n",
       "      <td>Yah watch #GrandCrew and tell ya friends to wa...</td>\n",
       "      <td>https://twitter.com/eclecticjunki3/status/1500...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         user user_type             tweet_id                 created_at  \\\n",
       "0  nicolebyer    comedy  1501332398092414978  2022-03-08 23:01:51+00:00   \n",
       "1  nicolebyer    comedy  1501241039440400385  2022-03-08 16:58:50+00:00   \n",
       "2  nicolebyer    comedy  1500957090990362625  2022-03-07 22:10:31+00:00   \n",
       "\n",
       "                                                text  \\\n",
       "0                                 @ashleyn1cole 🥰💜🥰💜   \n",
       "1                     😭😭😭😭😭😭 https://t.co/X7LngGNLlt   \n",
       "2  Yah watch #GrandCrew and tell ya friends to wa...   \n",
       "\n",
       "                                       expanded_urls  \n",
       "0                                                NaN  \n",
       "1  https://twitter.com/laurenlapkus/status/150124...  \n",
       "2  https://twitter.com/eclecticjunki3/status/1500...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Import data\n",
    "df = pd.read_csv(\"https://raw.githubusercontent.com/steve-wilson/nlpcss201-sm-preprocessing/main/celebrity_tweets_no_rts.csv\")\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e9ab9593",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@nicolebyer:\n",
      "@BisHilarious 💜💜💜🥳🥳🥳\n",
      "---\n",
      "@serenawilliams:\n",
      "I’m somehow watching IronMan 1 again 🤷🏿‍♀️🤩 @Marvel\n",
      "---\n",
      "@jvn:\n",
      "Watching Kaori cry is making me SOB #WinterOlympics\n",
      "---\n",
      "@AOC:\n",
      "Top winter picket line accessories:\n",
      "📢 Bullhorn\n",
      "🧤 Gloves\n",
      "♨️ Handwarmers\n",
      "🤝 Solidarity\n",
      "\n",
      "Shout out @TeenVogue for covering these important issues with the depth that they deserve.\n",
      "\n",
      "Let’s get these produce workers the buck they’re asking for. 💪🏽 https://t.co/uz6lMSQuPE\n",
      "Expanded URLs: https://twitter.com/TeamstersJC16/status/1352448474416107526\n",
      "---\n",
      "@Oprah:\n",
      "Do u all agree we are each versions of each other? #SuperSoulSunday\n",
      "---\n",
      "@usainbolt:\n",
      "Yessss @Cristiano @ManUtd 🙌🏿\n",
      "---\n",
      "@Zendaya:\n",
      "HAPPY #BeyDay !!!!!!🐝👑\n",
      "---\n",
      "@TheRock:\n",
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "Expanded URLs: https://twitter.com/NFL/status/1492561005984899072\n",
      "---\n",
      "@taylorswift13:\n",
      "ARE YOU KIDDING ME THIS IS THE NICEST THING ANYONE HAS EVER SAID - you’re just the best and so kind to say this 😭💕🙏😻 https://t.co/y3z1JzSejX\n",
      "Expanded URLs: https://twitter.com/KeithUrban/status/1162763090594865154\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# Examples from the dataset\n",
    "row_ids = [51, 2054, 7661, 12525, 29235, 30183, 50312, 52006, 54902]\n",
    "for row_id in row_ids:\n",
    "    row = df.iloc[row_id]\n",
    "    print(f\"@{row.user}:\")\n",
    "    print(row.text)\n",
    "    if type(row.expanded_urls) == str:\n",
    "        print('Expanded URLs:',row.expanded_urls)\n",
    "    print('---')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cc30956",
   "metadata": {},
   "source": [
    "### Removing Puncuation <a class=\"anchor\" id=\"remove_punc\"></a>\n",
    "\n",
    "Three methods to remove punctuation:\n",
    "- string.isalnum()\n",
    "- string.replace()\n",
    "- Regex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "63dd4a67",
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1b93f1a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "! \" # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \\ ] ^ _ ` { | } ~ "
     ]
    }
   ],
   "source": [
    "for punc in punctuation:\n",
    "    print(punc, end= \" \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "af592250",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "tweet = df.loc[52006, \"text\"]\n",
    "print(tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0c825a5",
   "metadata": {},
   "source": [
    "#### string.isalnum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1002fd42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "speci87al\n"
     ]
    }
   ],
   "source": [
    "special_string=\"spe@#$ci87al*&\"\n",
    "print(''.join([char for char in special_string if char.isalnum()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "87f82cd0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ImpumpeduplikeImgettingsnapsatDEndtomorrowCantwaitforyoutoseewhatmyselfNFLampNBChavecookinYouguysknowmyNFLdreamnevercametruesoisthisisafullcirclemomenttostandonthisfieldGratitudeRockAtThe50FINALLYSBLVIhttpstcoyyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "print(''.join([char for char in tweet if char.isalnum()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "127b6ba6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Im pumped up like Im getting snaps at DEnd tomorrow \n",
      "Cant wait for you to see what myself NFL amp NBC have cookin   \n",
      "You guys know my NFL dream never came true so is this is a full circle  moment to stand on this field\n",
      "Gratitude\n",
      "RockAtThe50\n",
      "FINALLY\n",
      "SBLVI httpstcoyyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "print(''.join([char for char in tweet if char.isalnum() or char.isspace()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91d18882",
   "metadata": {},
   "source": [
    "#### string.replace()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "36ca6d5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello world\n"
     ]
    }
   ],
   "source": [
    "text = \"Hello world!\"\n",
    "print(text.replace(\"!\", \"\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d6b1693a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_processed = tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "4017efb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for punc in list(punctuation):\n",
    "    tweet_processed = tweet_processed.replace(punc, \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "2f97e198",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tweet:\n",
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "\n",
      "Tweet Processed:\n",
      "I’m pumped up like I’m getting snaps at DEnd tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself NFL amp NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true so is this is a full circle ⭕️ moment to stand on this field\n",
      "Gratitude\n",
      "RockAtThe50\n",
      "FINALLY\n",
      "SBLVI httpstcoyyIhvBADt6\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"Tweet:\\n{tweet}\\n\")\n",
    "print(f\"Tweet Processed:\\n{tweet_processed}\\n\") ## Apostrophe (’) is not included in python's string.punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "af379973",
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_processed = tweet\n",
    "for punc in list(punctuation) + [\"’\"]:\n",
    "    tweet_processed = tweet_processed.replace(punc, \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "95306a1f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tweet:\n",
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "\n",
      "Tweet Processed:\n",
      "Im pumped up like Im getting snaps at DEnd tomorrow 🤣💪🏾🏈\n",
      "Cant wait for you to see what myself NFL amp NBC have cookin 😊💥 🎤 \n",
      "You guys know my NFL dream never came true so is this is a full circle ⭕️ moment to stand on this field\n",
      "Gratitude\n",
      "RockAtThe50\n",
      "FINALLY\n",
      "SBLVI httpstcoyyIhvBADt6\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(f\"Tweet:\\n{tweet}\\n\")\n",
    "\n",
    "print(f\"Tweet Processed:\\n{tweet_processed}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f24271",
   "metadata": {},
   "source": [
    "#### Regex (Regular Expression)\n",
    "\n",
    "Tutorials:\n",
    "- [Python RegEx](https://www.w3schools.com/python/python_regex.asp) from [W3Schools](https://www.w3schools.com)\n",
    "- [Regular Expressions: Regexes in Python (Part 1)](https://realpython.com/regex-python/) from [Real Python](https://realpython.com/)\n",
    "- [Regular Expressions: Regexes in Python (Part 2)](https://realpython.com/regex-python-part-2/) from [Real Python](https://realpython.com/)\n",
    "- [Python Regular Expression Tutorial](https://www.datacamp.com/tutorial/python-regular-expression-tutorial) from [DataCamp](https://www.datacamp.com/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "752a0ad7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ba2ec409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Im pumped up like Im getting snaps at DEnd tomorrow \n",
      "Cant wait for you to see what myself NFL amp NBC have cookin   \n",
      "You guys know my NFL dream never came true so is this is a full circle  moment to stand on this field\n",
      "Gratitude\n",
      "RockAtThe50\n",
      "FINALLY\n",
      "SBLVI httpstcoyyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "pattern = r'[^\\w\\s0-9]' # remove everything except words, space and numbers\n",
    "\n",
    "print(re.sub(pattern, \"\", tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bf6b759e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Im pumped up like Im getting snaps at DEnd tomorrow \n",
      "Cant wait for you to see what myself @NFL amp @NBC have cookin   \n",
      "You guys know my NFL dream never came true so is this is a full circle  moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI httpstcoyyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "pattern = r'[^\\w\\s0-9#@]' # remove everything except words, space, numbers, # and @\n",
    "\n",
    "print(re.sub(pattern, \"\", tweet))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dffe4051",
   "metadata": {},
   "source": [
    "### Removing URLs  <a class=\"anchor\" id=\"remove_url\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9823e2b1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "['https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "# URL pattern for shortened Twitter URLs\n",
    "url_pattern = \"https?://t\\.co/\\S+\"\n",
    "\n",
    "print(tweet)\n",
    "print(re.findall(url_pattern, tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "df293fbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# URL pattern for expanded URLs (source: https://uibakery.io/regex-library/url-regex-python)\n",
    "url_pattern = \"^https?:\\\\/\\\\/(?:www\\\\.)?[-a-zA-Z0-9@:%._\\\\+~#=]{1,256}\\\\.[a-zA-Z0-9()]{1,6}\\\\b(?:[-a-zA-Z0-9()@:%_\\\\+.~#?&\\\\/=]*)$\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2033067e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://twitter.com/NFL/status/1492561005984899072\n",
      "['https://twitter.com/NFL/status/1492561005984899072']\n"
     ]
    }
   ],
   "source": [
    "expanded_url = df.loc[52006, \"expanded_urls\"]\n",
    "print(expanded_url)\n",
    "print(re.findall(url_pattern, expanded_url))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b113cf6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://findtheone.byspotify.com/share/db2a80bd0f2dd362325cb5138cd6b9b555b43481\n",
      "['https://findtheone.byspotify.com/share/db2a80bd0f2dd362325cb5138cd6b9b555b43481']\n"
     ]
    }
   ],
   "source": [
    "expanded_url = df.loc[176, \"expanded_urls\"]\n",
    "print(expanded_url)\n",
    "print(re.findall(url_pattern, expanded_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a504170",
   "metadata": {},
   "source": [
    "### Lower Casing  <a class=\"anchor\" id=\"lower\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7d2f783c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n"
     ]
    }
   ],
   "source": [
    "print(tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7879418f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i’m pumped up like i’m getting snaps at d-end tomorrow 🤣💪🏾🏈\n",
      "can’t wait for you to see what myself, @nfl &amp; @nbc have cookin’ 😊💥 🎤 \n",
      "you guys know my nfl dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#gratitude\n",
      "#rockatthe50\n",
      "#finally\n",
      "#sblvi https://t.co/yyihvbadt6\n"
     ]
    }
   ],
   "source": [
    "print(tweet.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2fc5d73",
   "metadata": {},
   "source": [
    "- [unicode_tr](https://github.com/emre/unicode_tr): A python module to make unicode strings work as expected for Turkish characters. Solves the turkish \"İ\" problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "49dc13b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unicode_tr import unicode_tr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4b7c3557",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_true = unicode_tr(u\"istanbul\")\n",
    "text_wrong = u\"istanbul\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "43afc850",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unicode_tr.unicode_tr"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(text_true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "425b508c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(text_wrong)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ac01876d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "İSTANBUL ISTANBUL\n"
     ]
    }
   ],
   "source": [
    "# string.upper\n",
    "print(text_true.upper(), text_wrong.upper())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6a63de23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "İstanbul Istanbul\n"
     ]
    }
   ],
   "source": [
    "# string.capitalize\n",
    "print(text_true.capitalize(), text_wrong.capitalize())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "6aa9953c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "çınar çinar\n"
     ]
    }
   ],
   "source": [
    "# string.lower\n",
    "text_true  = unicode_tr(u\"ÇINAR\")\n",
    "text_false = u\"ÇINAR\"\n",
    "print(text_true.lower(), text_false.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "23b70cbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "İzmir İstanbul Izmir Istanbul\n"
     ]
    }
   ],
   "source": [
    "# string.title\n",
    "text_true  = unicode_tr(u\"izmir istanbul\")\n",
    "text_false = u\"izmir istanbul\"\n",
    "\n",
    "print(text_true.title(), text_false.title())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2d01c1",
   "metadata": {},
   "source": [
    "### Tokenization  <a class=\"anchor\" id=\"tokenize\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "ca01f3a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import TweetTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c4f1a009",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣💪🏾🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@', 'NFL', '&', 'amp', ';', '@', 'NBC', 'have', 'cookin', '’', '😊💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#', 'Gratitude', '#', 'RockAtThe50', '#', 'FINALLY', '#', 'SBLVI', 'https', ':', '//t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print(tweet)\n",
    "print(word_tokenize(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5f8c6d0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\n",
      "Can’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \n",
      "You guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\n",
      "#Gratitude\n",
      "#RockAtThe50\n",
      "#FINALLY\n",
      "#SBLVI https://t.co/yyIhvBADt6\n",
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "tt = TweetTokenizer()\n",
    "print(tweet)\n",
    "print(tt.tokenize(tweet))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221e6868",
   "metadata": {},
   "source": [
    "### Removing Stop Words  <a class=\"anchor\" id=\"remove_stop\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a916b172",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "a6ca64c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "English:\n",
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n",
      "\n",
      "Turkish:\n",
      "['acaba', 'ama', 'aslında', 'az', 'bazı', 'belki', 'biri', 'birkaç', 'birşey', 'biz', 'bu', 'çok', 'çünkü', 'da', 'daha', 'de', 'defa', 'diye', 'eğer', 'en', 'gibi', 'hem', 'hep', 'hepsi', 'her', 'hiç', 'için', 'ile', 'ise', 'kez', 'ki', 'kim', 'mı', 'mu', 'mü', 'nasıl', 'ne', 'neden', 'nerde', 'nerede', 'nereye', 'niçin', 'niye', 'o', 'sanki', 'şey', 'siz', 'şu', 'tüm', 've', 'veya', 'ya', 'yani']\n"
     ]
    }
   ],
   "source": [
    "print(\"English:\")\n",
    "print(stopwords.words('english'))\n",
    "print(\"\\nTurkish:\")\n",
    "print(stopwords.words('turkish'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4c18d26f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I’m pumped up like I’m getting snaps at D-End tomorrow 🤣💪🏾🏈\\nCan’t wait for you to see what myself, @NFL &amp; @NBC have cookin’ 😊💥 🎤 \\nYou guys know my NFL dream never came true, so is this is a full circle ⭕️ moment to stand on this field\\n#Gratitude\\n#RockAtThe50\\n#FINALLY\\n#SBLVI https://t.co/yyIhvBADt6'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3ba68148",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized Tweet:\n",
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n",
      "\n",
      "Tokenized Tweet without Stop Words:\n",
      "['’', 'pumped', 'like', '’', 'getting', 'snaps', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', '’', 'wait', 'see', ',', '@NFL', '&', '@NBC', 'cookin', '’', '😊', '💥', '🎤', 'guys', 'know', 'NFL', 'dream', 'never', 'came', 'true', ',', 'full', 'circle', '⭕', '️', 'moment', 'stand', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print(\"Tokenized Tweet:\")\n",
    "print([word for word in tt.tokenize(tweet)])\n",
    "print(\"\\nTokenized Tweet without Stop Words:\")\n",
    "print([word for word in tt.tokenize(tweet) if word.lower() not in stopwords.words('english')])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cecfc1a",
   "metadata": {},
   "source": [
    "### Removing Emoji  <a class=\"anchor\" id=\"remove_emoji\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "1207c175",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xf0\\x9f\\x98\\x8a'\n"
     ]
    }
   ],
   "source": [
    "# Emoji is represented as sequence of bytes\n",
    "print('😊'.encode(\"UTF-8\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e1ae49b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import emoji"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "e8e11482",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(emoji.is_emoji('😊'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8d6f69ff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":smiling_face_with_smiling_eyes:\n"
     ]
    }
   ],
   "source": [
    "print(emoji.demojize('😊'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "5a6406d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":american_football:\n"
     ]
    }
   ],
   "source": [
    "print(emoji.demojize('🏈'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "2d4a91f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tokenized tweet:\n",
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n",
      "\n",
      "Tokenized tweet with emojis demojized:\n",
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', ':rolling_on_the_floor_laughing:', ':flexed_biceps_medium-dark_skin_tone:', ':american_football:', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', ':smiling_face_with_smiling_eyes:', ':collision:', ':microphone:', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', ':hollow_red_circle:', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print(\"Tokenized tweet:\")\n",
    "print(tt.tokenize(tweet))\n",
    "print(\"\\nTokenized tweet with emojis demojized:\")\n",
    "print([emoji.demojize(token) if emoji.is_emoji(token) else token for token in tt.tokenize(tweet)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ebb375",
   "metadata": {},
   "source": [
    "### Stemming & Lemmatization  <a class=\"anchor\" id=\"stem_lemma\"></a>\n",
    "\n",
    "- They are both vocabulary reduction techniques.\n",
    "- **Stemming**: The process of reducing a word to its most basic form. Examples:\n",
    "    - \"party\", \"partying\", \"parties\" &#8594; \"parti\"\n",
    "    - \"programming\", \"programmer\", \"programs\" &#8594; \"program\"\n",
    "- **Lemmatization**: A technique to reduce inflected words to their root word. Examples:\n",
    "    - \"runs\", \"running\", \"ran\" &#8594; \"run\"\n",
    "    - \"am\", \"is\", \"are\" &#8594; \"be\"\n",
    "- [Stemming and Lemmatization in Python](https://www.datacamp.com/tutorial/stemming-lemmatization-python) by [DataCamp](https://www.datacamp.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "153007b0",
   "metadata": {},
   "source": [
    "#### Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "c96287ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "1c380ca8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print(tt.tokenize(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "cd349078",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', '’', 'm', 'pump', 'up', 'like', 'i', '’', 'm', 'get', 'snap', 'at', 'd-end', 'tomorrow', '🤣', '💪🏾', '🏈', 'can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@nfl', '&', '@nbc', 'have', 'cookin', '’', '😊', '💥', '🎤', 'you', 'guy', 'know', 'my', 'nfl', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'thi', 'is', 'a', 'full', 'circl', '⭕', '️', 'moment', 'to', 'stand', 'on', 'thi', 'field', '#gratitud', '#rockatthe50', '#final', '#sblvi', 'https://t.co/yyihvbadt6']\n"
     ]
    }
   ],
   "source": [
    "print([stemmer.stem(word) for word in tt.tokenize(tweet)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "318f22a2",
   "metadata": {},
   "source": [
    "#### Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "db8abae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet \n",
    "\n",
    "lemmatizer = WordNetLemmatizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "45d8a633",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snaps', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guys', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print(tt.tokenize(tweet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "4f941eb1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', '’', 'm', 'pumped', 'up', 'like', 'I', '’', 'm', 'getting', 'snap', 'at', 'D-End', 'tomorrow', '🤣', '💪🏾', '🏈', 'Can', '’', 't', 'wait', 'for', 'you', 'to', 'see', 'what', 'myself', ',', '@NFL', '&', '@NBC', 'have', 'cookin', '’', '😊', '💥', '🎤', 'You', 'guy', 'know', 'my', 'NFL', 'dream', 'never', 'came', 'true', ',', 'so', 'is', 'this', 'is', 'a', 'full', 'circle', '⭕', '️', 'moment', 'to', 'stand', 'on', 'this', 'field', '#Gratitude', '#RockAtThe50', '#FINALLY', '#SBLVI', 'https://t.co/yyIhvBADt6']\n"
     ]
    }
   ],
   "source": [
    "print([lemmatizer.lemmatize(word) for word in tt.tokenize(tweet)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "a78f222e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is\n",
      "be\n"
     ]
    }
   ],
   "source": [
    "print(lemmatizer.lemmatize(\"is\"))\n",
    "print(lemmatizer.lemmatize(\"is\",wordnet.VERB))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32aea5f7",
   "metadata": {},
   "source": [
    "### Apply All Preprocessing Steps to Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9c5d65a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_tweet(text):\n",
    "    tt = TweetTokenizer()\n",
    "    # Remove URLs\n",
    "    url_pattern = \"https?://t\\.co/\\S+\"\n",
    "    text = re.sub(url_pattern, \"\", text)\n",
    "    # Lower case\n",
    "    text = text.lower()\n",
    "    # Tokenization\n",
    "    text_tokenized = tt.tokenize(text)\n",
    "    # Remove stop words\n",
    "    text_tokenized = [token for token in text_tokenized if token not in stopwords.words('english')]\n",
    "    # Demojize\n",
    "    text_tokenized = [emoji.demojize(token) if emoji.is_emoji(token) else token for token in text_tokenized]\n",
    "    # Remove punctuation\n",
    "    text_tokenized = [token for token in text_tokenized if token not in punctuation]\n",
    "    # Stemming\n",
    "    text_tokenized = [stemmer.stem(token) for token in text_tokenized]\n",
    "    return ' '.join(text_tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d1a96ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "#df[\"preprocessed_text\"] = df[\"text\"].apply(lambda text: preprocess_tweet(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "1433ad05",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0/55,080 | 0.00%\n",
      "100/55,080 | 0.18%\n",
      "200/55,080 | 0.36%\n",
      "300/55,080 | 0.54%\n",
      "400/55,080 | 0.73%\n",
      "500/55,080 | 0.91%\n",
      "600/55,080 | 1.09%\n",
      "700/55,080 | 1.27%\n",
      "800/55,080 | 1.45%\n",
      "900/55,080 | 1.63%\n",
      "1,000/55,080 | 1.82%\n",
      "1,100/55,080 | 2.00%\n",
      "1,200/55,080 | 2.18%\n",
      "1,300/55,080 | 2.36%\n",
      "1,400/55,080 | 2.54%\n",
      "1,500/55,080 | 2.72%\n",
      "1,600/55,080 | 2.90%\n",
      "1,700/55,080 | 3.09%\n",
      "1,800/55,080 | 3.27%\n",
      "1,900/55,080 | 3.45%\n",
      "2,000/55,080 | 3.63%\n",
      "2,100/55,080 | 3.81%\n",
      "2,200/55,080 | 3.99%\n",
      "2,300/55,080 | 4.18%\n",
      "2,400/55,080 | 4.36%\n",
      "2,500/55,080 | 4.54%\n",
      "2,600/55,080 | 4.72%\n",
      "2,700/55,080 | 4.90%\n",
      "2,800/55,080 | 5.08%\n",
      "2,900/55,080 | 5.27%\n",
      "3,000/55,080 | 5.45%\n",
      "3,100/55,080 | 5.63%\n",
      "3,200/55,080 | 5.81%\n",
      "3,300/55,080 | 5.99%\n",
      "3,400/55,080 | 6.17%\n",
      "3,500/55,080 | 6.35%\n",
      "3,600/55,080 | 6.54%\n",
      "3,700/55,080 | 6.72%\n",
      "3,800/55,080 | 6.90%\n",
      "3,900/55,080 | 7.08%\n",
      "4,000/55,080 | 7.26%\n",
      "4,100/55,080 | 7.44%\n",
      "4,200/55,080 | 7.63%\n",
      "4,300/55,080 | 7.81%\n",
      "4,400/55,080 | 7.99%\n",
      "4,500/55,080 | 8.17%\n",
      "4,600/55,080 | 8.35%\n",
      "4,700/55,080 | 8.53%\n",
      "4,800/55,080 | 8.71%\n",
      "4,900/55,080 | 8.90%\n",
      "5,000/55,080 | 9.08%\n",
      "5,100/55,080 | 9.26%\n",
      "5,200/55,080 | 9.44%\n",
      "5,300/55,080 | 9.62%\n",
      "5,400/55,080 | 9.80%\n",
      "5,500/55,080 | 9.99%\n",
      "5,600/55,080 | 10.17%\n",
      "5,700/55,080 | 10.35%\n",
      "5,800/55,080 | 10.53%\n",
      "5,900/55,080 | 10.71%\n",
      "6,000/55,080 | 10.89%\n",
      "6,100/55,080 | 11.07%\n",
      "6,200/55,080 | 11.26%\n",
      "6,300/55,080 | 11.44%\n",
      "6,400/55,080 | 11.62%\n",
      "6,500/55,080 | 11.80%\n",
      "6,600/55,080 | 11.98%\n",
      "6,700/55,080 | 12.16%\n",
      "6,800/55,080 | 12.35%\n",
      "6,900/55,080 | 12.53%\n",
      "7,000/55,080 | 12.71%\n",
      "7,100/55,080 | 12.89%\n",
      "7,200/55,080 | 13.07%\n",
      "7,300/55,080 | 13.25%\n",
      "7,400/55,080 | 13.44%\n",
      "7,500/55,080 | 13.62%\n",
      "7,600/55,080 | 13.80%\n",
      "7,700/55,080 | 13.98%\n",
      "7,800/55,080 | 14.16%\n",
      "7,900/55,080 | 14.34%\n",
      "8,000/55,080 | 14.52%\n",
      "8,100/55,080 | 14.71%\n",
      "8,200/55,080 | 14.89%\n",
      "8,300/55,080 | 15.07%\n",
      "8,400/55,080 | 15.25%\n",
      "8,500/55,080 | 15.43%\n",
      "8,600/55,080 | 15.61%\n",
      "8,700/55,080 | 15.80%\n",
      "8,800/55,080 | 15.98%\n",
      "8,900/55,080 | 16.16%\n",
      "9,000/55,080 | 16.34%\n",
      "9,100/55,080 | 16.52%\n",
      "9,200/55,080 | 16.70%\n",
      "9,300/55,080 | 16.88%\n",
      "9,400/55,080 | 17.07%\n",
      "9,500/55,080 | 17.25%\n",
      "9,600/55,080 | 17.43%\n",
      "9,700/55,080 | 17.61%\n",
      "9,800/55,080 | 17.79%\n",
      "9,900/55,080 | 17.97%\n",
      "10,000/55,080 | 18.16%\n",
      "10,100/55,080 | 18.34%\n",
      "10,200/55,080 | 18.52%\n",
      "10,300/55,080 | 18.70%\n",
      "10,400/55,080 | 18.88%\n",
      "10,500/55,080 | 19.06%\n",
      "10,600/55,080 | 19.24%\n",
      "10,700/55,080 | 19.43%\n",
      "10,800/55,080 | 19.61%\n",
      "10,900/55,080 | 19.79%\n",
      "11,000/55,080 | 19.97%\n",
      "11,100/55,080 | 20.15%\n",
      "11,200/55,080 | 20.33%\n",
      "11,300/55,080 | 20.52%\n",
      "11,400/55,080 | 20.70%\n",
      "11,500/55,080 | 20.88%\n",
      "11,600/55,080 | 21.06%\n",
      "11,700/55,080 | 21.24%\n",
      "11,800/55,080 | 21.42%\n",
      "11,900/55,080 | 21.60%\n",
      "12,000/55,080 | 21.79%\n",
      "12,100/55,080 | 21.97%\n",
      "12,200/55,080 | 22.15%\n",
      "12,300/55,080 | 22.33%\n",
      "12,400/55,080 | 22.51%\n",
      "12,500/55,080 | 22.69%\n",
      "12,600/55,080 | 22.88%\n",
      "12,700/55,080 | 23.06%\n",
      "12,800/55,080 | 23.24%\n",
      "12,900/55,080 | 23.42%\n",
      "13,000/55,080 | 23.60%\n",
      "13,100/55,080 | 23.78%\n",
      "13,200/55,080 | 23.97%\n",
      "13,300/55,080 | 24.15%\n",
      "13,400/55,080 | 24.33%\n",
      "13,500/55,080 | 24.51%\n",
      "13,600/55,080 | 24.69%\n",
      "13,700/55,080 | 24.87%\n",
      "13,800/55,080 | 25.05%\n",
      "13,900/55,080 | 25.24%\n",
      "14,000/55,080 | 25.42%\n",
      "14,100/55,080 | 25.60%\n",
      "14,200/55,080 | 25.78%\n",
      "14,300/55,080 | 25.96%\n",
      "14,400/55,080 | 26.14%\n",
      "14,500/55,080 | 26.33%\n",
      "14,600/55,080 | 26.51%\n",
      "14,700/55,080 | 26.69%\n",
      "14,800/55,080 | 26.87%\n",
      "14,900/55,080 | 27.05%\n",
      "15,000/55,080 | 27.23%\n",
      "15,100/55,080 | 27.41%\n",
      "15,200/55,080 | 27.60%\n",
      "15,300/55,080 | 27.78%\n",
      "15,400/55,080 | 27.96%\n",
      "15,500/55,080 | 28.14%\n",
      "15,600/55,080 | 28.32%\n",
      "15,700/55,080 | 28.50%\n",
      "15,800/55,080 | 28.69%\n",
      "15,900/55,080 | 28.87%\n",
      "16,000/55,080 | 29.05%\n",
      "16,100/55,080 | 29.23%\n",
      "16,200/55,080 | 29.41%\n",
      "16,300/55,080 | 29.59%\n",
      "16,400/55,080 | 29.77%\n",
      "16,500/55,080 | 29.96%\n",
      "16,600/55,080 | 30.14%\n",
      "16,700/55,080 | 30.32%\n",
      "16,800/55,080 | 30.50%\n",
      "16,900/55,080 | 30.68%\n",
      "17,000/55,080 | 30.86%\n",
      "17,100/55,080 | 31.05%\n",
      "17,200/55,080 | 31.23%\n",
      "17,300/55,080 | 31.41%\n",
      "17,400/55,080 | 31.59%\n",
      "17,500/55,080 | 31.77%\n",
      "17,600/55,080 | 31.95%\n",
      "17,700/55,080 | 32.14%\n",
      "17,800/55,080 | 32.32%\n",
      "17,900/55,080 | 32.50%\n",
      "18,000/55,080 | 32.68%\n",
      "18,100/55,080 | 32.86%\n",
      "18,200/55,080 | 33.04%\n",
      "18,300/55,080 | 33.22%\n",
      "18,400/55,080 | 33.41%\n",
      "18,500/55,080 | 33.59%\n",
      "18,600/55,080 | 33.77%\n",
      "18,700/55,080 | 33.95%\n",
      "18,800/55,080 | 34.13%\n",
      "18,900/55,080 | 34.31%\n",
      "19,000/55,080 | 34.50%\n",
      "19,100/55,080 | 34.68%\n",
      "19,200/55,080 | 34.86%\n",
      "19,300/55,080 | 35.04%\n",
      "19,400/55,080 | 35.22%\n",
      "19,500/55,080 | 35.40%\n",
      "19,600/55,080 | 35.58%\n",
      "19,700/55,080 | 35.77%\n",
      "19,800/55,080 | 35.95%\n",
      "19,900/55,080 | 36.13%\n",
      "20,000/55,080 | 36.31%\n",
      "20,100/55,080 | 36.49%\n",
      "20,200/55,080 | 36.67%\n",
      "20,300/55,080 | 36.86%\n",
      "20,400/55,080 | 37.04%\n",
      "20,500/55,080 | 37.22%\n",
      "20,600/55,080 | 37.40%\n",
      "20,700/55,080 | 37.58%\n",
      "20,800/55,080 | 37.76%\n",
      "20,900/55,080 | 37.94%\n",
      "21,000/55,080 | 38.13%\n",
      "21,100/55,080 | 38.31%\n",
      "21,200/55,080 | 38.49%\n",
      "21,300/55,080 | 38.67%\n",
      "21,400/55,080 | 38.85%\n",
      "21,500/55,080 | 39.03%\n",
      "21,600/55,080 | 39.22%\n",
      "21,700/55,080 | 39.40%\n",
      "21,800/55,080 | 39.58%\n",
      "21,900/55,080 | 39.76%\n",
      "22,000/55,080 | 39.94%\n",
      "22,100/55,080 | 40.12%\n",
      "22,200/55,080 | 40.31%\n",
      "22,300/55,080 | 40.49%\n",
      "22,400/55,080 | 40.67%\n",
      "22,500/55,080 | 40.85%\n",
      "22,600/55,080 | 41.03%\n",
      "22,700/55,080 | 41.21%\n",
      "22,800/55,080 | 41.39%\n",
      "22,900/55,080 | 41.58%\n",
      "23,000/55,080 | 41.76%\n",
      "23,100/55,080 | 41.94%\n",
      "23,200/55,080 | 42.12%\n",
      "23,300/55,080 | 42.30%\n",
      "23,400/55,080 | 42.48%\n",
      "23,500/55,080 | 42.67%\n",
      "23,600/55,080 | 42.85%\n",
      "23,700/55,080 | 43.03%\n",
      "23,800/55,080 | 43.21%\n",
      "23,900/55,080 | 43.39%\n",
      "24,000/55,080 | 43.57%\n",
      "24,100/55,080 | 43.75%\n",
      "24,200/55,080 | 43.94%\n",
      "24,300/55,080 | 44.12%\n",
      "24,400/55,080 | 44.30%\n",
      "24,500/55,080 | 44.48%\n",
      "24,600/55,080 | 44.66%\n",
      "24,700/55,080 | 44.84%\n",
      "24,800/55,080 | 45.03%\n",
      "24,900/55,080 | 45.21%\n",
      "25,000/55,080 | 45.39%\n",
      "25,100/55,080 | 45.57%\n",
      "25,200/55,080 | 45.75%\n",
      "25,300/55,080 | 45.93%\n",
      "25,400/55,080 | 46.11%\n",
      "25,500/55,080 | 46.30%\n",
      "25,600/55,080 | 46.48%\n",
      "25,700/55,080 | 46.66%\n",
      "25,800/55,080 | 46.84%\n",
      "25,900/55,080 | 47.02%\n",
      "26,000/55,080 | 47.20%\n",
      "26,100/55,080 | 47.39%\n",
      "26,200/55,080 | 47.57%\n",
      "26,300/55,080 | 47.75%\n",
      "26,400/55,080 | 47.93%\n",
      "26,500/55,080 | 48.11%\n",
      "26,600/55,080 | 48.29%\n",
      "26,700/55,080 | 48.47%\n",
      "26,800/55,080 | 48.66%\n",
      "26,900/55,080 | 48.84%\n",
      "27,000/55,080 | 49.02%\n",
      "27,100/55,080 | 49.20%\n",
      "27,200/55,080 | 49.38%\n",
      "27,300/55,080 | 49.56%\n",
      "27,400/55,080 | 49.75%\n",
      "27,500/55,080 | 49.93%\n",
      "27,600/55,080 | 50.11%\n",
      "27,700/55,080 | 50.29%\n",
      "27,800/55,080 | 50.47%\n",
      "27,900/55,080 | 50.65%\n",
      "28,000/55,080 | 50.84%\n",
      "28,100/55,080 | 51.02%\n",
      "28,200/55,080 | 51.20%\n",
      "28,300/55,080 | 51.38%\n",
      "28,400/55,080 | 51.56%\n",
      "28,500/55,080 | 51.74%\n",
      "28,600/55,080 | 51.92%\n",
      "28,700/55,080 | 52.11%\n",
      "28,800/55,080 | 52.29%\n",
      "28,900/55,080 | 52.47%\n",
      "29,000/55,080 | 52.65%\n",
      "29,100/55,080 | 52.83%\n",
      "29,200/55,080 | 53.01%\n",
      "29,300/55,080 | 53.20%\n",
      "29,400/55,080 | 53.38%\n",
      "29,500/55,080 | 53.56%\n",
      "29,600/55,080 | 53.74%\n",
      "29,700/55,080 | 53.92%\n",
      "29,800/55,080 | 54.10%\n",
      "29,900/55,080 | 54.28%\n",
      "30,000/55,080 | 54.47%\n",
      "30,100/55,080 | 54.65%\n",
      "30,200/55,080 | 54.83%\n",
      "30,300/55,080 | 55.01%\n",
      "30,400/55,080 | 55.19%\n",
      "30,500/55,080 | 55.37%\n",
      "30,600/55,080 | 55.56%\n",
      "30,700/55,080 | 55.74%\n",
      "30,800/55,080 | 55.92%\n",
      "30,900/55,080 | 56.10%\n",
      "31,000/55,080 | 56.28%\n",
      "31,100/55,080 | 56.46%\n",
      "31,200/55,080 | 56.64%\n",
      "31,300/55,080 | 56.83%\n",
      "31,400/55,080 | 57.01%\n",
      "31,500/55,080 | 57.19%\n",
      "31,600/55,080 | 57.37%\n",
      "31,700/55,080 | 57.55%\n",
      "31,800/55,080 | 57.73%\n",
      "31,900/55,080 | 57.92%\n",
      "32,000/55,080 | 58.10%\n",
      "32,100/55,080 | 58.28%\n",
      "32,200/55,080 | 58.46%\n",
      "32,300/55,080 | 58.64%\n",
      "32,400/55,080 | 58.82%\n",
      "32,500/55,080 | 59.01%\n",
      "32,600/55,080 | 59.19%\n",
      "32,700/55,080 | 59.37%\n",
      "32,800/55,080 | 59.55%\n",
      "32,900/55,080 | 59.73%\n",
      "33,000/55,080 | 59.91%\n",
      "33,100/55,080 | 60.09%\n",
      "33,200/55,080 | 60.28%\n",
      "33,300/55,080 | 60.46%\n",
      "33,400/55,080 | 60.64%\n",
      "33,500/55,080 | 60.82%\n",
      "33,600/55,080 | 61.00%\n",
      "33,700/55,080 | 61.18%\n",
      "33,800/55,080 | 61.37%\n",
      "33,900/55,080 | 61.55%\n",
      "34,000/55,080 | 61.73%\n",
      "34,100/55,080 | 61.91%\n",
      "34,200/55,080 | 62.09%\n",
      "34,300/55,080 | 62.27%\n",
      "34,400/55,080 | 62.45%\n",
      "34,500/55,080 | 62.64%\n",
      "34,600/55,080 | 62.82%\n",
      "34,700/55,080 | 63.00%\n",
      "34,800/55,080 | 63.18%\n",
      "34,900/55,080 | 63.36%\n",
      "35,000/55,080 | 63.54%\n",
      "35,100/55,080 | 63.73%\n",
      "35,200/55,080 | 63.91%\n",
      "35,300/55,080 | 64.09%\n",
      "35,400/55,080 | 64.27%\n",
      "35,500/55,080 | 64.45%\n",
      "35,600/55,080 | 64.63%\n",
      "35,700/55,080 | 64.81%\n",
      "35,800/55,080 | 65.00%\n",
      "35,900/55,080 | 65.18%\n",
      "36,000/55,080 | 65.36%\n",
      "36,100/55,080 | 65.54%\n",
      "36,200/55,080 | 65.72%\n",
      "36,300/55,080 | 65.90%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "36,400/55,080 | 66.09%\n",
      "36,500/55,080 | 66.27%\n",
      "36,600/55,080 | 66.45%\n",
      "36,700/55,080 | 66.63%\n",
      "36,800/55,080 | 66.81%\n",
      "36,900/55,080 | 66.99%\n",
      "37,000/55,080 | 67.18%\n",
      "37,100/55,080 | 67.36%\n",
      "37,200/55,080 | 67.54%\n",
      "37,300/55,080 | 67.72%\n",
      "37,400/55,080 | 67.90%\n",
      "37,500/55,080 | 68.08%\n",
      "37,600/55,080 | 68.26%\n",
      "37,700/55,080 | 68.45%\n",
      "37,800/55,080 | 68.63%\n",
      "37,900/55,080 | 68.81%\n",
      "38,000/55,080 | 68.99%\n",
      "38,100/55,080 | 69.17%\n",
      "38,200/55,080 | 69.35%\n",
      "38,300/55,080 | 69.54%\n",
      "38,400/55,080 | 69.72%\n",
      "38,500/55,080 | 69.90%\n",
      "38,600/55,080 | 70.08%\n",
      "38,700/55,080 | 70.26%\n",
      "38,800/55,080 | 70.44%\n",
      "38,900/55,080 | 70.62%\n",
      "39,000/55,080 | 70.81%\n",
      "39,100/55,080 | 70.99%\n",
      "39,200/55,080 | 71.17%\n",
      "39,300/55,080 | 71.35%\n",
      "39,400/55,080 | 71.53%\n",
      "39,500/55,080 | 71.71%\n",
      "39,600/55,080 | 71.90%\n",
      "39,700/55,080 | 72.08%\n",
      "39,800/55,080 | 72.26%\n",
      "39,900/55,080 | 72.44%\n",
      "40,000/55,080 | 72.62%\n",
      "40,100/55,080 | 72.80%\n",
      "40,200/55,080 | 72.98%\n",
      "40,300/55,080 | 73.17%\n",
      "40,400/55,080 | 73.35%\n",
      "40,500/55,080 | 73.53%\n",
      "40,600/55,080 | 73.71%\n",
      "40,700/55,080 | 73.89%\n",
      "40,800/55,080 | 74.07%\n",
      "40,900/55,080 | 74.26%\n",
      "41,000/55,080 | 74.44%\n",
      "41,100/55,080 | 74.62%\n",
      "41,200/55,080 | 74.80%\n",
      "41,300/55,080 | 74.98%\n",
      "41,400/55,080 | 75.16%\n",
      "41,500/55,080 | 75.34%\n",
      "41,600/55,080 | 75.53%\n",
      "41,700/55,080 | 75.71%\n",
      "41,800/55,080 | 75.89%\n",
      "41,900/55,080 | 76.07%\n",
      "42,000/55,080 | 76.25%\n",
      "42,100/55,080 | 76.43%\n",
      "42,200/55,080 | 76.62%\n",
      "42,300/55,080 | 76.80%\n",
      "42,400/55,080 | 76.98%\n",
      "42,500/55,080 | 77.16%\n",
      "42,600/55,080 | 77.34%\n",
      "42,700/55,080 | 77.52%\n",
      "42,800/55,080 | 77.71%\n",
      "42,900/55,080 | 77.89%\n",
      "43,000/55,080 | 78.07%\n",
      "43,100/55,080 | 78.25%\n",
      "43,200/55,080 | 78.43%\n",
      "43,300/55,080 | 78.61%\n",
      "43,400/55,080 | 78.79%\n",
      "43,500/55,080 | 78.98%\n",
      "43,600/55,080 | 79.16%\n",
      "43,700/55,080 | 79.34%\n",
      "43,800/55,080 | 79.52%\n",
      "43,900/55,080 | 79.70%\n",
      "44,000/55,080 | 79.88%\n",
      "44,100/55,080 | 80.07%\n",
      "44,200/55,080 | 80.25%\n",
      "44,300/55,080 | 80.43%\n",
      "44,400/55,080 | 80.61%\n",
      "44,500/55,080 | 80.79%\n",
      "44,600/55,080 | 80.97%\n",
      "44,700/55,080 | 81.15%\n",
      "44,800/55,080 | 81.34%\n",
      "44,900/55,080 | 81.52%\n",
      "45,000/55,080 | 81.70%\n",
      "45,100/55,080 | 81.88%\n",
      "45,200/55,080 | 82.06%\n",
      "45,300/55,080 | 82.24%\n",
      "45,400/55,080 | 82.43%\n",
      "45,500/55,080 | 82.61%\n",
      "45,600/55,080 | 82.79%\n",
      "45,700/55,080 | 82.97%\n",
      "45,800/55,080 | 83.15%\n",
      "45,900/55,080 | 83.33%\n",
      "46,000/55,080 | 83.51%\n",
      "46,100/55,080 | 83.70%\n",
      "46,200/55,080 | 83.88%\n",
      "46,300/55,080 | 84.06%\n",
      "46,400/55,080 | 84.24%\n",
      "46,500/55,080 | 84.42%\n",
      "46,600/55,080 | 84.60%\n",
      "46,700/55,080 | 84.79%\n",
      "46,800/55,080 | 84.97%\n",
      "46,900/55,080 | 85.15%\n",
      "47,000/55,080 | 85.33%\n",
      "47,100/55,080 | 85.51%\n",
      "47,200/55,080 | 85.69%\n",
      "47,300/55,080 | 85.88%\n",
      "47,400/55,080 | 86.06%\n",
      "47,500/55,080 | 86.24%\n",
      "47,600/55,080 | 86.42%\n",
      "47,700/55,080 | 86.60%\n",
      "47,800/55,080 | 86.78%\n",
      "47,900/55,080 | 86.96%\n",
      "48,000/55,080 | 87.15%\n",
      "48,100/55,080 | 87.33%\n",
      "48,200/55,080 | 87.51%\n",
      "48,300/55,080 | 87.69%\n",
      "48,400/55,080 | 87.87%\n",
      "48,500/55,080 | 88.05%\n",
      "48,600/55,080 | 88.24%\n",
      "48,700/55,080 | 88.42%\n",
      "48,800/55,080 | 88.60%\n",
      "48,900/55,080 | 88.78%\n",
      "49,000/55,080 | 88.96%\n",
      "49,100/55,080 | 89.14%\n",
      "49,200/55,080 | 89.32%\n",
      "49,300/55,080 | 89.51%\n",
      "49,400/55,080 | 89.69%\n",
      "49,500/55,080 | 89.87%\n",
      "49,600/55,080 | 90.05%\n",
      "49,700/55,080 | 90.23%\n",
      "49,800/55,080 | 90.41%\n",
      "49,900/55,080 | 90.60%\n",
      "50,000/55,080 | 90.78%\n",
      "50,100/55,080 | 90.96%\n",
      "50,200/55,080 | 91.14%\n",
      "50,300/55,080 | 91.32%\n",
      "50,400/55,080 | 91.50%\n",
      "50,500/55,080 | 91.68%\n",
      "50,600/55,080 | 91.87%\n",
      "50,700/55,080 | 92.05%\n",
      "50,800/55,080 | 92.23%\n",
      "50,900/55,080 | 92.41%\n",
      "51,000/55,080 | 92.59%\n",
      "51,100/55,080 | 92.77%\n",
      "51,200/55,080 | 92.96%\n",
      "51,300/55,080 | 93.14%\n",
      "51,400/55,080 | 93.32%\n",
      "51,500/55,080 | 93.50%\n",
      "51,600/55,080 | 93.68%\n",
      "51,700/55,080 | 93.86%\n",
      "51,800/55,080 | 94.05%\n",
      "51,900/55,080 | 94.23%\n",
      "52,000/55,080 | 94.41%\n",
      "52,100/55,080 | 94.59%\n",
      "52,200/55,080 | 94.77%\n",
      "52,300/55,080 | 94.95%\n",
      "52,400/55,080 | 95.13%\n",
      "52,500/55,080 | 95.32%\n",
      "52,600/55,080 | 95.50%\n",
      "52,700/55,080 | 95.68%\n",
      "52,800/55,080 | 95.86%\n",
      "52,900/55,080 | 96.04%\n",
      "53,000/55,080 | 96.22%\n",
      "53,100/55,080 | 96.41%\n",
      "53,200/55,080 | 96.59%\n",
      "53,300/55,080 | 96.77%\n",
      "53,400/55,080 | 96.95%\n",
      "53,500/55,080 | 97.13%\n",
      "53,600/55,080 | 97.31%\n",
      "53,700/55,080 | 97.49%\n",
      "53,800/55,080 | 97.68%\n",
      "53,900/55,080 | 97.86%\n",
      "54,000/55,080 | 98.04%\n",
      "54,100/55,080 | 98.22%\n",
      "54,200/55,080 | 98.40%\n",
      "54,300/55,080 | 98.58%\n",
      "54,400/55,080 | 98.77%\n",
      "54,500/55,080 | 98.95%\n",
      "54,600/55,080 | 99.13%\n",
      "54,700/55,080 | 99.31%\n",
      "54,800/55,080 | 99.49%\n",
      "54,900/55,080 | 99.67%\n",
      "55,000/55,080 | 99.85%\n",
      "55,080/55,080 | 100.00%\n"
     ]
    }
   ],
   "source": [
    "preprocessed_texts = []\n",
    "\n",
    "for i, text in enumerate(list(df[\"text\"])):\n",
    "    if i % 100 == 0:\n",
    "        print(f\"{i:,}/{df.shape[0]:,} | {i/df.shape[0]*100:.2f}%\")\n",
    "    preprocessed_texts.append(preprocess_tweet(df.loc[i, \"text\"]))\n",
    "print(f\"{df.shape[0]:,}/{df.shape[0]:,} | {df.shape[0]/df.shape[0]*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "27f80db1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"preprocessed_text\"] = pd.Series(preprocessed_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "5abc4810",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- Text:\n",
      "As a man, it’s a whole other level when you hold another grown man with tears in his eyes. \n",
      "I’ve held a few. \n",
      "A few have held me. \n",
      "This man lost his mama and she used to compare us. \n",
      "You’re right brother,… https://t.co/Y8p0SsIvRS\n",
      "\n",
      "- Preprocessed Text:\n",
      "man ’ whole level hold anoth grown man tear eye ’ held held man lost mama use compar us ’ right brother …\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Of all the pictures @joethomas73 can asks me to sign. This is how I look at my wife... and cheat meals 😂. \n",
      "He’s an inspiration, good buddy and good man. \n",
      "10,363 (if you know, you know) \n",
      "#goat \n",
      "@NFL https://t.co/FwLDbsmgMZ\n",
      "\n",
      "- Preprocessed Text:\n",
      "pictur @joethomas73 ask sign look wife ... cheat meal :face_with_tears_of_joy: ’ inspir good buddi good man 10,363 know know #goat @nfl\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Introducing @SKIMS Teddy, our first all-weather collection. Drops Thursday, September 9 at 9AM PT. https://t.co/Qsy51S3rtD https://t.co/KD3nRjrTfg\n",
      "\n",
      "- Preprocessed Text:\n",
      "introduc @skim teddi first all-weath collect drop thursday septemb 9 9am pt\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Friday https://t.co/MEve0ZV9fI\n",
      "\n",
      "- Preprocessed Text:\n",
      "friday\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "We sprang forward Queens https://t.co/5E7lr1m5yA\n",
      "\n",
      "- Preprocessed Text:\n",
      "sprang forward queen\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "@i_hate_doug Advertising a child? No I didnt get paid to respond to a kid... it’s Christmas\n",
      "\n",
      "- Preprocessed Text:\n",
      "@i_hate_doug advertis child didnt get paid respond kid ... ’ christma\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "I always cover my phone microphone with my pinky...dumbass\n",
      "\n",
      "- Preprocessed Text:\n",
      "alway cover phone microphon pinki ... dumbass\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Election Day is TOMORROW!!\n",
      "If you haven’t voted yet, get your #VotingSquad NOW and make a plan to vote Tuesday. @WhenWeAllVote can help you find your polling place at https://t.co/SwQzGFXsHf\n",
      "\n",
      "- Preprocessed Text:\n",
      "elect day tomorrow ’ vote yet get #votingsquad make plan vote tuesday @whenweallvot help find poll place\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "आज यूपी में नए मेडिकल कॉलेज बन रहे हैं, एम्स बन रहे हैं, आधुनिक शिक्षा संस्थान बन रहे हैं।\n",
      " कुछ हफ्ते पहले ही कुशीनगर में अंतर्राष्ट्रीय हवाई अड्डे का लोकार्पण हुआ और आज पूर्वांचल एक्सप्रेस वे।\n",
      " \n",
      "इन सब विकास कार्यों से गरीबों, मध्यम वर्ग, किसानों, युवाओं, महिलाओं को फायदा मिलेगा। https://t.co/ImA3Zz3tn3\n",
      "\n",
      "- Preprocessed Text:\n",
      "आज यूपी में नए मेडिकल कॉलेज बन रहे हैं एम्स बन रहे हैं आधुनिक शिक्षा संस्थान बन रहे हैं । कुछ हफ्ते पहले ही कुशीनगर में अंतर्राष्ट्रीय हवाई अड्डे का लोकार्पण हुआ और आज पूर्वांचल एक्सप्रेस वे । इन सब विकास कार्यों से गरीबों मध्यम वर्ग किसानों युवाओं महिलाओं को फायदा मिलेगा ।\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "RAF x RIRI http://t.co/RSIzwxlNIH\n",
      "\n",
      "- Preprocessed Text:\n",
      "raf x riri\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Saddened by the boat accident in Assam. All possible efforts are being made to rescue the passengers. I pray for everyone’s safety and well-being.\n",
      "\n",
      "- Preprocessed Text:\n",
      "sadden boat accid assam possibl effort made rescu passeng pray everyon ’ safeti well-b\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Me... https://t.co/PmY7IP6Bh8\n",
      "\n",
      "- Preprocessed Text:\n",
      "...\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Caught this tonight....well kinda! #KingsOfLeon http://t.co/FdhApFUxZe\n",
      "\n",
      "- Preprocessed Text:\n",
      "caught tonight ... well kinda #kingsofleon\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "https://t.co/NmsBnG9USd \n",
      "\n",
      "https://t.co/ZiPb0rtfYF\n",
      "\n",
      "- Preprocessed Text:\n",
      "\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "@ItsTanbirC @biancagguerrero Methane lobby (among others I’m sure) was exerting a LOT of pressure against BBB, trying to delay + reopen for changes. I am very concerned we just locked in US emissions if BBB isn’t delivered. That said there are still incentives to pass BBB, but if gutted our future is at risk\n",
      "\n",
      "- Preprocessed Text:\n",
      "@itstanbirc @biancagguerrero methan lobbi among other ’ sure exert lot pressur bbb tri delay reopen chang concern lock us emiss bbb ’ deliv said still incent pass bbb gut futur risk\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Magnifique 👏 https://t.co/BGC9zdB2Dd\n",
      "\n",
      "- Preprocessed Text:\n",
      "magnifiqu :clapping_hands:\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Look forward to starting 2016 @Hopmancup in @WestAustralia with @JackSock #TeamUSA https://t.co/mkfxJevvqG\n",
      "\n",
      "- Preprocessed Text:\n",
      "look forward start 2016 @hopmancup @westaustralia @jacksock #teamusa\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "@jason_blum Jason this is horrible\n",
      "\n",
      "- Preprocessed Text:\n",
      "@jason_blum jason horribl\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Emily Blunt, myself &amp; our entire JUNGLE CRUISE family, invite you on the adventure of a lifetime.\n",
      "Enjoy our new world premiere of trailer #2. \n",
      "@JUNGLECRUISE \n",
      "@SevenBucksProd\n",
      "@DisneyStudios \n",
      "JULY 24th 🚢🗺🌴🐍🐆🥃 https://t.co/2aFfuXCQSy\n",
      "\n",
      "- Preprocessed Text:\n",
      "emili blunt entir jungl cruis famili invit adventur lifetim enjoy new world premier trailer 2 @junglecruis @sevenbucksprod @disneystudio juli 24th :ship: :world_map: :palm_tree: :snake: :leopard: :tumbler_glass:\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "We back. #CHANGESTOUR https://t.co/r5OG5OyceV\n",
      "\n",
      "- Preprocessed Text:\n",
      "back #changestour\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Wow. Thank u https://t.co/GRiOuL966A\n",
      "\n",
      "- Preprocessed Text:\n",
      "wow thank u\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Mahalo Mama. 🤙🏾 @HobbsAndShaw https://t.co/gy1gusTtV0\n",
      "\n",
      "- Preprocessed Text:\n",
      "mahalo mama :call_me_hand_medium-dark_skin_tone: @hobbsandshaw\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Had the best time @GeorgiaAquarium today! https://t.co/9btuydFPrU\n",
      "\n",
      "- Preprocessed Text:\n",
      "best time @georgiaaquarium today\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "The hierarchy of power in the DC UNIVERSE is about to change. \n",
      "BLACK ADAM arrives TOMORROW at #DCFanDome. \n",
      "He’s coming to crush them all. \n",
      "#ManInBlack \n",
      "#BLACKADAM https://t.co/2jiHT62jED\n",
      "\n",
      "- Preprocessed Text:\n",
      "hierarchi power dc univers chang black adam arriv tomorrow #dcfandom ’ come crush #maninblack #blackadam\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "New friends. #helpkellycharity #rogerfederersweater http://t.co/jbMkdYgeNA\n",
      "\n",
      "- Preprocessed Text:\n",
      "new friend #helpkellychar #rogerfederersweat\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "I could get used to playing on the same side of the net as @RafaelNadal 💪🏻👊🏻👯‍♂️🕺🏽🎯🔥 https://t.co/BxNvbFdDrn\n",
      "\n",
      "- Preprocessed Text:\n",
      "could get use play side net @rafaelnad :flexed_biceps_light_skin_tone: :oncoming_fist_light_skin_tone: :men_with_bunny_ears: ️ :man_dancing_medium_skin_tone: :bullseye: :fire:\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "Players like @manikabatra_TT have excelled in sports and at the same time are doing commendable community service. Kudos to  Manika. Sharing the video of our interaction. \n",
      "#Cheer4India https://t.co/WhYiXV98UY\n",
      "\n",
      "- Preprocessed Text:\n",
      "player like @manikabatra_tt excel sport time commend commun servic kudo manika share video interact #cheer4india\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "It’s a father and son story but it also serves as a cautionary tale that recognizes that our planet is in serious danger from climate change and that humanity is in serious danger because we’ve turned a blind eye to what doesn’t immediately affect us.\n",
      "\n",
      "- Preprocessed Text:\n",
      "’ father son stori also serv cautionari tale recogn planet seriou danger climat chang human seriou danger ’ turn blind eye ’ immedi affect us\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "2 hours. #INTENTIONS\n",
      "\n",
      "- Preprocessed Text:\n",
      "2 hour #intent\n",
      "------------------------------------------------------------\n",
      "- Text:\n",
      "If you don't have OWNtv SuperSoulSunday is streaming LIVE on my Facebook pg. right now. #SuperSoulSunday\n",
      "\n",
      "- Preprocessed Text:\n",
      "owntv supersoulsunday stream live facebook pg right #supersoulsunday\n",
      "------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for i in df.sample(30, random_state=0).index:\n",
    "    print(\"- Text:\")\n",
    "    print(df.loc[i, \"text\"])\n",
    "    print(\"\\n- Preprocessed Text:\")\n",
    "    print(df.loc[i, \"preprocessed_text\"])\n",
    "    print(\"-\"*60)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e47172d",
   "metadata": {},
   "source": [
    "## 2-  Converting Text to Numbers (Text Vectorization) <a class=\"anchor\" id=\"vectorization\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22690190",
   "metadata": {},
   "source": [
    "### Bag-of-Words (BoW) <a class=\"anchor\" id=\"bow\"></a>\n",
    "\n",
    "![bow](img/bow.png)\n",
    "\n",
    "Image Source: https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a2998661",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "8fc0dab5",
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vectorizer = CountVectorizer(min_df=50) # min_df=50 means \"ignore terms that appear in less than 50 documents\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "2c1ec6ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "BoW = count_vectorizer.fit_transform(df[\"preprocessed_text\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "c5d42c96",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>000</th>\n",
       "      <th>10</th>\n",
       "      <th>100</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>12pm</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>...</th>\n",
       "      <th>रह</th>\n",
       "      <th>वन</th>\n",
       "      <th>वर</th>\n",
       "      <th>शन</th>\n",
       "      <th>सभ</th>\n",
       "      <th>सम</th>\n",
       "      <th>सर</th>\n",
       "      <th>सरक</th>\n",
       "      <th>हम</th>\n",
       "      <th>हर</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preprocessed_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>@ashleyn1col :smiling_face_with_hearts: :purple_heart: :smiling_face_with_hearts: :purple_heart:</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>:loudly_crying_face: :loudly_crying_face: :loudly_crying_face:</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yah watch #grandcrew tell ya friend watch</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>@nanglish @grandcrewnbc :purple_heart: :purple_heart: :purple_heart:</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oooooh @ozzymo :purple_heart: :purple_heart: :purple_heart:</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax day they'r like eat cooki dough like ok i'll make sacrific art</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second glanc #reput ... readi</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>#lwymmdvideo</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>offici #lwymmdvideo world premier sunday 8/ 27 @vma</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>new singl #lookwhatyoumademedo pre-ord #reput merch ticket info</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>55080 rows × 1696 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    000  10  100  11  12  \\\n",
       "preprocessed_text                                                          \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...    0   0    0   0   0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...    0   0    0   0   0   \n",
       "yah watch #grandcrew tell ya friend watch             0   0    0   0   0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...    0   0    0   0   0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...    0   0    0   0   0   \n",
       "...                                                 ...  ..  ...  ..  ..   \n",
       "tax day they'r like eat cooki dough like ok i'l...    0   0    0   0   0   \n",
       "second glanc #reput ... readi                         0   0    0   0   0   \n",
       "#lwymmdvideo                                          0   0    0   0   0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...    0   0    0   0   0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...    0   0    0   0   0   \n",
       "\n",
       "                                                    12pm  13  14  15  16  ...  \\\n",
       "preprocessed_text                                                         ...   \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...     0   0   0   0   0  ...   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...     0   0   0   0   0  ...   \n",
       "yah watch #grandcrew tell ya friend watch              0   0   0   0   0  ...   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...     0   0   0   0   0  ...   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...     0   0   0   0   0  ...   \n",
       "...                                                  ...  ..  ..  ..  ..  ...   \n",
       "tax day they'r like eat cooki dough like ok i'l...     0   0   0   0   0  ...   \n",
       "second glanc #reput ... readi                          0   0   0   0   0  ...   \n",
       "#lwymmdvideo                                           0   0   0   0   0  ...   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...     0   0   0   0   0  ...   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...     0   0   0   0   0  ...   \n",
       "\n",
       "                                                    रह  वन  वर  शन  सभ  सम  \\\n",
       "preprocessed_text                                                            \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...   0   0   0   0   0   0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...   0   0   0   0   0   0   \n",
       "yah watch #grandcrew tell ya friend watch            0   0   0   0   0   0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...   0   0   0   0   0   0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...   0   0   0   0   0   0   \n",
       "...                                                 ..  ..  ..  ..  ..  ..   \n",
       "tax day they'r like eat cooki dough like ok i'l...   0   0   0   0   0   0   \n",
       "second glanc #reput ... readi                        0   0   0   0   0   0   \n",
       "#lwymmdvideo                                         0   0   0   0   0   0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...   0   0   0   0   0   0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...   0   0   0   0   0   0   \n",
       "\n",
       "                                                    सर  सरक  हम  हर  \n",
       "preprocessed_text                                                    \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...   0    0   0   0  \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...   0    0   0   0  \n",
       "yah watch #grandcrew tell ya friend watch            0    0   0   0  \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...   0    0   0   0  \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...   0    0   0   0  \n",
       "...                                                 ..  ...  ..  ..  \n",
       "tax day they'r like eat cooki dough like ok i'l...   0    0   0   0  \n",
       "second glanc #reput ... readi                        0    0   0   0  \n",
       "#lwymmdvideo                                         0    0   0   0  \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...   0    0   0   0  \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...   0    0   0   0  \n",
       "\n",
       "[55080 rows x 1696 columns]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(BoW.todense(), columns=count_vectorizer.get_feature_names_out(), index=df[\"preprocessed_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a12f9f8b",
   "metadata": {},
   "source": [
    "### Term Frequency–Inverse Document Frequency (TF-IDF) <a class=\"anchor\" id=\"tfidf\"></a>\n",
    "\n",
    "$t=term$\n",
    "\n",
    "$d=document$\n",
    "\n",
    "$n_{t, d}=\\text{number of times t occurs in d}$\n",
    "\n",
    "\n",
    "$\\textit{tf}_{t,d}=\\frac{n_{t, d}}{\\text{Number of terms in the document}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e863e2c2",
   "metadata": {},
   "source": [
    "![tfidf](img/tfidf.png)\n",
    "\n",
    "Image Source: https://www.analyticsvidhya.com/blog/2020/02/quick-introduction-bag-of-words-bow-tf-idf/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "c301c01c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "b1319d72",
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer(min_df=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "139e6b6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>000</th>\n",
       "      <th>10</th>\n",
       "      <th>100</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>12pm</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>...</th>\n",
       "      <th>रह</th>\n",
       "      <th>वन</th>\n",
       "      <th>वर</th>\n",
       "      <th>शन</th>\n",
       "      <th>सभ</th>\n",
       "      <th>सम</th>\n",
       "      <th>सर</th>\n",
       "      <th>सरक</th>\n",
       "      <th>हम</th>\n",
       "      <th>हर</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preprocessed_text</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>@ashleyn1col :smiling_face_with_hearts: :purple_heart: :smiling_face_with_hearts: :purple_heart:</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>:loudly_crying_face: :loudly_crying_face: :loudly_crying_face:</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>yah watch #grandcrew tell ya friend watch</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>@nanglish @grandcrewnbc :purple_heart: :purple_heart: :purple_heart:</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>oooooh @ozzymo :purple_heart: :purple_heart: :purple_heart:</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tax day they'r like eat cooki dough like ok i'll make sacrific art</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>second glanc #reput ... readi</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>#lwymmdvideo</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>offici #lwymmdvideo world premier sunday 8/ 27 @vma</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>new singl #lookwhatyoumademedo pre-ord #reput merch ticket info</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>55080 rows × 1696 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    000   10  100   11   12  \\\n",
       "preprocessed_text                                                             \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...  0.0  0.0  0.0  0.0  0.0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...  0.0  0.0  0.0  0.0  0.0   \n",
       "yah watch #grandcrew tell ya friend watch           0.0  0.0  0.0  0.0  0.0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...  0.0  0.0  0.0  0.0  0.0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...  0.0  0.0  0.0  0.0  0.0   \n",
       "...                                                 ...  ...  ...  ...  ...   \n",
       "tax day they'r like eat cooki dough like ok i'l...  0.0  0.0  0.0  0.0  0.0   \n",
       "second glanc #reput ... readi                       0.0  0.0  0.0  0.0  0.0   \n",
       "#lwymmdvideo                                        0.0  0.0  0.0  0.0  0.0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...  0.0  0.0  0.0  0.0  0.0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...  0.0  0.0  0.0  0.0  0.0   \n",
       "\n",
       "                                                    12pm   13   14   15   16  \\\n",
       "preprocessed_text                                                              \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...   0.0  0.0  0.0  0.0  0.0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...   0.0  0.0  0.0  0.0  0.0   \n",
       "yah watch #grandcrew tell ya friend watch            0.0  0.0  0.0  0.0  0.0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...   0.0  0.0  0.0  0.0  0.0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...   0.0  0.0  0.0  0.0  0.0   \n",
       "...                                                  ...  ...  ...  ...  ...   \n",
       "tax day they'r like eat cooki dough like ok i'l...   0.0  0.0  0.0  0.0  0.0   \n",
       "second glanc #reput ... readi                        0.0  0.0  0.0  0.0  0.0   \n",
       "#lwymmdvideo                                         0.0  0.0  0.0  0.0  0.0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...   0.0  0.0  0.0  0.0  0.0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...   0.0  0.0  0.0  0.0  0.0   \n",
       "\n",
       "                                                    ...   रह   वन   वर   शन  \\\n",
       "preprocessed_text                                   ...                       \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...  ...  0.0  0.0  0.0  0.0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...  ...  0.0  0.0  0.0  0.0   \n",
       "yah watch #grandcrew tell ya friend watch           ...  0.0  0.0  0.0  0.0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...  ...  0.0  0.0  0.0  0.0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...  ...  0.0  0.0  0.0  0.0   \n",
       "...                                                 ...  ...  ...  ...  ...   \n",
       "tax day they'r like eat cooki dough like ok i'l...  ...  0.0  0.0  0.0  0.0   \n",
       "second glanc #reput ... readi                       ...  0.0  0.0  0.0  0.0   \n",
       "#lwymmdvideo                                        ...  0.0  0.0  0.0  0.0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...  ...  0.0  0.0  0.0  0.0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...  ...  0.0  0.0  0.0  0.0   \n",
       "\n",
       "                                                     सभ   सम   सर  सरक   हम  \\\n",
       "preprocessed_text                                                             \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...  0.0  0.0  0.0  0.0  0.0   \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...  0.0  0.0  0.0  0.0  0.0   \n",
       "yah watch #grandcrew tell ya friend watch           0.0  0.0  0.0  0.0  0.0   \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...  0.0  0.0  0.0  0.0  0.0   \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...  0.0  0.0  0.0  0.0  0.0   \n",
       "...                                                 ...  ...  ...  ...  ...   \n",
       "tax day they'r like eat cooki dough like ok i'l...  0.0  0.0  0.0  0.0  0.0   \n",
       "second glanc #reput ... readi                       0.0  0.0  0.0  0.0  0.0   \n",
       "#lwymmdvideo                                        0.0  0.0  0.0  0.0  0.0   \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...  0.0  0.0  0.0  0.0  0.0   \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...  0.0  0.0  0.0  0.0  0.0   \n",
       "\n",
       "                                                     हर  \n",
       "preprocessed_text                                        \n",
       "@ashleyn1col :smiling_face_with_hearts: :purple...  0.0  \n",
       ":loudly_crying_face: :loudly_crying_face: :loud...  0.0  \n",
       "yah watch #grandcrew tell ya friend watch           0.0  \n",
       "@nanglish @grandcrewnbc :purple_heart: :purple_...  0.0  \n",
       "oooooh @ozzymo :purple_heart: :purple_heart: :p...  0.0  \n",
       "...                                                 ...  \n",
       "tax day they'r like eat cooki dough like ok i'l...  0.0  \n",
       "second glanc #reput ... readi                       0.0  \n",
       "#lwymmdvideo                                        0.0  \n",
       "offici #lwymmdvideo world premier sunday 8/ 27 ...  0.0  \n",
       "new singl #lookwhatyoumademedo pre-ord #reput m...  0.0  \n",
       "\n",
       "[55080 rows x 1696 columns]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = tfidf_vectorizer.fit_transform(df[\"preprocessed_text\"])\n",
    "\n",
    "pd.DataFrame(tfidf.todense(), columns=tfidf_vectorizer.get_feature_names_out(), index=df[\"preprocessed_text\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e46750df",
   "metadata": {},
   "source": [
    "### Build a Simple Classifier <a class=\"anchor\" id=\"classifier\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "b6b86a04",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "1973cbe0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_predict_score(X, y, vectorizer, classifier):\n",
    "    X = vectorizer.fit_transform(X)\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=530)\n",
    "    classifier.fit(X_train, y_train)\n",
    "    pred = classifier.predict(X_test)\n",
    "    return metrics.accuracy_score(y_test, pred), metrics.precision_score(y_test, pred, average=\"weighted\"), metrics.f1_score(y_test, pred, average=\"weighted\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "6e529e21",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizers = [\"Bag-of-Words\", \"TF-IDF\"]\n",
    "classifiers = [\"Logistic Regression\", \"Random Forest\", \"Support Vectors\", \"Multinomial Naive Bayes\"]\n",
    "\n",
    "vectorizer_name_to_vectorizer = {\"Bag-of-Words\":CountVectorizer(min_df=50), \"TF-IDF\":TfidfVectorizer(min_df=50)}\n",
    "classifier_name_to_classifier = {\"Logistic Regression\":LogisticRegression(max_iter=1000), \"Random Forest\":RandomForestClassifier(),\n",
    "                                 \"Support Vectors\": LinearSVC(max_iter=1000), \"Multinomial Naive Bayes\":MultinomialNB()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "436db655",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorizer: Bag-of-Words | Classifier: Logistic Regression\n",
      "\n",
      "Accuracy Score: 0.636\n",
      "Precision Score (weighted): 0.654\n",
      "F1 Score (weighted): 0.638\n",
      "------------------------------------------------------------\n",
      "Vectorizer: TF-IDF | Classifier: Logistic Regression\n",
      "\n",
      "Accuracy Score: 0.641\n",
      "Precision Score (weighted): 0.651\n",
      "F1 Score (weighted): 0.643\n",
      "------------------------------------------------------------\n",
      "Vectorizer: Bag-of-Words | Classifier: Random Forest\n",
      "\n",
      "Accuracy Score: 0.607\n",
      "Precision Score (weighted): 0.622\n",
      "F1 Score (weighted): 0.609\n",
      "------------------------------------------------------------\n",
      "Vectorizer: TF-IDF | Classifier: Random Forest\n",
      "\n",
      "Accuracy Score: 0.618\n",
      "Precision Score (weighted): 0.628\n",
      "F1 Score (weighted): 0.620\n",
      "------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\melihcanyardi\\anaconda3\\lib\\site-packages\\sklearn\\svm\\_base.py:1225: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorizer: Bag-of-Words | Classifier: Support Vectors\n",
      "\n",
      "Accuracy Score: 0.639\n",
      "Precision Score (weighted): 0.656\n",
      "F1 Score (weighted): 0.641\n",
      "------------------------------------------------------------\n",
      "Vectorizer: TF-IDF | Classifier: Support Vectors\n",
      "\n",
      "Accuracy Score: 0.638\n",
      "Precision Score (weighted): 0.647\n",
      "F1 Score (weighted): 0.639\n",
      "------------------------------------------------------------\n",
      "Vectorizer: Bag-of-Words | Classifier: Multinomial Naive Bayes\n",
      "\n",
      "Accuracy Score: 0.623\n",
      "Precision Score (weighted): 0.633\n",
      "F1 Score (weighted): 0.624\n",
      "------------------------------------------------------------\n",
      "Vectorizer: TF-IDF | Classifier: Multinomial Naive Bayes\n",
      "\n",
      "Accuracy Score: 0.624\n",
      "Precision Score (weighted): 0.637\n",
      "F1 Score (weighted): 0.626\n",
      "------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "for classifier in classifiers:\n",
    "    for vectorizer in vectorizers:\n",
    "        accuracy, precision, f1 = train_predict_score(X=df[\"preprocessed_text\"],\n",
    "                                                      y=df[\"user_type\"],\n",
    "                                                      vectorizer=vectorizer_name_to_vectorizer[vectorizer],\n",
    "                                                      classifier=classifier_name_to_classifier[classifier])\n",
    "        print(f\"Vectorizer: {vectorizer} | Classifier: {classifier}\\n\")\n",
    "        print(f\"Accuracy Score: {accuracy:.3f}\")\n",
    "        print(f\"Precision Score (weighted): {precision:.3f}\")\n",
    "        print(f\"F1 Score (weighted): {f1:.3f}\")\n",
    "        print(\"-\"*60)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}