tf-idf_modelling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Extraction w TF-IDF\n",
    "\n",
    "The purpose of this notebook is extract text features of movie reviews using **TF-IDF**. The idea being that this feature extraction method will be compared to **Doc2Vec** in the sentiment classification task. \n",
    "\n",
    "**Questions**\n",
    "\n",
    "What does the `TF-IDF` vectorizer do when the validation set has a word in it that does not appear in the train set? How does it scale this feature in the validation set? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## import packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from time import time\n",
    "import sys\n",
    "sys.path.append(\"/Users/lesleymi/data_science_tutorials/IMDB_Sentiment_Analysis/src\")\n",
    "import imdb_functions as imdb\n",
    "\n",
    "# feature extractor\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# models \n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# evaluation \n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.inspection import permutation_importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load clean train data \n",
    "X_train = pd.read_csv(\"data/train_clean.csv\").drop(labels='Unnamed: 0', axis=1).text\n",
    "y_train = pd.read_csv(\"data/Train.csv\").label\n",
    "X_valid = pd.read_csv(\"data/valid_clean.csv\").drop(labels='Unnamed: 0', axis=1)\n",
    "y_valid = X_valid.label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set has 40000 documents.\n",
      "Validation set has 5000 documents.\n"
     ]
    }
   ],
   "source": [
    "print(\"Train set has {} documents.\".format(X_train.shape[0]))\n",
    "print(\"Validation set has {} documents.\".format(X_valid.shape[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        grow watch love thunderbirds mate school watch...\n",
       "1        movie dvd player sit coke chip expectation hop...\n",
       "2        people know particular time past like feel nee...\n",
       "3        great interest biblical movie bore death minut...\n",
       "4        be die hard dad army fan change get tape dvd a...\n",
       "                               ...                        \n",
       "39995    western union forget classic western reason li...\n",
       "39996    movie incredible piece work explore nook crann...\n",
       "39997    wife watch movie plan visit sicily stromboli s...\n",
       "39998    watch flatliners amaze necessary feature good ...\n",
       "39999    film good gross estimate award nomination john...\n",
       "Name: text, Length: 40000, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF\n",
    "\n",
    "**Note**\n",
    "\n",
    "I didn't set a max features parameter thus, the vectorizer uses all the unique words in the training corpus as features. This results in many more features/unique words than training documents!!! This will likely pose a problem later. But most of the classifiers apply regularization by default anyway. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the vectorizer \n",
    "vectorizer = TfidfVectorizer()\n",
    "\n",
    "# fit and transform train set \n",
    "X_train_tfidf = vectorizer.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The TF-IDF transformed data structure is a <class 'scipy.sparse.csr.csr_matrix'>\n",
      "There are 73448 unique words in the vocabulary.\n"
     ]
    }
   ],
   "source": [
    "print(\"The TF-IDF transformed data structure is a {}\".format(type(X_train_tfidf)))\n",
    "print(\"There are {} unique words in the vocabulary.\".format(X_train_tfidf.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform the validation set \n",
    "X_valid_tfidf = vectorizer.transform(X_valid.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Classification "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### train set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training time for logistic regression... 3.18 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "lr = LogisticRegression(random_state=0, max_iter=1000).fit(X_train_tfidf, y_train)\n",
    "stop = time()\n",
    "print(\"training time for logistic regression... {} seconds\".format(round(stop - start, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 92.0 %\n"
     ]
    }
   ],
   "source": [
    "acc = round(lr.score(X_train_tfidf, y_train), 2)*100\n",
    "print('Train accuracy: {} %'.format(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x10eefe6d0>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAEGCAYAAADyuIefAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZyXZb3/8debYRdkGxcUSVLUg2tIinoyRVM0T9j5WWmLVJaWWq6Vdn5FuaVpmZ4SjyUpVuLakToqkemxDRRXBBdQXFAUYdhlmeVz/rivGb7gLN97mC+zvZ+Px/2Y+77u7bpn9MN93dd9Xx9FBGZmlunS2hUwM2tLHBTNzAo4KJqZFXBQNDMr4KBoZlaga2tXoFD5wLIYukubqpI1Yf7sPq1dBcthXaxhQ6zTlhzj2CO3iaUV1UVt+8Sz66dFxNgtOd/W1qYi0NBduvLoAzu2djUsh3/f7SOtXQXLYcb6B7b4GEsrqnls2tCiti0bPK98i0+4lbWpoGhmbV8ANdS0djVKxkHRzHIJgsoorvncHjkomlluvlM0M0uCoLoDfx7sV3LMLLcaoqipKZImSVos6bmCsgMkzZD0tKRZkg5K5ZJ0vaT5kp6VNLJgn/GS5qVpfEH5gZJmp32ul9Rkz7uDopnlEkA1UdRUhFuAzV/Z+THww4g4APh+WgY4DhieptOBiQCSBgITgIOBg4AJkgakfSYCXy3Yr8nXgxwUzSy3lrpTjIhHgYrNi4Ft03w/4K00Pw6YHJkZQH9Jg4FjgekRURERy4DpwNi0btuImBHZcGCTgRObqpOfKZpZLgFUFv9MsVzSrILlmyLipib2OReYJukashu3Q1P5zsAbBdstTGWNlS+sp7xRDopmlksU3zQGWBIRo3Ke4uvAeRFxj6RPAzcDR+c8RrO5+Wxm+QRUFzk103jg3jR/F9lzQoA3gV0KthuSyhorH1JPeaMcFM0sl+yLluKmZnoL+GiaHwPMS/NTgVNTL/RoYEVELAKmAcdIGpA6WI4BpqV1KyWNTr3OpwL3NXVyN5/NLCdRzRaNKbHxSNLtwBFkzx4XkvUifxW4TlJXYB1ZTzPA/cDxwHzgPeBLABFRIelS4PG03SURUdt5cyZZD3cv4IE0NcpB0cxyyTpaWiYoRsQpDaw6sJ5tAzirgeNMAibVUz4L2CdPnRwUzSyX7D3FlgmKbZGDopnlVtNCd4ptkYOimeXiO0UzswKBqO7AL644KJpZbm4+m5klgdgQZa1djZJxUDSzXLKXt918NjOr444WM7MkQlSH7xTNzOrU+E7RzCyTdbR03NDRca/MzErCHS1mZpup9nuKZmYZf9FiZraZGvc+m5llsgEhOm5Q7LhXZmYlEYjKKCtqaoqkSZIWS3pus/JvSHpB0hxJPy4ovzgltn9R0rEF5WNT2XxJFxWUD5M0M5XfIal7U3VyUDSzXCKgOroUNRXhFjZLUC/pSLIcz/tHxN7ANal8BHAysHfa5wZJZZLKgF8AxwEjgFPStgBXAddGxO7AMuC0pirkoGhmOYmaIqemRMSjQMVmxV8HroyI9Wmbxal8HDAlItZHxAKyXC0HpWl+RLwSERuAKcC4lKxqDHB32v9W4MSm6uSgaGa5BLnuFMslzSqYTm/i8AB7AB9Jzd7/lfThVN5Y0vv6ygcByyOiarPyRrmjxcxyy9HRsiQiRuU8fFdgIDAa+DBwp6QP5jxGszkomlkugUo9yOxC4N6Uve8xSTVAOQ0nvaeB8qVAf0ld091i4fYNcvPZzHLJUpx2LWpqpv8GjgSQtAfQHVgCTAVOltRD0jBgOPAYWb7n4amnuTtZZ8zUFFQfBk5Kxx0P3NfUyX2naGY5qcXGU5R0O3AE2bPHhcAEsvzNk9JrOhuA8SnAzZF0JzAXqALOiojqdJyzgWlAGTApIuakU3wHmCLpMuAp4Oam6uSgaGa5BC33RUtEnNLAqs83sP3lwOX1lN8P3F9P+StkvdNFc1A0s9w88raZWRIhf/tsZlYr62hxNj8zs8Q5WszM6mQdLX6maGZWpyMPHeagaGa5bIUvWlqVg6KZ5ebEVWZmSQRU1jgompkBtc1nB0Uzszr+osX4zwuGMevP/elXXsn1D2XpJBbM6c2NF+3KhvWirCucfvmr7PGhNcyc1p/brx6CugRlXeHLP3iNEQetZvHC7lz1leHU1EB1lTj+S+8w9gvvAvDys725/rwPsmFdFw4cs5zTLnkdddz/7ra68656hYPHLGf50m58bey+deWfGP82//aFxdRUi8ce7sfNVw6lrGsN5165gN33fo+yrsFD95Zzx8SdABj3xbc57uR3keCBKdvx37/esbUuqdX4lZwtIGkscB3ZyBW/iogrS3m+UhrzqSUc/8V3uO7cjWNd3nr5Lnz6vDc5cMwKnnioH5Mv34XL7n6B/f51JQcd8xwSvDq3F9d8fXd+/r+zGbB9JVfeN5duPYK1a7pwzlH7ctDHljNwx0puvHhXzvzxAvYYuYZLv7AHTz7cjwPHrGjFK+5Ypt9Tzh8m78CFP3mlrmy/0Ss55OjlnHn8PlRu6EK/QZUAfOT4Crp1D75+3L706FnNTdNn88jUQfTcpprjTn6Xc04cQWVlFy6/5UVm/qU/i17r2VqX1Uo6dvO5ZFfWRDKZdmfv0avo279qkzIJ1q7OPnd6b1UZA3fI/qfqtU1N3V3eurVl1LY0unUPuvUIACo3dCFqsvKKd7qxdnUZex64BgmOPGkJj00bUPqL6kSee2xbVi3f9B7ghM8v5s4bB1O5IfvfYMXSbtmKED1719ClLOjes4bKSrFmdRlDd1/Hi09vw/p1ZdRUi9mP9eWwscu29qW0CS2Vo6UtKuWdYl0yGQBJU8gSz8wt4Tm3qi//4DUu+dye3HLpLkSN+NF9Gy9txgMD+M2VQ1ixpBv/MfmluvIlb3XnslP3YNGrPRj//99g4I6VzH9mGwYN3lC3zaDBG1j6dpOZGG0L7TxsHXt/eBXjL1zIhvVd+NUVu/DSs3346wMDGP2xZfxu5lP07FXDf102lNUruvLqi70Yf+Eb9O1fyYZ1XfjwEct5afY2rX0ZW13W++xvn5ujvmQyB2++UUpkczrALju3r1/0tMnb8+UJr3PIx5fx9z8M5BcXDuOHU14EYPRxyxh93DLmzOjL7VfvXFdevtMGfvbn56h4uxs/Om04h35880RmtrWUlQV9+1dx7idHsMf+a/juz+fzxcP3Z8/911BTLT43+gD69KvmJ3c+z1N/25Y3Xu7FXTfuxBWTX2Td2jJenrsNNdXt825oS3T0l7db/cFARNwUEaMiYlT5oPYVFB++u5zRx2fNp0NPqGDe033et83eo1fxzus9WFmx6b8/A3esZOhea5k7sy8Dd9zA0kUb7wyXLurOoB03bH4oa2FL3u7O3x8cCIiXnulDTY3oN7CKI8ct5YlH+1Fd1YUVS7sxZ1Yfhu+3BoBpd27HNz6xD9/6zL+wekUZby7obM8TMy3VfJY0SdLiNMr25usukBSSytOyJF2fEts/K2lkwbbjJc1L0/iC8gMlzU77XJ/SnjaqlEGxsSQzHcKAHSqZ88++AMz++7YMHrYOgEULehDZo0Nent2byvVd6DugiiVvdWP92uxvsnp5Gc8/1pedd1vHwB0q6dWnmhef2IaILNgedEznfFa1Nf3jTwPY/5CVAOw8bC3dugUrKrqy+M3udeU9elWz14dWs/DlXgB1nTHb7bSew8Yu4+H7BrVO5VtRbe9zMVMRbiFLbL8JSbsAxwCvFxQfR5aXZThZ63Ji2nYgWRqDg8ke202QVPtQfiLw1YL93neuzZWy+VyXTIYsGJ4MfLaE5yupn5y1G3P+2ZeVFV35yqgDOPmChZz54wXcPOED1FSJbj1qOPOqBQD88/6BPHLPIMq6Bt17BhdMnI8EC+f34pZLhiJlz2VOPGMRH/iXtQCcccVrXH/+MDas68LII1Yw0j3PLeqi6+az3+hVbDugitv+8RS/+dkQ/nRXOef/eAE3PjibqkpxzYUfBMQfbtuBC65+hf+aNhsUTL97Oxa80BuA702cR9/+VVRXiV98/wOsWdU532prwXQEj0ratZ5V1wLfZtNEU+OAySlfywxJ/SUNJsvxMj0iKgAkTQfGSnoE2DYiZqTyycCJwAON1UlRe0tTApKOB37GxmQy78utUGjk/j3i0Qc633tf7dm/7/aR1q6C5TBj/QOsrFm6RQ8EB+y1fYyZdFLTGwL3HjbxNbJMfLVuioibCrdJQfGPEbFPWh4HjImIcyS9CoyKiCWS/ghcGRF/S9s9RJaY6gigZ0Rclsq/B6wFHknbH53KPwJ8JyJOaKzOJf1nrqFkMmbWvuXoaFkSEaOK3VhSb+C7ZE3nVtHqHS1m1r608DPFze0GDAOeSXeJQ4AnJe1Iw/0UjZUPqae8UQ6KZpZbqYJiRMyOiO0jYteI2JXsVb6REfE2MBU4NfVCjwZWRMQisnzPx0gakDpYjgGmpXUrJY1Ovc6nsukzynp1zqfEZtZsLfmeoqTbyZ4JlktaCEyIiIYS1t8PHA/MB94DvgQQERWSLiXr3AW4pLbTBTiTrIe7F1kHS6OdLOCgaGbN0FKf8EXEKU2s37VgPoCzGthuEjCpnvJZwD556uSgaGa5RECVB5k1M9uoI3/m56BoZrl09G+fHRTNLLdwUDQz26i9jpVYDAdFM8slws8UzcwKiGr3PpuZbeRnimZmibP5mZkVCijhiIOtzkHRzHJz77OZWRLuaDEz25Sbz2ZmBdz7bGaWRDgompltoiO/ktNxn5aaWclEFDc1RdIkSYslPVdQdrWkF1LC+99L6l+w7uKU2P5FSccWlI9NZfMlXVRQPkzSzFR+h6TuTdXJQdHMcglETU2XoqYi3ML7E9RPB/aJiP2Al4CLASSNIMsfv3fa5wZJZZLKgF8AxwEjgFPStgBXAddGxO7AMuC0pirkoGhmuUWRU5PHiXgUqNis7E8RUZUWZ7AxI984YEpErI+IBWS5Wg5K0/yIeCUiNgBTgHEpWdUY4O60/63AiU3VyUHRzPJJHS3FTGQJqWYVTKfnPNuX2ZhsamfgjYJ1C1NZQ+WDgOUFAba2vFHuaDGz/Ip/T3FJRIxqzikk/QdQBfy2Ofs3l4OimeVW6ldyJH0ROAE4KmXxg4aT3tNA+VKgv6Su6W6xcPsGNRgUJf0njfx7EBHfbOrgZtbxBFBTU7qgKGks8G3goxHxXsGqqcDvJP0U2AkYDjwGCBguaRhZ0DsZ+GxEhKSHgZPInjOOB+5r6vyN3SnOasb1mFlHF0AL3SlKuh04guzZ40JgAllvcw9getZXwoyI+FpEzJF0JzCXrFl9VkRUp+OcDUwDyoBJETEnneI7wBRJlwFPATc3VacGg2JE3LpZ5XtvFrXNrJNqqW+fI+KUeoobDFwRcTlweT3l9wP311P+ClnvdNGa7H2WdIikucALaXl/STfkOYmZdTAt9U5OG1TMKzk/A44le2hJRDwDHF7KSplZW1bc6zjt9fvoonqfI+KN1LavVV2a6phZu9BO7wKLUUxQfEPSoUBI6gacAzxf2mqZWZsVECXsfW5txTSfvwacRfYm+FvAAWnZzDotFTm1P03eKUbEEuBzW6EuZtZedODmczG9zx+U9AdJ76Yhfu6T9MGtUTkza6M6ee/z74A7gcFkb5HfBdxeykqZWRtW+/J2MVM7VExQ7B0Rt0VEVZp+A/QsdcXMrO1qqUFm26LGvn0emGYfSCPZTiH7N+Iz1PPmuJl1Ih2497mxjpYnyIJg7dWfUbAuSKPhmlnno3Z6F1iMxr59HrY1K2Jm7UQ77kQpRlFftEjahyz3Qd2zxIiYXKpKmVlb1n47UYrRZFCUNIFsaJ8RZM8SjwP+BjgomnVWHfhOsZje55OAo4C3I+JLwP5Av5LWyszatpoip3aomObz2oiokVQlaVtgMZsO/W1mnUkLDjLbFhVzpzgrJaP+JVmP9JPAP0taKzNr0xTFTU0eR5qUvpR7rqBsoKTpkualnwNSuSRdnxLbPytpZME+49P28ySNLyg/UNLstM/12my4r/o0GRQj4syIWB4RNwIfA8anZrSZdVYt95nfLWSJ7QtdBDwUEcOBh9IyZP0Zw9N0OjAR6t6pngAcTDbK9oTaQJq2+WrBfpuf630ae3l7ZGPrIuLJpg5uZtaYiHhU0q6bFY8j69yFLIH9I2S5VsYBk1N2vxmS+ksanLadHhEVAJKmA2MlPQJsGxEzUvlk4EQ25pGuV2PPFH/S2LUAYxo7cHPMf3YbPjkkVzoFa2XT3prZ2lWwHA46dk2LHCfHy9vlkgqT4N0UETc1sc8OEbEozb8N7JDmG0p631j5wnrKG9XYy9tHNrWzmXVCQZ7P/JZExKhmnypLU7pVXwAqpqPFzGxTpR067J3ULCb9XJzK36T+pPeNlQ+pp7xRDopmlltL9T43YCpZ4nrYNIH9VODU1As9GliRmtnTgGMkDUgdLMcA09K6lZJGp17nUwuO1aCiPvMzM9tECzVoJd1O1lFSLmkhWS/ylcCdkk4DXgM+nTa/HzgemA+8B3wJICIqJF0KPJ62u6S20wU4k6yHuxdZB0ujnSxQ3Gd+IktH8MGIuETSUGDHiHisqX3NrINqoaAYEac0sOqoerYNGsgPFRGTgEn1lM8C9slTp2KazzcAhwC1lV8F/CLPScys4yi26dxehxcrpvl8cESMlPQUQEQsk9S9xPUys7askw4yW6tSUhnphlnSdrTbT73NrCW017vAYhTTfL4e+D2wvaTLyYYNu6KktTKztq0DZ/MrJu/zbyU9QfbgU8CJEfF8yWtmZm1TO35eWIxiep+HknV//6GwLCJeL2XFzKwN68xBEfgfNiaw6gkMA14E9i5hvcysDVMH7lUopvm8b+FyGj3nzJLVyMysFeX+oiUinpR0cCkqY2btRGduPks6v2CxCzASeKtkNTKztq2zd7QAfQvmq8ieMd5TmuqYWbvQWYNiemm7b0RcuJXqY2btQWcMipK6RkSVpMO2ZoXMrG0Tnbf3+TGy54dPS5oK3AXUjWUeEfeWuG5m1hb5mSI9gaVkOVlq31cMwEHRrLPqpEFx+9Tz/Bwbg2GtDvwrMbMmdeAI0NiAEGVAnzT1LZivncysk2qp8RQlnSdpjqTnJN0uqaekYZJmpgT2d9QOVSipR1qen9bvWnCci1P5i5KO3ZJra+xOcVFEXLIlBzezDqoF7hQl7Qx8ExgREWsl3QmcTJZy4NqImCLpRuA0sqT2pwHLImJ3SScDVwGfkTQi7bc3sBPwZ0l7RER1c+rV2J1ixx1F0syaL7Le52KmInQFeknqCvQGFpH1X9yd1t9KlsAeYFxaJq0/KqVLGQdMiYj1EbGALIdLsxPINxYU35cjwcwMyDOeYrmkWQXT6XWHiHgTuAZ4nSwYrgCeAJZHRFXarDCBfV3S+7R+BTCosLyefXJrsPlckA3LzGwTOV7JWRIRo+o9RpaOdBzZyFvLyV77G9sS9dsSzvtsZvm1zMjbRwMLIuLdiKgke83vMKB/ak7Dpgns65Lep/X9yF4XrCuvZ5/cHBTNLJ9iA2LTQfF1YLSk3unZ4FHAXOBh4KS0zXg2JrCfmpZJ6/+S0p5OBU5OvdPDgOFkH580S+6hw8yscxMt80VLRMyUdDfwJNlgM08BN5ENOjNF0mWp7Oa0y83AbZLmAxVkPc5ExJzUcz03Hees5vY8g4OimTVDS33mFxETgAmbFb9CPb3HEbEO+FQDx7kcuLwl6uSgaGb5deAvWhwUzSw/B0Uzs8Sj5JiZbcZB0cxso846yKyZWb3cfDYzq1Xci9ntloOimeXnoGhmlmmpL1raKgdFM8tNNR03Kjoomlk+fqZoZrYpN5/NzAo5KJqZbeQ7RTOzQg6KZmZJdOzP/JyOwMxyqX1PsZipyWNJ/SXdLekFSc9LOkTSQEnTJc1LPwekbSXp+pT0/llJIwuOMz5tP0/S+IbP2DQHRTPLL6K4qWnXAQ9GxF7A/sDzwEXAQxExHHgoLQMcR5Z/ZThwOjARQNJAstG7DyYbsXtCbSBtDgdFM8utJe4UJfUDDiflYImIDRGxnE2T3t8KnJjmxwGTIzODLOvfYOBYYHpEVETEMmA6W5Aq1c8Um+H8n77OwUevYvmSrpwxZk8ATv3WIg45diURsHxJV645dygV73Rjl93Xcf5P32D3fddy61U7cveN29cd59aZc1m7uoyaGqiuEt84bo/WuqQO6Sfn7cLMP29L//Iqbnr4RQBefq4X1180hA3rulDWNTj7RwvZ60PvAfDMP/pw4/d3pqoK+g2s5pp75wOwekUZ1164C6++0BMp+/uPGJXtc9/N5Uy9pZwuZcHBR63kK99b1DoXuzXle3m7XNKsguWbIuKmND8MeBf4taT9gSeAc4AdIqL2F/k2sEOabyjpfUPlzVKyoChpEnACsDgi9inVeVrDn+4YyNRfl/Ot6zb+He6euD2Trx4MwLjT3uXz573D9RcNYeWyMiZ+b2cOHbui3mN9+1O7sbLC/zaVwjGfqeATX1rC1ecMrSv71WWD+fz5b/PhMat47KG+3HzZTlx9z3xWryjj5xcP4fLfvsz2QypZvmTj32Ti93dm1BEr+d4vX6Vyg1i/NmtgPf33PvxjWj8m/vlFuveITfbp6HJ0tCyJiFENrOsKjAS+kTL7XcfGpjIAERHS1n0BqJTN51vYglvYtuy5mX1YtWzT/wHeW11WN9+zV03d45QVS7vx0jO9qarS1qyiAfuOXkPfAZtmupRgzarsb7VmZRkDd6gE4OHf9+ew45ez/ZBsuX95VdqmC7NnbMPYz1YA0K170Kdfdsw/Th7EZ85+h+49YpN9OgPVFDc1YSGwMCJmpuW7yYLkO6lZTPq5OK1vKOl9Q+XNUrJ/2iLiUUm7lur4bdEXv7OIoz+1jDUry/j2Sbs1vUOIK25/BQL+57ZBPPDbQaWvZCf3tUve5Lun7MYvL9mJCLh26jwAFr7Sk+pK+Nb/2533VnfhxK+8y8c+tYy3X+9Bv0FV/OS8obwypyfD91vL1y99k569a3jz5Z48N7MPt1w1mO49gq9+/032PGBtK1/hVhAU24nS+GEi3pb0hqQ9I+JF4Ciy3M1zyZLeX5l+3pd2mQqcLWkKWafKiohYJGkacEVB58oxwMXNrVerd7RIOl3SLEmzKlnf2tXZIrdcNZjPjxrBX+7tzye+vKTJ7c8/cXfOPnYP/uNzw/jEF5ewz8Grt0ItO7c/3lrOGT98k98+MZczfvAWPz0/a1pXV8G82b259LZXuOJ3L/O7n+3Iwpd7UF0N82f35oRTl3DD9Jfo2buGO36ePReuroZVy8u47o/z+Mr33uLyM3ZtiVjRLrTUKznAN4DfSnoWOAC4giwYfkzSPODotAxwP1lO6PnAL4EzASKiArgUeDxNl6SyZmn1oBgRN0XEqIgY1Y0erV2dFvGX3w/gX4+v/xlioaVvdwOyJvbfH+xX98DfSmf6XQPr/jaH/9tyXnq6NwDbDa7kwI+uomfvGvoNqmbfg1fzytyelA+uZLvBlew1Mvvb/OsJy5k/uxcA5YMrOez4FUiw14feo0sXWFFRVv+JO5oocmrqMBFPp///94uIEyNiWUQsjYijImJ4RBxdG+BSr/NZEbFbROwbEbMKjjMpInZP06+35NJaPSh2FDsN23iXe8ixK3hjfuMBvkevanptU103f+BHV/HqCz1LWkeDQTtU8uw/+wDw9N/61P3dDhm7gjmPb0N1Fax7T7zwVG+GDl/PwO2rKN9pQ93f8+m/9mXo8GyfQ8eu4Jm/Z8da+HIPKjeIfgOr6zlrx9KSL2+3RZ2nu6wFXXTDa+x3yGr6DaziN7PmcttPduCgMasYstt6ampg8Zvduf47QwAYsF0l//nAPHr3rSZq4MSvLOH0I/Zk24FVTLj5VQDKugYP/34Asx7ZthWvquP50dc/wLP/7MOKiq587sARfOGCtzn36jeY+P2dqa4W3XvUcO7V2RsEQ4evZ9QRK/naUXuhLsHYz1aw617rADjrsje56uwPUFUpdhy6gQuufR2AY0+u4Kfn78LpR+5Jt27Bt657HXWG/rSIDj3IrKJED0Ek3Q4cAZQD7wATIuLmxvbZVgPjYB1VkvpYaUx76+nWroLlcNCxbzDrmXVbFLr79h8SHzr8nKK2/esfvv1EI6/ktEml7H0+pVTHNrPW1V6bxsVw89nM8gmgAzefHRTNLL+OGxMdFM0sPzefzcwKdOTeZwdFM8vHKU7NzDbKXt7uuFHRQdHM8uvAOVocFM0sN98pmpnV8jNFM7NCHfvbZwdFM8vPzWczsyRy5Whpdzyeopnl13J5n5FUJukpSX9My8MkzUxJ7++Q1D2V90jL89P6XQuOcXEqf1HSsVtyaQ6KZpZfC428nZwDPF+wfBVwbUTsDiwDTkvlpwHLUvm1aTskjQBOBvYmS5Z3g6RmD4HuoGhmuammpqipyeNIQ4CPA79KywLGkGX2A7gVODHNj0vLpPVHpe3HAVMiYn1ELCDL4XJQc6/NQdHM8gmyl7eLmaC8NjFdmk7f7Gg/A77NxtfBBwHLI6I2X2xhYvu6pPdp/Yq0fV15Pfvk5o4WM8tFRJ6Xt5c0NPK2pBOAxRHxhKQjWqp+W8pB0czya5lXcg4DPiHpeKAnsC1wHdBfUtd0N1iY2L426f1CSV2BfsDSgvJahfvk5uazmeXXAr3PEXFxRAyJiF3JOkr+EhGfAx4GTkqbjQfuS/NT0zJp/V8iSzI1FTg59U4PA4YDjzX30nynaGb51D5TLJ3vAFMkXQY8BdQmvLsZuE3SfKCCLJASEXMk3QnMBaqAsyKi2blmHRTNLLdiepbziIhHgEfS/CvU03scEeuATzWw/+XA5S1RFwdFM8up+Bez2yMHRTPLJ3BQNDPbRAf+9tlB0cxy8yCzZmaFHBTNzJIIqO647WcHRTPLz3eKZmYFHBTNzJIAnKPFzKxWQPiZoplZJnBHi5nZJvxM0cysgIOimVktDwhhZrZRAC08dFhb4qBoZvl14DtFpyMws5zSZ37FTI2QtIukhyXNlTRH0jmpfKCk6ZLmpXex5dUAAAXQSURBVJ8DUrkkXZ+S3j8raWTBscan7edJGt/QOYvhoGhm+QRE1BQ1NaEKuCAiRgCjgbNSYvuLgIciYjjwUFoGOI4s/8pw4HRgImRBFJgAHEw2YveE2kDaHA6KZpZfTRQ3NSIiFkXEk2l+FfA8Wb7mwqT3twInpvlxwOTIzCDL+jcYOBaYHhEVEbEMmA6Mbe6l+ZmimeVX/DPFckmzCpZvioibNt9I0q7Ah4CZwA4RsSitehvYIc03lPS+ofJmcVA0s3wi8vQ+L4mIUY1tIKkPcA9wbkSslFRwqghJW7VXx81nM8uvBfI+A0jqRhYQfxsR96bid1KzmPRzcSpvKOl9Q+XN4qBoZjkFUV1d1NQYZbeENwPPR8RPC1YVJr0fD9xXUH5q6oUeDaxIzexpwDGSBqQOlmNSWbO4+Wxm+bTc0GGHAV8AZkt6OpV9F7gSuFPSacBrwKfTuvuB44H5wHvAlwAiokLSpcDjabtLIqKiuZVyUDSz/Fpg6LCI+BugBlYfVc/2AZzVwLEmAZO2uFI4KJpZTgGEB5k1M0vCg8yamW2iqU6U9kzRhj7slvQu2YPVjqYcWNLalbBcOurf7AMRsd2WHEDSg2S/n2IsiYhmf13SGtpUUOyoJM1q6gVWa1v8N+u8/J6imVkBB0UzswIOilvH+z6AtzbPf7NOys8UzcwK+E7RzKyAg6KZWQEHxRKSNFbSiymnxEVN72GtTdIkSYslPdfadbHW4aBYIpLKgF+Q5ZUYAZyS8k9Y23YLWzCUvbV/DoqlcxAwPyJeiYgNwBSyHBPWhkXEo0Czh52y9s9BsXRaNG+EmW0dDopmZgUcFEunRfNGmNnW4aBYOo8DwyUNk9QdOJksx4SZtWEOiiUSEVXA2WQJdJ4H7oyIOa1bK2uKpNuBfwJ7SlqY8oRYJ+LP/MzMCvhO0cysgIOimVkBB0UzswIOimZmBRwUzcwKOCi2I5KqJT0t6TlJd0nqvQXHukXSSWn+V40NViHpCEmHNuMcr0p6X9a3hso322Z1znP9QNKFeetotjkHxfZlbUQcEBH7ABuArxWulNSsPN4R8ZWImNvIJkcAuYOiWXvkoNh+/RXYPd3F/VXSVGCupDJJV0t6XNKzks4AUObnaXzHPwPb1x5I0iOSRqX5sZKelPSMpIck7UoWfM9Ld6kfkbSdpHvSOR6XdFjad5CkP0maI+lXgJq6CEn/LemJtM/pm627NpU/JGm7VLabpAfTPn+VtFdL/DLNajXrzsJaV7ojPA54MBWNBPaJiAUpsKyIiA9L6gH8XdKfgA8Be5KN7bgDMBeYtNlxtwN+CRyejjUwIiok3Qisjohr0na/A66NiL9JGkr21c6/ABOAv0XEJZI+DhTzNciX0zl6AY9LuicilgLbALMi4jxJ30/HPpssodTXImKepIOBG4Axzfg1mtXLQbF96SXp6TT/V+BmsmbtYxGxIJUfA+xX+7wQ6AcMBw4Hbo+IauAtSX+p5/ijgUdrjxURDY0reDQwQqq7EdxWUp90jn9P+/6PpGVFXNM3JX0yze+S6roUqAHuSOW/Ae5N5zgUuKvg3D2KOIdZ0RwU25e1EXFAYUEKDmsKi4BvRMS0zbY7vgXr0QUYHRHr6qlL0SQdQRZgD4mI9yQ9AvRsYPNI512++e/ArCX5mWLHMw34uqRuAJL2kLQN8CjwmfTMcTBwZD37zgAOlzQs7Tswla8C+hZs9yfgG7ULkmqD1KPAZ1PZccCAJuraD1iWAuJeZHeqtboAtXe7nyVrlq8EFkj6VDqHJO3fxDnMcnFQ7Hh+Rfa88MmUfOm/yFoEvwfmpXWTyUaC2UREvAucTtZUfYaNzdc/AJ+s7WgBvgmMSh05c9nYC/5DsqA6h6wZ/XoTdX0Q6CrpeeBKsqBcaw1wULqGMcAlqfxzwGmpfnNwigdrYR4lx8ysgO8UzcwKOCiamRVwUDQzK+CgaGZWwEHRzKyAg6KZWQEHRTOzAv8HAW7ggL6v0BgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### confusion matrix\n",
    "plot_confusion_matrix(estimator=lr, X=X_train_tfidf, y_true=y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_reviews</th>\n",
       "      <th>pos_reivews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>precision</th>\n",
       "      <td>0.933062</td>\n",
       "      <td>0.917023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>0.915630</td>\n",
       "      <td>0.934187</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score</th>\n",
       "      <td>0.924264</td>\n",
       "      <td>0.925526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count_of_reviews</th>\n",
       "      <td>20019.000000</td>\n",
       "      <td>19981.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   neg_reviews   pos_reivews\n",
       "precision             0.933062      0.917023\n",
       "recall                0.915630      0.934187\n",
       "f1_score              0.924264      0.925526\n",
       "count_of_reviews  20019.000000  19981.000000"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get model predictions \n",
    "y_pred = lr.predict(X_train_tfidf)\n",
    "metrics = precision_recall_fscore_support(y_true=y_train, y_pred=y_pred)\n",
    "imdb.display_metrics(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### validation set "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation accuracy: 88.0 %\n"
     ]
    }
   ],
   "source": [
    "acc = round(lr.score(X_valid_tfidf, y_valid), 2)*100\n",
    "print('Validation accuracy: {} %'.format(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x12f513040>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT8AAAEICAYAAAAp2fO+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZwV1Z338c+XZlcQEEQQEI244IZi0MTEcclEcDJxiYmaSXSCCZK4JWMmT9QkJjpmnMeoMybRDCqPmrgrxiVuxCzojKhgEAU1NKKyKQJGkL27f88fVY0X6L59q+3bt7vr+/ZVr7731KmqU/Srf55Tp6p+igjMzPKmU6UbYGZWCQ5+ZpZLDn5mlksOfmaWSw5+ZpZLDn5mlksOfmZWEZKGSvqjpLmS5kg6Py2/UtKrkmZLul9Sn7R8uKR1kmaly68K9jVa0kuSqiVdK0lNHr8t3efXr1+n2GVIVaWbYRm8NbdPpZtgGayrW83GuvVNBoZijj1qu1ixsrakujNnb3g8IsY2tE7SIGBQRLwgqRcwEzgBGAL8ISJqJP0HQET8H0nDgYcjYr8G9vUccB7wLPAIcG1EPFqsbZ1LOoNWssuQKqb8rn+lm2EZnHvgP1S6CZbBM6se+Mj7WLGyluceH1ZS3apB8xr9g46IpcDS9PNqSa8Au0TEEwXVpgMnFztGGkR7R8T09PutJEG0aPDzsNfMMgmgrsT/SpX26g4i6bkVGs+WQWw3SX+R9GdJn07LdgEWFdRZlJYV1aZ6fmbW9gXBpiht2Av0lzSj4PukiJhUWEHS9sB9wLcjYlVB+cVADXBbWrQUGBYRKySNBn4rad/mnoeDn5lllqFXtzwiDmlspaQuJIHvtoiYUlD+z8DngGMinZiIiA3AhvTzTEnzgT2BxSTXCesNScuK8rDXzDIJgtoobSkmnZG9CXglIq4uKB8LfA/4fESsLSgfIKkq/bw7MAJ4Pb12uErSYek+TweavLjpnp+ZZVZHi9wlcjjwVeAlSbPSsouAa4FuwNT0jpXpETEROAK4VNImoA6YGBEr0+2+BdwM9CC5Rlh0sgMc/MwsowBqWyD4RcTTQEO33TzSSP37SIbIDa2bAWxzC0wxDn5mllkL9fwqysHPzDIJYFMbejiiuRz8zCyTIFpk2FtpDn5mlk1AbfuPfQ5+ZpZN8oRH++fgZ2YZidoGJ2nbFwc/M8skmfBw8DOznEnu83PwM7McqnPPz8zyxj0/M8ulQNR2gHeiOPiZWWYe9ppZ7gRiY7T/XDsOfmaWSXKTs4e9ZpZDnvAws9yJELXhnp+Z5VCde35mljfJhEf7Dx3tv+9qZq2qfsKjlKUYSUMl/VHSXElzJJ2flveTNFXSvPRn37Rckq6VVC1ptqSDC/Z1Rlp/nqQzSjkPBz8zy6w2VNLShBrggogYCRwGnC1pJPB94MmIGAE8mX4HGEeSsW0EMAG4HpJgCVwCHAqMAS6pD5jFOPiZWSb1T3iUshTdT8TSiHgh/bwaeAXYBTgeuCWtdgtwQvr5eODWSEwH+kgaBBwLTI2IlRHxHjAVGNvUebT/gbuZtbq60md7+0uaUfB9UkRM2rqSpOHAQcCzwMA0Fy/A28DA9PMuwMKCzRalZY2VF+XgZ2aZJC82KDn4LY+IQ4pVkLQ9SUrKb0fEqjRXb3KsiJBUlpfme9hrZpkEYlNUlbQ0RVIXksB3W0RMSYvfSYezpD+XpeWLgaEFmw9JyxorL8rBz8wyiYDa6FTSUoySLt5NwCsRcXXBqgeB+hnbM4AHCspPT2d9DwPeT4fHjwOfldQ3nej4bFpWlIe9ZpaRWuom58OBrwIvSZqVll0EXAHcLelM4E3gS+m6R4DjgGpgLfA1gIhYKeky4Pm03qURsbKpgzv4mVkmAS3yeFtEPA2NRtFjGqgfwNmN7GsyMDnL8R38zCwzv8zUzHInkF9mamb5k6SubP+ho/2fgZm1MictN7McCjI94dFmOfiZWWbu+ZlZ7kTIPT8zy59kwsPZ28wsd5zDw8xyKJnw8DU/M8shP+FhZrnjJzzMLLeaSk7UHjj4mVkmEbCpzsHPzHImGfY6+OXSyiVdueU7e7J6eVek4PAvv8PR45fwwu925HfXDOPt6p5878EX2fWADwBYsbAblx5zMAM/tg6A4Qet5ss/nb/FPq8/cx+Wv9WdH079S6ufT9506VrH/731Rbp0raOqc/D0E/257RfDOf+y1xix7wdIsPiNHlx98V6sX1vFiWcs4tiT36a2Rrz/Xhf+8wd7smxJ90qfRkX5CY8mSBoL/BdQBdwYEVeU83itpaoq+MIPFjBs/zWs/6CKKz43in0+9R6D9lzLhP9+ldsv2mObbfrvup6LHp3VwN7gL4/uSLeeteVutqU2bRQXjj+A9WurqOpcx89+8yIzpvVj0hUfY92a5E/iG9+bzz9+eTH33DiM+a9sz/lfPIgN66s47pQljL9gAVdcsE+Fz6JyOsqtLmXru0qqAn5Jkmh4JHBampC43dth4CaG7b8GgO7b17LzHmv52zvdGDRi3ebeXanWr+nEH24czLhzFzZd2VqIWL82eUKhc+egqnOSHKw+8EHQtXsdkf6Bz36uDxvWJ/Vfnd2b/gM3tHqL25Zk2FvK0paVs3VjgOqIeD0iNgJ3kiQd7lBWLOzGwjnbMXzU6ibqdeen40Zx9Zf2p/q53pvLH75qV475xhK69qgrd1OtQKdOwc+nzOT2p5/hL//bh9dmJ7+T71z+GrdNm86Q3dby0G2Dt9nu2JPeZsZTfVu7uW1OXZrHo6mlKZImS1om6eWCsrskzUqXN+rze0gaLmldwbpfFWwzWtJLkqolXavC/JeNKGfwa1Yi4fZk/ZpOTJq4Dyf/aAE9ejU+bO2900b+7ZnnuejRWZz8w9eZfN5erFtdxcI52/Hum90ZNXZFK7baAOrqxLknjeb0ow5jz/1Xs+seSU/+mov34qtHHsbC13tyxLh3t9jmqH98hxH7rebeyUMb2mVuJLO9VSUtJbgZGLvl/uOUiBgVEaNI0lpOKVg9v35dREwsKL8e+AYwIl222GdDKt4vlTRB0gxJM1aubD+9n9pN4oaJ+zDmhGUcNK548OrSLdi+bw0Aw/Zfw4Bd17NsQQ8WvNCLt2Zvzw8OP4SrTj6AZQt6cM0p+7dG8y21ZnVnZj/Xh9Gf/jDZV12dmPbIAA7/++Wby0Z94j1OmfAWPzl7X2o2VfzPpqLqb3IuZWlyXxHTgAYzraW9ty8BdxTbR5rbt3dETE+THN0KnNDUscs54VFSIuGImARMAtj/gC5lycze0iLg198bwc57rOWYbyxpsv7qFZ3Zrk8Nnapg+VvdWLagO/2HrWfXAz7giK++DSTD5+vGj+Q7d71U7ubnXu++G6mt6cSa1Z3p2q2Wgz75HvfeNJRBw9ax9K0eQHDo0StZuKAnALvv8wHnXjKPH561P++v7FrZxrcRGVJX9pc0o+D7pPRvvhSfBt6JiHkFZbtJ+guwCvhBRDxFMqJcVFCnpFFmOYPf88AISbuRBL1TgS+X8XitZv6M3jw3ZScG772Gn44bBcDn//VNajZ24u5LdueDlV247msjGTJyDef+eg7Vz+7Aw1cPo6pLIMFpP53Pdn1qKnwW+dVvwEYu+PfX6NQJ1Cl46rEBPP/nflz56xfpuX0NCBa8th2/+MkIAM787ut071nLhdfMBeDdJd249Jz9KnkKFZVxtnd5RBzSzEOdxpa9vqXAsIhYIWk08FtJ+zZz3+ULfhFRI+kckszpVcDkiJhTruO1pj0+vorr3ny6wXUNXb876LgVHHRc8aHxjkM3+B6/VvLGX7fn3C+M3qb8u18Z1WD9i888oNxNanfKPZMrqTNwErD5FxURG4AN6eeZkuYDe5J0roYUbN7gKHNrZb3PLyIeIcmybmYdRISoKf9tLJ8BXo2IzcNZSQOAlRFRK2l3komN1yNipaRVkg4DngVOB37e1AHyfeXWzJqlpSY8JN0BPAPsJWmRpDPTVaey7UTHEcDs9NaXe4GJEVE/WfIt4EagGpgPPNrUsf14m5ll0pJPeETEaY2U/3MDZfeR3PrSUP0ZQKYLsQ5+ZpZZR3i8zcHPzDLxy0zNLLcy3OfXZjn4mVkmEVDjl5maWR552GtmueNrfmaWW+HgZ2Z55AkPM8udCF/zM7NcErWe7TWzPPI1PzPLnY6Svc3Bz8yyieS6X3vn4GdmmXm218xyJzzhYWZ55WGvmeWSZ3vNLHciOkbwa/8DdzNrdS2Yw2OypGWSXi4o+7GkxZJmpctxBesulFQt6TVJxxaUj03LqiV9v5RzcPAzs8wiSltKcDMwtoHyayJiVLo8AiBpJElio33Tba6TVCWpCvglMA4YCZyW1i3Kw14zyyQQdS002xsR0yQNL7H68cCdaf7eBZKqgTHpuuqIeB1A0p1p3bnFduaen5llFiUuQH9JMwqWCSUe4hxJs9Nhcd+0bBdgYUGdRWlZY+VFuednZtlkm/BYHhGHZDzC9cBlyZG4DLgKGJ9xH01y8DOz7Mp4n19EvFP/WdINwMPp18XA0IKqQ9IyipQ3ysNeM8ssQiUtzSFpUMHXE4H6meAHgVMldZO0GzACeA54HhghaTdJXUkmRR5s6jiN9vwk/Zwi8T0izmvyLMyswwmgrq5l7vOTdAdwJMm1wUXAJcCRkkalh3oDOAsgIuZIuptkIqMGODsiatP9nAM8DlQBkyNiTlPHLjbsndHcEzKzDiyAFrrJOSJOa6D4piL1Lwcub6D8EeCRLMduNPhFxC2F3yX1jIi1WXZuZh1TR3i2t8lrfpI+IWku8Gr6/UBJ15W9ZWbWdmW416WtKmXC4z+BY4EVABHxInBEORtlZm1ZaZMdbf3535JudYmIhdIWJ1JbnuaYWbvQxnt1pSgl+C2U9EkgJHUBzgdeKW+zzKzNCogWmu2tpFKGvROBs0keF1kCjEq/m1luqcSl7Wqy5xcRy4F/aoW2mFl70QGGvaXM9u4u6SFJ76bv3XpA0u6t0Tgza6NyMtt7O3A3MAgYDNwD3FHORplZG1Z/k3MpSxtWSvDrGRG/joiadPkN0L3cDTOztqsFX2ZaMcWe7e2Xfnw0fS30nSQx/xQyPkZiZh1MB5jtLTbhMZMk2NWf5VkF6wK4sFyNMrO2TW28V1eKYs/27taaDTGzdqIdTGaUoqQnPCTtR5IYZPO1voi4tVyNMrO2rO1PZpSiyeAn6RKS922NJLnWNw54GnDwM8urDtDzK2W292TgGODtiPgacCCwQ1lbZWZtW12JSxtWyrB3XUTUSaqR1BtYxpbvyzezPGnBl5lWUinBb4akPsANJDPAHwDPlLVVZtamdYTZ3iaHvRHxrYj4W0T8Cvh74Ix0+GtmedVCj7eleXmXSXq5oOxKSa+meXvvTztfSBouaZ2kWenyq4JtRkt6SVK1pGu11Tv4GtJo8JN08NYL0A/onH42M/uobgbGblU2FdgvIg4A/sqW9xTPj4hR6TKxoPx64BskGd1GNLDPbRQb9l5VZF0ARze186zefKkX39r1Uy29Wyujx5f8udJNsAzGHLu6RfbTUsPeiJgmafhWZU8UfJ1OMunaeFuSVJe9I2J6+v1W4ATg0WLbFbvJ+aiirTazfAqyPN7WX1JhJshJETEpw9HGA3cVfN9N0l+AVcAPIuIpkneNLiqosygtK6qkm5zNzLZQes9veUQc0pxDSLqYJD/vbWnRUmBYRKyQNBr4raR9m7NvcPAzs2Yo92yvpH8GPgccE5G8HyYiNgAb0s8zJc0H9gQWA0MKNh+SlhVVyk3OZmZbKuPLTCWNBb4HfL4wV7ikAZKq0s+7k0xsvB4RS4FVkg5LZ3lPBx5o6jilvMlZkr4i6Ufp92GSxjTrrMysY2i5W13uILlveC9JiySdCfwC6AVM3eqWliOA2ZJmAfcCEyNiZbruW8CNQDUwnyYmO6C0Ye91JA+qHA1cCqwG7gM+XsK2ZtbBKFp0tve0BopvaqTufSSxp6F1M4D9shy7lOB3aEQcnM6wEBHvSeqa5SBm1sF08JeZ1tuUjrMDknE3bf6RZTMrp1w83gZcC9wP7CTpcpLXWf20rK0ys7atA2RvKyVv722SZpK81krACRHxStlbZmZtUwte86ukUl5mOgxYCzxUWBYRb5WzYWbWhuUh+AG/48NERt2B3YDXgGbfWW1m7Zs6wFX/Uoa9+xd+T9/o8q2ytcjMrBVkfrwtIl6QdGg5GmNm7UQehr2S/qXgayfgYGBJ2VpkZm1bXiY8SB4zqVdDcg2wwbuszSwnOnrwS29u7hUR322l9phZe9CRg5+kzhFRI+nw1myQmbVtouPP9j5Hcn1vlqQHgXuANfUrI2JKmdtmZm1Rjq75dQdWkLzVpf5+vwAc/MzyqoMHv53Smd6X+TDo1esAp25mzdYBIkCx4FcFbM+WQa9eBzh1M2uujj7sXRoRl7ZaS8ys/egAwa/YK63a/9sKzazlRTLbW8rSFEmTJS2T9HJBWT9JUyXNS3/2Tcsl6VpJ1ZJmp4/a1m9zRlp/nqQzSjmNYsHvmFJ2YGY51HLv87sZGLtV2feBJyNiBPBk+h1gHEnSohHABOB6SIIlcAlwKDAGuKQ+YBbTaPArSAxiZraF+jweTS1NiYhpwNax5njglvTzLcAJBeW3RmI60EfSIOBYYGpErIyI94CpbBtQt+G8vWaWXXmv+Q1M01ECvA0MTD/vAiwsqLcoLWusvCgHPzPLJtsr6vtLmlHwfVJETCr5UBEhlWdu2cHPzDIRmW51WR4Rh2Q8xDuSBkXE0nRYuywtXwwMLag3JC1bDBy5VfmfmjpIKQmMzMy20FLX/BrxIFA/Y3sG8EBB+enprO9hwPvp8Phx4LOS+qYTHZ9Ny4pyz8/MsmuhgaikO0h6bf0lLSKZtb0CuFvSmcCbwJfS6o8AxwHVJHmFvgbJ5Kyky4Dn03qXljJh6+BnZtm1UPCLiNMaWbXNrXYREcDZjexnMjA5y7Ed/Mwsmxy91cXMbEsOfmaWRx39ZaZmZg3ysNfM8ifbTc5tloOfmWXn4GdmeZPxCY82y8HPzDJTXfuPfg5+ZpaNr/mZWV552Gtm+eTgZ2Z55J6fmeWTg5+Z5U748TYzyyHf52dm+RXtP/o5+JlZZu75GQMGb+Rf/+st+gyogYBHfrMjv71pwOb1XzhrGRMuWcoX99uXVSs//Ofe88C1/OdD8/jpN3fl6d/1qUTTc2XZ4i5cef4w/vZuF1Bw3FdWcOLXl3PDpYOZPrU3XboGg3bdwAXXLGT7HWr5w5S+3HPdTpu3X/BKd375+F/52H7rmDe7Bz/79jA2rO/EmKNX8c3LFiNV8ORam29yLk7SZOBzwLKI2K9cx6m02hox6dLBVL/Ukx7b1fKLx/7KC9N68da87gwYvJGD/2417yzqssU2nToFZ168lJl/7lWhVudPVedgwo+WMOKAdaz9oBPnjN2Tg49YzcFHrGb8RUuo6gw3/tsg7vz5Tnz9B0s5+qT3OPqk94Ak8P1k/G58bL91AFz7/SF8+8qF7H3wWn7wld2Z8cdefPzo1ZU8vVbXEhMekvYC7ioo2h34EdAH+Abwblp+UUQ8km5zIXAmUAucFxFNJipqTDmzt91MCVnT27uVy7pQ/VJPANatqWJhdXf6D9oEwFk/XsJN/zZ4m8sjx49fztOP7MDflrvj3Vp2HFjDiAOS4NVz+zqG7rGB5Uu7MPrI1VSlv4Z9Rq9l+dIu22z7x9/25e+OTwLhinc6s3Z1FfuMXosEnzl5Jf/72A6tdh5thepKW4qJiNciYlREjAJGkyQluj9dfU39uoLANxI4FdiXJLZcJ6mquedQtuAXEdOAJjModSQDh2zkY/ut49UXevKJY99n+dtdeH1ujy3q7LjzJj457n0evmXHCrXS3l7Ylfkv92Dvg9duUf74Hf0a7MFNe7APR53wNwBWvN1l8//cAPoP3sTyt7cNmB1akEx4lLKU7hhgfkS8WaTO8cCdEbEhIhaQZHEb09zTqHjeXkkTJM2QNGMTGyrdnGbr3rOWH974Br/60WBqa8Wp5y7j1it33qbexJ8s5qbLBxGRp4tEbce6NZ247OvDmXjpYrbr9WHX5Pb/GkhV59g81K336gs96dajjuF7r2/tprZpGfL29q//+06XCY3s8lTgjoLv50iaLWlymosXYBdgYUGdRWlZs1R83BURk4BJAL3Vr11eRq3qHPzwxjf4w5S+/M+jfRi+9zp2HraR63//GgADBm3il4//lfOOG8GeB67jwuuT/7nt0K+WMcesprZWPJPDoVNrq9kEl319OEef9B6fOu79zeVP3NWP537fmyvuqt5m4uJPD/ThyBM+DIg77rxpi6Hx8iVd6L/zJnKn9L/U5RFxSLEKkroCnwcuTIuuBy5Lj3IZcBUwvlntLKLiwa/9C/7lqoUsnNedKZOSWd43Xu3BKQfsu7nGLc/O5dxxe7JqZWfOOGyfzeUXXPMWz/6+twNfK4iAqy8YxtARG/jCWe9uLn/+j72457qduHLKPLr33PIvuq4Opj3Uh6vur95ctuPAGnr2quWVmT3Z++C1/P7efhw//l3ypAw3OY8DXoiIdwDqfwJIugF4OP26GBhasN2QtKxZHPw+on3HrOEzX3yP1+d257qpSU/v//37IJ7/Q+8Kt8wKzXluO568tx+77bOOb35mLwC+duESrvvhEDZtEBeesgcAe49ew/n/sQiAl6Zvz4DBmxi068Yt9nXuvy/iZ98exsb1nTjkqFW5m+kloqVfZnoaBUNeSYMiYmn69UTg5fTzg8Dtkq4GBgMjgOeae1BFme7UlnQHcCTQH3gHuCQibiq2TW/1i0O1TaJ2a8MeXzKr0k2wDMYcu5AZL67/SBece/UZEgcdcX5JdZ966Hsziw17JW0HvAXsHhHvp2W/BkaRDHvfAM6qD4aSLiYZAtcA346IR5t7HmXr+UXEaeXat5lVVksNeyNiDbDjVmVfLVL/cuDylji2h71mlk0AzuFhZrnU/mOfg5+ZZecXG5hZLjl1pZnlj9/qYmZ5lNzk3P6jn4OfmWXnHB5mlkfu+ZlZ/vian5nlU4s/21sRDn5mlp2HvWaWO05abma55Z6fmeVS+499Dn5mlp3q2v+418HPzLIJfJOzmeWPCN/kbGY51QGCX8Xz9ppZO9RCScslvSHpJUmzJM1Iy/pJmippXvqzb1ouSddKqk5z+h78UU7Bwc/Msqm/5lfKUpqjImJUQaKj7wNPRsQI4Mn0OyQpLkekywSS/L7N5uBnZpmprq6kpZmOB25JP98CnFBQfmskpgN9JA1q7kEc/MwsoxKHvMmwt7+kGQXLhG13xhOSZhasG1iQt/dtYGD6eRdgYcG2i9KyZvGEh5llE2SZ8FheLG8v8KmIWCxpJ2CqpFe3OFRESOXJGOKen5ll10LX/CJicfpzGXA/MAZ4p344m/5cllZfDAwt2HxIWtYsDn5mlpkiSlqK7kPaTlKv+s/AZ4GXgQeBM9JqZwAPpJ8fBE5PZ30PA94vGB5n5mGvmWXXMvf5DQTulwRJLLo9Ih6T9Dxwt6QzgTeBL6X1HwGOA6qBtcDXPsrBHfzMLJsIqP3oz7dFxOvAgQ2UrwCOaaA8gLM/8oFTDn5mll0HeMLDwc/MsnPwM7PcCcA5PMwsfwKi/b/TysHPzLIJWmTCo9Ic/MwsO1/zM7NccvAzs/wp7V19bZ2Dn5llE4ATGJlZLrnnZ2b50zKPt1Wag5+ZZRMQvs/PzHLJT3iYWS75mp+Z5U6EZ3vNLKfc8zOz/AmitrbSjfjIHPzMLJsO8korJzAys+yirrSlCElDJf1R0lxJcySdn5b/WNJiSbPS5biCbS6UVC3pNUnHfpRTcM/PzDIJIFqm51cDXBARL6RZ3GZKmpquuyYiflZYWdJI4FRgX2Aw8HtJe0ZEs8bg7vmZWTYRLdLzi4ilEfFC+nk18AqwS5FNjgfujIgNEbGAJIvbmOaehoOfmWUWtbUlLUB/STMKlgkN7U/ScOAg4Nm06BxJsyVNltQ3LdsFWFiw2SKKB8uiFG1oylrSuyR5Ojua/sDySjfCMumov7NdI2LAR9mBpMdI/n1KsTwixjaxv+2BPwOXR8QUSQNJ/u0DuAwYFBHjJf0CmB4Rv0m3uwl4NCLubc55tKlrfh/1l9JWSZoREYdUuh1WOv/OGtdUMMtCUhfgPuC2iJiS7v+dgvU3AA+nXxcDQws2H5KWNYuHvWZWEZIE3AS8EhFXF5QPKqh2IvBy+vlB4FRJ3STtBowAnmvu8dtUz8/McuVw4KvAS5JmpWUXAadJGkUy7H0DOAsgIuZIuhuYSzJTfHZzZ3qhjV3z66gkTYiISZVuh5XOv7OOz8HPzHLJ1/zMLJcc/MpI0tj0MZxqSd+vdHusael9Zcskvdx0bWvPHPzKRFIV8EtgHDCS5CLuyMq2ykpwM9Bit3JY2+XgVz5jgOqIeD0iNgJ3kjyeY21YREwDVla6HVZ+Dn7l06KP4phZy3LwM7NccvArnxZ9FMfMWpaDX/k8D4yQtJukriTvIXuwwm0ys5SDX5lERA1wDvA4yXvK7o6IOZVtlTVF0h3AM8BekhZJOrPSbbLy8BMeZpZL7vmZWS45+JlZLjn4mVkuOfiZWS45+JlZLjn4tSOSatMkzi9LukdSz4+wr5slnZx+vrHYSxckHSnpk804xhuStkl001j5VnU+yHisH0v6btY2Wn45+LUv6yJiVETsB2wEJhaulNSstAQR8fWImFukypFA5uBn1pY5+LVfTwF7pL2ypyQ9CMyVVCXpSknPp3lPz4IkWYykX6TvF/w9sFP9jiT9SdIh6eexkl6Q9KKkJ9N8qhOB76S9zk9LGiDpvvQYz0s6PN12R0lPSJoj6UZATZ2EpN9KmpluM2Grddek5U9KGpCWfUzSY+k2T0nauyX+MS1/nMCoHUp7eOOAx9Kig4H9ImJBGkDej4iPS+oG/I+kJ0gSQu9F8m7BgSRJYCZvtd8BwA3AEem++kXESkm/Aj6IiJ+l9W4HromIpyUNI3mKZR/gEuDpiLhU0j8ApTwdMT49Rg/geUn3RcQKYDtgRkR8R9KP0n2fA/fyIIgAAAG6SURBVEwCJkbEPEmHAtcBRzfjn9FyzsGvfelRkOXqKZK0f58EnouIBWn5Z4ED6q/nATuQpPg7ArgjzXa1RNIfGtj/YcC0+n1FRGPvtfsMMDLJPAhA7zTx9BHASem2v5P0XgnndJ6kE9PPQ9O2rgDqgLvS8t8AU9JjfBK4p+DY3Uo4htk2HPzal3URMaqwIA0CawqLgHMj4vGt6h3Xgu3oBBwWEesbaEvJJB1JEkg/ERFrJf0J6N5I9UiP+7et/w3MmsPX/Dqex4FvSuoCIGlPSdsB04BT0muCg4CjGth2OnBEmhAaSf3S8tVAr4J6TwDn1n9Jc6ySHuPLadk4oG8Tbd0BeC8NfHuT9DzrdQLqe69fJhlOrwIWSPpiegxJOrCJY5g1yMGv47mR5HreC2kSnv8m6eHfD8xL191K8uaSLUTEu8AEkiHmi3w47HwIOLF+wgM4DzgknVCZy4ezzj8hCZ5zSIa/bzXR1seAzpJeAa4gCb711gBj0nM4Grg0Lf8n4My0fXNwagBrJr/VxcxyyT0/M8slBz8zyyUHPzPLJQc/M8slBz8zyyUHPzPLJQc/M8slBz8zy6X/DyFxkq/sQ9R5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "### confusion matrix\n",
    "plot_confusion_matrix(estimator=lr, X=X_valid_tfidf, y_true=y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg_reviews</th>\n",
       "      <th>pos_reivews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>precision</th>\n",
       "      <td>0.898249</td>\n",
       "      <td>0.872406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>0.866452</td>\n",
       "      <td>0.902944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score</th>\n",
       "      <td>0.882064</td>\n",
       "      <td>0.887412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count_of_reviews</th>\n",
       "      <td>2486.000000</td>\n",
       "      <td>2514.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  neg_reviews  pos_reivews\n",
       "precision            0.898249     0.872406\n",
       "recall               0.866452     0.902944\n",
       "f1_score             0.882064     0.887412\n",
       "count_of_reviews  2486.000000  2514.000000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get model predictions \n",
    "y_pred = lr.predict(X_valid_tfidf)\n",
    "metrics = precision_recall_fscore_support(y_true=y_valid, y_pred=y_pred)\n",
    "imdb.display_metrics(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-30-fb91d9c339fa>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpermutation_importance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mX_train_tfidf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_repeats\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     70\u001b[0m                           FutureWarning)\n\u001b[1;32m     71\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/sklearn/inspection/_permutation_importance.py\u001b[0m in \u001b[0;36mpermutation_importance\u001b[0;34m(estimator, X, y, scoring, n_repeats, n_jobs, random_state)\u001b[0m\n\u001b[1;32m    133\u001b[0m     \u001b[0mbaseline_score\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mestimator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    134\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 135\u001b[0;31m     scores = Parallel(n_jobs=n_jobs)(delayed(_calculate_permutation_scores)(\n\u001b[0m\u001b[1;32m    136\u001b[0m         \u001b[0mestimator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcol_idx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrandom_seed\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_repeats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscorer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m     ) for col_idx in range(X.shape[1]))\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, iterable)\u001b[0m\n\u001b[1;32m   1049\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_iterating\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_original_iterator\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1050\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispatch_one_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m                 \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36mdispatch_one_batch\u001b[0;34m(self, iterator)\u001b[0m\n\u001b[1;32m    864\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    865\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 866\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_dispatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtasks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    867\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    868\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m_dispatch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    782\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    783\u001b[0m             \u001b[0mjob_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 784\u001b[0;31m             \u001b[0mjob\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    785\u001b[0m             \u001b[0;31m# A job can complete so quickly than its callback is\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    786\u001b[0m             \u001b[0;31m# called before we get here, causing self._jobs to\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36mapply_async\u001b[0;34m(self, func, callback)\u001b[0m\n\u001b[1;32m    206\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mapply_async\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    207\u001b[0m         \u001b[0;34m\"\"\"Schedule a func to be run\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 208\u001b[0;31m         \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mImmediateResult\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    209\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/_parallel_backends.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    570\u001b[0m         \u001b[0;31m# Don't delay the application, to avoid keeping the input\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    571\u001b[0m         \u001b[0;31m# arguments in memory\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 572\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    573\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    574\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    260\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 262\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    263\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/joblib/parallel.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m    260\u001b[0m         \u001b[0;31m# change the default number of processes to -1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    261\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mparallel_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_backend\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_n_jobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 262\u001b[0;31m             return [func(*args, **kwargs)\n\u001b[0m\u001b[1;32m    263\u001b[0m                     for func, args, kwargs in self.items]\n\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.pyenv/versions/3.8.6/envs/imdb/lib/python3.8/site-packages/sklearn/inspection/_permutation_importance.py\u001b[0m in \u001b[0;36m_calculate_permutation_scores\u001b[0;34m(estimator, X, y, col_idx, random_state, n_repeats, scorer)\u001b[0m\n\u001b[1;32m     22\u001b[0m     \u001b[0;31m# (memmap). X.copy() on the other hand is always guaranteed to return a\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0;31m# writable data-structure whose columns can be shuffled inplace.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m     \u001b[0mX_permuted\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m     \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_repeats\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0mshuffling_idx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "result = permutation_importance(estimator=lr, X=X_train_tfidf.toarray(), y=y_train, n_repeats=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.05376199,  0.00242971,  0.02662237, ...,  0.04376444,\n",
       "       -0.03134004, -0.03134004])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.coef_[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73448"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(lr.coef_[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['10',\n",
       " '100',\n",
       " '1000',\n",
       " '1000000',\n",
       " '1000000000',\n",
       " '11',\n",
       " '12',\n",
       " '13',\n",
       " '15',\n",
       " '16',\n",
       " '17',\n",
       " '18',\n",
       " '19',\n",
       " '20',\n",
       " '70',\n",
       " 'aa',\n",
       " 'aaa',\n",
       " 'aaaaaaaaaaaahhhhhhhhhhhhhh',\n",
       " 'aaaaaaaargh',\n",
       " 'aaaaaaah',\n",
       " 'aaaaaaahhhhhhggg',\n",
       " 'aaaaagh',\n",
       " 'aaaaah',\n",
       " 'aaaaahhhh',\n",
       " 'aaaaargh',\n",
       " 'aaaaarrrrrrgggggghhhhhh',\n",
       " 'aaaaaw',\n",
       " 'aaaahhhhhh',\n",
       " 'aaaahhhhhhh',\n",
       " 'aaaand',\n",
       " 'aaaawwwwww',\n",
       " 'aaaggghhhhhhh',\n",
       " 'aaah',\n",
       " 'aaahhhhhhh',\n",
       " 'aaam',\n",
       " 'aaargh',\n",
       " 'aaawwwwnnn',\n",
       " 'aachen',\n",
       " 'aada',\n",
       " 'aadha',\n",
       " 'aag',\n",
       " 'aage',\n",
       " 'aagh',\n",
       " 'aaghh',\n",
       " 'aah',\n",
       " 'aahed',\n",
       " 'aahhhh',\n",
       " 'aahing',\n",
       " 'aaila',\n",
       " 'aailiyah']"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.get_feature_names()[:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'grow': 27165,\n",
       " 'watch': 70646,\n",
       " 'love': 38426,\n",
       " 'thunderbirds': 65017,\n",
       " 'mate': 40327,\n",
       " 'school': 56764,\n",
       " 'play': 49647,\n",
       " 'lunch': 38679,\n",
       " 'want': 70492,\n",
       " 'virgil': 69874,\n",
       " 'scott': 56943,\n",
       " 'alan': 1386,\n",
       " 'count': 14046,\n",
       " 'art': 3427,\n",
       " 'form': 23760,\n",
       " 'take': 63514,\n",
       " 'child': 11321,\n",
       " 'movie': 43337,\n",
       " 'hope': 30042,\n",
       " 'glimpse': 26041,\n",
       " 'bitterly': 6703,\n",
       " 'disappoint': 17398,\n",
       " 'high': 29313,\n",
       " 'point': 49889,\n",
       " 'snappy': 59725,\n",
       " 'theme': 64627,\n",
       " 'tune': 66743,\n",
       " 'compare': 12879,\n",
       " 'original': 46722,\n",
       " 'score': 56918,\n",
       " 'thankfully': 64513,\n",
       " 'early': 19268,\n",
       " 'saturday': 56307,\n",
       " 'morning': 43087,\n",
       " 'television': 64151,\n",
       " 'channel': 10809,\n",
       " 'rerun': 53843,\n",
       " 'series': 57660,\n",
       " 'gerry': 25525,\n",
       " 'anderson': 2272,\n",
       " 'wife': 71432,\n",
       " 'create': 14343,\n",
       " 'jonatha': 33925,\n",
       " 'frakes': 23958,\n",
       " 'hand': 28026,\n",
       " 'director': 17352,\n",
       " 'chair': 10711,\n",
       " 'version': 69502,\n",
       " 'completely': 12933,\n",
       " 'hopeless': 30048,\n",
       " 'waste': 70628,\n",
       " 'film': 22800,\n",
       " 'utter': 68913,\n",
       " 'rubbish': 55310,\n",
       " 'cgi': 10667,\n",
       " 'remake': 53536,\n",
       " 'acceptable': 348,\n",
       " 'replace': 53746,\n",
       " 'marionette': 39979,\n",
       " 'homo': 29905,\n",
       " 'sapiens': 56131,\n",
       " 'subsp': 62272,\n",
       " 'huge': 30430,\n",
       " 'error': 20779,\n",
       " 'judgment': 34101,\n",
       " 'dvd': 19132,\n",
       " 'player': 49656,\n",
       " 'sit': 59101,\n",
       " 'coke': 12520,\n",
       " 'chip': 11386,\n",
       " 'expectation': 21449,\n",
       " 'contain': 13452,\n",
       " 'strong': 61997,\n",
       " 'awsome': 4323,\n",
       " 'animation': 2456,\n",
       " 'good': 26385,\n",
       " 'flow': 23419,\n",
       " 'story': 61749,\n",
       " 'excellent': 21280,\n",
       " 'voice': 70070,\n",
       " 'cast': 10201,\n",
       " 'funny': 24566,\n",
       " 'comedy': 12721,\n",
       " 'kick': 35096,\n",
       " 'ass': 3650,\n",
       " 'soundtrack': 60353,\n",
       " 'disappointment': 17401,\n",
       " 'find': 22914,\n",
       " 'atlantis': 3857,\n",
       " 'milo': 41980,\n",
       " 'return': 54086,\n",
       " 'read': 52834,\n",
       " 'review': 54153,\n",
       " 'let': 37327,\n",
       " 'follow': 23563,\n",
       " 'paragraph': 47767,\n",
       " 'direct': 17341,\n",
       " 'see': 57273,\n",
       " 'enjoy': 20381,\n",
       " 'primarily': 50922,\n",
       " 'scene': 56563,\n",
       " 'appear': 2913,\n",
       " 'shock': 58411,\n",
       " 'pick': 49206,\n",
       " 'display': 17665,\n",
       " 'case': 10142,\n",
       " 'local': 38019,\n",
       " 'videoshop': 69663,\n",
       " 'music': 43761,\n",
       " 'feel': 22385,\n",
       " 'bad': 4574,\n",
       " 'imitation': 31164,\n",
       " 'fit': 23084,\n",
       " 'exception': 21287,\n",
       " 'character': 10855,\n",
       " 'like': 37611,\n",
       " 'sweet': 63094,\n",
       " 'actual': 567,\n",
       " 'drawing': 18562,\n",
       " 'not': 45458,\n",
       " 'particular': 47952,\n",
       " 'sad': 55682,\n",
       " 'sight': 58795,\n",
       " 'storyline': 61762,\n",
       " 'pretty': 50863,\n",
       " 'weak': 70770,\n",
       " 'episode': 20616,\n",
       " 'schooby': 56763,\n",
       " 'doo': 18189,\n",
       " 'single': 59007,\n",
       " 'adventurous': 826,\n",
       " 'get': 25548,\n",
       " 'time': 65169,\n",
       " 'misunderstand': 42372,\n",
       " 'laugh': 36755,\n",
       " 'snigger': 59770,\n",
       " 'audience': 3996,\n",
       " 'especially': 20869,\n",
       " 'care': 9872,\n",
       " 'similar': 58909,\n",
       " 'sequel': 57594,\n",
       " 'fast': 22175,\n",
       " 'stand': 61147,\n",
       " 'product': 51077,\n",
       " 'suspect': 62929,\n",
       " 'kid': 35108,\n",
       " 'know': 35565,\n",
       " 'well': 70940,\n",
       " 'cartoon': 10101,\n",
       " 'breakfast': 8105,\n",
       " 'tv': 66879,\n",
       " 'people': 48536,\n",
       " 'past': 48035,\n",
       " 'need': 44512,\n",
       " 'try': 66618,\n",
       " 'define': 16021,\n",
       " 'woodstock': 71970,\n",
       " 'civil': 11894,\n",
       " 'war': 70500,\n",
       " 'apollo': 2857,\n",
       " 'moon': 42930,\n",
       " 'land': 36463,\n",
       " 'titanic': 65316,\n",
       " 'sink': 59023,\n",
       " 'realistic': 52869,\n",
       " 'flick': 23300,\n",
       " 'formulaic': 23783,\n",
       " 'soap': 59862,\n",
       " 'opus': 46604,\n",
       " 'populate': 50201,\n",
       " 'entirely': 20506,\n",
       " 'low': 38460,\n",
       " 'life': 37546,\n",
       " 'trash': 66156,\n",
       " 'young': 72801,\n",
       " 'allow': 1665,\n",
       " 'fail': 21827,\n",
       " 'grade': 26661,\n",
       " 'composition': 12970,\n",
       " 'old': 46264,\n",
       " 'meanie': 40926,\n",
       " 'prove': 51350,\n",
       " 'nuttin': 45725,\n",
       " 'topic': 65666,\n",
       " 'money': 42721,\n",
       " 'yes': 72653,\n",
       " 'thing': 64788,\n",
       " 'little': 37914,\n",
       " 'insight': 32192,\n",
       " 'underclass': 67601,\n",
       " 'think': 64796,\n",
       " 'slut': 59557,\n",
       " 'bar': 5005,\n",
       " 'look': 38231,\n",
       " 'diane': 17002,\n",
       " 'lane': 36499,\n",
       " 'run': 55430,\n",
       " 'way': 70740,\n",
       " 'abuse': 305,\n",
       " 'parent': 47831,\n",
       " 'worthless': 72128,\n",
       " 'raise': 52395,\n",
       " 'simply': 58951,\n",
       " 'stick': 61546,\n",
       " 'moonlanding': 42937,\n",
       " 'ipso': 32720,\n",
       " 'facto': 21790,\n",
       " 'mean': 40919,\n",
       " 'portray': 50298,\n",
       " 'great': 26857,\n",
       " 'interest': 32405,\n",
       " 'biblical': 6401,\n",
       " 'bore': 7624,\n",
       " 'death': 15708,\n",
       " 'minute': 42144,\n",
       " 'long': 38183,\n",
       " 'act': 525,\n",
       " 'joke': 33896,\n",
       " 'script': 57030,\n",
       " 'horrible': 30123,\n",
       " 'mix': 42417,\n",
       " 'abraham': 224,\n",
       " 'noah': 45163,\n",
       " 'value': 69081,\n",
       " 'sanity': 56082,\n",
       " 'stay': 61326,\n",
       " 'away': 4290,\n",
       " 'horror': 30140,\n",
       " 'be': 5471,\n",
       " 'die': 17092,\n",
       " 'hard': 28185,\n",
       " 'dad': 15145,\n",
       " 'army': 3324,\n",
       " 'fan': 21962,\n",
       " 'change': 10795,\n",
       " 'tape': 63728,\n",
       " 'audiobooks': 3999,\n",
       " 'listen': 37850,\n",
       " 'brand': 8000,\n",
       " 'new': 44753,\n",
       " 'certain': 10632,\n",
       " 'man': 39570,\n",
       " 'hour': 30265,\n",
       " 'enemy': 20307,\n",
       " 'gate': 25102,\n",
       " 'battle': 5377,\n",
       " 'numerous': 45670,\n",
       " 'different': 17124,\n",
       " 'edge': 19462,\n",
       " 'introduction': 32577,\n",
       " 'general': 25322,\n",
       " 'instead': 32249,\n",
       " 'captain': 9793,\n",
       " 'square': 60965,\n",
       " 'brilliant': 8300,\n",
       " 'cash': 10151,\n",
       " 'cheque': 11163,\n",
       " 'rarely': 52630,\n",
       " 'year': 72589,\n",
       " 'equipment': 20669,\n",
       " 'uniform': 68020,\n",
       " 'start': 61264,\n",
       " 'train': 65994,\n",
       " 'sunday': 62567,\n",
       " 'afternoon': 999,\n",
       " 'draw': 18558,\n",
       " 'back': 4507,\n",
       " 'germans': 25507,\n",
       " 'bogus': 7306,\n",
       " 'dodgy': 17978,\n",
       " 'accent': 342,\n",
       " 'come': 12713,\n",
       " 'pronounce': 51206,\n",
       " 'letter': 37348,\n",
       " 'liz': 37953,\n",
       " 'frazer': 24093,\n",
       " 'familiar': 21943,\n",
       " 'janet': 33290,\n",
       " 'davis': 15590,\n",
       " 'carry': 10076,\n",
       " 'ons': 46426,\n",
       " 'correctly': 13870,\n",
       " 'choice': 11462,\n",
       " 'terrible': 64360,\n",
       " 'say': 56406,\n",
       " 'cameo': 9494,\n",
       " 'appearance': 2914,\n",
       " 'mcnealy': 40857,\n",
       " 'give': 25923,\n",
       " 'award': 4285,\n",
       " 'murder': 43694,\n",
       " 'programmer': 51139,\n",
       " 'wall': 70414,\n",
       " 'sun': 62556,\n",
       " 'logo': 38101,\n",
       " 'ceo': 10596,\n",
       " 'microsystem': 41748,\n",
       " 'company': 12872,\n",
       " 'practically': 50519,\n",
       " 'hatred': 28444,\n",
       " 'microsoft': 41747,\n",
       " 'instrumental': 32275,\n",
       " 'file': 22778,\n",
       " 'antitrust': 2695,\n",
       " 'complaint': 12925,\n",
       " 'silly': 58867,\n",
       " 'add': 628,\n",
       " 'fuel': 24443,\n",
       " 'public': 51548,\n",
       " 'record': 53089,\n",
       " 'involvement': 32676,\n",
       " 'clearly': 12061,\n",
       " 'maker': 39374,\n",
       " 'mystery': 43909,\n",
       " 'finally': 22902,\n",
       " 'night': 44978,\n",
       " 'disturb': 17787,\n",
       " 'unbelievably': 67381,\n",
       " 'bloody': 7057,\n",
       " 'unforgettable': 67896,\n",
       " 'total': 65789,\n",
       " 'assault': 3665,\n",
       " 'sense': 57507,\n",
       " 'mind': 42025,\n",
       " 'lynch': 38818,\n",
       " 'specifically': 60557,\n",
       " 'eraserhead': 20688,\n",
       " 'buttgereit': 9102,\n",
       " 'beget': 5740,\n",
       " 'guy': 27553,\n",
       " 'pregnant': 50698,\n",
       " 'sister': 59093,\n",
       " 'belief': 5849,\n",
       " 'happen': 28151,\n",
       " 'brain': 7955,\n",
       " 'leave': 36977,\n",
       " 'right': 54430,\n",
       " 'side': 58740,\n",
       " 'main': 39305,\n",
       " 'piece': 49256,\n",
       " 'fine': 22919,\n",
       " 'line': 37733,\n",
       " 'fantasy': 22030,\n",
       " 'reality': 52872,\n",
       " 'dream': 18580,\n",
       " 'fantasize': 22021,\n",
       " 'overtake': 47271,\n",
       " 'reason': 52919,\n",
       " 'logical': 38093,\n",
       " 'unbelievable': 67379,\n",
       " 'kind': 35250,\n",
       " 'electronic': 19831,\n",
       " 'absolutely': 266,\n",
       " 'perfect': 48601,\n",
       " 'totally': 65795,\n",
       " 'disgust': 17542,\n",
       " 'extreme': 21649,\n",
       " 'include': 31540,\n",
       " 'core': 13771,\n",
       " 'sex': 57778,\n",
       " 'view': 69704,\n",
       " 'work': 72051,\n",
       " 'amaze': 1923,\n",
       " 'available': 4187,\n",
       " 'subconscious': 62189,\n",
       " 'cruelty': 14682,\n",
       " 'real': 52860,\n",
       " 'event': 21117,\n",
       " 'viewer': 69707,\n",
       " 'easily': 19305,\n",
       " 'forget': 23736,\n",
       " 'note': 45465,\n",
       " 'gorehounds': 26489,\n",
       " 'careful': 9881,\n",
       " 'buy': 9122,\n",
       " 'print': 50956,\n",
       " 'fog': 23520,\n",
       " 'graphic': 26764,\n",
       " 'gore': 26485,\n",
       " 'copy': 13720,\n",
       " 'japanese': 33332,\n",
       " 'release': 53481,\n",
       " 'catch': 10279,\n",
       " 'azn': 4391,\n",
       " 'cable': 9227,\n",
       " 'sound': 60337,\n",
       " 'green': 26887,\n",
       " 'card': 9846,\n",
       " 'dislike': 17588,\n",
       " 'asian': 3604,\n",
       " 'contrary': 13544,\n",
       " 'incredible': 31624,\n",
       " 'korean': 35740,\n",
       " 'john': 33868,\n",
       " 'woo': 71936,\n",
       " 'hong': 29945,\n",
       " 'kong': 35688,\n",
       " 'adverse': 831,\n",
       " 'light': 37577,\n",
       " 'hearted': 28684,\n",
       " 'tampopo': 63647,\n",
       " 'chung': 11664,\n",
       " 'king': 35276,\n",
       " 'express': 21559,\n",
       " 'favourite': 22279,\n",
       " 'slit': 59466,\n",
       " 'wrist': 72210,\n",
       " 'drink': 18660,\n",
       " 'blood': 7025,\n",
       " 'laborious': 36197,\n",
       " 'badly': 4614,\n",
       " 'steven': 61526,\n",
       " 'okazaki': 46227,\n",
       " 'spike': 60676,\n",
       " 'water': 70670,\n",
       " 'quaalude': 51919,\n",
       " 'personality': 48751,\n",
       " 'oppose': 46568,\n",
       " 'mumble': 43612,\n",
       " 'performance': 48621,\n",
       " 'force': 23662,\n",
       " 'incredibly': 31626,\n",
       " 'fake': 21867,\n",
       " 'honestly': 29936,\n",
       " 'person': 48744,\n",
       " 'genuine': 25415,\n",
       " 'brenda': 8173,\n",
       " 'aoki': 2782,\n",
       " 'promote': 51193,\n",
       " 'surprise': 62869,\n",
       " 'cbs': 10445,\n",
       " 'news': 44788,\n",
       " 'call': 9411,\n",
       " 'refresh': 53284,\n",
       " 'breath': 8122,\n",
       " 'end': 20261,\n",
       " 'predictable': 50645,\n",
       " 'previous': 50882,\n",
       " 'reviewer': 54154,\n",
       " 'idiot': 30930,\n",
       " 'plot': 49752,\n",
       " 'wooden': 71946,\n",
       " 'actually': 571,\n",
       " 'bensonhurst': 6028,\n",
       " 'walk': 70402,\n",
       " 'half': 27866,\n",
       " 'autumn': 4176,\n",
       " 'tale': 63551,\n",
       " 'mabel': 38890,\n",
       " 'cheung': 11215,\n",
       " 'claim': 11916,\n",
       " 'employ': 20174,\n",
       " 'rock': 54752,\n",
       " 'medium': 41046,\n",
       " 'personal': 48749,\n",
       " 'attitude': 3941,\n",
       " 'desire': 16653,\n",
       " 'consequential': 13337,\n",
       " 'frustration': 24405,\n",
       " 'significantly': 58811,\n",
       " 'crucial': 14659,\n",
       " 'role': 54878,\n",
       " 'vehicle': 69311,\n",
       " 'convey': 13603,\n",
       " 'profound': 51120,\n",
       " 'sentiment': 57547,\n",
       " 'inappropriate': 31456,\n",
       " 'utilize': 68904,\n",
       " 'underground': 67629,\n",
       " 'musician': 43767,\n",
       " 'intense': 32350,\n",
       " 'ordinary': 46656,\n",
       " 'focus': 23512,\n",
       " 'depiction': 16479,\n",
       " 'subtle': 62301,\n",
       " 'affection': 907,\n",
       " 'ultimate': 67198,\n",
       " 'vanity': 69152,\n",
       " 'mere': 41406,\n",
       " 'father': 22211,\n",
       " 'son': 60139,\n",
       " 'lover': 38446,\n",
       " 'friend': 24268,\n",
       " 'delicately': 16176,\n",
       " 'touchingly': 65820,\n",
       " 'deliver': 16201,\n",
       " 'attempt': 3919,\n",
       " 'beautify': 5582,\n",
       " 'artist': 3476,\n",
       " 'reproduce': 53800,\n",
       " 'true': 66568,\n",
       " 'screen': 56997,\n",
       " 'make': 39368,\n",
       " 'effort': 19602,\n",
       " 'project': 51152,\n",
       " 'gather': 25110,\n",
       " 'information': 31953,\n",
       " 'beijing': 5791,\n",
       " 'probably': 51022,\n",
       " 'far': 22046,\n",
       " 'innate': 32093,\n",
       " 'dispiritedness': 17659,\n",
       " 'reticence': 54043,\n",
       " 'blue': 7105,\n",
       " 'mood': 42917,\n",
       " 'perfectly': 48606,\n",
       " 'super': 62618,\n",
       " 'ex': 21236,\n",
       " 'girlfriend': 25869,\n",
       " 'turn': 66810,\n",
       " 'pleasant': 49692,\n",
       " 'expect': 21444,\n",
       " 'stupid': 62115,\n",
       " 'wonderful': 71914,\n",
       " 'fun': 24521,\n",
       " 'cheesy': 11116,\n",
       " 'uma': 67222,\n",
       " 'anna': 2507,\n",
       " 'cool': 13642,\n",
       " 'steve': 61524,\n",
       " 'addition': 640,\n",
       " 'typical': 67008,\n",
       " 'average': 4220,\n",
       " 'librarian': 37489,\n",
       " 'woman': 71893,\n",
       " 'matt': 40403,\n",
       " 'male': 39441,\n",
       " 'ask': 3616,\n",
       " 'girl': 25866,\n",
       " 'date': 15531,\n",
       " 'hero': 29118,\n",
       " 'world': 72084,\n",
       " 'jealous': 33471,\n",
       " 'crazy': 14330,\n",
       " 'decide': 15829,\n",
       " 'idea': 30890,\n",
       " 'spend': 60619,\n",
       " 'apart': 2804,\n",
       " 'jenny': 33573,\n",
       " 'will': 71511,\n",
       " 'fall': 21897,\n",
       " 'hannah': 28108,\n",
       " 'live': 37927,\n",
       " 'hell': 28888,\n",
       " 'face': 21749,\n",
       " 'choose': 11492,\n",
       " 'break': 8097,\n",
       " 'effect': 19580,\n",
       " 'corny': 13836,\n",
       " 'seriously': 57665,\n",
       " 'quickly': 52077,\n",
       " 'step': 61453,\n",
       " 'prime': 50926,\n",
       " 'cute': 14992,\n",
       " 'romantic': 54932,\n",
       " 'believe': 5857,\n",
       " 'laural': 36793,\n",
       " 'hardy': 28221,\n",
       " 'lighten': 37579,\n",
       " 'riot': 54517,\n",
       " 'comic': 12745,\n",
       " 'genius': 25371,\n",
       " 'today': 65420,\n",
       " 'ago': 1091,\n",
       " 'filthy': 22886,\n",
       " 'word': 72033,\n",
       " 'mouth': 43321,\n",
       " 'able': 175,\n",
       " 'stitch': 61633,\n",
       " 'sophisticate': 60223,\n",
       " 'stretch': 61926,\n",
       " 'whoopee': 71332,\n",
       " 'cushion': 14961,\n",
       " 'grin': 27034,\n",
       " 'stuff': 62077,\n",
       " 'simple': 58940,\n",
       " 'guess': 27310,\n",
       " 'qualify': 51953,\n",
       " 'team': 63968,\n",
       " 'compute': 12999,\n",
       " 'fortunate': 23839,\n",
       " 'gong': 26367,\n",
       " 'amuse': 2140,\n",
       " 'help': 28938,\n",
       " 'obvious': 45943,\n",
       " 'snapshot': 59726,\n",
       " 'era': 20679,\n",
       " 'late': 36690,\n",
       " 'lot': 38343,\n",
       " 'chuck': 11636,\n",
       " 'host': 30212,\n",
       " 'air': 1227,\n",
       " 'bizarre': 6718,\n",
       " 'sort': 60282,\n",
       " 'weirdness': 70907,\n",
       " 'list': 37849,\n",
       " 'clip': 12177,\n",
       " 'tire': 65288,\n",
       " 'length': 37205,\n",
       " 'outside': 47042,\n",
       " 'segment': 57317,\n",
       " 'basically': 5259,\n",
       " 'halfway': 27877,\n",
       " 'decent': 15816,\n",
       " 'actor': 557,\n",
       " 'rest': 53974,\n",
       " 'homicide': 29895,\n",
       " 'street': 61892,\n",
       " 'hear': 28661,\n",
       " 'reunion': 54096,\n",
       " 'powerful': 50480,\n",
       " 'big': 6439,\n",
       " 'pleasure': 49705,\n",
       " 'andre': 2301,\n",
       " 'braugher': 8052,\n",
       " 'relationship': 53468,\n",
       " 'pembleton': 48444,\n",
       " 'bayliss': 5430,\n",
       " 'greg': 26935,\n",
       " 'bryan': 8600,\n",
       " 'daly': 15259,\n",
       " 'craze': 14327,\n",
       " 'statement': 61285,\n",
       " 'terrorist': 64382,\n",
       " 'commentary': 12784,\n",
       " 'bunch': 8857,\n",
       " 'uber': 67060,\n",
       " 'reactionary': 52829,\n",
       " 'ascribe': 3544,\n",
       " 'position': 50328,\n",
       " 'fundamentalist': 24535,\n",
       " 'moslems': 43191,\n",
       " 'planet': 49580,\n",
       " 'moslem': 43190,\n",
       " 'present': 50788,\n",
       " 'theology': 64655,\n",
       " 'islam': 32912,\n",
       " 'maybe': 40523,\n",
       " 'involve': 32675,\n",
       " 'interview': 32520,\n",
       " 'fred': 24115,\n",
       " 'phelps': 48987,\n",
       " 'congregation': 13232,\n",
       " 'westboro': 71021,\n",
       " 'baptist': 5000,\n",
       " 'church': 11679,\n",
       " 'militant': 41918,\n",
       " 'atheist': 3816,\n",
       " 'scholar': 56754,\n",
       " 'christianity': 11572,\n",
       " 'ultimately': 67199,\n",
       " 'suffer': 62405,\n",
       " 'poor': 50148,\n",
       " 'production': 51078,\n",
       " 'lack': 36229,\n",
       " 'attention': 3927,\n",
       " 'basic': 5258,\n",
       " 'standard': 61150,\n",
       " 'journalism': 34011,\n",
       " 'radio': 52270,\n",
       " 'rush': 55467,\n",
       " 'limbaugh': 37668,\n",
       " 'couple': 14088,\n",
       " 'day': 15612,\n",
       " 'free': 24141,\n",
       " 'message': 41502,\n",
       " 'level': 37375,\n",
       " 'intellectual': 32326,\n",
       " 'analysis': 2184,\n",
       " 'bring': 8313,\n",
       " 'elvis': 20006,\n",
       " 'presley': 50805,\n",
       " 'modern': 42528,\n",
       " 'despite': 16697,\n",
       " 'sexy': 57812,\n",
       " 'shower': 58536,\n",
       " 'pseudo': 51440,\n",
       " 'playboy': 49654,\n",
       " 'magazine': 39167,\n",
       " 'subplot': 62247,\n",
       " 'surround': 62891,\n",
       " 'coy': 14186,\n",
       " 'wink': 71643,\n",
       " 'cliché': 12125,\n",
       " 'beach': 5474,\n",
       " 'proceed': 51046,\n",
       " 'dick': 17045,\n",
       " 'sargent': 56188,\n",
       " 'grueling': 27192,\n",
       " 'sidebar': 58742,\n",
       " 'don': 18127,\n",
       " 'porter': 50274,\n",
       " 'rudy': 55357,\n",
       " 'vallee': 69055,\n",
       " 'boss': 7714,\n",
       " 'moonlight': 42938,\n",
       " 'song': 60154,\n",
       " 'loose': 38275,\n",
       " 'hipper': 29464,\n",
       " 'update': 68703,\n",
       " 'fatal': 22191,\n",
       " 'attraction': 3949,\n",
       " 'bed': 5632,\n",
       " 'board': 7174,\n",
       " 'romp': 54964,\n",
       " 'load': 37990,\n",
       " 'humor': 30528,\n",
       " 'understand': 67676,\n",
       " 'dennis': 16403,\n",
       " 'hopper': 30062,\n",
       " 'mess': 41501,\n",
       " 'head': 28598,\n",
       " 'shakespeare': 57903,\n",
       " 'ericka': 20717,\n",
       " 'eleniak': 19854,\n",
       " 'beautiful': 5576,\n",
       " 'hold': 29729,\n",
       " 'baywatch': 5441,\n",
       " 'show': 58528,\n",
       " 'knack': 35503,\n",
       " 'opportunity': 46566,\n",
       " 'expand': 21433,\n",
       " 'berenger': 6063,\n",
       " 'navy': 44398,\n",
       " 'william': 71530,\n",
       " 'mcnamara': 40853,\n",
       " 'solid': 60033,\n",
       " 'job': 33801,\n",
       " 'middle': 41757,\n",
       " 'chase': 10963,\n",
       " 'cherry': 11182,\n",
       " 'sundae': 62565,\n",
       " 'keep': 34819,\n",
       " 'soon': 60194,\n",
       " 'candy': 9626,\n",
       " 'favorite': 22269,\n",
       " 'action': 536,\n",
       " 'batman': 5356,\n",
       " 'schumacher': 56825,\n",
       " 'concern': 13054,\n",
       " 'subconsciously': 62190,\n",
       " 'contrarian': 13542,\n",
       " 'remember': 53559,\n",
       " 'conventional': 13585,\n",
       " 'wisdom': 71717,\n",
       " 'tim': 65159,\n",
       " 'burton': 9007,\n",
       " 'superior': 62685,\n",
       " 'problem': 51034,\n",
       " 'apologize': 2869,\n",
       " 'small': 59576,\n",
       " 'direction': 17344,\n",
       " 'scatter': 56536,\n",
       " 'justify': 34243,\n",
       " 'trump': 66591,\n",
       " 'sheer': 58130,\n",
       " 'style': 62148,\n",
       " 'flaw': 23245,\n",
       " 'common': 12826,\n",
       " 'nightmare': 45000,\n",
       " 'christmas': 11585,\n",
       " 'edward': 19526,\n",
       " 'scissorhands': 56892,\n",
       " 'begin': 5747,\n",
       " 'strongly': 62003,\n",
       " 'cobblepots': 12358,\n",
       " 'have': 28491,\n",
       " 'baby': 4462,\n",
       " 'dismay': 17600,\n",
       " 'birthing': 6654,\n",
       " 'room': 55003,\n",
       " 'ready': 52848,\n",
       " 'vomit': 70132,\n",
       " 'cage': 9294,\n",
       " 'even': 21110,\n",
       " 'stroll': 61991,\n",
       " 'dump': 18965,\n",
       " 'gotham': 26555,\n",
       " 'city': 11883,\n",
       " 'river': 54605,\n",
       " 'villain': 69771,\n",
       " 'penguin': 48482,\n",
       " 'danny': 15382,\n",
       " 'max': 40491,\n",
       " 'shreck': 58559,\n",
       " 'christopher': 11597,\n",
       " 'walken': 70405,\n",
       " 'evil': 21187,\n",
       " 'capitalist': 9746,\n",
       " 'comparable': 12874,\n",
       " 'grissom': 27067,\n",
       " 'jack': 33120,\n",
       " 'palance': 47518,\n",
       " 'plan': 49574,\n",
       " 'dupe': 19024,\n",
       " 'emotionally': 20150,\n",
       " 'secretary': 57224,\n",
       " 'timid': 65213,\n",
       " 'selina': 57400,\n",
       " 'kyle': 36144,\n",
       " 'michelle': 41705,\n",
       " 'pfeiffer': 48925,\n",
       " 'discover': 17476,\n",
       " 'nefarious': 44528,\n",
       " 'rid': 54351,\n",
       " 'rescue': 53849,\n",
       " 'cat': 10250,\n",
       " 'go': 26145,\n",
       " 'rumor': 55421,\n",
       " 'exist': 21399,\n",
       " 'dangerous': 15345,\n",
       " 'scheme': 56632,\n",
       " 'benevolent': 5983,\n",
       " 'ulterior': 67196,\n",
       " 'michael': 41678,\n",
       " 'keaton': 34788,\n",
       " 'surprisingly': 62871,\n",
       " 'necessarily': 44479,\n",
       " 'origin': 46721,\n",
       " 'major': 39355,\n",
       " 'complex': 12944,\n",
       " 'term': 64329,\n",
       " 'cadre': 9276,\n",
       " 'circus': 11863,\n",
       " 'performer': 48624,\n",
       " 'bid': 6418,\n",
       " 'frequently': 24213,\n",
       " 'learn': 36950,\n",
       " 'intrigue': 32569,\n",
       " 'tod': 65419,\n",
       " 'brown': 8516,\n",
       " 'freak': 24097,\n",
       " 'vincent': 69813,\n",
       " 'schiavelli': 56643,\n",
       " 'space': 60415,\n",
       " 'use': 68849,\n",
       " 'hardly': 28207,\n",
       " 'explore': 21529,\n",
       " 'catwoman': 10360,\n",
       " 'develope': 16832,\n",
       " 'feature': 22336,\n",
       " 'cut': 14988,\n",
       " 'bulk': 8795,\n",
       " 'excise': 21302,\n",
       " 'course': 14100,\n",
       " 'barely': 5088,\n",
       " 'dark': 15440,\n",
       " 'eye': 21678,\n",
       " 'unusually': 68633,\n",
       " 'bother': 7737,\n",
       " 'resolve': 53920,\n",
       " 'gothamites': 26556,\n",
       " 'lunatic': 38677,\n",
       " 'development': 16835,\n",
       " 'unfortunately': 67912,\n",
       " 'drop': 18698,\n",
       " 'minor': 42122,\n",
       " 'non': 45257,\n",
       " 'sequiturs': 57610,\n",
       " 'decision': 15838,\n",
       " 'logic': 38092,\n",
       " 'clear': 12053,\n",
       " 'writer': 72217,\n",
       " 'sam': 55919,\n",
       " 'hamm': 27987,\n",
       " 'daniel': 15354,\n",
       " 'exactly': 21240,\n",
       " 'tell': 64159,\n",
       " 'traditional': 65965,\n",
       " 'dialogue': 16983,\n",
       " 'pun': 51655,\n",
       " 'orient': 46713,\n",
       " 'fairly': 21849,\n",
       " 'blatant': 6882,\n",
       " 'transparent': 66117,\n",
       " 'poetic': 49871,\n",
       " 'intention': 32364,\n",
       " 'directorial': 17354,\n",
       " 'plenty': 49721,\n",
       " 'odd': 46025,\n",
       " 'service': 57694,\n",
       " 'easy': 19331,\n",
       " 'sum': 62505,\n",
       " 'positive': 50329,\n",
       " 'negative': 44537,\n",
       " 'depend': 16465,\n",
       " 'disposition': 17673,\n",
       " 'largely': 36596,\n",
       " 'transcend': 66047,\n",
       " 'insane': 32158,\n",
       " 'campy': 9564,\n",
       " 'beautifully': 5581,\n",
       " 'eerie': 19563,\n",
       " 'design': 16645,\n",
       " 'reference': 53253,\n",
       " 'vertigo': 69518,\n",
       " 'star': 61198,\n",
       " 'slight': 59438,\n",
       " 'invoke': 32671,\n",
       " 'nosferatu': 45441,\n",
       " 'schreck': 56799,\n",
       " 'dracula': 18482,\n",
       " 'motel': 43214,\n",
       " 'aforementioned': 967,\n",
       " 'willy': 71554,\n",
       " 'wonka': 71933,\n",
       " 'chocolate': 11456,\n",
       " 'factory': 21793,\n",
       " 'surreal': 62873,\n",
       " 'zombie': 73286,\n",
       " 'explicit': 21507,\n",
       " 'final': 22895,\n",
       " 'visual': 69954,\n",
       " 'atmosphere': 3864,\n",
       " 'power': 50475,\n",
       " 'pointless': 49897,\n",
       " 'whatsoever': 71135,\n",
       " 'awhile': 4312,\n",
       " 'wait': 70356,\n",
       " 'connect': 13264,\n",
       " 'drive': 18672,\n",
       " 'forward': 23850,\n",
       " 'pass': 48004,\n",
       " 'recycle': 53128,\n",
       " 'flat': 23209,\n",
       " 'uninteresting': 68074,\n",
       " 'lumet': 38651,\n",
       " 'support': 62772,\n",
       " 'blossom': 7076,\n",
       " 'hackman': 27706,\n",
       " 'concentrate': 13039,\n",
       " 'billings': 6527,\n",
       " 'political': 49981,\n",
       " 'thriller': 64945,\n",
       " 'jumble': 34168,\n",
       " 'lose': 38333,\n",
       " 'sea': 57108,\n",
       " 'write': 72215,\n",
       " 'fact': 21782,\n",
       " 'consider': 13352,\n",
       " 'gene': 25319,\n",
       " 'par': 47741,\n",
       " 'denzel': 16438,\n",
       " 'washington': 70605,\n",
       " 'gere': 25486,\n",
       " 'sidney': 58769,\n",
       " 'fierce': 22737,\n",
       " 'seventy': 57752,\n",
       " 'serpico': 57679,\n",
       " 'stop': 61712,\n",
       " 'recapture': 52986,\n",
       " 'success': 62334,\n",
       " 'angry': 2425,\n",
       " 'safe': 55722,\n",
       " 'likely': 37620,\n",
       " 'ensemble': 20447,\n",
       " 'drama': 18517,\n",
       " 'altman': 1824,\n",
       " 'folk': 23548,\n",
       " 'disney': 17616,\n",
       " 'explain': 21493,\n",
       " 'foremost': 23702,\n",
       " 'lesser': 37315,\n",
       " 'sitcom': 59105,\n",
       " 'material': 40336,\n",
       " 'motion': 43234,\n",
       " 'picture': 49236,\n",
       " 'kooky': 35717,\n",
       " 'teleplay': 64132,\n",
       " 'unique': 68092,\n",
       " 'martians': 40124,\n",
       " 'treatment': 66235,\n",
       " 'quaint': 51943,\n",
       " 'vision': 69941,\n",
       " 'dispel': 17648,\n",
       " 'close': 12221,\n",
       " 'encounter': 20244,\n",
       " 'alien': 1562,\n",
       " 'charm': 10936,\n",
       " 'property': 51234,\n",
       " 'mainly': 39316,\n",
       " 'endear': 20266,\n",
       " 'bill': 6518,\n",
       " 'bixby': 6715,\n",
       " 'ray': 52773,\n",
       " 'walston': 70445,\n",
       " 'unworldly': 68680,\n",
       " 'visitor': 69948,\n",
       " 'conceit': 13031,\n",
       " 'antenna': 2624,\n",
       " 'dopey': 18252,\n",
       " 'positively': 50330,\n",
       " 'idiotic': 30933,\n",
       " 'lloyd': 37973,\n",
       " 'uncle': 67458,\n",
       " 'martin': 40126,\n",
       " 'sport': 60863,\n",
       " 'metallic': 41545,\n",
       " 'appendage': 2927,\n",
       " 'shoot': 58457,\n",
       " ...}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.05376199,  0.00242971,  0.02662237, ...,  0.04376444,\n",
       "        -0.03134004, -0.03134004]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}