doc2vec_exploration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Pre-processing\n",
    "\n",
    "**Tasks**\n",
    "- remove html tags "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import sys\n",
    "sys.path.append(\"/Users/lesleymi/data_science_tutorials/IMDB_Sentiment_Analysis/src\")\n",
    "import imdb_functions as imdb\n",
    "\n",
    "# text modelling \n",
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument\n",
    "import spacy\n",
    "\n",
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_text(text):\n",
    "    \"\"\" \n",
    "    Produces the text with html tags removed and converts to all lower case. \n",
    "    \n",
    "    Arguments\n",
    "    ---------\n",
    "    text (pandas.core.series.Series) A series of text documents. \n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    pandas.core.series.Series A series of text with html tags removed & lower case letters. \n",
    "        \n",
    "    \"\"\"\n",
    "    # initialize a list for cleaned text \n",
    "    clean_text = []\n",
    "    for doc in text:\n",
    "        \n",
    "        ## remove html tags with beautifulsoup \n",
    "        soup = BeautifulSoup(doc)\n",
    "        text = soup.get_text().lower()\n",
    "\n",
    "        # append the text to a new series \n",
    "        clean_text.append(text)\n",
    "\n",
    "    # convert list to a pandas series \n",
    "    clean_text = pd.Series(clean_text)\n",
    "    \n",
    "    return clean_text\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize a dict to store the doc vectors \n",
    "vector_dict = {}\n",
    "colnames = []\n",
    "for i in range(len(model.docvecs)):\n",
    "    # build the dict of doc vectors \n",
    "    vector_dict[i] = model.docvecs[i]\n",
    "    \n",
    "# create the column names\n",
    "for dim in range(vec_size):\n",
    "    colname = \"dim_{0}\".format(dim)\n",
    "    colnames.append(colname)\n",
    "    \n",
    "# create a dataframe of doc vectors\n",
    "vector_df = pd.DataFrame(vector_dict).transpose()\n",
    "# set the col names to be number of dimensions\n",
    "vector_df.columns = colnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## load data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"data/train_clean.csv\")\n",
    "X_train = train.text\n",
    "train_raw = pd.read_csv(\"data/Train.csv\")\n",
    "y_train = train_raw.label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    grow watch love thunderbirds mate school watch...\n",
       "1    movie dvd player sit coke chip expectation hop...\n",
       "2    people know particular time past like feel nee...\n",
       "3    great interest biblical movie bore death minut...\n",
       "4    be die hard dad army fan change get tape dvd a...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 40000 training documents.\n"
     ]
    }
   ],
   "source": [
    "print(\"There are {} training documents.\".format(len(X_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    0\n",
       "4    1\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenize text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert documents into tokens\n",
    "docs = imdb.tokenize(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Doc2Vec Model \n",
    "The `Doc2Vec` instances take 2 inputs. A single document that is represented as a list of unicode strings (tokens) and a unique `tag` for the document. Can just be an integer index. \n",
    "\n",
    "The data structure input into `Doc2Vec` should be a list of `TaggedDocument`. \n",
    "\n",
    "**How the model is trained**\n",
    "\n",
    "The `dm=1` model param indicates which training algorithm should be used along with its underlying model architecture. In this case the 1 means it uses the `distributed memory (PV-DM)` version. This is the version of the model that learns not only the document vector but the individual word vectors as well. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tag the documents \n",
    "tagged_docs = [TaggedDocument(words= doc, tags=[tag]) for tag, doc in enumerate(docs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set number of processing cores \n",
    "cores = multiprocessing.cpu_count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set model params\n",
    "max_epochs = 100\n",
    "vec_size = 100\n",
    "min_count=2\n",
    "alpha = 0.025\n",
    "dm=1\n",
    "window=10\n",
    "\n",
    "# initialize the model \n",
    "model = Doc2Vec(vector_size=vec_size,\n",
    "               min_count=min_count,\n",
    "               dm=dm,\n",
    "               epochs=max_epochs,\n",
    "               window=window, \n",
    "               workers=cores)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.build_vocab` builds a dictionary for the model. It consists of all the unique words from the training corpus along with their word count frequency in the corpus. \n",
    "\n",
    "The vocabulary can be access by: `model.wv.vocab`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build the vocobulary \n",
    "model.build_vocab(tagged_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word love is used 10403 times throughout the corpus.\n"
     ]
    }
   ],
   "source": [
    "# get the number of times \"love\" is used in the corpus\n",
    "print(\"Word love is used {} times throughout the corpus.\".format(model.wv.vocab['love'].count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train the model\n",
    "model.train(documents=tagged_docs, \n",
    "            total_examples=model.corpus_count, \n",
    "            epochs=model.epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the model \n",
    "model.save(\"results/d2v.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Model \n",
    "\n",
    "**Most similar documents**\n",
    "\n",
    "My main thoughts right now are that I do not understand the most similar docs. The scores are low; the most similar doc for either a positive or negative review is below 0.5 similarity. So perhaps that explains why the documents don't seem related at all. There is nothing about the words being used at a superficial glance that gives me any intution why they are most similar. \n",
    "\n",
    "\n",
    "**most similar words**\n",
    "\n",
    "For the most part, these make more sense. The top most similar words are usually synonyms or antonyms. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the saved model \n",
    "model = Doc2Vec.load(\"results/d2v_train_clean.model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load spacy model \n",
    "nlp = spacy.load(\"en_core_web_sm\", disable=[\"tagger\", \"parser\", \"ner\"])\n",
    "\n",
    "# infer doc vector for a new document \n",
    "test_data = nlp(\"i love the imdb.\")\n",
    "\n",
    "# get the text from the document\n",
    "test_data_text = [token.text for token in test_data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'love', 'the', 'imdb', '.']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the test_data\n",
    "test_data_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Infer a vector "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-8.7601978e-01, -7.1176410e-02,  7.0591187e-01, -4.8143679e-01,\n",
       "       -4.8137760e-01, -3.5408694e-02, -6.3677318e-02,  1.2787600e-01,\n",
       "        1.5174484e-02, -7.7707690e-01, -5.6983024e-01,  1.5190088e+00,\n",
       "       -5.5980694e-01,  1.0904826e+00, -6.5231264e-01,  2.8780508e-01,\n",
       "       -4.6022081e-01,  5.5990957e-02,  3.1254226e-01,  2.3393029e-01,\n",
       "       -2.3947120e-01,  3.2985991e-01, -6.5190800e-02,  3.7434554e-01,\n",
       "        1.0477492e+00,  6.2189631e-02, -2.0642418e-01, -4.9563262e-01,\n",
       "       -8.6489081e-02, -7.7440524e-01, -2.5099444e-01,  5.6423777e-01,\n",
       "        5.8801621e-01,  4.6902564e-01,  1.4091048e+00,  1.9119181e-01,\n",
       "       -8.9552552e-01,  1.0647823e+00, -1.1273570e-01,  6.1967146e-01,\n",
       "       -3.8818705e-01,  1.7704718e-01,  7.8034365e-01,  3.7679043e-01,\n",
       "       -2.9321137e-01, -7.2484118e-01, -1.0252564e-01, -5.6680793e-01,\n",
       "        2.7603558e-01,  8.9655185e-01, -1.1634492e+00,  1.1330364e+00,\n",
       "       -3.0282557e-03, -2.1823885e-01,  1.8622139e-02,  9.7461119e-02,\n",
       "        5.2186996e-01, -1.4987764e-03,  6.1184567e-01, -7.9365540e-01,\n",
       "       -7.8892566e-02,  2.9105002e-01, -6.6677368e-01,  4.2381916e-02,\n",
       "       -1.9970247e-01, -6.7343667e-02,  4.8318934e-01, -6.2124461e-01,\n",
       "        4.5558158e-02, -1.8616958e-01, -1.1898289e+00, -2.7813023e-01,\n",
       "        1.5125978e-01, -1.3641568e-01,  5.4830265e-01, -2.8631943e-01,\n",
       "        6.0150701e-01, -5.2603197e-01, -2.4480064e-01, -2.7532336e-01,\n",
       "       -7.2442603e-01, -3.6999661e-01,  3.3055925e-01,  5.9283894e-01,\n",
       "        8.4529974e-02, -1.4992806e-01,  1.3040055e-01, -8.2607943e-01,\n",
       "        2.8514495e-01,  2.7968121e-01,  2.1431322e-01,  2.9854095e-01,\n",
       "       -3.3667493e-01,  2.2647020e-01,  7.3300518e-02, -1.8920484e-01,\n",
       "        1.2214462e-01,  2.6514393e-01,  2.3088410e-01,  3.8411549e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the inferred vector for the test document \n",
    "test_data_vector = model.infer_vector(test_data_text)\n",
    "test_data_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that the vector is 100 dimensions \n",
    "len(test_data_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get doc vector for a document in training data \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.3663208 , -0.8338305 , -1.5876216 , -0.4484202 ,  0.37763113,\n",
       "        0.18997066, -0.15291765, -0.33705813,  1.0348196 , -1.830415  ,\n",
       "       -1.8290714 , -0.59267575,  2.2594986 ,  0.47995684,  0.94431174,\n",
       "        3.1104918 ,  1.8962055 ,  0.25603727, -0.9457041 ,  0.39135978,\n",
       "        2.0950372 ,  1.123366  ,  1.5927137 ,  0.23922084,  2.2375321 ,\n",
       "       -1.6106455 , -1.9829898 , -1.9768217 ,  0.24138217, -2.785498  ,\n",
       "        2.7964442 , -1.17678   ,  0.20068131,  3.0975914 ,  0.41910535,\n",
       "        2.8714657 , -0.15959644, -0.39355338, -2.368877  ,  0.85839856,\n",
       "        0.4438577 , -0.56905115, -0.01014811, -1.033843  ,  0.42319435,\n",
       "       -1.2470948 ,  1.8577793 ,  0.6409478 , -3.4192915 , -1.5226547 ,\n",
       "        0.82558554,  1.6715181 , -1.1165489 , -1.0488758 , -2.9167726 ,\n",
       "        0.6252657 ,  3.7159734 , -1.2257144 , -1.5467651 , -0.8403192 ,\n",
       "        0.9729997 , -1.0113367 , -2.298275  , -2.788685  , -1.6347834 ,\n",
       "        0.18489991, -2.1888363 ,  0.8171673 ,  0.91485846, -2.8338616 ,\n",
       "       -0.00982627, -0.74751115,  1.1341369 ,  0.13319439,  2.0180352 ,\n",
       "        1.1607494 , -3.140825  , -0.07270265,  0.03409767, -0.27616808,\n",
       "        0.05139452,  0.29801506,  0.59058577, -2.0151532 ,  1.5595578 ,\n",
       "        0.13652885, -0.9572642 , -1.0586988 ,  3.2113593 ,  3.0636656 ,\n",
       "       -2.1120572 , -1.0664217 , -1.3094448 ,  1.6558317 ,  1.4419686 ,\n",
       "       -1.468999  , -0.02334668,  0.7569472 ,  1.3024312 ,  0.7462779 ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get doc vector for document with tag 0\n",
    "model.docvecs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.6520679e+00, -3.8588434e-01, -6.7569941e-01,  3.4696057e+00,\n",
       "       -1.9679284e+00, -2.2057221e+00, -1.4142027e+00, -4.7539362e-01,\n",
       "       -7.5158638e-01, -1.0545136e+00, -1.4270027e+00, -3.2239280e+00,\n",
       "       -1.2632240e+00,  3.6208701e+00, -2.3817019e+00,  1.4604803e+00,\n",
       "       -1.2440633e+00, -1.5918282e+00, -2.2776539e+00,  2.8869450e+00,\n",
       "        4.9006929e+00, -3.4761369e-01, -1.0802556e+00,  1.8751051e+00,\n",
       "        9.4934928e-01,  1.8222005e+00, -6.7758745e-01, -3.0567009e+00,\n",
       "        7.6635689e-02,  1.9879228e+00, -5.4529065e-01,  1.3167815e-02,\n",
       "        2.3686938e+00, -5.3301048e-01,  8.2191074e-01,  6.8047595e-01,\n",
       "       -1.7336306e-01,  2.6591143e-01,  2.5094647e+00, -6.7854130e-01,\n",
       "       -6.3020778e-01,  2.7396691e+00,  2.2822950e+00, -1.8262483e-01,\n",
       "       -1.6745805e+00, -3.9558086e-01,  1.2627271e+00, -8.4782064e-01,\n",
       "       -1.9645272e+00, -2.8780472e+00, -5.5718642e-01,  4.7494566e-01,\n",
       "       -1.8261996e-01,  1.3231115e+00, -1.6106341e+00,  6.6597298e-02,\n",
       "        2.3647397e+00,  2.0748298e+00, -1.3173913e+00, -2.7609813e+00,\n",
       "        1.4107221e+00, -2.8465359e+00,  1.2536379e+00,  6.2747471e-02,\n",
       "       -4.7419758e+00,  1.4302489e-01, -1.3553225e+00,  2.3649932e-01,\n",
       "       -1.7723687e+00,  1.1034976e+00,  1.3962588e+00, -4.6060290e+00,\n",
       "       -1.1303442e+00,  3.6200447e+00,  2.7311218e+00, -1.1943829e+00,\n",
       "       -3.1171376e-01,  1.4391632e+00, -1.5203526e+00,  3.6632748e+00,\n",
       "        6.1850023e-01, -2.6685531e+00,  2.0601497e+00,  1.9221529e+00,\n",
       "       -2.7256141e+00, -1.7259015e-03,  3.1477077e+00,  1.3882947e+00,\n",
       "        5.8766758e-01,  1.4788194e+00, -1.1702196e-01, -2.2788112e+00,\n",
       "        5.2659768e-01,  1.3298284e-01, -5.5604851e-01,  1.3337315e+00,\n",
       "        8.9039600e-01,  5.5021608e-01, -1.9811816e+00, -5.3283697e-01],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the doc vector for the document with tag 1\n",
    "model.docvecs[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most similar document: negative\n",
    "This to returns the document tags along with the cosine similarity score to `doc 1` a negative review. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(24141, 0.4895094037055969),\n",
       " (7551, 0.4757860004901886),\n",
       " (7709, 0.4681105613708496),\n",
       " (28398, 0.46748244762420654),\n",
       " (37855, 0.4662354588508606),\n",
       " (38331, 0.4656111001968384),\n",
       " (26914, 0.45377689599990845),\n",
       " (37692, 0.4481343626976013),\n",
       " (32846, 0.44798168540000916),\n",
       " (5742, 0.445815771818161)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_sim_docs = model.docvecs.most_similar(1)\n",
    "most_sim_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'movie dvd player sit coke chip expectation hope movie contain strong point movie awsome animation good flow story excellent voice cast funny comedy kick ass soundtrack disappointment find atlantis milo return read review let follow paragraph direct see movie enjoy primarily point scene appear shock pick atlantis milo return display case local videoshop expectation music feel bad imitation movie voice cast replace fit exception character like voice sweet actual drawing not bad animation particular sad sight storyline pretty weak like episode schooby doo single adventurous story get time not misunderstand good schooby doo episode not laugh single time snigger audience see movie especially care similar sequel fast review movie stand product like schooby doo like movie enjoy movie suspect good kid movie know well milo return episode series cartoon channel breakfast tv'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at doc 1 \n",
    "query_text = \" \".join(tagged_docs[1].words)\n",
    "query_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"When I put this movie in my DVD player, and sat down with a coke and some chips, I had some expectations. I was hoping that this movie would contain some of the strong-points of the first movie: Awsome animation, good flowing story, excellent voice cast, funny comedy and a kick-ass soundtrack. But, to my disappointment, not any of this is to be found in Atlantis: Milo's Return. Had I read some reviews first, I might not have been so let down. The following paragraph will be directed to those who have seen the first movie, and who enjoyed it primarily for the points mentioned.<br /><br />When the first scene appears, your in for a shock if you just picked Atlantis: Milo's Return from the display-case at your local videoshop (or whatever), and had the expectations I had. The music feels as a bad imitation of the first movie, and the voice cast has been replaced by a not so fitting one. (With the exception of a few characters, like the voice of Sweet). The actual drawings isnt that bad, but the animation in particular is a sad sight. The storyline is also pretty weak, as its more like three episodes of Schooby-Doo than the single adventurous story we got the last time. But dont misunderstand, it's not very good Schooby-Doo episodes. I didnt laugh a single time, although I might have sniggered once or twice.<br /><br />To the audience who haven't seen the first movie, or don't especially care for a similar sequel, here is a fast review of this movie as a stand-alone product: If you liked schooby-doo, you might like this movie. If you didn't, you could still enjoy this movie if you have nothing else to do. And I suspect it might be a good kids movie, but I wouldn't know. It might have been better if Milo's Return had been a three-episode series on a cartoon channel, or on breakfast TV.\""
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get original raw text of doc 1 \n",
    "train_raw.text[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sentiment of Doc 1 is: 0\n"
     ]
    }
   ],
   "source": [
    "# get the sentiment of the query doc \n",
    "print(\"The sentiment of Doc 1 is: {}\".format(y_train[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize a dictionary to hold most similar texts \n",
    "# keyed by their index in the TaggedDocuments lists \n",
    "most_similar_texts = {}\n",
    "\n",
    "# get the texts of the most similar docs\n",
    "for most_sim_doc in most_sim_docs:\n",
    "    # get the tagged doc index \n",
    "    index = most_sim_doc[0]\n",
    "    \n",
    "    # convert the tokens from most similar into text\n",
    "    most_sim_text = \" \".join(tagged_docs[index].words)\n",
    "    \n",
    "    # append the text to the list \n",
    "    most_similar_texts[index] = most_sim_text\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>most_similar_texts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24141</th>\n",
       "      <td>new bear big fan surface think script computer...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7551</th>\n",
       "      <td>warn movie scary horror movie fan especially c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7709</th>\n",
       "      <td>contain spoiler inuyasha good anime actually o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28398</th>\n",
       "      <td>shock surprise negative review see web think c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37855</th>\n",
       "      <td>despite disney well effort enjoyable movie fol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38331</th>\n",
       "      <td>entertain random love hate expect sophisticate...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26914</th>\n",
       "      <td>despite review angel outfield pretty good movi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37692</th>\n",
       "      <td>lion king doubt favorite disney movie time fig...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32846</th>\n",
       "      <td>lot music see movie time tonight road picture ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5742</th>\n",
       "      <td>see movie teenager come theater way see nearly...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      most_similar_texts\n",
       "24141  new bear big fan surface think script computer...\n",
       "7551   warn movie scary horror movie fan especially c...\n",
       "7709   contain spoiler inuyasha good anime actually o...\n",
       "28398  shock surprise negative review see web think c...\n",
       "37855  despite disney well effort enjoyable movie fol...\n",
       "38331  entertain random love hate expect sophisticate...\n",
       "26914  despite review angel outfield pretty good movi...\n",
       "37692  lion king doubt favorite disney movie time fig...\n",
       "32846  lot music see movie time tonight road picture ...\n",
       "5742   see movie teenager come theater way see nearly..."
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert the most similar texts into a dataframe \n",
    "most_similar_df = pd.DataFrame(most_similar_texts, index=[1]).transpose().rename(columns={1:'most_similar_texts'})\n",
    "most_similar_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Most similar doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'new bear big fan surface think script computer graphic exceptional good sci fi flick see theater february tv guide say season finale announcer say effect season finale surface season finale series finale wait fall go happen fall get find nbc go pick sci fi usa bay watch long abc usa pick go gang buster bet abc chock ha series mini series loyal fan closure happen guy trap church steeple creature chaple nim grouth spert clone guy come unanswered question thank listen babble'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# most similar text 25471\n",
    "most_similar_df.loc[24141][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am new at this, so bear with me please. I am a big fan of Surface. I thought the script and the computer graphics were exceptional, as good as any Sci Fi flick I\\'ve seen at the theater. In February the TV guide said Season Finale, the announcer for the show said something to the effect of, \"...and now for the season finale of Surface.\" Season Finale, not series finale! I couldn\\'t wait for fall to get here, to see was going to happen next. So fall gets here and it\\'s nowhere to be found! If NBC isn\\'t going to pick it up, what about Sci Fi or USA? It seems to me that Bay Watch didn\\'t last long on ABC & then USA picked it up, and it went gang busters! (I bet ABC was chocking) Ha! If not a series, then at least a mini series, to give all us loyal fans closure. What happened to our guy\\'s trapped in the church steeple? Was the creature in the chaple Nim? Did he have a grouth spert? Does the cloned guy come over to our side? There are so many unanswered questions. Thank\\'s for listening to me babble!'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# original text \n",
    "train_raw.text[24141]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of most similar doc is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of most similar doc is: {}\".format(y_train[24141]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2nd most similar doc "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'warn movie scary horror movie fan especially child play fan think incredibly funny will scare bad movie scary'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar_df.loc[7551][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I'm warning you -- this movie is not scary. If you're a horror movie fan, especially a Child's Play fan, you'll think it's incredibly funny, but you won't be scared. It's not a bad movie, but it's not scary.\""
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# original text\n",
    "train_raw.text[7551]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of 2nd most similar doc is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of 2nd most similar doc is: {}\".format(y_train[7551]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3rd most similar doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'contain spoiler inuyasha good anime actually overrate absolutely story line plot drag story filler episode plot progress filler story repeat episode plot kagome sense jewel shard worm slime tentacle demon thing pop inuyasha say wind scar iron reaver soul stealer etc kill demon jewel shard repeat scene repeat episode repeat comedic device funny anymore wait sexual harassment funny viz rate series old teen idea rate bad call funny sexual harassment kind suggestive arrest know inuyasha overrate videogame suck especially mask game play friend house interest game slow bore nintendo like graphic magazine get rate say role play game slow milkshake move cocktail straw stupid inuyasha toy action figure trade card sticker color book color book think inuyasha maybe member inuyasha group msn half people guess inuyasha little kid anime think small bite edit show toonami manga volume help wonder mile forest cut sad music music annoy hear song episode episode music get annoy anime music fit mood hear song different inuyasha soundtrack waste money think know fan inuyasha feel ashamed watch kim possible pokemon instead sadly show romance main protagonist inuyasha'"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar_df.loc[7709][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of 3rd most similar doc is: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of 3rd most similar doc is: {}\".format(y_train[7709]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'May or may not contain spoilers. <br /><br />Inuyasha is not a good anime. It\\'s actually very overrated. Why? There\\'s absolutely no story line, no plot, and the show just drags on... and on... and on... That\\'s because there are more side stories and fillers than episodes that make the plot progress. And the fillers are just the same stories being repeated over and over again. The same episodes seem to go with the same plot: Kagome sensing a jewel shard, a worm/slime/tentacle demon thing pops up, Inuyasha says \"Wind-Scar\", \"Iron Reaver Soul Stealer\", etc. and kills the demon, they get the jewel shard, and then we just repeat this scene 160 or more times.<br /><br />Besides the repeating of episodes, there\\'s the repeating of comedic devices, and they\\'re not funny anymore. Wait, they never were. Sexual harassment is NOT funny. Viz rated the series Older Teens, 16+. I have no idea why they rated it that. There\\'s nothing bad about it except for the so-called funny sexual harassment, which is kind of suggestive, and that could get you arrested these days.<br /><br />Now, this is how we know Inuyasha is overrated. The videogames. They all sucked. Especially the Mask game. I played that at my friend\\'s house. It wasn\\'t anything interesting. The game was slow, boring, and it had Nintendo 64 like graphics. In a magazine, it got a rating of 4/10, saying \"...this role-playing game is slower than milkshake moving up a cocktail straw.\" Then, there\\'s all these stupid Inuyasha toys, action figures, trading cards, stickers, and coloring books. COLORING BOOKS! We thought Inuyasha was 16+! Maybe not... But after being a member of Inuyasha groups on MSN, about half the people on there were 10 to 13. I guess Inuyasha is a little kid anime after all. (I think that just a small bit of editing done to this show, it could be shown on Toonami.) There are over 40 manga volumes. I can only help but wonder how many miles of forest that have been cut down to make them. Sad...<br /><br />Then there\\'s the music. The music is so annoying. We hear the same 5 songs every episode. After 10 episodes, the music gets really annoying. In other anime, they have music to fit the mood and we don\\'t hear some songs very often. There are about 15 different Inuyasha soundtracks. Don\\'t waste your money on that garbage!<br /><br />And how do you think I know all this? Because I used to be a fan of Inuyasha. I feel ashamed of myself. I\\'d rather watch Kim Possible or Pokemon instead. Sadly, those two shows have more romance between the two main protagonists than Inuyasha will ever have...'"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get original text \n",
    "train_raw.text[7709]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Most similar document: positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(5435, 0.4771555960178375),\n",
       " (18429, 0.4644291400909424),\n",
       " (27699, 0.44662174582481384),\n",
       " (9161, 0.4408680498600006),\n",
       " (36322, 0.43742644786834717),\n",
       " (186, 0.43081188201904297),\n",
       " (13058, 0.42809751629829407),\n",
       " (16188, 0.4186444878578186),\n",
       " (25912, 0.41499632596969604),\n",
       " (23975, 0.4139326214790344)]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# query document \n",
    "most_sim_docs = model.docvecs.most_similar(4)\n",
    "most_sim_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'be die hard dad army fan change get tape dvd audiobooks time watch listen brand new film film run certain episode man hour enemy gate battle school numerous different edge introduction new general instead captain square brilliant especially cash cheque rarely follow early year get equipment uniform start train great film bore sunday afternoon draw back germans bogus dodgy accent come germans not pronounce letter w like cast liz frazer instead familiar janet davis like liz film like carry ons carry correctly janet davis well choice'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at doc 4\n",
    "query_text = \" \".join(tagged_docs[4].words)\n",
    "query_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of doc 4 is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of doc 4 is: {}\".format(y_train[4]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>most_similar_texts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5435</th>\n",
       "      <td>music laurence olivier sombre delivery set ton...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18429</th>\n",
       "      <td>watch series avidly wonder lengthy break tune ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27699</th>\n",
       "      <td>sitcom big screen spin offs come list serve bl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9161</th>\n",
       "      <td>think series go fun action series dynamic plot...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36322</th>\n",
       "      <td>like idea female turtle know tmnt brother teac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>good movie typical war flick bite different mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13058</th>\n",
       "      <td>episode man man dean learner air scratch episo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16188</th>\n",
       "      <td>memorable line short live view episode line in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25912</th>\n",
       "      <td>movie kick ass bar bam crue film get dvd day a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23975</th>\n",
       "      <td>well martial fu movie time u love martial art ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      most_similar_texts\n",
       "5435   music laurence olivier sombre delivery set ton...\n",
       "18429  watch series avidly wonder lengthy break tune ...\n",
       "27699  sitcom big screen spin offs come list serve bl...\n",
       "9161   think series go fun action series dynamic plot...\n",
       "36322  like idea female turtle know tmnt brother teac...\n",
       "186    good movie typical war flick bite different mo...\n",
       "13058  episode man man dean learner air scratch episo...\n",
       "16188  memorable line short live view episode line in...\n",
       "25912  movie kick ass bar bam crue film get dvd day a...\n",
       "23975  well martial fu movie time u love martial art ..."
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get most similar texts\n",
    "most_similar_df = imdb.get_most_similar_docs(tagged_docs=tagged_docs, most_sim_docs=most_sim_docs)\n",
    "most_similar_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'music laurence olivier sombre delivery set tone perfectly outstanding documentary ww ii buff descendant participant conflict politician think thing way extend foreign policy deck aircraft carrier hear george bush curious need know s s s aspect conflict episode roughly chronological order see sequence self contain bind new insight new viewer sheer volume present actual footage battle intersperse interview involve story interview 2 line authority support personnel main character private captain secretary eyewitness like real upfront taste war presently watch dvd version original television documentary strongly recommend wear gaptoothed overpriced vhs offering available ebay pay cdn dvd disc new release include bonus material screen mode menu easy follow choice episode want view select give option chapter episode play episode understandable comprehensive presentation tiny navigation menu impact diminish year nay year war remember watch broadcast buffalo pbs station move london wish right time copy wish finally come documentary tell friend buy copy library remember honour sacrifice challenge overcome america russia britain canada nation people involve final victory eye opener'"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar_df.loc[5435][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of most similar doc is: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of most similar doc is: {}\".format(y_train[39119]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'watch series avidly wonder lengthy break tune episode series hook excellent telly grind break stuff like mission impossible character round expand series go change adapt readily new surrounding cleverly remain strictly character possible sympathy think feel sorry crush happen hope year lose look forward actually mind answer mystery'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar_df.loc[18429][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment of 2nd most similar doc is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentiment of 2nd most similar doc is: {}\".format(y_train[18429]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'watch series avidly wonder lengthy break tune episode series hook excellent telly grind break stuff like mission impossible character round expand series go change adapt readily new surrounding cleverly remain strictly character possible sympathy think feel sorry crush happen hope year lose look forward actually mind answer mystery'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar_df.loc[18429][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentiment of 3rd most similar doc is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"sentiment of 3rd most similar doc is: {}\".format(y_train[18429]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Most similar words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('flat', 0.49725016951560974),\n",
       " ('asleep', 0.4771397113800049),\n",
       " ('apart', 0.46345919370651245),\n",
       " ('category', 0.45624393224716187),\n",
       " ('lust', 0.4409942030906677),\n",
       " ('lover', 0.4356623888015747),\n",
       " ('vassar', 0.41631919145584106),\n",
       " ('short', 0.40895459055900574),\n",
       " ('donna', 0.40465378761291504),\n",
       " ('ect', 0.390438973903656)]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('love')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('brutal', 0.5618649125099182),\n",
       " ('vicious', 0.5486468076705933),\n",
       " ('violence', 0.5431817770004272),\n",
       " ('gory', 0.5042787194252014),\n",
       " ('cannibalism', 0.44713398814201355),\n",
       " ('tame', 0.44368988275527954),\n",
       " ('explicit', 0.4399290680885315),\n",
       " ('gruesome', 0.4369996190071106),\n",
       " ('exploit', 0.41187113523483276),\n",
       " ('levres', 0.4106130003929138)]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('violent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('decieve', 0.43368470668792725),\n",
       " ('facepaint', 0.40532469749450684),\n",
       " ('occassionaly', 0.4004054665565491),\n",
       " ('unspoken', 0.39546847343444824),\n",
       " ('mindbogglingly', 0.3859484791755676),\n",
       " ('abstinence', 0.382354736328125),\n",
       " ('temple', 0.3801541328430176),\n",
       " ('beret', 0.3800097405910492),\n",
       " ('walrus', 0.3783597946166992),\n",
       " ('curtain', 0.37587088346481323)]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('grass')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('nancy', 0.4182920455932617),\n",
       " ('fenchurch', 0.40522515773773193),\n",
       " ('dog', 0.39421069622039795),\n",
       " ('mammy', 0.38828516006469727),\n",
       " ('michael', 0.3875651955604553),\n",
       " ('mutt', 0.3847420811653137),\n",
       " ('monkey', 0.3802471160888672),\n",
       " ('oog', 0.3756932020187378),\n",
       " ('bobbi', 0.37538284063339233),\n",
       " ('edgar', 0.3712940216064453)]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('cat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('villain', 0.6178774833679199),\n",
       " ('heroic', 0.5215709805488586),\n",
       " ('protagonist', 0.5046625137329102),\n",
       " ('guy', 0.46219581365585327),\n",
       " ('soldier', 0.4537964463233948),\n",
       " ('criminal', 0.4315299093723297),\n",
       " ('evil', 0.4274206757545471),\n",
       " ('cop', 0.4197346866130829),\n",
       " ('character', 0.4120820164680481),\n",
       " ('gun', 0.4019153416156769)]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar('hero')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract Doc Vectors "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize a dict to store the doc vectors \n",
    "vector_dict = {}\n",
    "colnames = []\n",
    "for i in range(len(model.docvecs)):\n",
    "    # build the dict of doc vectors \n",
    "    vector_dict[i] = model.docvecs[i]\n",
    "    \n",
    "# create the column names\n",
    "for dim in range(vec_size):\n",
    "    colname = \"dim_{0}\".format(dim)\n",
    "    colnames.append(colname)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dataframe of doc vectors\n",
    "vector_df = pd.DataFrame(vector_dict).transpose()\n",
    "# set the col names to be number of dimensions\n",
    "vector_df.columns = colnames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 40000 documents; each document is represented in 100 dimensions.\n"
     ]
    }
   ],
   "source": [
    "print(\"There are {0} documents; each document is represented in {1} dimensions.\".format(vector_df.shape[0], vector_df.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dim_0</th>\n",
       "      <th>dim_1</th>\n",
       "      <th>dim_2</th>\n",
       "      <th>dim_3</th>\n",
       "      <th>dim_4</th>\n",
       "      <th>dim_5</th>\n",
       "      <th>dim_6</th>\n",
       "      <th>dim_7</th>\n",
       "      <th>dim_8</th>\n",
       "      <th>dim_9</th>\n",
       "      <th>...</th>\n",
       "      <th>dim_90</th>\n",
       "      <th>dim_91</th>\n",
       "      <th>dim_92</th>\n",
       "      <th>dim_93</th>\n",
       "      <th>dim_94</th>\n",
       "      <th>dim_95</th>\n",
       "      <th>dim_96</th>\n",
       "      <th>dim_97</th>\n",
       "      <th>dim_98</th>\n",
       "      <th>dim_99</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.784555</td>\n",
       "      <td>-0.397502</td>\n",
       "      <td>-0.331985</td>\n",
       "      <td>0.965592</td>\n",
       "      <td>2.012932</td>\n",
       "      <td>1.887488</td>\n",
       "      <td>0.510781</td>\n",
       "      <td>3.288674</td>\n",
       "      <td>3.433098</td>\n",
       "      <td>-3.791230</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.244087</td>\n",
       "      <td>1.192884</td>\n",
       "      <td>0.891242</td>\n",
       "      <td>2.866597</td>\n",
       "      <td>8.549128</td>\n",
       "      <td>-0.478778</td>\n",
       "      <td>1.692087</td>\n",
       "      <td>2.417104</td>\n",
       "      <td>1.771905</td>\n",
       "      <td>-1.312251</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.027664</td>\n",
       "      <td>-3.786560</td>\n",
       "      <td>-1.583889</td>\n",
       "      <td>3.124440</td>\n",
       "      <td>-0.892188</td>\n",
       "      <td>-1.558566</td>\n",
       "      <td>3.319089</td>\n",
       "      <td>1.988021</td>\n",
       "      <td>2.089380</td>\n",
       "      <td>-0.524098</td>\n",
       "      <td>...</td>\n",
       "      <td>0.397586</td>\n",
       "      <td>2.595253</td>\n",
       "      <td>2.952577</td>\n",
       "      <td>-2.684180</td>\n",
       "      <td>1.753658</td>\n",
       "      <td>-1.200642</td>\n",
       "      <td>-0.714399</td>\n",
       "      <td>-1.440350</td>\n",
       "      <td>1.549200</td>\n",
       "      <td>-1.613970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.591049</td>\n",
       "      <td>-1.170077</td>\n",
       "      <td>0.830686</td>\n",
       "      <td>3.815988</td>\n",
       "      <td>-0.730998</td>\n",
       "      <td>0.646123</td>\n",
       "      <td>4.216710</td>\n",
       "      <td>3.768117</td>\n",
       "      <td>1.581684</td>\n",
       "      <td>0.660360</td>\n",
       "      <td>...</td>\n",
       "      <td>2.918566</td>\n",
       "      <td>0.399827</td>\n",
       "      <td>-1.357417</td>\n",
       "      <td>-0.124357</td>\n",
       "      <td>0.657655</td>\n",
       "      <td>-2.304216</td>\n",
       "      <td>-3.644226</td>\n",
       "      <td>-0.174466</td>\n",
       "      <td>2.901997</td>\n",
       "      <td>0.991823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1.579851</td>\n",
       "      <td>0.370804</td>\n",
       "      <td>0.325788</td>\n",
       "      <td>2.724860</td>\n",
       "      <td>0.076738</td>\n",
       "      <td>0.271273</td>\n",
       "      <td>2.576356</td>\n",
       "      <td>1.767729</td>\n",
       "      <td>-1.268051</td>\n",
       "      <td>-3.105602</td>\n",
       "      <td>...</td>\n",
       "      <td>1.757116</td>\n",
       "      <td>-3.508960</td>\n",
       "      <td>-0.034007</td>\n",
       "      <td>0.554777</td>\n",
       "      <td>2.150062</td>\n",
       "      <td>1.295318</td>\n",
       "      <td>-0.126439</td>\n",
       "      <td>2.238556</td>\n",
       "      <td>1.850339</td>\n",
       "      <td>-1.993807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.058910</td>\n",
       "      <td>-1.690887</td>\n",
       "      <td>-0.004952</td>\n",
       "      <td>-2.187342</td>\n",
       "      <td>-3.583359</td>\n",
       "      <td>2.466982</td>\n",
       "      <td>2.651749</td>\n",
       "      <td>0.484369</td>\n",
       "      <td>1.344823</td>\n",
       "      <td>-2.241074</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.418538</td>\n",
       "      <td>-4.106030</td>\n",
       "      <td>2.691415</td>\n",
       "      <td>-2.924291</td>\n",
       "      <td>-0.589641</td>\n",
       "      <td>-2.774781</td>\n",
       "      <td>-1.087569</td>\n",
       "      <td>-1.415747</td>\n",
       "      <td>-0.630574</td>\n",
       "      <td>0.065931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 100 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      dim_0     dim_1     dim_2     dim_3     dim_4     dim_5     dim_6  \\\n",
       "0  0.784555 -0.397502 -0.331985  0.965592  2.012932  1.887488  0.510781   \n",
       "1  1.027664 -3.786560 -1.583889  3.124440 -0.892188 -1.558566  3.319089   \n",
       "2  0.591049 -1.170077  0.830686  3.815988 -0.730998  0.646123  4.216710   \n",
       "3 -1.579851  0.370804  0.325788  2.724860  0.076738  0.271273  2.576356   \n",
       "4 -1.058910 -1.690887 -0.004952 -2.187342 -3.583359  2.466982  2.651749   \n",
       "\n",
       "      dim_7     dim_8     dim_9  ...    dim_90    dim_91    dim_92    dim_93  \\\n",
       "0  3.288674  3.433098 -3.791230  ... -1.244087  1.192884  0.891242  2.866597   \n",
       "1  1.988021  2.089380 -0.524098  ...  0.397586  2.595253  2.952577 -2.684180   \n",
       "2  3.768117  1.581684  0.660360  ...  2.918566  0.399827 -1.357417 -0.124357   \n",
       "3  1.767729 -1.268051 -3.105602  ...  1.757116 -3.508960 -0.034007  0.554777   \n",
       "4  0.484369  1.344823 -2.241074  ... -2.418538 -4.106030  2.691415 -2.924291   \n",
       "\n",
       "     dim_94    dim_95    dim_96    dim_97    dim_98    dim_99  \n",
       "0  8.549128 -0.478778  1.692087  2.417104  1.771905 -1.312251  \n",
       "1  1.753658 -1.200642 -0.714399 -1.440350  1.549200 -1.613970  \n",
       "2  0.657655 -2.304216 -3.644226 -0.174466  2.901997  0.991823  \n",
       "3  2.150062  1.295318 -0.126439  2.238556  1.850339 -1.993807  \n",
       "4 -0.589641 -2.774781 -1.087569 -1.415747 -0.630574  0.065931  \n",
       "\n",
       "[5 rows x 100 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the first few documents \n",
    "# each row represents a movie review \n",
    "vector_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the document vector dataset\n",
    "vector_df.to_csv(\"data/train_d2v.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}