Docx/CS224N/Tutorial/Gensim word vector visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim word vector visualization of various word vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Get the interactive Tools for Matplotlib\n",
    "%matplotlib notebook\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "from gensim.test.utils import datapath, get_tmpfile\n",
    "from gensim.models import KeyedVectors\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For looking at word vectors, I'll use Gensim. We also use it in hw1 for word vectors. Gensim isn't really a deep learning package. It's a package for for word and text similarity modeling, which started with (LDA-style) topic models and grew into SVD and neural word representations. But its efficient and scalable, and quite widely used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our homegrown Stanford offering is GloVe word vectors. Gensim doesn't give them first class support, but allows you to convert a file of GloVe vectors into word2vec format. You can download the GloVe vectors from [the Glove page](https://nlp.stanford.edu/projects/glove/). They're inside [this zip file](https://nlp.stanford.edu/data/glove.6B.zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(I use the 100d vectors below as a mix between speed and smallness vs. quality. If you try out the 50d vectors, they basically work for similarity but clearly aren't as good for analogy problems. If you load the 300d vectors, they're even better than the 100d vectors.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_file = datapath('/Users/manning/Corpora/GloVe/glove.6B.100d.txt')\n",
    "word2vec_glove_file = get_tmpfile(\"glove.6B.100d.word2vec.txt\")\n",
    "glove2word2vec(glove_file, word2vec_glove_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = KeyedVectors.load_word2vec_format(word2vec_glove_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar('obama')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar('banana')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(negative='banana')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = model.most_similar(positive=['woman', 'king'], negative=['man'])\n",
    "print(\"{}: {:.4f}\".format(*result[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def analogy(x1, x2, y1):\n",
    "    result = model.most_similar(positive=[y1, x2], negative=[x1])\n",
    "    return result[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Analogy](imgs/word2vec-king-queen-composition.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analogy('japan', 'japanese', 'australia')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analogy('australia', 'beer', 'france')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analogy('obama', 'clinton', 'reagan')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analogy('tall', 'tallest', 'long')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "analogy('good', 'fantastic', 'bad')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.doesnt_match(\"breakfast cereal dinner lunch\".split()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_pca_scatterplot(model, words=None, sample=0):\n",
    "    if words == None:\n",
    "        if sample > 0:\n",
    "            words = np.random.choice(list(model.vocab.keys()), sample)\n",
    "        else:\n",
    "            words = [ word for word in model.vocab ]\n",
    "        \n",
    "    word_vectors = np.array([model[w] for w in words])\n",
    "\n",
    "    twodim = PCA().fit_transform(word_vectors)[:,:2]\n",
    "    \n",
    "    plt.figure(figsize=(6,6))\n",
    "    plt.scatter(twodim[:,0], twodim[:,1], edgecolors='k', c='r')\n",
    "    for word, (x,y) in zip(words, twodim):\n",
    "        plt.text(x+0.05, y+0.05, word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_pca_scatterplot(model, \n",
    "                        ['coffee', 'tea', 'beer', 'wine', 'brandy', 'rum', 'champagne', 'water',\n",
    "                         'spaghetti', 'borscht', 'hamburger', 'pizza', 'falafel', 'sushi', 'meatballs',\n",
    "                         'dog', 'horse', 'cat', 'monkey', 'parrot', 'koala', 'lizard',\n",
    "                         'frog', 'toad', 'monkey', 'ape', 'kangaroo', 'wombat', 'wolf',\n",
    "                         'france', 'germany', 'hungary', 'luxembourg', 'australia', 'fiji', 'china',\n",
    "                         'homework', 'assignment', 'problem', 'exam', 'test', 'class',\n",
    "                         'school', 'college', 'university', 'institute'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_pca_scatterplot(model, sample=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}