11_bayes/20_newsgroups.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The 20 newsgroups text dataset (sklearn)\n",
    "\n",
    "\n",
    "The **[20 newsgroups dataset](http://qwone.com/~jason/20Newsgroups/)** is a common ML dataset for comparison and\n",
    "comprises around 18,000 newsgroups posts on 20 topics. The `sklearn` package offers functionality to easily download, parse and analyze this dataset. The set has already been split into a training and a test set, based upon a messages posted before and after a specific date.\n",
    "\n",
    "\n",
    "## Loading the data\n",
    "\n",
    "Let's download the dataset (~14MB) to disk. It will be saved in your `~/scikit_learn_data/20news_home` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "newsgroups_train = fetch_20newsgroups(subset='train')\n",
    "print newsgroups_train.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We just called `fetch_20newsgroups`, which returns a list of the raw texts. You could also call `fetch_20newsgroups_vectorized`, which returns ready-to-use features for modelling, i.e., it is not necessary to use such feature extractors as `CountVectorizer`.\n",
    "\n",
    "The real data lies in the ``filenames`` and ``target`` attributes. The target attribute is the integer index of the category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11314,) (11314,)\n",
      "[ '/Users/ruben/scikit_learn_data/20news_home/20news-bydate-train/rec.autos/102994'\n",
      " '/Users/ruben/scikit_learn_data/20news_home/20news-bydate-train/comp.sys.mac.hardware/51861'\n",
      " '/Users/ruben/scikit_learn_data/20news_home/20news-bydate-train/comp.sys.mac.hardware/51879'\n",
      " '/Users/ruben/scikit_learn_data/20news_home/20news-bydate-train/comp.graphics/38242'\n",
      " '/Users/ruben/scikit_learn_data/20news_home/20news-bydate-train/sci.space/60880']\n",
      "[ 7  4  4  1 14]\n",
      "['rec.autos', 'comp.sys.mac.hardware', 'comp.sys.mac.hardware', 'comp.graphics', 'sci.space']\n"
     ]
    }
   ],
   "source": [
    "print newsgroups_train.filenames.shape, newsgroups_train.target.shape\n",
    "print newsgroups_train.filenames[:5]\n",
    "print newsgroups_train.target[:5]\n",
    "print [newsgroups_train.target_names[no] for no in newsgroups_train.target[:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The actual raw text can be found in the `data` attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u\"From: lerxst@wam.umd.edu (where's my thing)\\nSubject: WHAT car is this!?\\nNntp-Posting-Host: rac3.wam.umd.edu\\nOrganization: University of Maryland, College Park\\nLines: 15\\n\\n I was wondering if anyone out there could enlighten me on this car I saw\\nthe other day. It was a 2-door sports car, looked to be from the late 60s/\\nearly 70s. It was called a Bricklin. The doors were really small. In addition,\\nthe front bumper was separate from the rest of the body. This is \\nall I know. If anyone can tellme a model name, engine specs, years\\nof production, where this car is made, history, or whatever info you\\nhave on this funky looking car, please e-mail.\\n\\nThanks,\\n- IL\\n   ---- brought to you by your neighborhood Lerxst ----\\n\\n\\n\\n\\n\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_train.data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to load only a sub-selection of the categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']\n",
      "(2034,) (2034,)\n"
     ]
    }
   ],
   "source": [
    "categories = ['alt.atheism', 'sci.space', 'talk.religion.misc', 'comp.graphics']\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', categories=categories)\n",
    "print newsgroups_train.target_names\n",
    "print newsgroups_train.filenames.shape, newsgroups_train.target.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2034"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(newsgroups_train.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting text to vectors\n",
    "\n",
    "`sklearn` comes with many built-in feature extraction and manipulation tools. For dealing with text data, there is the  `sklearn.feature_extraction.text` module, which contains the **`CountVectorizer`**, which we've already seen, and the `TfidfVectorizer`, which we will discuss later on.\n",
    "\n",
    "### `CountVectorizer`\n",
    "\n",
    "This class transforms an array-like (list, dataframe column, array) of strings into a matrix where each column represents a token (word or phrase) and each row represents the sample.\n",
    "\n",
    "For example, if we had a two-element array [\"Hello good day\", \"Good day to you\"], we would create a matrix with 2 rows (one for each sample) and 5 columns (one for every unique word). The matrix would look like this:\n",
    "\n",
    "hello|good|day|to|you\n",
    "--|--|--|--|--\n",
    "1|1|1|0|0\n",
    "0|1|1|1|1\n",
    "\n",
    "The `CountVectorizer` (and most feature extraction methods in sklearn) follows a very simple interface:\n",
    "- `fit` takes a dataset and learns the features it's trying to extract. In this case that means that the algorithm learns the vocabulary of all samples\n",
    "- `transform` takes a dataset and produces the matrix as described above, based on the vocabulary (or feature elements) it learned.\n",
    "- `fit_transform` combines the two steps at once.\n",
    "\n",
    "For example, you may want to fit a vocabulary to a training set, transform the training set to train a model and then continually transform any new incoming examples you want to classify. You will generally only perform the fit step once but the transform step many times for any new datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2034x34118 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 323433 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "cv = CountVectorizer()\n",
    "X_train = cv.fit_transform(newsgroups_train.data)\n",
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the returned matrix is a sparse matrix, i.e., with almost only zeros except for a few entries. This is stored differently than an ordinary array to save memory space.\n",
    "\n",
    "Instead of word counts, we can set the `ngram_range` to include sequences of words as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "cv = CountVectorizer(ngram_range=(1,3))\n",
    "X_train = cv.fit_transform(newsgroups_train.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `TfidfVectorizer`\n",
    "\n",
    "The `Tfidf` stands for _Term Frequency - Inverse Document Frequency_, or TF-IDF representation.\n",
    "\n",
    "- The _Term Frequency_ is simply the number of times that a word appear in a sample. This is equivalent to the `CountVectorizer` features, and is our most basic representation of text.\n",
    "- The _Document Frequency_ is the percentage of samples that a particular word appears in.  Note that 'document' means sample here.  For example, you could assume `the` appears in 100% of samples, while words like `Syria` would have low document frequency.  The _Inverse Document Frequency_ is simply 1 / Document Frequency (although frequently the log is also taken). \n",
    "\n",
    "The TF-IDF representation computes the Term Frequency / Document Frequency ratio.  Words that appear a lot in a specific sample, or appear in very few other samples, will get a high score.  The intuition is that these words are somehow very characteristic for the specific sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidfv = TfidfVectorizer(stop_words='english', ngram_range=(1,3))\n",
    "X_train = tfidfv.fit_transform(newsgroups_train.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes\n",
    "\n",
    "Let's combine the above text representations with a Naive Bayes model. (Please see the other course materials for an extensive explanation of Bayes' Theorem and the Naive Bayes algorithm.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.91421569,  0.93382353,  0.93120393,  0.92364532,  0.92839506])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "\n",
    "model = MultinomialNB()\n",
    "cross_val_score(model, X_train, newsgroups_train.target, cv=5)\n",
    "# model.fit(X_train, newsgroups_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now try with the `CountVectorizer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.93382353,  0.9754902 ,  0.96314496,  0.95566502,  0.97037037])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv = CountVectorizer(stop_words='english', ngram_range=(1,3))\n",
    "X_train = cv.fit_transform(newsgroups_train.data)\n",
    "cross_val_score(model, X_train, newsgroups_train.target, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fit the model and analyze the coeficients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, newsgroups_train.target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc']\n",
      "[0 1 2 3]\n",
      "[ 480.  584.  593.  377.]\n",
      "[-1.44397347 -1.24785859 -1.23256518 -1.68551439]\n"
     ]
    }
   ],
   "source": [
    "print newsgroups_train.target_names\n",
    "print model.classes_\n",
    "print model.class_count_\n",
    "print model.class_log_prior_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each class, we have the relative log probabilities per feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  3.   0.   0. ...,   0.   0.   0.]\n",
      " [ 34.   2.   1. ...,   0.   0.   0.]\n",
      " [ 42.   1.   1. ...,   0.   0.   0.]\n",
      " [  4.   0.   0. ...,   1.   1.   1.]]\n",
      "[[-12.14041884 -13.5267132  -13.5267132  ..., -13.5267132  -13.5267132\n",
      "  -13.5267132 ]\n",
      " [ -9.97298563 -12.4297214  -12.83518651 ..., -13.52833369 -13.52833369\n",
      "  -13.52833369]\n",
      " [ -9.86476123 -12.93281416 -12.93281416 ..., -13.62596134 -13.62596134\n",
      "  -13.62596134]\n",
      " [-11.85321754 -13.46265545 -13.46265545 ..., -12.76950827 -12.76950827\n",
      "  -12.76950827]]\n"
     ]
    }
   ],
   "source": [
    "print model.feature_count_\n",
    "print model.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 498763)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coef_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "498763"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cv.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "coef = pd.DataFrame(model.coef_, columns=cv.get_feature_names(), index=newsgroups_train.target_names).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alt.atheism           (+) writes, people, god, edu\n",
      "                      (-) know good book, noticed article messianic, noticed article, noticeably worse viewers\n",
      "comp.graphics         (+) graphics, subject, lines, edu\n",
      "                      (-) ºnd sun eclipsed, launch coverage, launch countdown abort, launch countdown\n",
      "sci.space             (+) subject, nasa, edu, space\n",
      "                      (-) know good book, multiverse feel, multiverse, multitudes commoners said\n",
      "talk.religion.misc    (+) subject, god, com, edu\n",
      "                      (-) know good book, logically conclusion example, logically conclusion, talking natural\n"
     ]
    }
   ],
   "source": [
    "top = 4\n",
    "for newsgroup in coef:\n",
    "    s = coef[[newsgroup]].sort(newsgroup)\n",
    "    print \"%-20s  (+) %s\" % (newsgroup, \", \".join(s.iloc[-top:].index))\n",
    "    print \"%-20s  (-) %s\" % (\"\", \", \".join(s.iloc[:top].index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading \n",
    "\n",
    "- [20 Newsgroups Dataset](http://qwone.com/~jason/20Newsgroups/)\n",
    "- [sklearn's 20 newsgroups](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html)\n",
    "- [sklearn's `CountVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n",
    "- [sklearn's `TfidfVectorizer`](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html)\n",
    "- [sklearn's `MultinomialNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}