Stemming-words-from-multiple-languages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stemming (and Inverse Stemming) words from multiple languages\n",
    "\n",
    "For more information on the inner workings of the algorithm, refer to: \n",
    "http://snowball.tartarus.org/algorithms/french/stemmer.html\n",
    "\n",
    "The following content is derived from the quickstart guide [here](https://github.com/snowballstem/pystemmer/blob/master/docs/quickstart_python3.txt), which is [licensed](https://github.com/snowballstem/pystemmer/blob/master/LICENSE) under the MIT License and contains traces of the 3-Clause BSD License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip3 install pystemmer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian', 'italian', 'norwegian', 'porter', 'portuguese', 'romanian', 'russian', 'spanish', 'swedish', 'turkish']\n",
      "cod\n",
      "['cod', 'cod']\n",
      "['cod', b'cod']\n",
      "10000\n",
      "1000\n"
     ]
    }
   ],
   "source": [
    "# Quickstart\n",
    "# This is a very brief introduction to the use of PyStemmer.\n",
    "# First, import the library:\n",
    "\n",
    "import Stemmer\n",
    "\n",
    "# Just for show, we'll display a list of the available stemming algorithms:\n",
    "\n",
    "print(Stemmer.algorithms())\n",
    "# ['danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian', 'italian', 'norwegian', 'porter', 'portuguese', 'romanian', 'russian', 'spanish', 'swedish', 'turkish']\n",
    "\n",
    "# Now, we'll get an instance of the french stemming algorithm:\n",
    "stemmer = Stemmer.Stemmer('french')\n",
    "\n",
    "# Stem a single word:\n",
    "print(stemmer.stemWord('coder'))\n",
    "# cod\n",
    "\n",
    "# Stem a list of words:\n",
    "print(stemmer.stemWords(['coder', 'codera']))\n",
    "# ['cod', 'cod']\n",
    "\n",
    "# Strings which are supplied are assumed to be unicode.\n",
    "# We can use UTF-8 encoded input, too:\n",
    "print(stemmer.stemWords(['coder', b'codera']))\n",
    "# ['cod', b'cod']\n",
    "\n",
    "# Each instance of the stemming algorithms uses a cache to speed up processing of\n",
    "# common words.  By default, the cache holds 10000 words, but this may be\n",
    "# modified.  The cache may be disabled entirely by setting the cache size to 0:\n",
    "print(stemmer.maxCacheSize)\n",
    "# 10000\n",
    "\n",
    "stemmer.maxCacheSize = 1000\n",
    "print(stemmer.maxCacheSize)\n",
    "# 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to do Inverse Stemming?\n",
    "\n",
    "We'll need to be able to do the backward pass too to convert the topics from the LDA back to words. For more information, see how the above Stemmer class is wrapped into the code of the current [project](https://github.com/ArtificiAI/Multilingual-Latent-Dirichlet-Allocation-LDA). This is done in [lda_service/logic/stemmer.py](https://github.com/ArtificiAI/Multilingual-Latent-Dirichlet-Allocation-LDA/blob/master/lda_service/logic/stemmer.py)\n",
    "\n",
    "Quickly explained, inverse stemming can be done by keeping track of each original words that pointed to the stemmed version of that word, and their count. When doing the inverse stemming, the word with the top count can then be retrieved. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Original comments:\n",
    "['Un super-chat marche sur le trottoir',\n",
    " 'Les super-chats aiment ronronner',\n",
    " 'Les chats sont ronrons',\n",
    " 'Un super-chien aboie',\n",
    " 'Deux super-chiens',\n",
    " \"Combien de chiens sont en train d'aboyer?\"]\n",
    "\n",
    "# Original comments without stop words:\n",
    "['super-chat marche trottoir',\n",
    " 'super-chats aiment ronronner',\n",
    " 'chats ronrons',\n",
    " 'super-chien aboie',\n",
    " 'Deux super-chiens',\n",
    " 'Combien chiens train aboyer?']\n",
    "\n",
    "# Stemmed comments:\n",
    "['sup chat march trottoir',\n",
    " 'sup chat aiment ronron',\n",
    " 'chat ronron',\n",
    " 'sup chien aboi',\n",
    " 'deux sup chien',\n",
    " 'combien chien train aboi']\n",
    "\n",
    "# Custom stemmer's cache that was saved for the inverse pass later on which will need to choose the top corresponding words back from their counts:\n",
    "{'aboi': {'aboie': 1, 'aboyer': 1},\n",
    " 'aiment': {'aiment': 1},\n",
    " 'chat': {'chat': 1, 'chats': 2},\n",
    " 'chien': {'chien': 1, 'chiens': 2},\n",
    " 'combien': {'Combien': 1},\n",
    " 'deux': {'Deux': 1},\n",
    " 'march': {'marche': 1},\n",
    " 'ronron': {'ronronner': 1, 'ronrons': 1},\n",
    " 'sup': {'super': 4},\n",
    " 'train': {'train': 1},\n",
    " 'trottoir': {'trottoir': 1}}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}