Basic_NLP_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.5.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/todnewman/coe_training/blob/master/Basic_NLP_Tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CFOkVVi7RVP9"
      },
      "source": [
        "# Basic Natural Language Processing\n",
        "**Author**: W. Tod Newman\n",
        "\n",
        "**Updates**: New release\n",
        "\n",
        "## Learning Objectives\n",
        "\n",
        "\n",
        "*   Learn the basics of the Python Natural Language Toolkit\n",
        "*   Explore concepts of language processing: parts of speech, corpora, stemming, lemmatizing, etc.\n",
        "*   Overview simple neural network classification\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LTgfZdlwRVQJ"
      },
      "source": [
        "# About Python's Natural Language Toolkit (NLTK)\n",
        "\n",
        "NLTK is the most widely used NLP module for Python.  It comes with the Anaconda distribution, so it's very easy to start working once Anaconda is in place.  From the NLTK site:\n",
        "\n",
        "*NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.*\n",
        "\n",
        "NLTK has a very large set of open data that can be used to train the NLTK learner.  This NLTK data includes a lot of corpora, grammars, models and etc. Without NLTK Data, NLTK is not extremely useful. You can find the complete nltk data list here: http://nltk.org/nltk_data/\n",
        "\n",
        "The simplest way to install NLTK Data is run the Python interpreter and type the commands:\n",
        "'>>> import nltk\n",
        "'>>> nltk_download()\n",
        "\n",
        "This should open the NLTK Downloader window and you can select which modules to download.  The Brown University corpus is one of the most cited artifacts in the field of corpus linguistics.  We'll start by exploring how we can make use of it in our own text classification tasks.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tgd-6YXESPMg",
        "outputId": "42d88069-4a1c-44d3-aff9-e1a0a5ebc1d5"
      },
      "source": [
        "# use natural language toolkit\n",
        "import nltk\n",
        "\n",
        "#\n",
        "# Use the nltk downloader to download corpora, tools, and dictionaries\n",
        "#\n",
        "nltk.download('brown')\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')\n",
        "nltk.download('names')\n",
        "nltk.download('tagsets')\n",
        "nltk.download('maxent_ne_chunker')\n",
        "nltk.download('words')\n",
        "nltk.download('omw-1.4')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]   Package brown is already up-to-date!\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
            "[nltk_data]       date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n",
            "[nltk_data] Downloading package names to /root/nltk_data...\n",
            "[nltk_data]   Package names is already up-to-date!\n",
            "[nltk_data] Downloading package tagsets to /root/nltk_data...\n",
            "[nltk_data]   Package tagsets is already up-to-date!\n",
            "[nltk_data] Downloading package maxent_ne_chunker to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
            "[nltk_data] Downloading package words to /root/nltk_data...\n",
            "[nltk_data]   Package words is already up-to-date!\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n",
            "[nltk_data]   Package omw-1.4 is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l-IbJGPSVMz3"
      },
      "source": [
        "## Corpora\n",
        "\n",
        "### Brown University corpus\n",
        "\n",
        "The Brown Corpus was compiled in the 1960s by Henry Kučera and W. Nelson Francis at Brown University in Providence, Rhode Island, as a general corpus (text collection) in the field of corpus linguistics. It contains 500 samples of English-language text, totaling roughly one million words, compiled from works published in the United States in 1961.\n",
        "\n",
        "for more information: https://en.wikipedia.org/wiki/Brown_Corpus\n",
        "\n",
        "### What will we do here?\n",
        "\n",
        "We will load the corpus (which we downloaded with the nltk downloader above) and print the first 10 works along with their parts-of-speech (POS) tags.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XnZoAJPJRVQK",
        "outputId": "5ada9616-c7b2-4daa-d2b4-7313ca88b392"
      },
      "source": [
        "# Import the Brown University Corpus and print the first ten words\n",
        "from nltk.corpus import brown\n",
        "print (\"\\nPrinting the first 10 words in the Brown University Corpora:\\n\")\n",
        "print (brown.words()[0:10])\n",
        "print (\"\\nNow printing the POS tags for the first 10 words:\\n\")\n",
        "print (brown.tagged_words()[0:10])\n",
        "print (\"\\nNote the u'WORD' is the UNICODE UTF-8 encoding\")\n",
        "print (len(brown.words()))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Printing the first 10 words in the Brown University Corpora:\n",
            "\n",
            "['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of']\n",
            "\n",
            "Now printing the POS tags for the first 10 words:\n",
            "\n",
            "[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'), ('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation', 'NN'), ('of', 'IN')]\n",
            "\n",
            "Note the u'WORD' is the UNICODE UTF-8 encoding\n",
            "1161192\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uxXo7YHzRVQQ"
      },
      "source": [
        "## Overview of Sentence, Word, and Part of Speech Processing\n",
        "\n",
        "### What will we do here?\n",
        "\n",
        "We will bring in a large block of text (Wikipedia entry on Signal Processing) and do work to it.\n",
        "\n",
        "*  Tokenize the text into sentences\n",
        "*  Tokenize the sentences into words\n",
        "*  Tag the words with part of speech and demonstrate use cases for POS tags\n",
        "*  Show how to print out the \"key\" for NLTK POS tags"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bJwTCpw-RVQR",
        "outputId": "bc0b6624-932f-4ced-c19d-d46a29199830"
      },
      "source": [
        "from nltk import sent_tokenize, word_tokenize, pos_tag\n",
        "from nltk.chunk import ne_chunk\n",
        "\n",
        "text = \"\"\" The legendary Norman Lykes house, designed by architect Frank Lloyd Wright, will be up for auction on Oct. 16. It was the last residence designed by Wright, \n",
        "    who designed many iconic homes, including \"Fallingwater\" in Pennsylvania, as well as the Solomon Guggenheim Museum in New York City.  \n",
        "\"\"\"\n",
        "sents = sent_tokenize(text) # This will break the text into sentences.\n",
        "\n",
        "for i,s in enumerate(sents):\n",
        "    print(f\"Sentence{i}: {s}\\n\")\n",
        "\n",
        "print (\"*** The # of Sentences in the last example is %s\" % len(sents))\n",
        "\n",
        "tokens = word_tokenize(text)\n",
        "\n",
        "print (\"\\n*** Printing the tokens (words) out of the sentences\\n\")\n",
        "print (tokens)  # Breaks into tokens.  \n",
        "\n",
        "tagged_tokens = pos_tag(tokens)\n",
        "\n",
        "print (\"\\n*** Printing the POS TAGGED tokens (words) out of the sentences\\n\")\n",
        "\n",
        "print (tagged_tokens) # Breaks into (Token, POS Tag) tuples\n",
        "\n",
        "# Lets walk through the tuple and do some grouping\n",
        "\n",
        "print (\"\\n*** NOW we'll be printing only the tokens (words) that are Nouns\\n\")\n",
        "\n",
        "\n",
        "for token, pos_tag in tagged_tokens:\n",
        "    if pos_tag == 'NNP' or pos_tag == 'NN':\n",
        "        print(token)\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sentence0:  The legendary Norman Lykes house, designed by architect Frank Lloyd Wright, will be up for auction on Oct. 16.\n",
            "\n",
            "Sentence1: It was the last residence designed by Wright, \n",
            "    who designed many iconic homes, including \"Fallingwater\" in Pennsylvania, as well as the Solomon Guggenheim Museum in New York City.\n",
            "\n",
            "*** The # of Sentences in the last example is 2\n",
            "\n",
            "*** Printing the tokens (words) out of the sentences\n",
            "\n",
            "['The', 'legendary', 'Norman', 'Lykes', 'house', ',', 'designed', 'by', 'architect', 'Frank', 'Lloyd', 'Wright', ',', 'will', 'be', 'up', 'for', 'auction', 'on', 'Oct.', '16', '.', 'It', 'was', 'the', 'last', 'residence', 'designed', 'by', 'Wright', ',', 'who', 'designed', 'many', 'iconic', 'homes', ',', 'including', '``', 'Fallingwater', \"''\", 'in', 'Pennsylvania', ',', 'as', 'well', 'as', 'the', 'Solomon', 'Guggenheim', 'Museum', 'in', 'New', 'York', 'City', '.']\n",
            "\n",
            "*** Printing the POS TAGGED tokens (words) out of the sentences\n",
            "\n",
            "[('The', 'DT'), ('legendary', 'JJ'), ('Norman', 'NNP'), ('Lykes', 'NNP'), ('house', 'NN'), (',', ','), ('designed', 'VBN'), ('by', 'IN'), ('architect', 'NN'), ('Frank', 'NNP'), ('Lloyd', 'NNP'), ('Wright', 'NNP'), (',', ','), ('will', 'MD'), ('be', 'VB'), ('up', 'RP'), ('for', 'IN'), ('auction', 'NN'), ('on', 'IN'), ('Oct.', 'NNP'), ('16', 'CD'), ('.', '.'), ('It', 'PRP'), ('was', 'VBD'), ('the', 'DT'), ('last', 'JJ'), ('residence', 'NN'), ('designed', 'VBN'), ('by', 'IN'), ('Wright', 'NNP'), (',', ','), ('who', 'WP'), ('designed', 'VBD'), ('many', 'JJ'), ('iconic', 'JJ'), ('homes', 'NNS'), (',', ','), ('including', 'VBG'), ('``', '``'), ('Fallingwater', 'NNP'), (\"''\", \"''\"), ('in', 'IN'), ('Pennsylvania', 'NNP'), (',', ','), ('as', 'RB'), ('well', 'RB'), ('as', 'IN'), ('the', 'DT'), ('Solomon', 'NNP'), ('Guggenheim', 'NNP'), ('Museum', 'NNP'), ('in', 'IN'), ('New', 'NNP'), ('York', 'NNP'), ('City', 'NNP'), ('.', '.')]\n",
            "\n",
            "*** NOW we'll be printing only the tokens (words) that are Nouns\n",
            "\n",
            "Norman\n",
            "Lykes\n",
            "house\n",
            "architect\n",
            "Frank\n",
            "Lloyd\n",
            "Wright\n",
            "auction\n",
            "Oct.\n",
            "residence\n",
            "Wright\n",
            "Fallingwater\n",
            "Pennsylvania\n",
            "Solomon\n",
            "Guggenheim\n",
            "Museum\n",
            "New\n",
            "York\n",
            "City\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1YOdjgW5Tiiz",
        "outputId": "a0464f79-3718-4ecf-9ebf-29ea82860437"
      },
      "source": [
        "print('Here\\'s how we can figure out what these Part of Speech Tags mean!')\n",
        "print('__________________________________________________________________')\n",
        "\n",
        "# Print out pos_tag as a unique list first - TODO\n",
        "for token, pos_tag in tagged_tokens:\n",
        "    nltk.help.upenn_tagset(pos_tag)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Here's how we can figure out what these Part of Speech Tags mean!\n",
            "__________________________________________________________________\n",
            "DT: determiner\n",
            "    all an another any both del each either every half la many much nary\n",
            "    neither no some such that the them these this those\n",
            "JJ: adjective or numeral, ordinal\n",
            "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
            "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
            "    multilingual multi-disciplinary ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NN: noun, common, singular or mass\n",
            "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
            "    investment slide humour falloff slick wind hyena override subhumanity\n",
            "    machinist ...\n",
            ",: comma\n",
            "    ,\n",
            "VBN: verb, past participle\n",
            "    multihulled dilapidated aerosolized chaired languished panelized used\n",
            "    experimented flourished imitated reunifed factored condensed sheared\n",
            "    unsettled primed dubbed desired ...\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NN: noun, common, singular or mass\n",
            "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
            "    investment slide humour falloff slick wind hyena override subhumanity\n",
            "    machinist ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            ",: comma\n",
            "    ,\n",
            "MD: modal auxiliary\n",
            "    can cannot could couldn't dare may might must need ought shall should\n",
            "    shouldn't will would\n",
            "VB: verb, base form\n",
            "    ask assemble assess assign assume atone attention avoid bake balkanize\n",
            "    bank begin behold believe bend benefit bevel beware bless boil bomb\n",
            "    boost brace break bring broil brush build ...\n",
            "RP: particle\n",
            "    aboard about across along apart around aside at away back before behind\n",
            "    by crop down ever fast for forth from go high i.e. in into just later\n",
            "    low more off on open out over per pie raising start teeth that through\n",
            "    under unto up up-pp upon whole with you\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NN: noun, common, singular or mass\n",
            "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
            "    investment slide humour falloff slick wind hyena override subhumanity\n",
            "    machinist ...\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "CD: numeral, cardinal\n",
            "    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-\n",
            "    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025\n",
            "    fifteen 271,124 dozen quintillion DM2,000 ...\n",
            ".: sentence terminator\n",
            "    . ! ?\n",
            "PRP: pronoun, personal\n",
            "    hers herself him himself hisself it itself me myself one oneself ours\n",
            "    ourselves ownself self she thee theirs them themselves they thou thy us\n",
            "VBD: verb, past tense\n",
            "    dipped pleaded swiped regummed soaked tidied convened halted registered\n",
            "    cushioned exacted snubbed strode aimed adopted belied figgered\n",
            "    speculated wore appreciated contemplated ...\n",
            "DT: determiner\n",
            "    all an another any both del each either every half la many much nary\n",
            "    neither no some such that the them these this those\n",
            "JJ: adjective or numeral, ordinal\n",
            "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
            "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
            "    multilingual multi-disciplinary ...\n",
            "NN: noun, common, singular or mass\n",
            "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
            "    investment slide humour falloff slick wind hyena override subhumanity\n",
            "    machinist ...\n",
            "VBN: verb, past participle\n",
            "    multihulled dilapidated aerosolized chaired languished panelized used\n",
            "    experimented flourished imitated reunifed factored condensed sheared\n",
            "    unsettled primed dubbed desired ...\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            ",: comma\n",
            "    ,\n",
            "WP: WH-pronoun\n",
            "    that what whatever whatsoever which who whom whosoever\n",
            "VBD: verb, past tense\n",
            "    dipped pleaded swiped regummed soaked tidied convened halted registered\n",
            "    cushioned exacted snubbed strode aimed adopted belied figgered\n",
            "    speculated wore appreciated contemplated ...\n",
            "JJ: adjective or numeral, ordinal\n",
            "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
            "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
            "    multilingual multi-disciplinary ...\n",
            "JJ: adjective or numeral, ordinal\n",
            "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
            "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
            "    multilingual multi-disciplinary ...\n",
            "NNS: noun, common, plural\n",
            "    undergraduates scotches bric-a-brac products bodyguards facets coasts\n",
            "    divestitures storehouses designs clubs fragrances averages\n",
            "    subjectivists apprehensions muses factory-jobs ...\n",
            ",: comma\n",
            "    ,\n",
            "VBG: verb, present participle or gerund\n",
            "    telegraphing stirring focusing angering judging stalling lactating\n",
            "    hankerin' alleging veering capping approaching traveling besieging\n",
            "    encrypting interrupting erasing wincing ...\n",
            "``: opening quotation mark\n",
            "    ` ``\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "'': closing quotation mark\n",
            "    ' ''\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            ",: comma\n",
            "    ,\n",
            "RB: adverb\n",
            "    occasionally unabatingly maddeningly adventurously professedly\n",
            "    stirringly prominently technologically magisterially predominately\n",
            "    swiftly fiscally pitilessly ...\n",
            "RB: adverb\n",
            "    occasionally unabatingly maddeningly adventurously professedly\n",
            "    stirringly prominently technologically magisterially predominately\n",
            "    swiftly fiscally pitilessly ...\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "DT: determiner\n",
            "    all an another any both del each either every half la many much nary\n",
            "    neither no some such that the them these this those\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "IN: preposition or conjunction, subordinating\n",
            "    astride among uppon whether out inside pro despite on by throughout\n",
            "    below within for towards near behind atop around if like until below\n",
            "    next into if beside ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            "NNP: noun, proper, singular\n",
            "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
            "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
            "    Shannon A.K.C. Meltex Liverpool ...\n",
            ".: sentence terminator\n",
            "    . ! ?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IQcVAviERG9I"
      },
      "source": [
        "## Named Entity Recognition (NER)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SctBv5fiQYcY",
        "outputId": "3aca328d-424d-404c-d07d-6df7e72357e7"
      },
      "source": [
        "# Reusing our tagged_tokens from the above block.  Now we'll do NER on it.\n",
        "print('\\n*** First we will print out the entire NER-tagged tree.\\n')\n",
        "ne_tree = ne_chunk(tagged_tokens)\n",
        "print(ne_tree)\n",
        "print ('\\n*** Extracting the NER Labels below ***\\n')\n",
        "for chunk in ne_chunk(tagged_tokens):\n",
        "    if hasattr(chunk, 'label'):\n",
        "        print(chunk.label(), ' '.join(c[0] for c in chunk))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "*** First we will print out the entire NER-tagged tree.\n",
            "\n",
            "(S\n",
            "  The/DT\n",
            "  legendary/JJ\n",
            "  (PERSON Norman/NNP Lykes/NNP)\n",
            "  house/NN\n",
            "  ,/,\n",
            "  designed/VBN\n",
            "  by/IN\n",
            "  architect/NN\n",
            "  (PERSON Frank/NNP Lloyd/NNP Wright/NNP)\n",
            "  ,/,\n",
            "  will/MD\n",
            "  be/VB\n",
            "  up/RP\n",
            "  for/IN\n",
            "  auction/NN\n",
            "  on/IN\n",
            "  Oct./NNP\n",
            "  16/CD\n",
            "  ./.\n",
            "  It/PRP\n",
            "  was/VBD\n",
            "  the/DT\n",
            "  last/JJ\n",
            "  residence/NN\n",
            "  designed/VBN\n",
            "  by/IN\n",
            "  (PERSON Wright/NNP)\n",
            "  ,/,\n",
            "  who/WP\n",
            "  designed/VBD\n",
            "  many/JJ\n",
            "  iconic/JJ\n",
            "  homes/NNS\n",
            "  ,/,\n",
            "  including/VBG\n",
            "  ``/``\n",
            "  Fallingwater/NNP\n",
            "  ''/''\n",
            "  in/IN\n",
            "  (GPE Pennsylvania/NNP)\n",
            "  ,/,\n",
            "  as/RB\n",
            "  well/RB\n",
            "  as/IN\n",
            "  the/DT\n",
            "  (ORGANIZATION Solomon/NNP Guggenheim/NNP Museum/NNP)\n",
            "  in/IN\n",
            "  (GPE New/NNP York/NNP City/NNP)\n",
            "  ./.)\n",
            "\n",
            "*** Extracting the NER Labels below ***\n",
            "\n",
            "PERSON Norman Lykes\n",
            "PERSON Frank Lloyd Wright\n",
            "PERSON Wright\n",
            "GPE Pennsylvania\n",
            "ORGANIZATION Solomon Guggenheim Museum\n",
            "GPE New York City\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hfpitEE9RVQX"
      },
      "source": [
        "# Stemming and Lemmatization (what???)\n",
        "\n",
        "Stemming and Lemmatization are the basic text processing methods for English text. The goal of both stemming and lemmatization is to *reduce inflectional forms of a word to a common base form*. Here is the definition from wikipedia for stemming and lemmatization:\n",
        "\n",
        "In linguistic morphology (i.e., the structure of words) and information retrieval, **stemming** is the process for reducing inflected (or sometimes derived) words to their stem, base or root form\n",
        "\n",
        "**Lemmatization** in linguistics, is the process of grouping together the different inflected forms of a word so they can be analysed as a single item.\n",
        "\n",
        "For English, which has a fairly simple morphology, this task is generally simple.  For other languages (Turkish is a good example) it is absolutely necessary.\n",
        "\n",
        "### What will we do here?\n",
        "We'll instantiate a Lancaster Stemmer and demonstrate what a stemmer does.  Then we will instantiate a Lemmatizer and demonstrate what a lemmatizer does."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zpuf7kbORVQY",
        "outputId": "bcd7ccd8-2f6a-43ea-e67f-320a8daf3a72"
      },
      "source": [
        "from nltk.stem.lancaster import LancasterStemmer\n",
        "# word stemmer\n",
        "stemmer = LancasterStemmer()\n",
        "print (stemmer.stem('quickly'))\n",
        "print (stemmer.stem('challenging'))\n",
        "print (stemmer.stem('challenges'))\n",
        "print (stemmer.stem('wolves'))\n",
        "print (stemmer.stem('centre'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "quick\n",
            "challeng\n",
            "challeng\n",
            "wolv\n",
            "cent\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TRJD9id1RVQd",
        "outputId": "41cdc30a-0f7e-4e65-cc02-052258a982ee"
      },
      "source": [
        "from nltk.stem import WordNetLemmatizer\n",
        "wordnet_lemmatizer = WordNetLemmatizer()\n",
        "print (wordnet_lemmatizer.lemmatize('dogs'))\n",
        "print (wordnet_lemmatizer.lemmatize('wolves'))\n",
        "# Note that the default POS for lemmatize is Noun.  Lets see how it handles verbs.\n",
        "print (wordnet_lemmatizer.lemmatize('does', pos='n'))\n",
        "print (wordnet_lemmatizer.lemmatize('centre'))\n",
        "print (wordnet_lemmatizer.lemmatize('challenging'))\n",
        "print (wordnet_lemmatizer.lemmatize('challenges'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dog\n",
            "wolf\n",
            "doe\n",
            "centre\n",
            "challenging\n",
            "challenge\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hN5QNiFzRVQl"
      },
      "source": [
        "# What can we do with these NLP techniques??"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hDtTPEhiRVQm"
      },
      "source": [
        "## Toy Example: Gender-based name classifier\n",
        "\n",
        "Use the NLTK Name corpus to train a Gender Identification classifier.  This approach determines the likelihood that a name is associated with the 'male name' section of the corpus or the 'female name' section.  In this case, this is a lightweight form of supervised machine learning.\n",
        "\n",
        "This approach is the basis for more complex classifiers that I have developed.\n",
        "\n",
        "### What will we do here?\n",
        "\n",
        "we're going to take the male and female names from the NLTK names function, shuffle these names, and then"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SAATLVpMRVQo"
      },
      "source": [
        "# Grab names out of the nltk name corpus.\n",
        "\n",
        "from nltk.corpus import names\n",
        "import random\n",
        "\n",
        "# Look for the likelihood that a name is contained in the male or the female name corpus.\n",
        "classified_names = ([(name, 'male') for name in names.words('male.txt')] \n",
        "         + [(name, 'female') for name in names.words('female.txt')])\n",
        "\n",
        "random.shuffle(classified_names)\n",
        "\n",
        "print (\"\\nLets output our simple Bayesian name-gender classifications:\")\n",
        "classified_names[0:17]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MBa5drRiRVQw"
      },
      "source": [
        "## Improve the Name Classifier and Return Scores\n",
        "\n",
        "Using some built-in utilities from NLTK, we will train a classifier (using Scikit-learn, another great Python module) to classify names that were held out from the training set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XWgk_N80RVQx"
      },
      "source": [
        "from nltk.classify.scikitlearn import SklearnClassifier\n",
        "import numpy as np\n",
        "from nltk.classify.util import names_demo, binary_names_demo_features\n",
        "try:\n",
        "    from sklearn.linear_model.sparse import LogisticRegression\n",
        "except ImportError:     # separate sparse LR to be removed in 0.12\n",
        "    from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.naive_bayes import BernoulliNB\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "# Classify names using nltk built in names demo\n",
        "\n",
        "print(\"\\nClassify names using scikit-learn Naive Bayes:\\n\")\n",
        "names_demo(SklearnClassifier(BernoulliNB(binarize=False), dtype=bool).train,\n",
        "               features=binary_names_demo_features)\n",
        "\n",
        "print(\"\\nClassify names using scikit-learn logistic regression:\\n\")\n",
        "names_demo(SklearnClassifier(LogisticRegression(), dtype=np.float64).train,\n",
        "               features=binary_names_demo_features)\n",
        "\n",
        "print(\"\\nClassify names using scikit-learn Random Forest Classifier:\\n\")\n",
        "names_demo(SklearnClassifier(RandomForestClassifier(), dtype=np.float64).train,\n",
        "               features=binary_names_demo_features)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PyOUFHoSn7rf"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}