Descriptive-statistics-length.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Copyright 2021 Andrew M. Olney and made available under [CC BY-SA](https://creativecommons.org/licenses/by-sa/4.0) for text and [Apache-2.0](http://www.apache.org/licenses/LICENSE-2.0) for code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Descriptive statistics: length-based metrics\n",
    "\n",
    "Text affords different opportunities for descriptive statistics than typical variables.\n",
    "\n",
    "Why not analyze text like any other variable?\n",
    "Suppose we consider the primary unit of analysis to be the word.\n",
    "Then we could consider characterizing words as categorical (also known as nominal) variables.\n",
    "As a categorical variable, we might expect a word variable to have millions of levels if we include dictionary words (about 250 thousand words in a college dictionary) as well as proper nouns (Wikipedia has over 4 million entries).\n",
    "That's just for a single word.\n",
    "Since words typically don't occur in isolation but rather in the company of other words, we are forced to consider using a sequence of words as a categorical variable.\n",
    "However, we clearly run into sparsity issues very quickly (consider sequences of 2 words could have in the ballpark of 4 million-squared levels) - we will never observe many levels of our variable of interest.\n",
    "\n",
    "For these reasons, it is not useful or practical to try to directly calculate descriptive statistics as we would for typical variables. \n",
    "Instead, we typically **transform the text into numeric form** using some kind of metric.\n",
    "<!-- or we **transform the text into a distribution.** -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What you will learn\n",
    "\n",
    "You will learn about text-oriented descriptive statistics based on text length.\n",
    "  \n",
    "We will cover:\n",
    "\n",
    "- Length-based metrics\n",
    "    - text length\n",
    "    - text length in words\n",
    "    - text length in sentences\n",
    "    - average word/sentence length\n",
    "    - readability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use length-based metrics\n",
    "\n",
    "Descriptive statistics are helpful for exploring the data and considering other potential analyses.\n",
    "The transformations on text that we discuss may also be useful as features in later modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Length-based metrics\n",
    "\n",
    "We'll continue with the built-in `gutenberg` corpus from NLTK.\n",
    "Let's import the `gutenberg` corpus:\n",
    "\n",
    "- from `nltk.corpus` import `gutenberg`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import gutenberg\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"importFrom\" id=\"XD9YVa/m9vX;ax-@^}(K\" x=\"16\" y=\"64\"><field name=\"libraryName\">nltk.corpus</field><field name=\"libraryAlias\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text length in characters\n",
    "\n",
    "When we consider text length as a metric, we can clearly consider it on multiple scales.\n",
    "At the coarsest level we can consider the length of the entire text in characters.\n",
    "We've previously seen how to use NLTK to get a list of texts in a corpus and the raw form of the text (i.e. string, or sequence of characters).\n",
    "Let's combine those operations and also get the length of the raw text:\n",
    "\n",
    "- Set rawLengths to a list with one element containing\n",
    "    - for each item `i` in list with `gutenberg` do `fileids` (see LOOPS)\n",
    "        - yield length of (see TEXT) with `gutenberg` do `raw` using `i`\n",
    "- Display rawLengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[887071, 466292, 673022, 4332554, 38153, 249439, 84663, 144395, 457450, 406629, 320525, 935158, 1242990, 468220, 112310, 162881, 100351, 711215]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rawLengths = [(len(gutenberg.raw(i))) for i in (gutenberg.fileids())]\n",
    "\n",
    "rawLengths\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"AO?GdNQ:*92|iDWB.)YR\">rawLengths</variable><variable id=\"LZ#}.J~9XYczA[nu4?|Q\">i</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"ZR=zxJ`,Px9$cDQ}u?!N\" x=\"29\" y=\"316\"><field name=\"VAR\" id=\"AO?GdNQ:*92|iDWB.)YR\">rawLengths</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"8eO0[B%0~mtEmLX:+=dw\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"qHXxT3|WBM:kMw~muDTN\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">fileids</field><data>gutenberg:fileids</data></block></value><value name=\"YIELD\"><block type=\"text_length\" id=\"C[M$vurF8L;mQ|g`!((h\"><value name=\"VALUE\"><shadow type=\"text\" id=\"OWYJxoN6q~fXVN0HS0QW\"><field name=\"TEXT\">abc</field></shadow><block type=\"varDoMethod\" id=\"#IalxaHkKH5=q1de@8Ar\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">raw</field><data>gutenberg:raw</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"u[Lug07Y.B({Q]G-6du|\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\")DU}BKs8923`-}.@eA)Q\" x=\"8\" y=\"402\"><field name=\"VAR\" id=\"AO?GdNQ:*92|iDWB.)YR\">rawLengths</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each one of these is the length (in characters) of a book in the `gutenberg` corpus."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text length in words\n",
    "\n",
    "Let's repeat this operation but retrieve words instead of text length.\n",
    "Since we've already covered how to do word tokenization manually, we'll use the built in `gutenberg` tokenization to focus on the new concept:\n",
    "\n",
    "- Set wordLengths to a list with one element containing\n",
    "    - for each item `i` in list with `gutenberg` do `fileids` (see LOOPS)\n",
    "        - yield length of (see LISTS) with `gutenberg` do `words` using `i`\n",
    "- Display wordLengths\n",
    "\n",
    "Note the only changes are `words` instead of `raw` and `length of` from LISTS instead of TEXT.\n",
    "This is because while `raw` gives us one big string, `words` gives us a list of words.\n",
    "However, the logic of the loop is the same (we sometimes call this a traversal, because we are traversing the data to calculate something)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[192427, 98171, 141576, 1010654, 8354, 55563, 18963, 34110, 96996, 86063, 69213, 210663, 260819, 96825, 25833, 37360, 23140, 154883]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wordLengths = [(len(gutenberg.words(i))) for i in (gutenberg.fileids())]\n",
    "\n",
    "wordLengths\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</variable><variable id=\"LZ#}.J~9XYczA[nu4?|Q\">i</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"SKZpRLP{wcl/g*{^W3WV\" x=\"4\" y=\"319\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"8eO0[B%0~mtEmLX:+=dw\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"qHXxT3|WBM:kMw~muDTN\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">fileids</field><data>gutenberg:fileids</data></block></value><value name=\"YIELD\"><block type=\"lists_length\" id=\"b5(0SiwR87=9]SU8IBvy\"><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"#IalxaHkKH5=q1de@8Ar\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">words</field><data>gutenberg:words</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"u[Lug07Y.B({Q]G-6du|\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"aJWMl/VaGmo=-c|`$nxp\" x=\"8\" y=\"436\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the number or words is much shorter than the number of characters.\n",
    "We will return to this shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text length in sentences\n",
    "\n",
    "Let's look at the same text, but this time in sentences:\n",
    "\n",
    "- Set sentenceLengths to a list with one element containing\n",
    "    - for each item `i` in list with `gutenberg` do `fileids` (see LOOPS)\n",
    "        - yield length of (see LISTS) with `gutenberg` do `sents` using `i`\n",
    "- Display sentenceLengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[7752, 3747, 4999, 30103, 438, 2863, 1054, 1703, 4779, 3806, 3742, 10230, 10059, 1851, 2163, 3106, 1907, 4250]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentenceLengths = [(len(gutenberg.sents(i))) for i in (gutenberg.fileids())]\n",
    "\n",
    "sentenceLengths\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</variable><variable id=\"LZ#}.J~9XYczA[nu4?|Q\">i</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable></variables><block type=\"variables_set\" id=\"SKZpRLP{wcl/g*{^W3WV\" x=\"4\" y=\"319\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field><value name=\"VALUE\"><block type=\"lists_create_with\" id=\"8eO0[B%0~mtEmLX:+=dw\"><mutation items=\"1\"></mutation><value name=\"ADD0\"><block type=\"comprehensionForEach\" id=\"qHXxT3|WBM:kMw~muDTN\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field><value name=\"LIST\"><block type=\"varDoMethod\" id=\"tLR@!_zful,@toy1e3E(\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">fileids</field><data>gutenberg:fileids</data></block></value><value name=\"YIELD\"><block type=\"lists_length\" id=\"b5(0SiwR87=9]SU8IBvy\"><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"#IalxaHkKH5=q1de@8Ar\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">sents</field><data>gutenberg:sents</data><value name=\"INPUT\"><block type=\"variables_get\" id=\"u[Lug07Y.B({Q]G-6du|\"><field name=\"VAR\" id=\"LZ#}.J~9XYczA[nu4?|Q\">i</field></block></value></block></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"aJWMl/VaGmo=-c|`$nxp\" x=\"8\" y=\"436\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again, the number of sentences is quite a bit lower than the number of words, as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average word/sentence length\n",
    "\n",
    "There are at least two ways we could calculate average word length using what we've covered so far.\n",
    "We could to a traversal of words and calculate the average word length for each text.\n",
    "Alternatively, we could use the values we've already compute and divide.\n",
    "The same applies to sentence length.\n",
    "\n",
    "Conceptually what we want to do is perform operations with the first element of `wordLengths`, `sentenceLengths`, and `rawLengths`, the second element of these lists, and so on.\n",
    "Rather than create our own data structures for facilitating these operations, let's put these lists into a dataframe.\n",
    "Start by importing `pandas`:\n",
    "\n",
    "- import `pandas` as `pd`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"_V`RIwppcpbRKT:m6^qH\">pd</variable></variables><block type=\"importAs\" id=\"Gy5)p-`[BHUUWE}k1DeL\" x=\"16\" y=\"10\"><field name=\"libraryName\">pandas</field><field name=\"libraryAlias\" id=\"_V`RIwppcpbRKT:m6^qH\">pd</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to create a dataframe with these lists using the `zip` operator in LISTS:\n",
    "\n",
    "- Set `dataframe` to with `pd` create `DataFrame` using a list containing\n",
    "    - `zip` a list containing\n",
    "        - with `gutenberg` do `fileids`,`wordLengths`, `sentenceLengths`, `rawLengths`\n",
    "    - freestyle `columns=['corpus','words','sentences','characters']`\n",
    "- Display `dataframe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>words</th>\n",
       "      <th>sentences</th>\n",
       "      <th>characters</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>austen-emma.txt</td>\n",
       "      <td>192427</td>\n",
       "      <td>7752</td>\n",
       "      <td>887071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>austen-persuasion.txt</td>\n",
       "      <td>98171</td>\n",
       "      <td>3747</td>\n",
       "      <td>466292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>austen-sense.txt</td>\n",
       "      <td>141576</td>\n",
       "      <td>4999</td>\n",
       "      <td>673022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bible-kjv.txt</td>\n",
       "      <td>1010654</td>\n",
       "      <td>30103</td>\n",
       "      <td>4332554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>blake-poems.txt</td>\n",
       "      <td>8354</td>\n",
       "      <td>438</td>\n",
       "      <td>38153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bryant-stories.txt</td>\n",
       "      <td>55563</td>\n",
       "      <td>2863</td>\n",
       "      <td>249439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>burgess-busterbrown.txt</td>\n",
       "      <td>18963</td>\n",
       "      <td>1054</td>\n",
       "      <td>84663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>carroll-alice.txt</td>\n",
       "      <td>34110</td>\n",
       "      <td>1703</td>\n",
       "      <td>144395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>chesterton-ball.txt</td>\n",
       "      <td>96996</td>\n",
       "      <td>4779</td>\n",
       "      <td>457450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>chesterton-brown.txt</td>\n",
       "      <td>86063</td>\n",
       "      <td>3806</td>\n",
       "      <td>406629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>chesterton-thursday.txt</td>\n",
       "      <td>69213</td>\n",
       "      <td>3742</td>\n",
       "      <td>320525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>edgeworth-parents.txt</td>\n",
       "      <td>210663</td>\n",
       "      <td>10230</td>\n",
       "      <td>935158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>melville-moby_dick.txt</td>\n",
       "      <td>260819</td>\n",
       "      <td>10059</td>\n",
       "      <td>1242990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>milton-paradise.txt</td>\n",
       "      <td>96825</td>\n",
       "      <td>1851</td>\n",
       "      <td>468220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>shakespeare-caesar.txt</td>\n",
       "      <td>25833</td>\n",
       "      <td>2163</td>\n",
       "      <td>112310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>shakespeare-hamlet.txt</td>\n",
       "      <td>37360</td>\n",
       "      <td>3106</td>\n",
       "      <td>162881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>shakespeare-macbeth.txt</td>\n",
       "      <td>23140</td>\n",
       "      <td>1907</td>\n",
       "      <td>100351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>whitman-leaves.txt</td>\n",
       "      <td>154883</td>\n",
       "      <td>4250</td>\n",
       "      <td>711215</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/latex": [
       "None"
      ],
      "text/plain": [
       "                     corpus    words  sentences  characters\n",
       "0           austen-emma.txt   192427       7752      887071\n",
       "1     austen-persuasion.txt    98171       3747      466292\n",
       "2          austen-sense.txt   141576       4999      673022\n",
       "3             bible-kjv.txt  1010654      30103     4332554\n",
       "4           blake-poems.txt     8354        438       38153\n",
       "5        bryant-stories.txt    55563       2863      249439\n",
       "6   burgess-busterbrown.txt    18963       1054       84663\n",
       "7         carroll-alice.txt    34110       1703      144395\n",
       "8       chesterton-ball.txt    96996       4779      457450\n",
       "9      chesterton-brown.txt    86063       3806      406629\n",
       "10  chesterton-thursday.txt    69213       3742      320525\n",
       "11    edgeworth-parents.txt   210663      10230      935158\n",
       "12   melville-moby_dick.txt   260819      10059     1242990\n",
       "13      milton-paradise.txt    96825       1851      468220\n",
       "14   shakespeare-caesar.txt    25833       2163      112310\n",
       "15   shakespeare-hamlet.txt    37360       3106      162881\n",
       "16  shakespeare-macbeth.txt    23140       1907      100351\n",
       "17       whitman-leaves.txt   154883       4250      711215"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = pd.DataFrame(zip(gutenberg.fileids(), wordLengths, sentenceLengths, rawLengths), columns=['corpus','words','sentences','characters'])\n",
    "\n",
    "dataframe\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</variable><variable id=\"_V`RIwppcpbRKT:m6^qH\">pd</variable><variable id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</variable><variable id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</variable><variable id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</variable><variable id=\"AO?GdNQ:*92|iDWB.)YR\">rawLengths</variable></variables><block type=\"variables_set\" id=\"Nnj3jwtJa6+~cV1=RDn_\" x=\"-149\" y=\"237\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varCreateObject\" id=\"*C0V~q-fS]kDUMRq`O-N\"><field name=\"VAR\" id=\"_V`RIwppcpbRKT:m6^qH\">pd</field><field name=\"MEMBER\">DataFrame</field><data>pd:DataFrame</data><value name=\"INPUT\"><block type=\"lists_create_with\" id=\"DoM~-@qI)6TgbDc;vBMb\"><mutation items=\"2\"></mutation><value name=\"ADD0\"><block type=\"zipBlock\" id=\"nv7/65]-+;=,M.B)yU%U\"><value name=\"x\"><block type=\"lists_create_with\" id=\"@PN$;KCRy[Jv;QJ+d#($\"><mutation items=\"4\"></mutation><value name=\"ADD0\"><block type=\"varDoMethod\" id=\"FJ#DraHg!(/g)_[-^Dn]\"><field name=\"VAR\" id=\"`$^y`^v4:G8DO1QCBCw8\">gutenberg</field><field name=\"MEMBER\">fileids</field><data>gutenberg:fileids</data></block></value><value name=\"ADD1\"><block type=\"variables_get\" id=\"1UbO_xg6I)qqO#p=]Trh\"><field name=\"VAR\" id=\"nb}L;_W{,H)*Jc!qq]@S\">wordLengths</field></block></value><value name=\"ADD2\"><block type=\"variables_get\" id=\"%~eQRKbkLX{by+raZ$LQ\"><field name=\"VAR\" id=\"$OD[:+1843Cn0O3j8JiE\">sentenceLengths</field></block></value><value name=\"ADD3\"><block type=\"variables_get\" id=\"}U=.2[w^C@TMLrYf)B-o\"><field name=\"VAR\" id=\"AO?GdNQ:*92|iDWB.)YR\">rawLengths</field></block></value></block></value></block></value><value name=\"ADD1\"><block type=\"dummyOutputCodeBlock\" id=\"o35uqta54?^UVk|[,.|(\"><field name=\"CODE\">columns=['corpus','words','sentences','characters']</field></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"(gZ^x=Q!@}~:|xjc+ZXy\" x=\"-133\" y=\"397\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This nicely brings together and displays everything we've done so far.\n",
    "\n",
    "To calculate average word length and sentence length, just add columns:\n",
    "\n",
    "- set `dataframe` to with `dataframe` to `assign` using\n",
    "    - freestyle `avg_wl =` `dataframe[\"characters\"]` / `dataframe[\"words\"]`\n",
    "- set `dataframe` to with `dataframe` to `assign` using\n",
    "    - freestyle `avg_sl =` `dataframe[\"words\"]` / `dataframe[\"sentences\"]`\n",
    "- Display dataframe\n",
    "\n",
    "Note that the standard unit for word length is characters but that the standard unit for sentence length is words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>words</th>\n",
       "      <th>sentences</th>\n",
       "      <th>characters</th>\n",
       "      <th>avg_wl</th>\n",
       "      <th>avg_sl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>austen-emma.txt</td>\n",
       "      <td>192427</td>\n",
       "      <td>7752</td>\n",
       "      <td>887071</td>\n",
       "      <td>4.609909</td>\n",
       "      <td>24.822884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>austen-persuasion.txt</td>\n",
       "      <td>98171</td>\n",
       "      <td>3747</td>\n",
       "      <td>466292</td>\n",
       "      <td>4.749794</td>\n",
       "      <td>26.199893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>austen-sense.txt</td>\n",
       "      <td>141576</td>\n",
       "      <td>4999</td>\n",
       "      <td>673022</td>\n",
       "      <td>4.753786</td>\n",
       "      <td>28.320864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bible-kjv.txt</td>\n",
       "      <td>1010654</td>\n",
       "      <td>30103</td>\n",
       "      <td>4332554</td>\n",
       "      <td>4.286882</td>\n",
       "      <td>33.573199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>blake-poems.txt</td>\n",
       "      <td>8354</td>\n",
       "      <td>438</td>\n",
       "      <td>38153</td>\n",
       "      <td>4.567034</td>\n",
       "      <td>19.073059</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bryant-stories.txt</td>\n",
       "      <td>55563</td>\n",
       "      <td>2863</td>\n",
       "      <td>249439</td>\n",
       "      <td>4.489300</td>\n",
       "      <td>19.407265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>burgess-busterbrown.txt</td>\n",
       "      <td>18963</td>\n",
       "      <td>1054</td>\n",
       "      <td>84663</td>\n",
       "      <td>4.464642</td>\n",
       "      <td>17.991461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>carroll-alice.txt</td>\n",
       "      <td>34110</td>\n",
       "      <td>1703</td>\n",
       "      <td>144395</td>\n",
       "      <td>4.233216</td>\n",
       "      <td>20.029360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>chesterton-ball.txt</td>\n",
       "      <td>96996</td>\n",
       "      <td>4779</td>\n",
       "      <td>457450</td>\n",
       "      <td>4.716174</td>\n",
       "      <td>20.296296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>chesterton-brown.txt</td>\n",
       "      <td>86063</td>\n",
       "      <td>3806</td>\n",
       "      <td>406629</td>\n",
       "      <td>4.724783</td>\n",
       "      <td>22.612454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>chesterton-thursday.txt</td>\n",
       "      <td>69213</td>\n",
       "      <td>3742</td>\n",
       "      <td>320525</td>\n",
       "      <td>4.630994</td>\n",
       "      <td>18.496259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>edgeworth-parents.txt</td>\n",
       "      <td>210663</td>\n",
       "      <td>10230</td>\n",
       "      <td>935158</td>\n",
       "      <td>4.439118</td>\n",
       "      <td>20.592669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>melville-moby_dick.txt</td>\n",
       "      <td>260819</td>\n",
       "      <td>10059</td>\n",
       "      <td>1242990</td>\n",
       "      <td>4.765719</td>\n",
       "      <td>25.928919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>milton-paradise.txt</td>\n",
       "      <td>96825</td>\n",
       "      <td>1851</td>\n",
       "      <td>468220</td>\n",
       "      <td>4.835735</td>\n",
       "      <td>52.309562</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>shakespeare-caesar.txt</td>\n",
       "      <td>25833</td>\n",
       "      <td>2163</td>\n",
       "      <td>112310</td>\n",
       "      <td>4.347540</td>\n",
       "      <td>11.943135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>shakespeare-hamlet.txt</td>\n",
       "      <td>37360</td>\n",
       "      <td>3106</td>\n",
       "      <td>162881</td>\n",
       "      <td>4.359770</td>\n",
       "      <td>12.028332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>shakespeare-macbeth.txt</td>\n",
       "      <td>23140</td>\n",
       "      <td>1907</td>\n",
       "      <td>100351</td>\n",
       "      <td>4.336690</td>\n",
       "      <td>12.134242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>whitman-leaves.txt</td>\n",
       "      <td>154883</td>\n",
       "      <td>4250</td>\n",
       "      <td>711215</td>\n",
       "      <td>4.591950</td>\n",
       "      <td>36.443059</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/latex": [
       "None"
      ],
      "text/plain": [
       "                     corpus    words  ...    avg_wl     avg_sl\n",
       "0           austen-emma.txt   192427  ...  4.609909  24.822884\n",
       "1     austen-persuasion.txt    98171  ...  4.749794  26.199893\n",
       "2          austen-sense.txt   141576  ...  4.753786  28.320864\n",
       "3             bible-kjv.txt  1010654  ...  4.286882  33.573199\n",
       "4           blake-poems.txt     8354  ...  4.567034  19.073059\n",
       "5        bryant-stories.txt    55563  ...  4.489300  19.407265\n",
       "6   burgess-busterbrown.txt    18963  ...  4.464642  17.991461\n",
       "7         carroll-alice.txt    34110  ...  4.233216  20.029360\n",
       "8       chesterton-ball.txt    96996  ...  4.716174  20.296296\n",
       "9      chesterton-brown.txt    86063  ...  4.724783  22.612454\n",
       "10  chesterton-thursday.txt    69213  ...  4.630994  18.496259\n",
       "11    edgeworth-parents.txt   210663  ...  4.439118  20.592669\n",
       "12   melville-moby_dick.txt   260819  ...  4.765719  25.928919\n",
       "13      milton-paradise.txt    96825  ...  4.835735  52.309562\n",
       "14   shakespeare-caesar.txt    25833  ...  4.347540  11.943135\n",
       "15   shakespeare-hamlet.txt    37360  ...  4.359770  12.028332\n",
       "16  shakespeare-macbeth.txt    23140  ...  4.336690  12.134242\n",
       "17       whitman-leaves.txt   154883  ...  4.591950  36.443059\n",
       "\n",
       "[18 rows x 6 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = dataframe.assign(avg_wl= (dataframe['characters'] / dataframe['words']))\n",
    "dataframe = dataframe.assign(avg_sl= (dataframe['words'] / dataframe['sentences']))\n",
    "\n",
    "dataframe\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</variable></variables><block type=\"variables_set\" id=\"{UO)w}M?tYx?A02OPAw9\" x=\"-83\" y=\"249\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"L%G*;r8*$i5SLn{,Cc$T\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><field name=\"MEMBER\">assign</field><data>dataframe:assign</data><value name=\"INPUT\"><block type=\"valueOutputCodeBlock\" id=\"Ux2OR.~,)cCrIxgQW6VI\"><field name=\"CODE\">avg_wl=</field><value name=\"INPUT\"><block type=\"math_arithmetic\" id=\"^JEFtmhs2.cv#;80c/nT\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\" id=\"syWxT)`[^TBT:IsIQGMf\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"5zNI-WP@PpW0doRlek8W\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"3@IMZDY0GOmgS:YQgx?C\"><field name=\"TEXT\">characters</field></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"t1.s:%LN=uK/vv%zl:f:\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"QeW+!Bpy|dosxiHiI(Vq\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"sg3+/)wO!$kLNdtDKOJN\"><field name=\"TEXT\">words</field></block></value></block></value></block></value></block></value></block></value><next><block type=\"variables_set\" id=\";EOx!PoNSJEXe3}Nc1AU\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"-{X4*i^Z:O4lkNW=F]rs\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><field name=\"MEMBER\">assign</field><data>dataframe:assign</data><value name=\"INPUT\"><block type=\"valueOutputCodeBlock\" id=\"P+/y;r)NaM2k#OEIf08~\"><field name=\"CODE\">avg_sl=</field><value name=\"INPUT\"><block type=\"math_arithmetic\" id=\"Xl4rhP=?Xbe$O90]f;b1\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"fp`q$osSW7CCT3EC^-jh\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"M/QDQE2#j$}uzL!/L{ow\"><field name=\"TEXT\">words</field></block></value></block></value><value name=\"B\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"u.cc%cZ]S6NC+|6@7MZc\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"FaX@#$GZ_F/S]$=oXR@?\"><field name=\"TEXT\">sentences</field></block></value></block></value></block></value></block></value></block></value></block></next></block><block type=\"variables_get\" id=\"uzxkeHUiih]mUfldHAMe\" x=\"-98\" y=\"405\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Readability\n",
    "\n",
    "What we've calculated so far may seem simplistic and perhaps not that useful.\n",
    "However, several of these metrics are components of perhaps the most well known readability formula, [Flesch Kincaid Grade Level (FKGL)](https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests):\n",
    "\n",
    "\\begin{equation*}\n",
    "0.39 \\left( \\frac{\\mbox{total words}}{\\mbox{total sentences}} \\right) +11.8 \\left( \\frac{\\mbox{total syllables}}{\\mbox{total words}} \\right) - 15.59\n",
    "\\end{equation*}\n",
    "\n",
    "FKGL gives us a sense of how difficult text is to read, which could be an important/useful predictor as well as an interesting descriptive statistic.\n",
    "\n",
    "We don't have syllable length, however.\n",
    "Syllable length is a bit of a pain to calculate because English has a deep orthography, so the best way is to use a pronunciation dictionary like [this](https://github.com/steveash/jg2p).\n",
    "For now, we will just assume that English has 1.5 syllables per word and estimate this component:\n",
    "\n",
    "- set `dataframe` to with `dataframe` to `assign` using\n",
    "    - freestyle `fkgl =` 0.39 * `dataframe[\"words\"]` / `dataframe[\"sentences\"]` + 11.8 * 1.5 - 15.59\n",
    "- Display dataframe\n",
    "\n",
    "*Note 1.5 * words/words = 1.5*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>words</th>\n",
       "      <th>sentences</th>\n",
       "      <th>characters</th>\n",
       "      <th>avg_wl</th>\n",
       "      <th>avg_sl</th>\n",
       "      <th>fkgl</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>austen-emma.txt</td>\n",
       "      <td>192427</td>\n",
       "      <td>7752</td>\n",
       "      <td>887071</td>\n",
       "      <td>4.609909</td>\n",
       "      <td>24.822884</td>\n",
       "      <td>11.790925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>austen-persuasion.txt</td>\n",
       "      <td>98171</td>\n",
       "      <td>3747</td>\n",
       "      <td>466292</td>\n",
       "      <td>4.749794</td>\n",
       "      <td>26.199893</td>\n",
       "      <td>12.327958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>austen-sense.txt</td>\n",
       "      <td>141576</td>\n",
       "      <td>4999</td>\n",
       "      <td>673022</td>\n",
       "      <td>4.753786</td>\n",
       "      <td>28.320864</td>\n",
       "      <td>13.155137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>bible-kjv.txt</td>\n",
       "      <td>1010654</td>\n",
       "      <td>30103</td>\n",
       "      <td>4332554</td>\n",
       "      <td>4.286882</td>\n",
       "      <td>33.573199</td>\n",
       "      <td>15.203547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>blake-poems.txt</td>\n",
       "      <td>8354</td>\n",
       "      <td>438</td>\n",
       "      <td>38153</td>\n",
       "      <td>4.567034</td>\n",
       "      <td>19.073059</td>\n",
       "      <td>9.548493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bryant-stories.txt</td>\n",
       "      <td>55563</td>\n",
       "      <td>2863</td>\n",
       "      <td>249439</td>\n",
       "      <td>4.489300</td>\n",
       "      <td>19.407265</td>\n",
       "      <td>9.678833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>burgess-busterbrown.txt</td>\n",
       "      <td>18963</td>\n",
       "      <td>1054</td>\n",
       "      <td>84663</td>\n",
       "      <td>4.464642</td>\n",
       "      <td>17.991461</td>\n",
       "      <td>9.126670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>carroll-alice.txt</td>\n",
       "      <td>34110</td>\n",
       "      <td>1703</td>\n",
       "      <td>144395</td>\n",
       "      <td>4.233216</td>\n",
       "      <td>20.029360</td>\n",
       "      <td>9.921450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>chesterton-ball.txt</td>\n",
       "      <td>96996</td>\n",
       "      <td>4779</td>\n",
       "      <td>457450</td>\n",
       "      <td>4.716174</td>\n",
       "      <td>20.296296</td>\n",
       "      <td>10.025556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>chesterton-brown.txt</td>\n",
       "      <td>86063</td>\n",
       "      <td>3806</td>\n",
       "      <td>406629</td>\n",
       "      <td>4.724783</td>\n",
       "      <td>22.612454</td>\n",
       "      <td>10.928857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>chesterton-thursday.txt</td>\n",
       "      <td>69213</td>\n",
       "      <td>3742</td>\n",
       "      <td>320525</td>\n",
       "      <td>4.630994</td>\n",
       "      <td>18.496259</td>\n",
       "      <td>9.323541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>edgeworth-parents.txt</td>\n",
       "      <td>210663</td>\n",
       "      <td>10230</td>\n",
       "      <td>935158</td>\n",
       "      <td>4.439118</td>\n",
       "      <td>20.592669</td>\n",
       "      <td>10.141141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>melville-moby_dick.txt</td>\n",
       "      <td>260819</td>\n",
       "      <td>10059</td>\n",
       "      <td>1242990</td>\n",
       "      <td>4.765719</td>\n",
       "      <td>25.928919</td>\n",
       "      <td>12.222279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>milton-paradise.txt</td>\n",
       "      <td>96825</td>\n",
       "      <td>1851</td>\n",
       "      <td>468220</td>\n",
       "      <td>4.835735</td>\n",
       "      <td>52.309562</td>\n",
       "      <td>22.510729</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>shakespeare-caesar.txt</td>\n",
       "      <td>25833</td>\n",
       "      <td>2163</td>\n",
       "      <td>112310</td>\n",
       "      <td>4.347540</td>\n",
       "      <td>11.943135</td>\n",
       "      <td>6.767822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>shakespeare-hamlet.txt</td>\n",
       "      <td>37360</td>\n",
       "      <td>3106</td>\n",
       "      <td>162881</td>\n",
       "      <td>4.359770</td>\n",
       "      <td>12.028332</td>\n",
       "      <td>6.801050</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>shakespeare-macbeth.txt</td>\n",
       "      <td>23140</td>\n",
       "      <td>1907</td>\n",
       "      <td>100351</td>\n",
       "      <td>4.336690</td>\n",
       "      <td>12.134242</td>\n",
       "      <td>6.842354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>whitman-leaves.txt</td>\n",
       "      <td>154883</td>\n",
       "      <td>4250</td>\n",
       "      <td>711215</td>\n",
       "      <td>4.591950</td>\n",
       "      <td>36.443059</td>\n",
       "      <td>16.322793</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/latex": [
       "None"
      ],
      "text/plain": [
       "                     corpus    words  sentences  ...    avg_wl     avg_sl       fkgl\n",
       "0           austen-emma.txt   192427       7752  ...  4.609909  24.822884  11.790925\n",
       "1     austen-persuasion.txt    98171       3747  ...  4.749794  26.199893  12.327958\n",
       "2          austen-sense.txt   141576       4999  ...  4.753786  28.320864  13.155137\n",
       "3             bible-kjv.txt  1010654      30103  ...  4.286882  33.573199  15.203547\n",
       "4           blake-poems.txt     8354        438  ...  4.567034  19.073059   9.548493\n",
       "5        bryant-stories.txt    55563       2863  ...  4.489300  19.407265   9.678833\n",
       "6   burgess-busterbrown.txt    18963       1054  ...  4.464642  17.991461   9.126670\n",
       "7         carroll-alice.txt    34110       1703  ...  4.233216  20.029360   9.921450\n",
       "8       chesterton-ball.txt    96996       4779  ...  4.716174  20.296296  10.025556\n",
       "9      chesterton-brown.txt    86063       3806  ...  4.724783  22.612454  10.928857\n",
       "10  chesterton-thursday.txt    69213       3742  ...  4.630994  18.496259   9.323541\n",
       "11    edgeworth-parents.txt   210663      10230  ...  4.439118  20.592669  10.141141\n",
       "12   melville-moby_dick.txt   260819      10059  ...  4.765719  25.928919  12.222279\n",
       "13      milton-paradise.txt    96825       1851  ...  4.835735  52.309562  22.510729\n",
       "14   shakespeare-caesar.txt    25833       2163  ...  4.347540  11.943135   6.767822\n",
       "15   shakespeare-hamlet.txt    37360       3106  ...  4.359770  12.028332   6.801050\n",
       "16  shakespeare-macbeth.txt    23140       1907  ...  4.336690  12.134242   6.842354\n",
       "17       whitman-leaves.txt   154883       4250  ...  4.591950  36.443059  16.322793\n",
       "\n",
       "[18 rows x 7 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = dataframe.assign(fkgl= ((0.39 * (dataframe['words'] / dataframe['sentences']) + 11.8 * 1.5) - 15.59))\n",
    "\n",
    "dataframe\n",
    "\n",
    "#<xml xmlns=\"https://developers.google.com/blockly/xml\"><variables><variable id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</variable></variables><block type=\"variables_set\" id=\"d3ELqhcW@UA^R%PLy3cV\" x=\"-83\" y=\"308\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"VALUE\"><block type=\"varDoMethod\" id=\"!?g`gMP):imN8F)T,@V|\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><field name=\"MEMBER\">assign</field><data>dataframe:assign</data><value name=\"INPUT\"><block type=\"valueOutputCodeBlock\" id=\"k^_3ol/yS5*#Dha3rPVq\"><field name=\"CODE\">fkgl=</field><value name=\"INPUT\"><block type=\"math_arithmetic\" id=\"Md5u*{OHNtri0AjDj],6\"><field name=\"OP\">MINUS</field><value name=\"A\"><shadow type=\"math_number\" id=\"oOOHvHpa1xT0^Jls*1X+\"><field name=\"NUM\">0.39</field></shadow><block type=\"math_arithmetic\" id=\"QGJQ@m3mx5!l=,g9-R+d\"><field name=\"OP\">ADD</field><value name=\"A\"><shadow type=\"math_number\" id=\"X]w:+`6NY)?+=r#3v?Aj\"><field name=\"NUM\">0.39</field></shadow><block type=\"math_arithmetic\" id=\",gsasO{}sRYc]60t@#ud\"><field name=\"OP\">MULTIPLY</field><value name=\"A\"><shadow type=\"math_number\" id=\"^KWa~qEgh3Q5R?==H)+9\"><field name=\"NUM\">0.39</field></shadow></value><value name=\"B\"><shadow type=\"math_number\" id=\"?4DQu[d7x;oy^4iY|Y3(\"><field name=\"NUM\">1</field></shadow><block type=\"math_arithmetic\" id=\"b1}M?uM#pV~iqEz`%Dh~\"><field name=\"OP\">DIVIDE</field><value name=\"A\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"/t1A;EYsDfw+BYK-f{P8\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"/`cSF)tsx%3]HefEnKlV\"><field name=\"TEXT\">words</field></block></value></block></value><value name=\"B\"><shadow type=\"math_number\"><field name=\"NUM\">1</field></shadow><block type=\"indexer\" id=\"9_HN7=?;UGbK`ZFpBD)n\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field><value name=\"INDEX\"><block type=\"text\" id=\"SDk,ItV,c;N.~+Z%vOH]\"><field name=\"TEXT\">sentences</field></block></value></block></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"osy!WzPZ^Ac.Li?,M]Sq\"><field name=\"NUM\">1</field></shadow><block type=\"math_arithmetic\" id=\"pns2jEFkEFV2potV{wzJ\"><field name=\"OP\">MULTIPLY</field><value name=\"A\"><shadow type=\"math_number\" id=\"~z95el]}Uct`QV~*3@LC\"><field name=\"NUM\">11.8</field></shadow></value><value name=\"B\"><shadow type=\"math_number\" id=\"L,wO%x|x/NAv$%-5KtZc\"><field name=\"NUM\">1.5</field></shadow></value></block></value></block></value><value name=\"B\"><shadow type=\"math_number\" id=\"N6x=21,Sww{nHR%@JF|@\"><field name=\"NUM\">15.59</field></shadow></value></block></value></block></value></block></value></block><block type=\"variables_get\" id=\"F^RD,W:_|bYpb/e8Wq3b\" x=\"-90\" y=\"410\"><field name=\"VAR\" id=\"d*P53^Ni!VyA[RubgfYr\">dataframe</field></block></xml>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The readability differences generally make sense - `Alice and Wonderland` lower than the `King James Bible` lower than `Paradise Lost`, but we also see Shakespeare is the least difficult.\n",
    "This is perhaps expected given the formula, since Shakespeare has the lowest `avg_sl`, but seems intuitively incorrect for those who have experienced Shakespeare. \n",
    "Note however, that FKGL does not take into account the frequency of the words themselves (how rare they are), which is a question of *distribution*.\n",
    "\n",
    "This are just a few common length-based text metrics.\n",
    "There are a potentially infinite number - As FKGL shows, basic length-based text metrics can be combined in arbitrary ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "xpython",
   "language": "python",
   "name": "xpython"
  },
  "language_info": {
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}