Chapter19_Comparing_categorical_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up a fancy stylesheet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>\n",
       "\n",
       "<style>\n",
       "\n",
       "@font-face {\n",
       "    font-family: \"Computer Modern\";\n",
       "    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "\n",
       "#notebook_panel { /* main background */\n",
       "    background: #ddd;\n",
       "    color: #000000;\n",
       "}\n",
       "\n",
       "\n",
       "div.text_cell_render{\n",
       "    font-family: 'Arvo' sans-serif;\n",
       "    line-height: 130%;\n",
       "    font-size: 115%;\n",
       "    width:1000px;\n",
       "    margin-left:auto;\n",
       "    margin-right:auto;\n",
       "}\n",
       "\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "    font-family: 'Philosopher', sans-serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 2.2em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(0, 80, 120);\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\t\n",
       ".text_cell_render h2 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 1.9em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(200,100,0);\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\t\n",
       "\n",
       ".text_cell_render h3 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "    margin-top:12px;\n",
       "    margin-bottom: 3px;\n",
       "    font-style: italic;\n",
       "    color: rgb(94,127,192);\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {\n",
       "    font-family: 'Alegreya Sans', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 16pt;\n",
       "    color: grey;\n",
       "    font-style: italic;\n",
       "    margin-bottom: .1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 {\n",
       "    font-family: 'PT Mono', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 10pt;\n",
       "    color: grey;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       ".CodeMirror{\n",
       "        font-family: \"PT Mono\";\n",
       "        font-size: 100%;\n",
       "}\n",
       "\n",
       "</style>\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "css_file = 'style.css'\n",
    "HTML(open(css_file, 'r').read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting up the required python &#8482; environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "from scipy.stats import bayes_mvs\n",
    "from math import factorial\n",
    "import scikits.bootstrap as bs\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from warnings import filterwarnings\n",
    "%matplotlib inline\n",
    "sns.set_style('whitegrid')\n",
    "sns.set_context('paper', font_scale = 2.0, rc = {'lines.linewidth': 1.5, 'figure.figsize' : (10, 8)})\n",
    "filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing categorical data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>In the previous chapter we created two groups.  These groups were categorical in nature.  Group A was those without histological evidence of appendicitis and group B included those with evidence of appendicitis.  Furthermore, these two groups are nominal.  There is no order to with and without appendicitis.</p>\n",
    "<p>The variable we examined was white cell count.  This represented continuous data of the ratio numerical type.  Now, let's construct two groups, once again of nominal ordinal type, but this time the data variable is also categorical.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing and examining the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>File</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Delay</th>\n",
       "      <th>Stay</th>\n",
       "      <th>ICU</th>\n",
       "      <th>RVD</th>\n",
       "      <th>CD4</th>\n",
       "      <th>HR</th>\n",
       "      <th>Temp</th>\n",
       "      <th>CRP</th>\n",
       "      <th>WCC</th>\n",
       "      <th>HB</th>\n",
       "      <th>Rupture</th>\n",
       "      <th>Histo</th>\n",
       "      <th>Comp</th>\n",
       "      <th>MASS</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "      <td>Female</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>97</td>\n",
       "      <td>35.2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.49</td>\n",
       "      <td>10.4</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>Male</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>57</td>\n",
       "      <td>109</td>\n",
       "      <td>38.8</td>\n",
       "      <td>45.3</td>\n",
       "      <td>7.08</td>\n",
       "      <td>19.8</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "      <td>Female</td>\n",
       "      <td>1</td>\n",
       "      <td>16</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>120</td>\n",
       "      <td>36.3</td>\n",
       "      <td>10.7</td>\n",
       "      <td>13.00</td>\n",
       "      <td>8.7</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>20</td>\n",
       "      <td>Female</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>NaN</td>\n",
       "      <td>120</td>\n",
       "      <td>35.7</td>\n",
       "      <td>77.8</td>\n",
       "      <td>4.45</td>\n",
       "      <td>8.8</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>28</td>\n",
       "      <td>Female</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>491</td>\n",
       "      <td>115</td>\n",
       "      <td>37.1</td>\n",
       "      <td>51.6</td>\n",
       "      <td>21.98</td>\n",
       "      <td>13.4</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   File  Age  Gender  Delay  Stay ICU  RVD  CD4   HR  Temp   CRP    WCC    HB  \\\n",
       "0     1   38  Female      3     6  No   No  NaN   97  35.2   NaN  10.49  10.4   \n",
       "1     2   32    Male      6    10  No  Yes   57  109  38.8  45.3   7.08  19.8   \n",
       "2     3   19  Female      1    16  No   No  NaN  120  36.3  10.7  13.00   8.7   \n",
       "3     4   20  Female      2     9  No  Yes  NaN  120  35.7  77.8   4.45   8.8   \n",
       "4     5   28  Female      3     3  No  Yes  491  115  37.1  51.6  21.98  13.4   \n",
       "\n",
       "  Rupture Histo Comp  MASS  \n",
       "0      No   Yes  Yes     5  \n",
       "1      No    No  Yes     8  \n",
       "2      No    No   No     3  \n",
       "3      No    No   No     0  \n",
       "4      No   Yes   No     7  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('MOOC_Mock.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The chi-squared (*&#935;<sup>2</sup>*) test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>For this example I want to know if there is a difference in the incidence of histologically proven appendicitis between those with and without retroviral disease (RVD).</p>\n",
    "<p>Think about it for a moment.  I will have two groups: Group A without appendicitis and Group B with. Within each of these I will have two groups: Group I without RVD and Group II with.  Form this I can create a little table called a *contingency table*.  In this example the table will have two rows and two columns.  For a &#935;*<sup>2</sup>* test, the table can have more rows and more columns.</p>\n",
    "<p>In order to do this test, we will have to get the values to fill in our column, manually.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Histo  RVD\n",
       "No     No     16\n",
       "       Yes    14\n",
       "Yes    No     80\n",
       "       Yes    40\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "histo_group = data.groupby(data['Histo'])\n",
    "histo_group['RVD'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Here, I have made use of the powerful *.groupby()* function.  It splits my DataFrame into parts according to the values found in the specified column (here I chose **Histo**).  I attached the new (split) DataFrame to a computer variable name *histo_group*.</p>\n",
    "<p>I then moved on and asked the software to give me the value counts for the **RVD** column of this new DataFrame.  Note how the results tell me that the DataFrame is split by **Histo** and then gives me a breakdown of results found in the requested column.  It found values for *yes* and *No* and told me how many of each it found.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the contingency table (matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Now we have to construct our little two-by-two matrix.  Here I will use *numpy*.  Note the use of square brackets.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[16, 14],\n",
       "       [80, 40]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "histo_RVD_observed = np.array([[16, 14], [80, 40]])\n",
    "histo_RVD_observed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>From the *scipy.stats* library I am going to use the *chi2_contingency()* function.  It takes a single argument (my observed table above) and returns for values, hence my use of four computer variable names before the equal sign.  They are:\n",
    "* The &#935;*<sup>2</sup>* value\n",
    "* The *p*-value\n",
    "* The degrees of freedom\n",
    "* The expected table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "chi_val, p, df, expected = stats.chi2_contingency(histo_RVD_observed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>Let's print each of these to the screen.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3183593750000002"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25088670393543944"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 19.2,  10.8],\n",
       "       [ 76.8,  43.2]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>So, our *p*-value was more than *0.05* and we can say that there was no difference in the rate of RVD between those with and without appendicitis.<p>\n",
    "<p>The expected table is quite interesting.  It calculates what we would have expected given the total number in each group.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}