KULKARNI_MANASI_PS6 (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Set 5\n",
    "\n",
    "## Trees, Bagging and RandomForest\n",
    "\n",
    "### Manasi Kulkarni\n",
    "\n",
    "### INFX 574\n",
    "\n",
    "### Collaborated with: Prem Shah, Aditya Wakade, Pratik Damania, Gaurav Gohil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import entropy\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "import math\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Importing the data\n",
    "data = pd.read_csv('/Users/manasi/Desktop/titanic.csv')\n",
    "data['sex'] = np.where(data.sex == 'female', 1, 0)\n",
    "\n",
    "# Making a family siz by adding the columns parch and sibsp. And then classifying\n",
    "data['family size'] = data.parch + data.sibsp\n",
    "data['family class'] =  np.where(data['family size'].isin([0]), 'Single', np.where(data['family size'].isin([1,2]), 'Small', np.where(data['family size'].isin([3,4,5]), 'Medium', 'large')))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We grouped the family sizes into bundles. Accounting for 0 are singles. For 1 and 2, we have a Small family. For 3,4 and 5 ; we have a medium sized family and large accounts for the rest. Based on an initial analysis, we see that Singles account for the largest in the titanic population. This might lead to revelations later that solo travelers had a much higher chance to die than to survive. In addition, people traveling in families of 2-4 people actually could have a relatively high chance to survive. This chance could be significantly lower among 5+ families."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 421,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Splitting the data in survived and not survived\n",
    "sur = data[data.survived == 1]\n",
    "n_sur = data[data.survived == 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 423,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pclass         0\n",
       "survived       0\n",
       "sex            0\n",
       "age          263\n",
       "sibsp          0\n",
       "parch          0\n",
       "fare           1\n",
       "body        1188\n",
       "dtype: int64"
      ]
     },
     "execution_count": 423,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding null values for every column\n",
    "nulls = data.isnull().sum()\n",
    "nl = nulls[[0,1,3,4,5,6,8,12]]\n",
    "nl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 263 null values in age and 1188 null values in body. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Finding mean, max and min for survived\n",
    "x1 = sur.describe().T[['mean', 'min', 'max']]\n",
    "x1.columns = ['survived_average', 'survived_min', 'survived_max']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Finding mean, max and min for not survived\n",
    "x2 = n_sur.describe().T[['mean', 'min', 'max']]\n",
    "x2.columns = ['not survived_average', 'not survived_min', 'not survived_max']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived_average</th>\n",
       "      <th>not survived_average</th>\n",
       "      <th>nulls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pclass</th>\n",
       "      <td>1.962000</td>\n",
       "      <td>2.500618</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>survived</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <td>0.678000</td>\n",
       "      <td>0.156984</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>28.918228</td>\n",
       "      <td>30.545369</td>\n",
       "      <td>263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sibsp</th>\n",
       "      <td>0.462000</td>\n",
       "      <td>0.521632</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>0.476000</td>\n",
       "      <td>0.328801</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fare</th>\n",
       "      <td>49.361184</td>\n",
       "      <td>23.353831</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>NaN</td>\n",
       "      <td>160.809917</td>\n",
       "      <td>1188</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          survived_average  not survived_average  nulls\n",
       "pclass            1.962000              2.500618      0\n",
       "survived          1.000000              0.000000      0\n",
       "sex               0.678000              0.156984      0\n",
       "age              28.918228             30.545369    263\n",
       "sibsp             0.462000              0.521632      0\n",
       "parch             0.476000              0.328801      0\n",
       "fare             49.361184             23.353831      1\n",
       "body                   NaN            160.809917   1188"
      ]
     },
     "execution_count": 429,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_data = pd.DataFrame(list(zip(sur.mean(),n_sur.mean(),nl)))\n",
    "sum_data.columns = ['survived_average', 'not survived_average', 'nulls']\n",
    "sum_data.index = ['pclass', 'survived', 'sex','age', 'sibsp', 'parch', 'fare', 'body']\n",
    "sum_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we join the datasets to find average and min max values for survived, not survived along with null values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived_average</th>\n",
       "      <th>not survived_average</th>\n",
       "      <th>nulls</th>\n",
       "      <th>survived_min</th>\n",
       "      <th>survived_max</th>\n",
       "      <th>not survived_min</th>\n",
       "      <th>not survived_max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pclass</th>\n",
       "      <td>1.962000</td>\n",
       "      <td>2.500618</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>3.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>survived</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <td>0.678000</td>\n",
       "      <td>0.156984</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>28.918228</td>\n",
       "      <td>30.545369</td>\n",
       "      <td>263</td>\n",
       "      <td>0.1667</td>\n",
       "      <td>80.0000</td>\n",
       "      <td>0.3333</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sibsp</th>\n",
       "      <td>0.462000</td>\n",
       "      <td>0.521632</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>0.476000</td>\n",
       "      <td>0.328801</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>5.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fare</th>\n",
       "      <td>49.361184</td>\n",
       "      <td>23.353831</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>512.3292</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>263.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>body</th>\n",
       "      <td>NaN</td>\n",
       "      <td>160.809917</td>\n",
       "      <td>1188</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>328.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          survived_average  not survived_average  nulls  survived_min  \\\n",
       "pclass            1.962000              2.500618      0        1.0000   \n",
       "survived          1.000000              0.000000      0        1.0000   \n",
       "sex               0.678000              0.156984      0        0.0000   \n",
       "age              28.918228             30.545369    263        0.1667   \n",
       "sibsp             0.462000              0.521632      0        0.0000   \n",
       "parch             0.476000              0.328801      0        0.0000   \n",
       "fare             49.361184             23.353831      1        0.0000   \n",
       "body                   NaN            160.809917   1188           NaN   \n",
       "\n",
       "          survived_max  not survived_min  not survived_max  \n",
       "pclass          3.0000            1.0000               3.0  \n",
       "survived        1.0000            0.0000               0.0  \n",
       "sex             1.0000            0.0000               1.0  \n",
       "age            80.0000            0.3333              74.0  \n",
       "sibsp           4.0000            0.0000               8.0  \n",
       "parch           5.0000            0.0000               9.0  \n",
       "fare          512.3292            0.0000             263.0  \n",
       "body               NaN            1.0000             328.0  "
      ]
     },
     "execution_count": 431,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp = pd.merge(sum_data, x1, on = 'survived_average')\n",
    "final = pd.merge(temp, x2, on = 'not survived_average')\n",
    "final.index = ['pclass', 'survived','sex', 'age', 'sibsp', 'parch', 'fare', 'body']\n",
    "final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the describe table above, we can see that the class 1 people survived the least. The class 3 survived the max in fact. Could it be for the fact that the class3 passengers had better facilities or were given first preferences of life boats?\n",
    "\n",
    "Speaking of age, the younger people seem to have survived more but the range for not survived is smaller and spans till 74. Age has a lot of null values so we cannot be sure about its results.\n",
    "\n",
    "In alignment with the above analysis, mid sized families had a better chance of survival than singles and large families. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Subsetting the data for including only ordinal and categorical variables which show strong differences between survived and not survived\n",
    "dt_data = data[['pclass', 'sex','age', 'family class', 'embarked', 'survived']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 435,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Splitting the dataset based on 75% 25%\n",
    "dt_test = data.sample(frac = 0.25)\n",
    "dt_train = data.drop(dt_test.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make a function to calculate the entropy based on 2 columns. For eg, this function will calculate the entropy based on lengths of columns to result into a weighted entropy later. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 437,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def entropy_finder(x,y):\n",
    "    sum = x + y\n",
    "    if(x==0 or y==0):\n",
    "        entropy = 0\n",
    "    else:\n",
    "        entropy = -(x/sum)*np.log2(x/sum) - (y/sum)*np.log2(y/sum)\n",
    "    return(entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 439,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Entropy of survived in training data 0.9549950637253497\n"
     ]
    }
   ],
   "source": [
    "# Finding the entropy of the target variable in the training data\n",
    "entropy_target = entropy_finder(len(dt_train[dt_train['survived'] == 1]), len(dt_train[dt_train['survived'] == 0]))\n",
    "print('Entropy of survived in training data',entropy_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Making a function for finding entropy and information gain\n",
    "def col_entropy_finder(col_tar_data, col_name):\n",
    "    res = pd.DataFrame(columns = ['value', 'entropy', 'weight'])\n",
    "    len_tot = len(col_tar_data)\n",
    "    ig = 0\n",
    "    for elem in col_tar_data[col_name].unique():\n",
    "        x = col_tar_data[col_tar_data[col_name] == elem]\n",
    "        col_weight = len(x)\n",
    "        e1 = len(x[x['survived'] == 1])\n",
    "        e2 = len(x[x['survived'] == 0])\n",
    "        entr = entropy_finder(e1, e2)\n",
    "        idx = len(res) + 1\n",
    "        res.loc[idx] = [elem,entr,col_weight]\n",
    "        ig = ig + (entr*col_weight/len_tot)\n",
    "    \n",
    "    return(res,ig)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   value   entropy  weight\n",
      "1    1.0  0.958609   323.0\n",
      "2    2.0  0.985653   277.0\n",
      "3    3.0  0.819554   709.0\n",
      "Column entropy is 0.8890147580167741\n"
     ]
    }
   ],
   "source": [
    "f, col_entr = col_entropy_finder(data, 'pclass')\n",
    "print(f)\n",
    "print(\"Column entropy is\", col_entr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding an optimal value for splitting age "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating an ordered vector of unique age values (a1, a2, . . . , ak). These will form the potential split\n",
    "points for age. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 445,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We drop all NaN values for age and embarked\n",
    "data_new = data.drop(index = data[data['age'] == float('nan')].index)\n",
    "data_new = data.drop(index = data[data['age'].isnull() == True].index)\n",
    "\n",
    "data_new = data.drop(index = data[data['embarked'] == float('nan')].index)\n",
    "data_new = data.drop(index = data[data['embarked'].isnull() == True].index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 447,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Redefining test train for combined analysis\n",
    "dt_test = data_new.iloc[0:300]\n",
    "dt_train = data_new.drop(index = dt_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>entropy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>36.5000</td>\n",
       "      <td>0.886966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>36.0000</td>\n",
       "      <td>0.887311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>34.0000</td>\n",
       "      <td>0.887564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.887750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>37.0000</td>\n",
       "      <td>0.887764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>38.0000</td>\n",
       "      <td>0.887887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.887913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>34.5000</td>\n",
       "      <td>0.887918</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>33.0000</td>\n",
       "      <td>0.888051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>32.5000</td>\n",
       "      <td>0.888069</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        age   entropy\n",
       "62  36.5000  0.886966\n",
       "15  36.0000  0.887311\n",
       "20  34.0000  0.887564\n",
       "73   0.3333  0.887750\n",
       "37  37.0000  0.887764\n",
       "42  38.0000  0.887887\n",
       "74   0.1667  0.887913\n",
       "79  34.5000  0.887918\n",
       "51  33.0000  0.888051\n",
       "59  32.5000  0.888069"
      ]
     },
     "execution_count": 448,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Making an empty dataframe for storing the emtropy values for every splitting value of age\n",
    "age_entropy  = pd.DataFrame(columns = ['age', 'entropy'])\n",
    "# Dropping the NAN rows from training data\n",
    "for elem in dt_train['age'].unique():\n",
    "    # Splitting age according to threshold in this iteration\n",
    "    dt_train['age class'] = np.where(dt_train['age'] > elem, 1,0)\n",
    "    # Finding entropy of age column\n",
    "    f, en = col_entropy_finder(dt_train, 'age class')\n",
    "    idx = len(age_entropy) + 1\n",
    "    age_entropy.loc[idx] = [elem, en]\n",
    "\n",
    "age_entropy.nsmallest(10, columns = 'entropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have only found out this information for a single column and we need to know if this is the split based on the information gain. Just like we performed the above analysis for a single column, we perform this on all columns. Basically, we iterate this function over our dataset to find the entropy for all columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>entropy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.869473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.888770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.891828</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pclass   entropy\n",
       "2     2.0  0.869473\n",
       "1     1.0  0.888770\n",
       "3     3.0  0.891828"
      ]
     },
     "execution_count": 449,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Consider column pclass\n",
    "# Making an empty dataframe for storing the emtropy values for every splitting value of\n",
    "pclass_entropy  = pd.DataFrame(columns = ['pclass', 'entropy'])\n",
    "# Dropping the NAN rows from training data\n",
    "for elem in dt_train['pclass'].unique():\n",
    "    # Splitting age according to threshold in this iteration\n",
    "    dt_train['pclass class'] = np.where(dt_train['pclass'] > elem, 1,0)\n",
    "    # Finding entropy of age column\n",
    "    f, en = col_entropy_finder(dt_train, 'pclass class')\n",
    "    idx = len(pclass_entropy) + 1\n",
    "    pclass_entropy.loc[idx] = [elem, en]\n",
    "\n",
    "pclass_entropy.nsmallest(10, columns = 'entropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that for pclass, the entropy was much smaller than age, so that is probably a better column to start as a root. Now we create a loop to run this analysis for all columns except age, considered for the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create an analysis for all the possible column values and their thresholds to determine which combination would be the best for splitting the root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Converting all categorical values into ordinal values\n",
    "\n",
    "# Family class\n",
    "data_new['family class'] = np.where(data_new['family class'] == 'single', 0, np.where(data_new['family class'] == 'small',1, np.where(data_new['family class'] == 'medium', 2, 3)))\n",
    "data_new['embarked'] = np.where(data_new['embarked'] =='Q', 0, np.where(data_new['embarked'] =='C',1,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Redefining test train for combined analysis\n",
    "dt_test = data_new.iloc[0:300]\n",
    "dt_train = data_new.drop(index = dt_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([35.    , 64.    , 60.    , 54.    , 21.    , 55.    , 31.    ,\n",
       "       57.    , 45.    , 50.    , 27.    , 51.    ,     nan, 62.    ,\n",
       "       36.    , 30.    , 28.    , 18.    , 25.    , 34.    , 23.    ,\n",
       "       32.    , 19.    ,  1.    ,  4.    , 12.    , 26.    , 42.    ,\n",
       "       24.    , 15.    , 40.    , 20.    ,  0.8333, 22.    , 44.    ,\n",
       "       52.    , 37.    , 29.    ,  8.    , 48.    , 17.    , 38.    ,\n",
       "       16.    , 47.    ,  0.6667,  6.    ,  7.    , 43.    , 49.    ,\n",
       "       63.    , 33.    ,  3.    , 61.    , 46.    , 13.    , 41.    ,\n",
       "       39.    , 70.    , 32.5   , 14.    ,  2.    , 36.5   , 59.    ,\n",
       "       18.5   ,  0.9167,  5.    , 66.    ,  9.    , 11.    ,  0.75  ,\n",
       "       70.5   , 22.5   ,  0.3333,  0.1667, 65.    , 40.5   , 10.    ,\n",
       "       23.5   , 34.5   , 20.5   , 30.5   , 55.5   , 28.5   , 38.5   ,\n",
       "       14.5   , 24.5   , 60.5   , 74.    ,  0.4167, 11.5   , 45.5   ,\n",
       "       26.5   ])"
      ]
     },
     "execution_count": 452,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Finding unique values from the age\n",
    "dt_train['age'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 453,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>split_value</th>\n",
       "      <th>entropy</th>\n",
       "      <th>column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0.722999</td>\n",
       "      <td>sex</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.869473</td>\n",
       "      <td>pclass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>1</td>\n",
       "      <td>0.883848</td>\n",
       "      <td>embarked</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>36.5</td>\n",
       "      <td>0.886966</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>36</td>\n",
       "      <td>0.887311</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>34</td>\n",
       "      <td>0.887564</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0.3333</td>\n",
       "      <td>0.887750</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>37</td>\n",
       "      <td>0.887764</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>38</td>\n",
       "      <td>0.887887</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>0.1667</td>\n",
       "      <td>0.887913</td>\n",
       "      <td>age</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   split_value   entropy    column\n",
       "5            0  0.722999       sex\n",
       "2            2  0.869473    pclass\n",
       "99           1  0.883848  embarked\n",
       "67        36.5  0.886966       age\n",
       "20          36  0.887311       age\n",
       "25          34  0.887564       age\n",
       "78      0.3333  0.887750       age\n",
       "42          37  0.887764       age\n",
       "47          38  0.887887       age\n",
       "79      0.1667  0.887913       age"
      ]
     },
     "execution_count": 453,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = ['pclass', 'sex','age', 'family class', 'embarked']\n",
    "\n",
    "\n",
    "# Making an empty dataframe for storing the emtropy values for every splitting value of a given column\n",
    "result_entropy  = pd.DataFrame(columns = ['split_value', 'entropy', 'column'])\n",
    "for col in cols:\n",
    "        # Unique values\n",
    "        for elem in dt_train[col].unique():\n",
    "            # Splitting age according to threshold in this iteration\n",
    "            dt_train['class'] = np.where(dt_train[col] > elem, 1,0)\n",
    "            # Finding entropy of current column\n",
    "            f, en = col_entropy_finder(dt_train, 'class')\n",
    "            idx = len(result_entropy) + 1\n",
    "            result_entropy.loc[idx] = [elem, en, col]\n",
    "\n",
    "result_entropy.nsmallest(10, columns = 'entropy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence the 10 smallest entropies and thresholds are given in the column above. Let's make a function out of this analysis. So basically, the tree can be split on sex as the root node. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 454,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split_finder(data, cols):\n",
    "    result_entropy  = pd.DataFrame(columns = ['split_value', 'entropy', 'column'])\n",
    "    for col in cols:\n",
    "        # Unique values\n",
    "        for elem in data[col].unique():\n",
    "            # Splitting age according to threshold in this iteration\n",
    "            data['class'] = np.where(data[col] > elem, 1,0)\n",
    "            # Finding entropy of current column\n",
    "            f, en = col_entropy_finder(data, 'class')\n",
    "            idx = len(result_entropy) + 1\n",
    "            result_entropy.loc[idx] = [elem, en, col]\n",
    "    return(result_entropy[result_entropy['entropy'] == result_entropy['entropy'].min()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 455,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>split_value</th>\n",
       "      <th>entropy</th>\n",
       "      <th>column</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0.722999</td>\n",
       "      <td>sex</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  split_value   entropy column\n",
       "5           0  0.722999    sex"
      ]
     },
     "execution_count": 455,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(result_entropy[result_entropy['entropy'] == result_entropy['entropy'].min()].iloc[0]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minimum entropy was found on sex column. This could be our root node. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make a function that prints the percentage of survived and not survived on each branch given data and split point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 456,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38.19709702062643"
      ]
     },
     "execution_count": 456,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.groupby(['survived']).survived.count()[1]/data.survived.count()*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 457,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def per_finder(data):\n",
    "    if(len(data['survived'].unique()) > 0):\n",
    "        per_1 = len(data[data['survived'] == 1])/len(data)*100\n",
    "        per_0 = len(data[data['survived'] == 0])/len(data)*100\n",
    "    else:   \n",
    "        if(data['survived'].unique() == 0):\n",
    "            per_0 = 100\n",
    "            per_1 = 0\n",
    "        else:\n",
    "            per_1 = 100\n",
    "            per_0 = 0\n",
    "\n",
    "    return(per_1, per_0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 458,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Preparing data\n",
    "d_final = dt_train[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 459,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/manasi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "tree = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "path = 'root'\n",
    "def tree_maker(data, path):\n",
    "    cols = data.columns.drop(['survived'])\n",
    "    re = split_finder(data, cols)\n",
    "    if(np.shape(re)[0]>1):\n",
    "        return(1)\n",
    "    per_sur, per_nsur = per_finder(data)\n",
    "    tc = re['column'].item()\n",
    "    sp = re['split_value'].item()\n",
    "    data_left = data[ data[tc] > sp]\n",
    "    data_right = data.drop(index = data_left.index)\n",
    "    idx = len(tree) + 1\n",
    "    tree.loc[idx] = [path, len(data),per_sur, per_nsur, tc, sp]\n",
    "    if(per_sur == 0 or per_nsur == 0):\n",
    "        idx = len(tree) + 1\n",
    "        pathleaf = path + 'leaf'\n",
    "        tree.loc[idx] = [pathleaf, len(data),per_sur, per_nsur, tc, sp]\n",
    "        return(1)\n",
    "    else:\n",
    "        pathl = path + 'L'\n",
    "        pathr = path + 'R'\n",
    "        tree_maker(data_left, pathl)\n",
    "        tree_maker(data_right, pathr)  \n",
    "\n",
    "x = tree_maker(d_final, path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decision tree classification model holds the path from root for different values of % surived, % not survived along with column name used for splitting and the threshold for splitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets prepare the testing data. To demonstrate how we will use a prediction function to use this model we have built we shall apply the function on a couple of rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Testing data\n",
    "d_final_test = dt_test[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 461,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Building a function to predict the class of a given row\n",
    "# Initializing a path from root\n",
    "path = 'root'\n",
    "def predict_class(row, path):\n",
    "    rules = tree[tree['path'] == path]\n",
    "    #print(rules)\n",
    "    if(len(rules.index) == 0):\n",
    "        path = path[:-1]\n",
    "        rules = tree[tree['path'] == path]\n",
    "        x = np.where(rules['% sur'].item() > rules['% not_sur'].item(), 1, 0)\n",
    "        return(x.item())\n",
    "    else:\n",
    "        if(row[rules['columnOfSplit'].item()].item() > rules['thresholdOfSplit'].item()):\n",
    "            path = path + 'L'\n",
    "            return(predict_class(row, path))\n",
    "        else:\n",
    "            path = path + 'R'\n",
    "            return(predict_class(row, path))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have built the predictor function, we initialize a 'root' variable and pass a test row with some values into the function to see if the prediction happens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>family class</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pclass  sex   age  family class  embarked\n",
       "0       1    1  29.0             0         2"
      ]
     },
     "execution_count": 462,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = 'root'\n",
    "test = pd.DataFrame([[1,1,29.00,0,2]], columns = ['pclass', 'sex', 'age', 'family class', 'embarked'])\n",
    "test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 463,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "path = 'root'\n",
    "label = predict_class(test, path)\n",
    "print(label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 464,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>path</th>\n",
       "      <th>data_length</th>\n",
       "      <th>% sur</th>\n",
       "      <th>% not_sur</th>\n",
       "      <th>columnOfSplit</th>\n",
       "      <th>thresholdOfSplit</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>rootLR</td>\n",
       "      <td>112</td>\n",
       "      <td>90.178571</td>\n",
       "      <td>9.821429</td>\n",
       "      <td>age</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      path data_length      % sur  % not_sur columnOfSplit thresholdOfSplit\n",
       "48  rootLR         112  90.178571   9.821429           age               55"
      ]
     },
     "execution_count": 464,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree[tree['path'] == 'rootLR']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output value by the function is 1, that is survived and is a true positive as the test data for that row also has survived as the outcome. Now we run the function on all rows in testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 465,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 465,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = list()\n",
    "for elem in d_final_test.iterrows():\n",
    "    temp = pd.DataFrame(list(elem)[1:], columns = ['pclass', 'sex', 'age', 'family class', 'embarked'])\n",
    "    #print(list(elem)[1:])\n",
    "    #print('*******')\n",
    "    path = 'root'\n",
    "    t = predict_class(temp, path)\n",
    "    results.append(t)\n",
    "\n",
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 466,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test['predictions'] = results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the results, we can find the accuracy of the labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 467,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Making a function for finding accuracy score\n",
    "def accuracy_finder(data):\n",
    "    tp = 0\n",
    "    fp = 0\n",
    "    tn = 0\n",
    "    fn = 0\n",
    "    tp = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    fp = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    tn = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    fn = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    precision = tp.sum()/(tp.sum()+fp.sum())\n",
    "    recall = tp.sum()/(tp.sum()+fn.sum())\n",
    "    fsc = 2/((1/precision) + (1/recall))\n",
    "    accuracy = (tp.sum()+tn.sum())/(tp.sum()+tn.sum()+fp.sum()+fn.sum())\n",
    "    return(accuracy,fsc,recall,precision)\n",
    "\n",
    "#accuracy_finder(d_final_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 468,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is 0.7908745247148289 FSC is 0.8135593220338982 Recall is 0.7100591715976331 Precision is 0.9523809523809523\n"
     ]
    }
   ],
   "source": [
    "a,b,c,d = accuracy_finder(d_final_test)\n",
    "print('Accuracy is',a,'FSC is',b,'Recall is',c,'Precision is',d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, our algorithm has highest rate of true positives out of predicted positives. The accuracy is about 80%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bagging\n",
    "We implement bagging by iterating over a loop and storing the results of every bag into a dataframe. Then we calculate the majority and find final predictions.\n",
    "\n",
    "We create 5 bags of size 150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 469,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test = dt_test[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 470,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "path1 = 'root'\n",
    "def tree_maker1(data, path):\n",
    "    cols = data.columns.drop(['survived'])\n",
    "    re = split_finder(data, cols)\n",
    "    if(np.shape(re)[0]>1):\n",
    "        return(1)\n",
    "    per_sur, per_nsur = per_finder(data)\n",
    "    tc = re['column'].item()\n",
    "    sp = re['split_value'].item()\n",
    "    data_left = data[ data[tc] > sp]\n",
    "    data_right = data.drop(index = data_left.index)\n",
    "    idx = len(tree1) + 1\n",
    "    tree1.loc[idx] = [path, len(data),per_sur, per_nsur, tc, sp]\n",
    "    if(per_sur == 0 or per_nsur == 0):\n",
    "        idx = len(tree1) + 1\n",
    "        pathleaf = path + 'leaf'\n",
    "        tree1.loc[idx] = [pathleaf, len(data),per_sur, per_nsur, tc, sp]\n",
    "        return(1)\n",
    "    else:\n",
    "        pathl = path + 'L'\n",
    "        pathr = path + 'R'\n",
    "        tree_maker1(data_left, pathl)\n",
    "        tree_maker1(data_right, pathr)  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Building a function to predict the class of a given row\n",
    "# Initializing a path from root\n",
    "path1 = 'root'\n",
    "def predict_class1(row, path):\n",
    "    rules = tree1[tree1['path'] == path]\n",
    "    if(len(rules.index) == 0):\n",
    "        path = path[:-1]\n",
    "        rules = tree1[tree1['path'] == path]\n",
    "        x = np.where(rules['% sur'].item() > rules['% not_sur'].item(), 1, 0)\n",
    "        return(x.item())\n",
    "    else:\n",
    "        if(row[rules['columnOfSplit'].item()].item() > rules['thresholdOfSplit'].item()):\n",
    "            path = path + 'L'\n",
    "            return(predict_class1(row, path))\n",
    "        else:\n",
    "            path = path + 'R'\n",
    "            return(predict_class1(row, path))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/manasi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "#results_bag = pd.DataFrame(columns = ['bag1', 'bag2', 'bag3', 'bag4', 'bag5'])\n",
    "#for elem in ['bag1', 'bag2', 'bag3', 'bag4', 'bag5']:\n",
    "tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "path1 = 'root'\n",
    "bag_train = d_final.sample(n=100, replace = True)\n",
    "tree_maker1(bag_train, path1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 473,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = list()\n",
    "for elem in d_final_test.iterrows():\n",
    "    temp = pd.DataFrame(list(elem)[1:], columns = ['pclass', 'sex', 'age', 'family class', 'embarked'])\n",
    "    path = 'root'\n",
    "    t = predict_class1(temp, path)\n",
    "    results.append(t)\n",
    "\n",
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 474,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test['predictions'] = results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 475,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Making a function for finding accuracy score\n",
    "def accuracy_finder(data):\n",
    "    tp = 0\n",
    "    fp = 0\n",
    "    tn = 0\n",
    "    fn = 0\n",
    "    tp = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    fp = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    tn = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    fn = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    precision = tp.sum()/(tp.sum()+fp.sum())\n",
    "    recall = tp.sum()/(tp.sum()+fn.sum())\n",
    "    fsc = 2/((1/precision) + (1/recall))\n",
    "    accuracy = (tp.sum()+tn.sum())/(tp.sum()+tn.sum()+fp.sum()+fn.sum())\n",
    "    return(accuracy, fsc, recall, precision)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/manasi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "bagging_data = pd.DataFrame(columns = ['bag1', 'bag2','bag3','bag4','bag5'])\n",
    "for elem in bagging_data.columns:\n",
    "    tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "    bag_train = d_final.sample(n=300, replace = True)\n",
    "    tree_maker1(bag_train, 'root')\n",
    "    result = list()\n",
    "    d_final_test = dt_test[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()\n",
    "    for e in d_final_test.iterrows():\n",
    "        temp = pd.DataFrame(list(e)[1:], columns = ['pclass', 'sex', 'age', 'family class', 'embarked','survived'])\n",
    "        t = predict_class1(temp,'root')\n",
    "        result.append(t)\n",
    "    bagging_data[elem] = result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "majority = list()\n",
    "for dx in range(0,len(bagging_data)):\n",
    "    vals = bagging_data.iloc[dx].value_counts()\n",
    "    if(len(vals) == 1):\n",
    "        if(bagging_data.iloc[dx].unique().item() == 1):\n",
    "            majority.append(1)\n",
    "        else:\n",
    "            majority.append(0)\n",
    "    else:\n",
    "        maj = np.where(vals[0] < vals[1], 1, 0)\n",
    "        majority.append(maj)\n",
    "bagging_data['majority'] = majority"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 478,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test['predictions'] = majority"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 479,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a,b,c,d = accuracy_finder(d_final_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 480,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7832699619771863 0.8041237113402062 0.6923076923076923 0.9590163934426229\n"
     ]
    }
   ],
   "source": [
    "print(a,b,c,d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A difficulty of this method is that even though deep trees are constructed, the bagged trees that are created are very similar. In turn, the predictions made by these trees are also similar, and the high variance we desire among the trees trained on different samples of the training dataset is diminished.\n",
    "\n",
    "This is because of the greedy algorithm used in the construction of the trees selecting the same or similar split points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 481,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test = dt_test[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()\n",
    "cols = ['pclass', 'sex','age', 'family class', 'embarked', 'survived']\n",
    "x = np.random.choice(5,2,replace=False)\n",
    "cols = [cols[x[0]],cols[x[1]]\n",
    "       ]\n",
    "cols\n",
    "d_final_test = d_final_test[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 482,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "path1 = 'root'\n",
    "def tree_maker1(data, path):\n",
    "    cols = data.columns.drop(['survived'])\n",
    "    re = split_finder(data, cols)\n",
    "    if(np.shape(re)[0]>1):\n",
    "        return(1)\n",
    "    per_sur, per_nsur = per_finder(data)\n",
    "    tc = re['column'].item()\n",
    "    sp = re['split_value'].item()\n",
    "    data_left = data[ data[tc] > sp]\n",
    "    data_right = data.drop(index = data_left.index)\n",
    "    idx = len(tree1) + 1\n",
    "    tree1.loc[idx] = [path, len(data),per_sur, per_nsur, tc, sp]\n",
    "    if(per_sur == 0 or per_nsur == 0):\n",
    "        idx = len(tree1) + 1\n",
    "        pathleaf = path + 'leaf'\n",
    "        tree1.loc[idx] = [pathleaf, len(data),per_sur, per_nsur, tc, sp]\n",
    "        return(1)\n",
    "    else:\n",
    "        pathl = path + 'L'\n",
    "        pathr = path + 'R'\n",
    "        tree_maker1(data_left, pathl)\n",
    "        tree_maker1(data_right, pathr)  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 483,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Building a function to predict the class of a given row\n",
    "# Initializing a path from root\n",
    "path1 = 'root'\n",
    "def predict_class1(row, path):\n",
    "    rules = tree1[tree1['path'] == path]\n",
    "    if(len(rules.index) == 0):\n",
    "        path = path[:-1]\n",
    "        rules = tree1[tree1['path'] == path]\n",
    "        x = np.where(rules['% sur'].item() > rules['% not_sur'].item(), 1, 0)\n",
    "        return(x.item())\n",
    "    else:\n",
    "        if(row[rules['columnOfSplit'].item()].item() > rules['thresholdOfSplit'].item()):\n",
    "            path = path + 'L'\n",
    "            return(predict_class1(row, path))\n",
    "        else:\n",
    "            path = path + 'R'\n",
    "            return(predict_class1(row, path))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 484,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/manasi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "#results_bag = pd.DataFrame(columns = ['bag1', 'bag2', 'bag3', 'bag4', 'bag5'])\n",
    "#for elem in ['bag1', 'bag2', 'bag3', 'bag4', 'bag5']:\n",
    "tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "path1 = 'root'\n",
    "bag_train = d_final.sample(n=100, replace = True)\n",
    "tree_maker1(bag_train, path1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 485,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = list()\n",
    "for elem in d_final_test.iterrows():\n",
    "    temp = pd.DataFrame(list(elem)[1:], columns = ['pclass', 'sex', 'age', 'family class', 'embarked'])\n",
    "    path = 'root'\n",
    "    t = predict_class1(temp, path)\n",
    "    results.append(t)\n",
    "\n",
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 486,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test['predictions'] = results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 487,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Making a function for finding accuracy score\n",
    "def accuracy_finder(data):\n",
    "    tp = 0\n",
    "    fp = 0\n",
    "    tn = 0\n",
    "    fn = 0\n",
    "    tp = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    fp = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==1,1,0),0)\n",
    "    tn = np.where(data['survived'] == data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    fn = np.where(data['survived'] != data['predictions'], np.where(data['predictions']==0,1,0),0)\n",
    "    precision = tp.sum()/(tp.sum()+fp.sum())\n",
    "    recall = tp.sum()/(tp.sum()+fn.sum())\n",
    "    fsc = 2/((1/precision) + (1/recall))\n",
    "    accuracy = (tp.sum()+tn.sum())/(tp.sum()+tn.sum()+fp.sum()+fn.sum())\n",
    "    return(accuracy, fsc, recall, precision)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 488,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>family class</th>\n",
       "      <th>embarked</th>\n",
       "      <th>survived</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>302</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>304</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>60.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>305</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>306</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>309</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>31.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>310</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>45.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     pclass  sex   age  family class  embarked  survived  class\n",
       "302       1    1  35.0             3         1         1      1\n",
       "303       1    0  64.0             3         1         0      1\n",
       "304       1    1  60.0             3         1         1      1\n",
       "305       1    0  60.0             3         2         0      1\n",
       "306       1    0  54.0             3         2         0      1\n",
       "307       1    0  21.0             3         2         0      1\n",
       "308       1    1  55.0             3         1         1      1\n",
       "309       1    1  31.0             3         2         1      1\n",
       "310       1    0  57.0             3         2         0      1\n",
       "311       1    1  45.0             3         2         1      1"
      ]
     },
     "execution_count": 488,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d_final.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 489,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/manasi/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  import sys\n"
     ]
    }
   ],
   "source": [
    "bagging_data = pd.DataFrame(columns = ['bag1', 'bag2','bag3','bag4','bag5'])\n",
    "for elem in bagging_data.columns:\n",
    "    tree1 = pd.DataFrame(columns = ['path', 'data_length', '% sur', '% not_sur', 'columnOfSplit', 'thresholdOfSplit'])\n",
    "    d_final = dt_train[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()\n",
    "    cols = ['pclass', 'sex','age', 'family class', 'embarked']\n",
    "    x = np.random.choice(5,2,replace=False)\n",
    "    cols = [cols[x[0]],cols[x[1]],'survived']\n",
    "    d_final = d_final[cols]\n",
    "    bag_train = d_final.sample(n=500, replace = True)\n",
    "    tree_maker1(bag_train, 'root')\n",
    "    result = list()\n",
    "    #d_final_test = dt_test[cols].dropna()\n",
    "    d_final_test = dt_test[['pclass', 'sex','age', 'family class', 'embarked', 'survived']].dropna()\n",
    "    d_final_test = d_final_test[cols]\n",
    "    for e in d_final_test.iterrows():\n",
    "        temp = pd.DataFrame(list(e)[1:], columns = cols)\n",
    "        t = predict_class1(temp,'root')\n",
    "        result.append(int(t))\n",
    "    bagging_data[elem] = result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "majority = list()\n",
    "for dx in range(0,len(bagging_data)):\n",
    "    vals = bagging_data.iloc[dx].value_counts()\n",
    "    if(len(vals) == 1):\n",
    "        if(bagging_data.iloc[dx].unique().item() == 1):\n",
    "            majority.append(1)\n",
    "        else:\n",
    "            majority.append(0)\n",
    "    else:\n",
    "        maj = np.where(vals[0] < vals[1], 1, 0)\n",
    "        majority.append(maj)\n",
    "bagging_data['majority'] = majority"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "int_list = list()\n",
    "for l in majority:\n",
    "    int_list.append(int(l))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0]"
      ]
     },
     "execution_count": 492,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "int_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "d_final_test['predictions'] = int_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is 0.596958174904943 FScore is 0.5508474576271187 Recall is 0.38461538461538464 Precision is 0.9701492537313433\n"
     ]
    }
   ],
   "source": [
    "a,b,c,d = accuracy_finder(d_final_test)\n",
    "print('Accuracy is',a,'FScore is',b,'Recall is',c,'Precision is',d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the number of observations is large, but the number of trees is too small, then some observations will be predicted only once or even not at all. If the number of predictors is large but the number of trees is too small, then some features can (theoretically) be missed in all subspaces used. Both cases results in the decrease of random forest predictive power. But the last is a rather extreme case, since the selection of subspace is performed at each node."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}