Hyper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.politecnicos.com.br/img/075.jpg\" alt=\"Grupo Turing\" height=\"420\" width=\"420\">\n",
    "Autor: Ariel Guerreiro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introdução\n",
    "Hiperparâmetros são parâmetros cujo valor é determinado antes da aprendizagem de um modelo. Para a biblioteca [sklearn](https://docs.google.com/document/u/1/d/1W7Z0SvUGHGU3YkTo1rECUMJervk7FhPlfCxfNkgiOdI/edit?usp=drive_open&ouid=110097037757552537186), os hiperparâmetros são fornecidos como argumentos da função responsável pela construção de um modelo, com cada modelo possuindo hiperparâmetros específicos.\n",
    "\n",
    "Para obter um melhor modelo de IA, é interessante obter os melhores hiperparâmetros. Esse notebook busca mostrar duas formas de se obter os melhores hiperparâmetros, dado um modelo e um dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construindo um modelo\n",
    "Antes da busca de hiperparâmetros, vamos primeiro criar o dataframe, separá-lo em treino e teste e criar um modelo. Neste notebook, será utilizado o [breast cancer wisconsin dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html), um dos datasets já presentes no sklearn, além da Decision Tree como modelo. Esse modelo já foi abordado em um **[Turing Talks](https://medium.com/turing-talks)**, cujo texto pode ser conferido [neste link](https://medium.com/turing-talks/turing-talks-17-modelos-de-predi%C3%A7%C3%A3o-decision-tree-610aa484cb05).\n",
    "\n",
    "Para a execução deste processo, é necessária a importação das bibliotecas [pandas](https://drive.google.com/drive/u/1/folders/1FlpRFtXXMytmqjW_ZPVd_U76jwHwcYY1) e a partes da já mencionada scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.30010</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>0.07871</td>\n",
       "      <td>...</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.08690</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>0.05667</td>\n",
       "      <td>...</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.19740</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>0.05999</td>\n",
       "      <td>...</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.24140</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>0.09744</td>\n",
       "      <td>...</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.19800</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>0.05883</td>\n",
       "      <td>...</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>12.45</td>\n",
       "      <td>15.70</td>\n",
       "      <td>82.57</td>\n",
       "      <td>477.1</td>\n",
       "      <td>0.12780</td>\n",
       "      <td>0.17000</td>\n",
       "      <td>0.15780</td>\n",
       "      <td>0.08089</td>\n",
       "      <td>0.2087</td>\n",
       "      <td>0.07613</td>\n",
       "      <td>...</td>\n",
       "      <td>23.75</td>\n",
       "      <td>103.40</td>\n",
       "      <td>741.6</td>\n",
       "      <td>0.1791</td>\n",
       "      <td>0.5249</td>\n",
       "      <td>0.5355</td>\n",
       "      <td>0.1741</td>\n",
       "      <td>0.3985</td>\n",
       "      <td>0.12440</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>18.25</td>\n",
       "      <td>19.98</td>\n",
       "      <td>119.60</td>\n",
       "      <td>1040.0</td>\n",
       "      <td>0.09463</td>\n",
       "      <td>0.10900</td>\n",
       "      <td>0.11270</td>\n",
       "      <td>0.07400</td>\n",
       "      <td>0.1794</td>\n",
       "      <td>0.05742</td>\n",
       "      <td>...</td>\n",
       "      <td>27.66</td>\n",
       "      <td>153.20</td>\n",
       "      <td>1606.0</td>\n",
       "      <td>0.1442</td>\n",
       "      <td>0.2576</td>\n",
       "      <td>0.3784</td>\n",
       "      <td>0.1932</td>\n",
       "      <td>0.3063</td>\n",
       "      <td>0.08368</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>13.71</td>\n",
       "      <td>20.83</td>\n",
       "      <td>90.20</td>\n",
       "      <td>577.9</td>\n",
       "      <td>0.11890</td>\n",
       "      <td>0.16450</td>\n",
       "      <td>0.09366</td>\n",
       "      <td>0.05985</td>\n",
       "      <td>0.2196</td>\n",
       "      <td>0.07451</td>\n",
       "      <td>...</td>\n",
       "      <td>28.14</td>\n",
       "      <td>110.60</td>\n",
       "      <td>897.0</td>\n",
       "      <td>0.1654</td>\n",
       "      <td>0.3682</td>\n",
       "      <td>0.2678</td>\n",
       "      <td>0.1556</td>\n",
       "      <td>0.3196</td>\n",
       "      <td>0.11510</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>13.00</td>\n",
       "      <td>21.82</td>\n",
       "      <td>87.50</td>\n",
       "      <td>519.8</td>\n",
       "      <td>0.12730</td>\n",
       "      <td>0.19320</td>\n",
       "      <td>0.18590</td>\n",
       "      <td>0.09353</td>\n",
       "      <td>0.2350</td>\n",
       "      <td>0.07389</td>\n",
       "      <td>...</td>\n",
       "      <td>30.73</td>\n",
       "      <td>106.20</td>\n",
       "      <td>739.3</td>\n",
       "      <td>0.1703</td>\n",
       "      <td>0.5401</td>\n",
       "      <td>0.5390</td>\n",
       "      <td>0.2060</td>\n",
       "      <td>0.4378</td>\n",
       "      <td>0.10720</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>12.46</td>\n",
       "      <td>24.04</td>\n",
       "      <td>83.97</td>\n",
       "      <td>475.9</td>\n",
       "      <td>0.11860</td>\n",
       "      <td>0.23960</td>\n",
       "      <td>0.22730</td>\n",
       "      <td>0.08543</td>\n",
       "      <td>0.2030</td>\n",
       "      <td>0.08243</td>\n",
       "      <td>...</td>\n",
       "      <td>40.68</td>\n",
       "      <td>97.65</td>\n",
       "      <td>711.4</td>\n",
       "      <td>0.1853</td>\n",
       "      <td>1.0580</td>\n",
       "      <td>1.1050</td>\n",
       "      <td>0.2210</td>\n",
       "      <td>0.4366</td>\n",
       "      <td>0.20750</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean radius  mean texture  mean perimeter  mean area  mean smoothness  \\\n",
       "0        17.99         10.38          122.80     1001.0          0.11840   \n",
       "1        20.57         17.77          132.90     1326.0          0.08474   \n",
       "2        19.69         21.25          130.00     1203.0          0.10960   \n",
       "3        11.42         20.38           77.58      386.1          0.14250   \n",
       "4        20.29         14.34          135.10     1297.0          0.10030   \n",
       "5        12.45         15.70           82.57      477.1          0.12780   \n",
       "6        18.25         19.98          119.60     1040.0          0.09463   \n",
       "7        13.71         20.83           90.20      577.9          0.11890   \n",
       "8        13.00         21.82           87.50      519.8          0.12730   \n",
       "9        12.46         24.04           83.97      475.9          0.11860   \n",
       "\n",
       "   mean compactness  mean concavity  mean concave points  mean symmetry  \\\n",
       "0           0.27760         0.30010              0.14710         0.2419   \n",
       "1           0.07864         0.08690              0.07017         0.1812   \n",
       "2           0.15990         0.19740              0.12790         0.2069   \n",
       "3           0.28390         0.24140              0.10520         0.2597   \n",
       "4           0.13280         0.19800              0.10430         0.1809   \n",
       "5           0.17000         0.15780              0.08089         0.2087   \n",
       "6           0.10900         0.11270              0.07400         0.1794   \n",
       "7           0.16450         0.09366              0.05985         0.2196   \n",
       "8           0.19320         0.18590              0.09353         0.2350   \n",
       "9           0.23960         0.22730              0.08543         0.2030   \n",
       "\n",
       "   mean fractal dimension  ...  worst texture  worst perimeter  worst area  \\\n",
       "0                 0.07871  ...          17.33           184.60      2019.0   \n",
       "1                 0.05667  ...          23.41           158.80      1956.0   \n",
       "2                 0.05999  ...          25.53           152.50      1709.0   \n",
       "3                 0.09744  ...          26.50            98.87       567.7   \n",
       "4                 0.05883  ...          16.67           152.20      1575.0   \n",
       "5                 0.07613  ...          23.75           103.40       741.6   \n",
       "6                 0.05742  ...          27.66           153.20      1606.0   \n",
       "7                 0.07451  ...          28.14           110.60       897.0   \n",
       "8                 0.07389  ...          30.73           106.20       739.3   \n",
       "9                 0.08243  ...          40.68            97.65       711.4   \n",
       "\n",
       "   worst smoothness  worst compactness  worst concavity  worst concave points  \\\n",
       "0            0.1622             0.6656           0.7119                0.2654   \n",
       "1            0.1238             0.1866           0.2416                0.1860   \n",
       "2            0.1444             0.4245           0.4504                0.2430   \n",
       "3            0.2098             0.8663           0.6869                0.2575   \n",
       "4            0.1374             0.2050           0.4000                0.1625   \n",
       "5            0.1791             0.5249           0.5355                0.1741   \n",
       "6            0.1442             0.2576           0.3784                0.1932   \n",
       "7            0.1654             0.3682           0.2678                0.1556   \n",
       "8            0.1703             0.5401           0.5390                0.2060   \n",
       "9            0.1853             1.0580           1.1050                0.2210   \n",
       "\n",
       "   worst symmetry  worst fractal dimension  target  \n",
       "0          0.4601                  0.11890       0  \n",
       "1          0.2750                  0.08902       0  \n",
       "2          0.3613                  0.08758       0  \n",
       "3          0.6638                  0.17300       0  \n",
       "4          0.2364                  0.07678       0  \n",
       "5          0.3985                  0.12440       0  \n",
       "6          0.3063                  0.08368       0  \n",
       "7          0.3196                  0.11510       0  \n",
       "8          0.4378                  0.10720       0  \n",
       "9          0.4366                  0.20750       0  \n",
       "\n",
       "[10 rows x 31 columns]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Criação do dataframe\n",
    "breast = datasets.load_breast_cancer()\n",
    "df = pd.DataFrame(breast.data, columns=breast.feature_names)\n",
    "df['target'] = pd.Series(breast.target)\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Separando os dados em 'train' e 'test'\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = df.drop('target', axis=1)\n",
    "y = df.target\n",
    "\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=0,test_size=0.2) #reservamos 20% dos dados para teste"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Criação de um modelo Decision Tree\n",
    "from sklearn import tree\n",
    "\n",
    "clf = tree.DecisionTreeClassifier() # vamos utilizar os hiperparâmetros default inicialmente"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Com isso feito, podemos começar a busca pelos hiperparâmetros ideais para estes dados.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Busca pelos melhores hiperparâmetros"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1º Método: Grid Search\n",
    "Grid search consiste no teste de todas as combinações de hiperparâmetros, dentro dos limites que estabelecemos. Podemos ver os hiperparâmetros de um knn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['class_weight', 'criterion', 'max_depth', 'max_features', 'max_leaf_nodes', 'min_impurity_decrease', 'min_impurity_split', 'min_samples_leaf', 'min_samples_split', 'min_weight_fraction_leaf', 'presort', 'random_state', 'splitter'])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.get_params().keys() #Podemos ver os parâmetros que o modelo possui"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Queremos descobrir se estes são os valores ideais, e é neste momento que o grid search é útil. Primeiro, criaremos um dicionário que possui os nomes dos hiperparâmetros e os valores que queremos testar. Caso haja dúvida sobre o significado de cada um, o ideal é buscar na [documentação](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html). Para esse exemplo, testaremos somente alguns do hiperparâmetros do modelo, deixando os outros no padrão."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': array([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       " 'min_samples_split': array([2, 3, 4, 5, 6, 7, 8, 9]),\n",
       " 'min_samples_leaf': array([1, 2, 3, 4, 5, 6, 7, 8, 9])}"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#criação do dicionário, com o auxilio da biblioteca NumPy\n",
    "import numpy as np\n",
    "\n",
    "max_depth = np.arange(1, 10)\n",
    "min_samples_split = np.arange(2, 10)\n",
    "min_samples_leaf = np.arange(1, 10)\n",
    "\n",
    "param_grid = {'max_depth' : max_depth,\n",
    "              'min_samples_split' : min_samples_split,\n",
    "              'min_samples_leaf' : min_samples_leaf\n",
    "}\n",
    "\n",
    "param_grid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Com o dicionário criado, podemos começar o teste de todas as permutações possíveis dentro do determinado em 'param_grid'. Importaremos então o Grid Search do sklearn, usando como critério de avaliação 'scoring = accuracy'. Para obtenção deste valor, utilizaremos [validação cruzada](https://scikit-learn.org/stable/modules/cross_validation.html), separando o dataframe de treino em 10 partes (cv = 10). Além desses argumentos, também são passados o modelo e o dicionário de valores possíveis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Ariel\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:813: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, error_score='raise-deprecating',\n",
       "             estimator=DecisionTreeClassifier(class_weight=None,\n",
       "                                              criterion='gini', max_depth=None,\n",
       "                                              max_features=None,\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              presort=False, random_state=None,\n",
       "                                              splitter='best'),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'max_depth': array([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       "                         'min_samples_leaf': array([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       "                         'min_samples_split': array([2, 3, 4, 5, 6, 7, 8, 9])},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "grid = GridSearchCV(clf, param_grid, cv=10, scoring='accuracy')\n",
    "\n",
    "grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Podemos então utilizar o próprio 'grid' para a previsão, que já se utiliza dos melhores hiperparâmetros encontrados nos limites estabelecidos. É possível visualizar os hiperparâmetros encontrados utilizando-se do argumento **.best_params_**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': 7, 'min_samples_leaf': 1, 'min_samples_split': 8}"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Para comparar, vamos criar um modelo com os hiperparâmetros padrão e compará-lo com o 'grid', utilizando-se do **accuracy_score** do sklearn para obter os resultados."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Avaliação dos modelos\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "#modelo com GridSearch\n",
    "y_pred_grid = grid.predict(X_test)\n",
    "accuracy_grid= accuracy_score(y_test, y_pred_grid)\n",
    "\n",
    "#modelo com hiperparâmetros padrão\n",
    "y_pred_std = tree_std.predict(X_test)\n",
    "accuracy_std = accuracy_score(y_test, y_pred_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Modelo com hiperparâmetros: 0.9473684210526315\n",
      "Modelo padrão: 0.9122807017543859\n"
     ]
    }
   ],
   "source": [
    "print(\"Modelo com hiperparâmetros: {}\".format(accuracy_grid))\n",
    "print(\"Modelo padrão: {}\".format(accuracy_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como podemos observar, os hiperparâmetros aumentaram a acurácia do modelo em relação ao modelo padrão.\n",
    "\n",
    "Porém, este método de busca, testando todas as permutações possíveis, tende a demorar muito para ser executado, à medida que o número de hiperparâmetros testados e suas possibilidades aumenta. Neste exemplo, onde foram testados somente 3 hiperparâmetros, o número de combinações que tiveram que ser testadas foi de 9 x 9 x 8 = **648 combinações**, o que já é demorado para se executar. Para modelos mais complexos, o número de combinações cresce muito além disso, tornando este método de busca de hiperparâmetros inviável. Para ajudar neste problema, surge outro método:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2º Método: Random Search\n",
    "O Random Search se assimila ao Grid Search, mas, ao invés de serem testadas todas as permutações possíveis de forma exaustiva, são executadas um número determinado de combinações. O tradeoff existente entre os métodos é de que o Random Search é mais rápido de ser executado, pelo menor número de testes, mas não garante que a combinação de hiperparâmetros é a ideal.\n",
    "\n",
    "Para o Random Search, usaremos o mesmo dicionário 'param_grid' utilizado no Grid Search, porém, é possível aumentar os limites dos valores possíveis para os hiperparâmetros, já que não serão testadas todas as possibilidades"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Criação de uma nova decision tree (essa etapa é desnecessária neste caso)\n",
    "clf = tree.DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Ariel\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:813: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=10, error_score='raise-deprecating',\n",
       "                   estimator=DecisionTreeClassifier(class_weight=None,\n",
       "                                                    criterion='gini',\n",
       "                                                    max_depth=None,\n",
       "                                                    max_features=None,\n",
       "                                                    max_leaf_nodes=None,\n",
       "                                                    min_impurity_decrease=0.0,\n",
       "                                                    min_impurity_split=None,\n",
       "                                                    min_samples_leaf=1,\n",
       "                                                    min_samples_split=2,\n",
       "                                                    min_weight_fraction_leaf=0.0,\n",
       "                                                    presort=False,\n",
       "                                                    random_state=None,\n",
       "                                                    splitter='best'),\n",
       "                   iid='warn', n_iter=300, n_jobs=None,\n",
       "                   param_distributions={'max_depth': array([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       "                                        'min_samples_leaf': array([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n",
       "                                        'min_samples_split': array([2, 3, 4, 5, 6, 7, 8, 9])},\n",
       "                   pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
       "                   return_train_score=False, scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "#É possível perceber a semelhança com o GridSearchCV\n",
    "random = RandomizedSearchCV(clf, param_grid, cv=10, scoring='accuracy', n_iter=300)\n",
    "\n",
    "random.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'min_samples_split': 5, 'min_samples_leaf': 1, 'max_depth': 8}\n"
     ]
    }
   ],
   "source": [
    "# Podemos ver os parâmetros encontrados e avaliar o score do modelo\n",
    "print(random.best_params_)\n",
    "\n",
    "y_pred_rand = random.predict(X_test)\n",
    "accuracy_rand = accuracy_score(y_test, y_pred_rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Podemos então comparar os 3 resultados, com os dois métodos utilizados:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Modelo com GridSearchCV: 0.9473684210526315\n",
      "Modelo com RandomizedSearchCV: 0.9385964912280702\n",
      "Modelo padrão: 0.9122807017543859\n"
     ]
    }
   ],
   "source": [
    "print(\"Modelo com GridSearchCV: {}\".format(accuracy_grid))\n",
    "print(\"Modelo com RandomizedSearchCV: {}\".format(accuracy_rand))\n",
    "print(\"Modelo padrão: {}\".format(accuracy_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusão\n",
    "É possível observar que ambas as formas de hyperparameter tuning foram capazes de gerar hiperparâmetros melhores que os obtidos pelo padrão do modelo, mas o 'score' do modelo que se utilizou do Random Search não obteve o resultado do Grid Search, pois o teste de forma aleatória não garante que os melhores hiperparâmetros serão encontrados, o que possui maior chance de ser encontrado pelo Grid Search.\n",
    "\n",
    "Em uma situação real, com datasets e modelos mais complexos, o ideal é se executar um Random Search com possibilidades mais abrangentes e realizar um Grid Search com valores próximos aos encontrados pelo Random Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}