07-[TALLER]_Metricas_de_Evaluacion_de_modelos_de_ML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/UDEA-Esp-Analitica-y-Ciencia-de-Datos/EACD-04-MACHINE-LEARNING-1/blob/master/07-%5BTALLER%5D_Metricas_de_Evaluacion_de_modelos_de_ML.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "\n",
    "**Recuerda que una vez abierto, Da clic en \"Copiar en Drive\", de lo contrario no podras alamancenar tu progreso**\n",
    "\n",
    "Nota: no olvide ir ejecutando las celdas de código de arriba hacia abajo para que no tenga errores de importación de librerías o por falta de definición de variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget --no-cache -O init.py -q https://raw.githubusercontent.com/UDEA-Esp-Analitica-y-Ciencia-de-Datos/EACD-04-MACHINE-LEARNING-1/master/init.py\n",
    "import init; init.init(force_download=False); "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scikit-learn==0.24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ejercicio 1: Contextualización del problema\n",
    "\n",
    "\n",
    "Usaremos el dataset breast_cancer para el problema de clasificación. En el repositorio de sklearn se encuentra más información en el siguiente [link](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "data = load_breast_cancer()\n",
    "X = data.data\n",
    "Y = data.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Número de muestras 569\n",
      "Número de variables 30\n",
      "Número de clases 2\n"
     ]
    }
   ],
   "source": [
    "print (\"Número de muestras\", X.shape[0])\n",
    "print (\"Número de variables\", X.shape[1])\n",
    "print (\"Número de clases\", len(np.unique(Y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grafique un histograma de la variables de salida:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Ejercicio de código\n",
    "\n",
    "plt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ejercicio 2: Entrenamiento y evaluación de modelos\n",
    "\n",
    "En este ejercicio se usarán los modelos de regresión logística, Naïve Bayes y Discriminante Cuadrático para resolver el problema de predicción de cancer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a usar una partición similar a las usados en talleres anteriores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recuerde estandarizar variables..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Ejercicio de código\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = ...\n",
    "X_train_n = ...\n",
    "X_test_n = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instancie un objeto de cada uno de los modelos a comparar, use los parámetros por defecto para cada modelo y entrene el modelo con el conjunto X_train_n, y_train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Ejercicio de código\n",
    "\n",
    "clf1 = ...\n",
    "clf2 = ...\n",
    "clf3 = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A continuación cargaremos las métricas para evaluar los modelos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, balanced_accuracy_score, f1_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ejecute la siguiente celda para ver los resultados de los modelos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = ['modelo_1', 'modelo_2', 'modelo_3']\n",
    "measures = ['acc','ppv','recall','f1','bacc']\n",
    "Performance = []\n",
    "Performance.append([accuracy_score(y_test,clf1.predict(X_test_n)), accuracy_score(y_test,clf2.predict(X_test_n)), accuracy_score(y_test,clf3.predict(X_test_n))])\n",
    "Performance.append([precision_score(y_test,clf1.predict(X_test_n)), precision_score(y_test,clf2.predict(X_test_n)), precision_score(y_test,clf3.predict(X_test_n))])\n",
    "Performance.append([recall_score(y_test,clf1.predict(X_test_n)), recall_score(y_test,clf2.predict(X_test_n)), recall_score(y_test,clf3.predict(X_test_n))])\n",
    "Performance.append([f1_score(y_test,clf1.predict(X_test_n)), f1_score(y_test,clf2.predict(X_test_n)), f1_score(y_test,clf3.predict(X_test_n))])\n",
    "Performance.append([balanced_accuracy_score(y_test,clf1.predict(X_test_n)), balanced_accuracy_score(y_test,clf2.predict(X_test_n)), balanced_accuracy_score(y_test,clf3.predict(X_test_n))])\n",
    "\n",
    "x = np.arange(len(labels))*2  # the label locations\n",
    "width = 0.2  # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "for i in range(len(measures)):\n",
    "    ax.bar(x + i*width+0.3 -0.7, Performance[i],width, label=measures[i])\n",
    "    \n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Scores')\n",
    "ax.set_title('Scores by measure')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend(ncol=1,loc='center right', fontsize='small',bbox_to_anchor=(1.2, 0.5))\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Pregunta Abierta\n",
    "#@markdown  ¿El desbalance observado en el conjunto de datos afectó el desempeño de los modelos?\n",
    "respuesta_1 = \"\" #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#@title Pregunta Abierta\n",
    "#@markdown  Escoja una medida de desempeño global para evaluar el modelo. ¿Qué medida seleccionó? Justifique su respuesta.\n",
    "respuesta_2 = \"\" #@param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grafique la matriz de confusión normalizada para el mejor modelo de acuerdo con la medida de desempeño seleccionada  en la respuesta a la pregunta anterior. Revise la documentación del método en [link](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ConfusionMatrixDisplay.html#sklearn.metrics.ConfusionMatrixDisplay.from_estimator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Ejercicio de código\n",
    "from sklearn.metrics import ConfusionMatrixDisplay\n",
    "\n",
    "...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Grafique en una misma figura las curvas ROC de los tres modelos e incluya en las etiquetas de la figura, el Area Bajo la curva obtenida para cada modelo. Revise la documentación del método en [link](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Ejercicio de código\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "fpr = np.zeros(len(labels))\n",
    "tpr = np.zeros(len(labels))\n",
    "\n",
    "#Complete el código\n",
    "for i in range(len(labels)):\n",
    "    fpr[i], tpr[i] = ..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}