songjiang_liu_finaltermproj.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "485d6be4f4d8c491",
   "metadata": {},
   "source": [
    "# FA24-CS634101 Data Mining\n",
    "## Final Term Project Report\n",
    "Professor: Yasser Abduallah"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e94b6a",
   "metadata": {},
   "source": [
    "Student: Songjiang Liu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b64dce3b",
   "metadata": {},
   "source": [
    "UCID: sl947"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4139cdb0",
   "metadata": {},
   "source": [
    "GitHub Repository: https://github.com/youjustlook/CS634_songjiang_liu_finaltermproj"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e12c953",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "This report uses three algorithms, namely random forest, decision tree, and convolutional neural network, to do binary prediction and then use several performance evaluation metrics to compare their performance and propose possible explanations for their results. The data set used is from https://archive.ics.uci.edu/dataset/222/bank+marketing, which contains marketing campaign data from a Portuguese bank, and the target is to predict whether the client would subscribe to a term deposit. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65bd604c",
   "metadata": {},
   "source": [
    "### About Algorithms Used\n",
    "**Random Forest**: an ensemble algorithm which combines multiple decision trees to improve accuracy, robustness, and generalization. <br>\n",
    "**Decision Tree**: a tree-like model to split data into branches based on feature conditions to make predictions or classify data. <br>\n",
    "**Convolutional Neural Network**: a deep learning algorithm uses filters (convolutions) to detect patterns, which are then combined to identify more complex structures in the data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f1ce42",
   "metadata": {},
   "source": [
    "### Step 0: How to Run this program / Install Necessary Packages\n",
    "The below packages are required to run the code for using random forest, decision tree, and converlutional neural network algorithms. The metrics used to evaluate the performance of the aforementioned algorithms are implemented manually with self-built functions. Run the following command to install the required packages if missing. Be sure to remove \"#\" before running the command\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-14T01:07:38.443599Z",
     "start_time": "2024-11-14T01:07:36.858885Z"
    },
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "# Run the following command to install the required packages if missing. Be sure to remove \"#\" before running the command\n",
    "\n",
    "# pip install pandas numpy scikit-learn tensorflow seaborn matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ede5b4e",
   "metadata": {},
   "source": [
    "### Step 1. Data Preparation, Metrics Calculation, and Tabular Display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d3008c7914e62f03",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T00:11:39.724827Z",
     "start_time": "2024-10-31T00:11:39.500197Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-------------------\n",
      "\n",
      "Data Preparation Started...\n",
      "\n",
      "-------------------\n",
      "\n",
      "Data Preparation Finished...\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.metrics import brier_score_loss\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "print(\"\\n-------------------\\n\\nData Preparation Started...\")\n",
    "\n",
    "data = pd.read_csv(\"bank.csv\", delimiter=\";\")\n",
    "\n",
    "# Encode categorical features\n",
    "categorical_columns = [\"job\", \"marital\", \"education\", \"default\", \"housing\", \"loan\", \"contact\", \"month\", \"poutcome\"]\n",
    "for col in categorical_columns:\n",
    "    data[col] = LabelEncoder().fit_transform(data[col])\n",
    "\n",
    "# Encode target\n",
    "data['y'] = LabelEncoder().fit_transform(data['y'])\n",
    "\n",
    "# Normalize\n",
    "numerical_columns = [\"age\", \"balance\", \"day\", \"duration\", \"campaign\", \"pdays\", \"previous\"]\n",
    "scaler = StandardScaler()\n",
    "data[numerical_columns] = scaler.fit_transform(data[numerical_columns])\n",
    "\n",
    "# Separate features and target\n",
    "X = data.drop(\"y\", axis=1)\n",
    "y = data[\"y\"]\n",
    "\n",
    "# Split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=66)\n",
    "\n",
    "# Setting up KFold\n",
    "n_splits = 10\n",
    "kf = KFold(n_splits=n_splits, shuffle=True, random_state=66)\n",
    "\n",
    "print(\"\\n-------------------\\n\\nData Preparation Finished...\")\n",
    "\n",
    "def calculate_metrics(cm, y_true=None, y_prob=None):\n",
    "    TN, FP, FN, TP = cm.ravel()\n",
    "    P = TP + FN  \n",
    "    N = TN + FP  \n",
    "    TPR = TP / P  \n",
    "    TNR = TN / N  \n",
    "    FPR = FP / N  \n",
    "    FNR = FN / P  \n",
    "    Recall = TPR  \n",
    "    Precision = TP / (TP + FP)  \n",
    "    F1 = 2 * (Precision * Recall) / (Precision + Recall)  \n",
    "    Accuracy = (TP + TN) / (P + N)  \n",
    "    Error_Rate = (FP + FN) / (P + N)  \n",
    "    BACC = (TPR + TNR) / 2  \n",
    "    TSS = TPR - FPR  \n",
    "    HSS = 2 * (TP * TN - FP * FN) / ((TP + FN) * (FN + TN) + (TP + FP) * (FP + TN))  \n",
    "    \n",
    "    # Calculate Brier Score and Brier Skill Score if probabilities are provided\n",
    "    if y_true is not None and y_prob is not None:\n",
    "        BS = brier_score_loss(y_true, y_prob)\n",
    "        BS_ref = brier_score_loss(y_true, [y_true.mean()] * len(y_true)) \n",
    "        BSS = 1 - (BS / BS_ref)\n",
    "    else:\n",
    "        BS = None\n",
    "        BSS = None\n",
    "    \n",
    "    return {\n",
    "        'TP': TP, 'TN': TN, 'FP': FP, 'FN': FN, 'P': P, 'N': N,\n",
    "        'TPR': TPR, 'TNR': TNR, 'FPR': FPR, 'FNR': FNR,\n",
    "        'Recall': Recall, 'Precision': Precision, 'F1': F1,\n",
    "        'Accuracy': Accuracy, 'Error Rate': Error_Rate,\n",
    "        'BACC': BACC, 'TSS': TSS, 'HSS': HSS,\n",
    "        'Brier Score': BS, 'Brier Skill Score': BSS\n",
    "    }\n",
    "\n",
    "# Function to display metrics\n",
    "def display_metrics(metrics_list, model_name):\n",
    "    metrics_df = pd.DataFrame(metrics_list).set_index(' ').T\n",
    "    metrics_df['Average'] = metrics_df.mean(axis=1)\n",
    "    metrics_df = metrics_df.round(2)\n",
    "    print(f\"\\n-------------------\\n{model_name} - Metrics for Each Fold and Average of 10 Folds\")\n",
    "    print(metrics_df.to_string())\n",
    "    return metrics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25adab2c",
   "metadata": {},
   "source": [
    "### Step 2. 10-Fold Validation on 3 Algorithms: Random Forest, Decision Tree, and ConvNet1D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8a4386ca",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-31T00:11:55.369208Z",
     "start_time": "2024-10-31T00:11:39.732844Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-------------------\n",
      "\n",
      "Model Running...\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 5ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 912us/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 4ms/step\n",
      "\u001b[1m15/15\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n",
      "\n",
      "-------------------\n",
      "\n",
      "Model Running Finished...\n",
      "\n",
      "-------------------\n",
      "Random Forest - Metrics for Each Fold and Average of 10 Folds\n",
      "                   fold_1  fold_2  fold_3  fold_4  fold_5  fold_6  fold_7  fold_8  fold_9  fold_10  Average\n",
      "TP                  17.00   15.00   14.00   15.00   17.00   20.00   14.00   15.00   18.00    16.00    16.10\n",
      "TN                 401.00  387.00  391.00  398.00  400.00  384.00  388.00  395.00  374.00   387.00   390.50\n",
      "FP                   1.00   10.00    7.00   10.00    4.00   14.00   11.00   13.00   18.00     7.00     9.50\n",
      "FN                  34.00   40.00   40.00   29.00   31.00   34.00   39.00   29.00   42.00    42.00    36.00\n",
      "P                   51.00   55.00   54.00   44.00   48.00   54.00   53.00   44.00   60.00    58.00    52.10\n",
      "N                  402.00  397.00  398.00  408.00  404.00  398.00  399.00  408.00  392.00   394.00   400.00\n",
      "TPR                  0.33    0.27    0.26    0.34    0.35    0.37    0.26    0.34    0.30     0.28     0.31\n",
      "TNR                  1.00    0.97    0.98    0.98    0.99    0.96    0.97    0.97    0.95     0.98     0.98\n",
      "FPR                  0.00    0.03    0.02    0.02    0.01    0.04    0.03    0.03    0.05     0.02     0.02\n",
      "FNR                  0.67    0.73    0.74    0.66    0.65    0.63    0.74    0.66    0.70     0.72     0.69\n",
      "Recall               0.33    0.27    0.26    0.34    0.35    0.37    0.26    0.34    0.30     0.28     0.31\n",
      "Precision            0.94    0.60    0.67    0.60    0.81    0.59    0.56    0.54    0.50     0.70     0.65\n",
      "F1                   0.49    0.37    0.37    0.43    0.49    0.45    0.36    0.42    0.37     0.40     0.42\n",
      "Accuracy             0.92    0.89    0.90    0.91    0.92    0.89    0.89    0.91    0.87     0.89     0.90\n",
      "Error Rate           0.08    0.11    0.10    0.09    0.08    0.11    0.11    0.09    0.13     0.11     0.10\n",
      "BACC                 0.67    0.62    0.62    0.66    0.67    0.67    0.62    0.65    0.63     0.63     0.64\n",
      "TSS                  0.33    0.25    0.24    0.32    0.34    0.34    0.24    0.31    0.25     0.26     0.29\n",
      "HSS                  0.46    0.32    0.33    0.39    0.46    0.40    0.31    0.37    0.31     0.35     0.37\n",
      "Brier Score          0.06    0.07    0.07    0.06    0.06    0.07    0.08    0.06    0.08     0.08     0.07\n",
      "Brier Skill Score    0.38    0.34    0.29    0.33    0.35    0.33    0.23    0.28    0.29     0.30     0.31\n",
      "\n",
      "-------------------\n",
      "Decision Tree - Metrics for Each Fold and Average of 10 Folds\n",
      "                   fold_1  fold_2  fold_3  fold_4  fold_5  fold_6  fold_7  fold_8  fold_9  fold_10  Average\n",
      "TP                  18.00   21.00   22.00   18.00   22.00   19.00   19.00   22.00   31.00    24.00    21.60\n",
      "TN                 375.00  372.00  368.00  375.00  381.00  366.00  369.00  379.00  346.00   359.00   369.00\n",
      "FP                  27.00   25.00   30.00   33.00   23.00   32.00   30.00   29.00   46.00    35.00    31.00\n",
      "FN                  33.00   34.00   32.00   26.00   26.00   35.00   34.00   22.00   29.00    34.00    30.50\n",
      "P                   51.00   55.00   54.00   44.00   48.00   54.00   53.00   44.00   60.00    58.00    52.10\n",
      "N                  402.00  397.00  398.00  408.00  404.00  398.00  399.00  408.00  392.00   394.00   400.00\n",
      "TPR                  0.35    0.38    0.41    0.41    0.46    0.35    0.36    0.50    0.52     0.41     0.42\n",
      "TNR                  0.93    0.94    0.92    0.92    0.94    0.92    0.92    0.93    0.88     0.91     0.92\n",
      "FPR                  0.07    0.06    0.08    0.08    0.06    0.08    0.08    0.07    0.12     0.09     0.08\n",
      "FNR                  0.65    0.62    0.59    0.59    0.54    0.65    0.64    0.50    0.48     0.59     0.58\n",
      "Recall               0.35    0.38    0.41    0.41    0.46    0.35    0.36    0.50    0.52     0.41     0.42\n",
      "Precision            0.40    0.46    0.42    0.35    0.49    0.37    0.39    0.43    0.40     0.41     0.41\n",
      "F1                   0.38    0.42    0.42    0.38    0.47    0.36    0.37    0.46    0.45     0.41     0.41\n",
      "Accuracy             0.87    0.87    0.86    0.87    0.89    0.85    0.86    0.89    0.83     0.85     0.86\n",
      "Error Rate           0.13    0.13    0.14    0.13    0.11    0.15    0.14    0.11    0.17     0.15     0.14\n",
      "BACC                 0.64    0.66    0.67    0.66    0.70    0.64    0.64    0.71    0.70     0.66     0.67\n",
      "TSS                  0.29    0.32    0.33    0.33    0.40    0.27    0.28    0.43    0.40     0.32     0.34\n",
      "HSS                  0.30    0.34    0.34    0.31    0.41    0.28    0.29    0.40    0.36     0.32     0.34\n",
      "Brier Score          0.13    0.13    0.14    0.13    0.11    0.15    0.14    0.11    0.17     0.15     0.14\n",
      "Brier Skill Score   -0.33   -0.22   -0.30   -0.49   -0.14   -0.41   -0.37   -0.28   -0.44    -0.36    -0.33\n",
      "\n",
      "-------------------\n",
      "Conv1D - Metrics for Each Fold and Average of 10 Folds\n",
      "                   fold_1  fold_2  fold_3  fold_4  fold_5  fold_6  fold_7  fold_8  fold_9  fold_10  Average\n",
      "TP                   0.00    1.00    0.00    1.00    0.00    3.00    0.00    0.00    0.00     1.00     0.60\n",
      "TN                 402.00  396.00  398.00  407.00  404.00  396.00  399.00  408.00  392.00   394.00   399.60\n",
      "FP                   0.00    1.00    0.00    1.00    0.00    2.00    0.00    0.00    0.00     0.00     0.40\n",
      "FN                  51.00   54.00   54.00   43.00   48.00   51.00   53.00   44.00   60.00    57.00    51.50\n",
      "P                   51.00   55.00   54.00   44.00   48.00   54.00   53.00   44.00   60.00    58.00    52.10\n",
      "N                  402.00  397.00  398.00  408.00  404.00  398.00  399.00  408.00  392.00   394.00   400.00\n",
      "TPR                  0.00    0.02    0.00    0.02    0.00    0.06    0.00    0.00    0.00     0.02     0.01\n",
      "TNR                  1.00    1.00    1.00    1.00    1.00    0.99    1.00    1.00    1.00     1.00     1.00\n",
      "FPR                  0.00    0.00    0.00    0.00    0.00    0.01    0.00    0.00    0.00     0.00     0.00\n",
      "FNR                  1.00    0.98    1.00    0.98    1.00    0.94    1.00    1.00    1.00     0.98     0.99\n",
      "Recall               0.00    0.02    0.00    0.02    0.00    0.06    0.00    0.00    0.00     0.02     0.01\n",
      "Precision             NaN    0.50     NaN    0.50     NaN    0.60     NaN     NaN     NaN     1.00     0.65\n",
      "F1                    NaN    0.04     NaN    0.04     NaN    0.10     NaN     NaN     NaN     0.03     0.05\n",
      "Accuracy             0.89    0.88    0.88    0.90    0.89    0.88    0.88    0.90    0.87     0.87     0.89\n",
      "Error Rate           0.11    0.12    0.12    0.10    0.11    0.12    0.12    0.10    0.13     0.13     0.11\n",
      "BACC                 0.50    0.51    0.50    0.51    0.50    0.53    0.50    0.50    0.50     0.51     0.51\n",
      "TSS                  0.00    0.02    0.00    0.02    0.00    0.05    0.00    0.00    0.00     0.02     0.01\n",
      "HSS                  0.00    0.03    0.00    0.04    0.00    0.08    0.00    0.00    0.00     0.03     0.02\n",
      "Brier Score          0.09    0.09    0.09    0.08    0.08    0.09    0.09    0.08    0.10     0.10     0.09\n",
      "Brier Skill Score    0.12    0.13    0.16    0.09    0.11    0.12    0.10    0.12    0.12     0.13     0.12\n",
      "\n",
      "-------------------\n",
      "\n",
      "Combined Average Results of Random Forest, Decision Tree, and Conv1D\n",
      "                    Random Forest  Decision Tree  Conv1D\n",
      "TP                         16.10          21.60    0.60\n",
      "TN                        390.50         369.00  399.60\n",
      "FP                          9.50          31.00    0.40\n",
      "FN                         36.00          30.50   51.50\n",
      "P                          52.10          52.10   52.10\n",
      "N                         400.00         400.00  400.00\n",
      "TPR                         0.31           0.42    0.01\n",
      "TNR                         0.98           0.92    1.00\n",
      "FPR                         0.02           0.08    0.00\n",
      "FNR                         0.69           0.58    0.99\n",
      "Recall                      0.31           0.42    0.01\n",
      "Precision                   0.65           0.41    0.65\n",
      "F1                          0.42           0.41    0.05\n",
      "Accuracy                    0.90           0.86    0.89\n",
      "Error Rate                  0.10           0.14    0.11\n",
      "BACC                        0.64           0.67    0.51\n",
      "TSS                         0.29           0.34    0.01\n",
      "HSS                         0.37           0.34    0.02\n",
      "Brier Score                 0.07           0.14    0.09\n",
      "Brier Skill Score           0.31          -0.33    0.12\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import pandas as pd\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Input, Conv1D, GlobalMaxPooling1D, Dense, Dropout\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "dt_metrics_list = []\n",
    "rf_metrics_list = []\n",
    "conv1d_metrics_list = []\n",
    "\n",
    "print(\"\\n-------------------\\n\\nModel Running...\")\n",
    "# Perform K-Fold Cross Validation\n",
    "for i, (train_index, test_index) in enumerate(kf.split(X), start=1):\n",
    "    # Splitting data\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "    X_train_cnn = np.expand_dims(X_train, axis=2)\n",
    "    X_test_cnn = np.expand_dims(X_test, axis=2)\n",
    "    \n",
    "    # Model Training - Random Forest\n",
    "    rf_model = RandomForestClassifier(n_estimators=100, random_state=66)\n",
    "    rf_model.fit(X_train, y_train)\n",
    "\n",
    "    rf_preds = rf_model.predict(X_test)\n",
    "    rf_probs = rf_model.predict_proba(X_test)[:, 1]  # Get probabilities for the positive class\n",
    "\n",
    "    # Calculate and store metrics\n",
    "    rf_cm = confusion_matrix(y_test, rf_preds)\n",
    "    rf_metrics = calculate_metrics(rf_cm, y_true=y_test, y_prob=rf_probs)\n",
    "    rf_metrics[' '] = f'fold_{i}'\n",
    "    rf_metrics_list.append(rf_metrics)\n",
    "\n",
    "    # Model Training - Decision Tree\n",
    "    dt_model = DecisionTreeClassifier(random_state=66)\n",
    "    dt_model.fit(X_train, y_train)\n",
    "    # Predictions and Probabilities\n",
    "    dt_preds = dt_model.predict(X_test)\n",
    "    dt_probs = dt_model.predict_proba(X_test)[:, 1]  # Get probabilities for the positive class\n",
    "\n",
    "    # Calculate and store metrics\n",
    "    dt_cm = confusion_matrix(y_test, dt_preds)\n",
    "    dt_metrics = calculate_metrics(dt_cm, y_true=y_test, y_prob=dt_probs)\n",
    "    dt_metrics[' '] = f'fold_{i}'  # Label each fold\n",
    "    dt_metrics_list.append(dt_metrics)\n",
    "\n",
    "    # Model Training - Conv1D\n",
    "    # Conv1D Model Definition\n",
    "    conv1d_model = Sequential()\n",
    "    conv1d_model.add(Input(shape=(X_train_cnn.shape[1], 1)))  # Define input shape using Input layer\n",
    "    conv1d_model.add(Conv1D(64, kernel_size=4, activation='relu'))\n",
    "    conv1d_model.add(GlobalMaxPooling1D())\n",
    "    conv1d_model.add(Dropout(0.3))\n",
    "    conv1d_model.add(Dense(64, activation='relu'))\n",
    "    conv1d_model.add(Dropout(0.3))\n",
    "    conv1d_model.add(Dense(1, activation='sigmoid'))  # Sigmoid for binary classification\n",
    "\n",
    "    # Compile and Train Model\n",
    "    conv1d_model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
    "    conv1d_model.fit(X_train_cnn, y_train, epochs=5, batch_size=64, verbose=0);\n",
    "    \n",
    "    # Predictions and Confusion Matrix\n",
    "    conv1d_preds = (conv1d_model.predict(X_test_cnn) > 0.5).astype(int).flatten();\n",
    "    conv1d_probs = conv1d_model.predict(X_test_cnn).flatten()\n",
    "    conv1d_cm = confusion_matrix(y_test, conv1d_preds)\n",
    "    \n",
    "    # Calculate and store metrics\n",
    "    conv1d_metrics = calculate_metrics(conv1d_cm, y_true=y_test, y_prob=conv1d_probs)\n",
    "    conv1d_metrics[' '] = f'fold_{i}'  # Label each fold\n",
    "    conv1d_metrics_list.append(conv1d_metrics)\n",
    "\n",
    "print(\"\\n-------------------\\n\\nModel Running Finished...\")   \n",
    "\n",
    "# Display metrics and capture returned DataFrames with 'Average' column\n",
    "rf_metrics_df = display_metrics(rf_metrics_list, \"Random Forest\")\n",
    "dt_metrics_df = display_metrics(dt_metrics_list, \"Decision Tree\")\n",
    "conv1d_metrics_df = display_metrics(conv1d_metrics_list, \"Conv1D\")\n",
    "\n",
    "# Combine and print the Average columns in one line\n",
    "print(\"\\n-------------------\\n\\nCombined Average Results of Random Forest, Decision Tree, and Conv1D\\n\", \n",
    "      pd.DataFrame({\n",
    "          'Random Forest': rf_metrics_df['Average'], \n",
    "          'Decision Tree': dt_metrics_df['Average'], \n",
    "          'Conv1D': conv1d_metrics_df['Average']\n",
    "      }).round(2).to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4800746d",
   "metadata": {},
   "source": [
    "### Discussion of Results\n",
    "**Random Forest**: Achieved the best overall balance between sensitivity (TPR = 0.31), specificity (TNR = 0.98), and accuracy (90%). It also had the highest precision (0.65) and F1 score (0.42). Error rate and Brier score were relatively low, indicating robust performance.<br>\n",
    "**Decision Tree**: Demonstrated moderate performance with a slightly better sensitivity (TPR = 0.42) compared to Random Forest but lower specificity (TNR = 0.92). Its accuracy (86%) and precision (0.41) were inferior to Random Forest, and the higher Brier score (0.14) reflects less reliability.<br>\n",
    "**Convolutional Neural Network**: Performed poorly for true positive detection (TP = 0.10, TPR = 0.00) but excelled in specificity (TNR = 1.00). Its accuracy (88%) was decent due to the high true negative detection but failed to balance positive predictions, leading to the lowest F1 score (0.04).<br>\n",
    "\n",
    "### Which Performed Better and Why\n",
    "Random Forest performed better overall due to its superior balance between sensitivity, specificity, and accuracy. Its ensemble approach mitigates overfitting, making it more robust in various scenarios. It aggregates predictions from multiple decision trees, effectively reducing variance and improving generalization. The higher Brier Skill Score (0.31) further emphasizes its reliability and predictive skill in scenarios with class imbalance or rare events. Decision Tree is relatively poorly performing as it suffers from overfitting to the training data and its inability to handle complexity. Convolutional Neural Network failed to detect true positives effectively, likely due to insufficient data for the deep learning model or improper tuning, which is critical for neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97d1472",
   "metadata": {},
   "source": [
    "### "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}