week1_session4_intro_to_ml_and_scikit_learn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI in Medicine: Data Science - Machine Learning\n",
    "## Python programming: Machine learning using *scikit-learn*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Tutor:** Roshan Prakash Rane, AG Ritter, Charité - Universitätsmedizin Berlin (roshan-prakash.rane@charite.de)\n",
    "- **Target audience**: Medical students from Charité"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:** If you are using this notebook outside of Google Colab, **please ensure that you're using a Python version 3.6 or higher**. You can check that in the top right corner of your browser window. If you're using a different Version, go to the Tab \"Kernel\" --> \"Change Kernel\" and select \"Python 3.6\" or higher. <br>\n",
    "\n",
    "**Executing the below cell should return a version higher than 3.6.0**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Aims of this session\n",
    "\n",
    "This session will serve as a programming tutorial following the previous theoretical session on machine learning. We will revisit the basic concepts of **machine learning** using a practical example. We will use the Python programming language and learn to use Python's machine learning package '*scikit-learn*' (along with *pandas*, *numpy* and *matplotlib* packages), to train different types of machine learning models and compare them.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Learning goals\n",
    "\n",
    "By the end of this session, you should be familiar with:\n",
    "\n",
    "- How to read a dataset, explore the different data columns and clean it, if necessary, for training machine learning models.\n",
    "- Cross validation: Why we should split our dataset into 'training' data and 'test' data and how to do it.\n",
    "- Learn to identify if the task at hand is a *classification task* or a *regression task*.\n",
    "- Training 2 different machine learning models: \n",
    "    - Support Vector Machine Classifiers (SVC)\n",
    "    - Logistic Regression\n",
    "- Compare the performance of different machine learning models and determine which one is better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. References\n",
    "\n",
    "Documentation for python libraries used in this notebook:\n",
    "\n",
    "- https://scikit-learn.org/stable/\n",
    "- https://pandas.pydata.org/pandas-docs/stable/\n",
    "- https://numpy.org/doc/\n",
    "- https://matplotlib.org/\n",
    "\n",
    "Documentation for classifiers:\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Support_vector_machine\n",
    "- https://en.wikipedia.org/wiki/Logistic_regression\n",
    "\n",
    "Documentation for metrics used:\n",
    "\n",
    "- https://en.wikipedia.org/wiki/Accuracy_and_precision\n",
    "- https://towardsdatascience.com/understanding-confusion-matrix-a9ad42dcfd62\n",
    "- https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5\n",
    "\n",
    "Further learning material:\n",
    "\n",
    "- StatQuest's Machine learning lecture series on youtube: ([click here](https://www.youtube.com/playlist?list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF))<br>(Episodes: Introduction, Cross-validation, Confusion Matrix, Sensitivity and Specificity, k-nearest neighbor, linear / logistic regression, support vector machines)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Theory\n",
    "The theory will be refreshed alongside, as we walk-through the practical sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Practical\n",
    "5.1 Data loading and exploration<br>\n",
    "5.2 Data preprocessing<br>\n",
    "5.3 Classification or regression?<br>\n",
    "5.4 Splitting the data into 'training' set and a 'test' set<br>\n",
    "5.5 Train machine learning models<br>\n",
    "5.6 Evaluate model performance<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Data loading and exploration\n",
    "\n",
    "* In this tutorial, we will look at the dataset provided by *Pima Indians Diabetes Database*. It is freely [available here](https://www.kaggle.com/uciml/pima-indians-diabetes-database). \n",
    "* Our objective is to build a machine learning model that can accurately predict (diagnose) whether or not a patient has diabetes or not, based on several diagnostic measurements provided in the dataset.\n",
    "* We will be using a subset of the larger database for ease-of-use. In our subset we have selected only females who are at least 21 years old, and of Pima Indian heritage. You can find this data subset in our github repository: https://github.com/volkamerlab/ai_in_medicine/raw/master/data/ .\n",
    "\n",
    "*In this part, you will get a chance to apply the lessons learned in the previous programming sessions too.*\n",
    "\n",
    "The source publication of the data for reference:<br>\n",
    "*Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., & Johannes, R.S. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care (pp. 261--265). IEEE Computer Society Press.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first load the dataset and try to understand the different columns provided: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, import the Pandas, NumPy and matplotlib libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the csv file as a pandas dataframe\n",
    "df = pd.read_csv(\"https://github.com/volkamerlab/ai_in_medicine/raw/master/data/diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#see the loaded dataframe \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can observe,\n",
    "- the dataset has 768 subjects and each subject have 9 variables:\n",
    "    1. Pregnancies\n",
    "    2. Glucose \n",
    "    3. BloodPressure \n",
    "    4. SkinThickness \n",
    "    5. Insulin \n",
    "    6. BMI \n",
    "    7. DiabetesPedigreeFunction \n",
    "    8. Age \n",
    "    9. Outcome\n",
    "- The last column \"Outcome\" denotes the diabetic information. A value of '1' here denotes that the subject has diabetes.\n",
    "- The rest of the 8 columns are medical predictor variables that are known to be associated with diabetes and they *might* help us to more accurately predict diabetes.\n",
    "\n",
    "**Our task here is to use these 8 medical predictor variables and try and predict if the subject has diabetes or not** (i.e. to predict the 'outcome' variable):\n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/MLflowchart.png\">\n",
    "\n",
    "(*[link to the source diagram](https://github.com/volkamerlab/ai_in_medicine/raw/master/images/MLflowchart.png)*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** \n",
    "1. What variables among these do you think are the most informative, medically, for diabetes diagnosis? \n",
    "2. Would a machine learning model that uses these variables and diagnoses diabetes accurately be beneficial for the medical field?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us try to visualize the 'distribution' of the diabetes column in our data. <br>\n",
    "We can use the `values_counts()` method of a `pandas.Dataframe`, to print the count of diabetic/ non-diabetic subjects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# count the subject with and without diabetes\n",
    "label = df[\"Outcome\"] # first, select the 'Outcome' column from the dataframe\n",
    "label_counts = label.value_counts() # next, use the pandas 'value_counts()' method \n",
    "# print the result\n",
    "label_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make these numbers more intuitive by plotting a 'bar' graph. <br>\n",
    "We can use the `plot()` function of pandas.Dataframe to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = label_counts.plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, we can definitely improve this graph a bit more and make it readable for anyone else who would look at it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = label_counts.plot(kind=\"bar\", \n",
    "                       title=\"Count of diabetic vs healthy subjects\",\n",
    "                       ylabel=\"number of subjects\", \n",
    "                       rot=0)\n",
    "tick_names = ax.set_xticklabels([\"healthy\", \"diabetic\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's visualize the distributions of the different medical variables that will be provided as input to our machine learning model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first, create a canvas on which 1 x 4 graphs can be drawn\n",
    "f, axes = plt.subplots(1, 4, sharey=True, figsize=(16,4))\n",
    "\n",
    "df.plot(y=\"Age\", kind=\"hist\", ax=axes[0])\n",
    "df.plot(y=\"Glucose\", kind=\"hist\", ax=axes[1])\n",
    "df.plot(y=\"DiabetesPedigreeFunction\", kind=\"hist\", ax=axes[2])\n",
    "df.plot(y=\"Pregnancies\", kind=\"hist\", ax=axes[3])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Side note: We can also use the 'density' plot to visualize the same data.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Now it's your turn. Similar to what we did in the previous cell, plot the distributions of the remaining 4 variables in the dataset:\n",
    "1. BMI\n",
    "2. SkinThickness\n",
    "3. BloodPressure\n",
    "4. Insulin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "# hint: start by copying the code from the above cell. \n",
    "# Next, provide the column names you want to visualize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- f, axes = plt.subplots(1, 4, sharey=True, figsize=(16,4))\n",
    "\n",
    "df.plot(y=\"BMI\", kind=\"hist\", ax=axes[0])\n",
    "df.plot(y=\"SkinThickness\", kind=\"hist\", ax=axes[1])\n",
    "df.plot(y=\"BloodPressure\", kind=\"hist\", ax=axes[2])\n",
    "df.plot(y=\"Insulin\", kind=\"hist\", ax=axes[3])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show() -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Preprocessing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, large datasets are messy with missing information or incorrect and invalid values due to human errors or systemic issues in the data collection process. Therefore, in practice, you need to preprocess the data. Otherwise, they wouldn't be reliable enough to train a machine learning model.\n",
    "\n",
    "Some of the prominent preprocessing steps include:\n",
    "* Dropping invalid/non-numeric/gibberish values\n",
    "* Cleaning unneeded columns\n",
    "* Converting measurement units\n",
    "* Reducing noise using smoothing functions\n",
    "* Standardizing and normalizing variable values: covered in the previous theory session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** Do you find any such discrepancies in our dataset so far?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some subjects have 'Glucose' value as '0' which can't be the case. Maybe the data was not collected for these subjects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the count of subjects with Glucose == 0\n",
    "(df['Glucose'] == 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(df['Glucose'] == 0).value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's drop these subjects from our data as they are probably not very reliable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select only the subjects who have a 'Glucose' value \n",
    "df_clean = df[(df['Glucose'] != 0)]\n",
    "df_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the same case with 'BMI' where some subjects have '0'. Let's remove them too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = df_clean[(df_clean['BMI'] != 0)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Similarly, lets also remove subjects with BloodPressure=0 as those are also probably incoherent data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--  df_clean = df_clean[(df_clean['BMI'] != 0)] -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Classification or regression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is our task a classification task or a regression task?\n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/classandregress.png\" width=\"700\" />\n",
    "\n",
    "*([link to source image](http://tonyeiyalla.com/images/classandregress.png))*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to look at our data again. <br>\n",
    "This time, let's use a 'scatter plot' to compare how 2 input variable relate to each others. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, axes = plt.subplots(1, 2, figsize=(10,4))\n",
    "\n",
    "# plot BMI vs Glucose \n",
    "ax1 = df_clean.plot(x=\"BMI\", y=\"Glucose\", kind=\"scatter\", ax=axes[0])\n",
    "\n",
    "# plot DiabetesPedigreeFunction vs Glucose \n",
    "ax2 = df_clean.plot(x=\"DiabetesPedigreeFunction\", y=\"Glucose\", kind=\"scatter\", ax=axes[1])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see how the output 'diabetes' variable relates to these relations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, axes = plt.subplots(1, 2, figsize=(10,4))\n",
    "\n",
    "# set a red color for diabetic subjects and blue for healthy subjects\n",
    "label = df_clean['Outcome'].map({0:'blue', 1:'red'})\n",
    "\n",
    "# plot BMI vs Glucose \n",
    "ax1 = df_clean.plot(x=\"BMI\", y=\"Glucose\", kind=\"scatter\", ax=axes[0], c=label)\n",
    "ax1.legend([\"Diabetic\", \"Healthy\"])\n",
    "\n",
    "# plot DiabetesPedigreeFunction vs Glucose \n",
    "ax2 = df_clean.plot(x=\"DiabetesPedigreeFunction\", y=\"Glucose\", kind=\"scatter\", ax=axes[1], c=label)\n",
    "ax2.legend([\"Diabetic\", \"Healthy\"])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** \n",
    "1. Do you see a correlation between the \"BMI\" and \"Glucose\" scores?\n",
    "2. What about between \"BMI\" and \"DiabetesPedigreeFunction\"?\n",
    "3. If you had to draw a line to differentiate between diabetic and healthy subjects in the first plot, where would you put it?\n",
    "4. Are we doing classification or regression?\n",
    "5. If we were trying to predict 'Glucose' from 'BMI', would we be doing a classification or a regression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the 'X' and y for our machine learning model as numpy arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df_clean[[\"Pregnancies\",\"Glucose\",\"BloodPressure\",\"SkinThickness\",\n",
    "              \"Insulin\",\"BMI\",\"DiabetesPedigreeFunction\",\"Age\"]].values\n",
    "y = df_clean[[\"Outcome\"]].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the shapes of our numpy arrays\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4 Splitting the data into 'training' set and 'test' set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why do we need a test set?\n",
    "\n",
    "Our goal is to be learn a (machine learning) model that **generalizes** well. <br>\n",
    "Over-fitting problem in a classification task vs a regression task:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/tuning.png\" width=\"300\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/fitting_data.png\" width=\"800\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "*([link to source](https://github.com/volkamerlab/ai_in_medicine/raw/update-2021.02/images/tuning.png))*\n",
    "*([link to source](https://github.com/volkamerlab/ai_in_medicine/raw/update-2021.02/images/fitting_data.png))*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, we split our dataset into 2 subsets: \n",
    "1. A larger subset on which we will train our model called the 'training' set. \n",
    "2. A smaller subset on which we will 'test' the model. It is important that we evaluate our model on a subset of the data it has never seen before to ensure that we are not overfitting the data and that our model can generalize well to unseen data.\n",
    "\n",
    "Let us use 20% of our data as the test set. The remainding 80% can be used to train the classifer. These ratios may vary depending on the size of the dataset we are using, but 20% to 80% is a good starting point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find out how many subject form '80%' in our data\n",
    "round(len(X)*80/100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = X[:579]\n",
    "x_test = X[579:]\n",
    "\n",
    "y_train = y[:579]\n",
    "y_test = y[579:]\n",
    "\n",
    "(x_train.shape), (x_test.shape), (y_train.shape), (y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has a machine learning library called '*sklearn*' that provides several convenient functions for machine learning:<br>\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/Scikit_learn_logo.png\" width=\"300\" /> \n",
    "*([link to source](https://en.wikipedia.org/wiki/Scikit-learn#/media/File:Scikit_learn_logo_small.svg))* <br>\n",
    "\n",
    "*sklearn* library has a function called `train_test_split()` that can be used for splitting our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import function for splitting our data \n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split the features and labels intro training and test sets by setting the test_size variable to 20%\n",
    "x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)\n",
    "\n",
    "(x_train.shape), (x_test.shape), (y_train.shape), (y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.5 Train machine learning models\n",
    "\n",
    "There are many different machine learning models. Just to name a few popular ones:\n",
    "1. Linear Regression\n",
    "2. Logistic Regression\n",
    "3. Decision Tree\n",
    "4. Linear Support Vector Machine Classifier (LinearSVC)\n",
    "5. Non-linear Support Vector Machine Classifier (SVC)\n",
    "6. Deep learning methods: \n",
    "    1. Feed-forward networks\n",
    "    2. Convolutional neural networks (CNN)\n",
    "    3. Recurrent neural networks (RNN)\n",
    "    \n",
    "We will try 3 different machine learning models for our task:\n",
    "1. Linear Support Vector Machine Classifier (LinearSVC)\n",
    "2. Non-linear Support Vector Machine Classifier (SVC)\n",
    "3. Logistic Regression\n",
    "\n",
    "Why do we need so many different models?\n",
    "- In machine learning, the idea of using an algorithm is to figure out the relationship or function mapping between the features X and target y.\n",
    "- Every machine learning algorithm makes its assumptions about the data based on which it generalizes and makes predictions. Its performance depends on how well the assumptions correlate with the underlying patterns in the data.\n",
    "- No free lunch theorem: There is \"no free lunch\" (best predictions) without having the best knowledge of the underlying data. It is the job of the data scientist to determine what fits best with the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.5.1 Linear Support Vector Machine Classifier (LinearSVC)\n",
    "To separate the two classes of data points, there are many possible hyperplanes that could be chosen. Our objective is to find a plane that has the maximum margin, i.e the maximum distance between data points of both classes. \n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/hyperplane.png\" width=\"400\" /> \n",
    "\n",
    "*([link to source](https://github.com/volkamerlab/ai_in_medicine/raw/update-2021.02/images/hyperplane.png))* \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import model classe from sklearn\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "# Create a model (also called creating an 'instance' of a model class in programming lingo) \n",
    "linsvc = LinearSVC(max_iter=2000)\n",
    "\n",
    "# fit the model to our train data using a class method\n",
    "linsvc.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = y_train.ravel()\n",
    "y_test = y_test.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linearly seperable task vs non-linearly seperable task:\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/linear_sep.png\" width=\"300\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/non-linear_sep.png\" width=\"300\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "(*[link to source](https://github.com/volkamerlab/ai_in_medicine/raw/master/images/linear_sep.png)*)\n",
    "(*[link to source](https://github.com/volkamerlab/ai_in_medicine/raw/master/images/non-linear_sep.png)*)\n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw//master/images/sphx_glr_plot_iris_svc_001.png\" width=\"700\" />\n",
    "\n",
    "(*[link to source](https://github.com/volkamerlab/ai_in_medicine/raw/master/images/sphx_glr_plot_iris_svc_001.png)*)\n",
    "\n",
    "Reading the warning message from python, it appears as though our linear SVC did not converge. This may imply that our data is not linearly separable and maybe a different classifier is better suited for the classification task. Lets forget the linear SVC and try the other classifier methods.\n",
    "\n",
    "\n",
    "#### 5.5.2 Non-linear Support Vector Machine Classifier (SVC) \n",
    "This is good for separating non-linearly separable data.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import model classe from sklearn\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# Instantiate an object of the model class\n",
    "svc = SVC(probability=True) \n",
    "# We set probability to True when instantiating our SVC model to get a probability estimate of the labels.\n",
    "\n",
    "# fit the model to our train data using a class method\n",
    "svc.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.5.3 Logistic Regression Classifier\n",
    "\n",
    "A logistic regression classifier is also good for separating non-linearly separable data.\n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/logistic_reg.png\" width=\"400\" /> *([link to source](https://github.com/volkamerlab/ai_in_medicine/raw/update-2021.02/images/logistic_reg.png))* \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import model from sklearn\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Instantiate an object of the model class\n",
    "logreg = LogisticRegression(max_iter=200) \n",
    "# We set the max iterations of our logistic classigier to 200 when we instantiate our class because it did not converges with the default value of 100.\n",
    "\n",
    "# fit the model to our train data using a class method\n",
    "logreg.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have two model that have been fit to our training data, we can use our test data to evaluate them.\n",
    "\n",
    "### 5.6 Evaluate model performance\n",
    "\n",
    "\n",
    "#### 5.6.1 Make predictions\n",
    "\n",
    "Lets make predictions on the test set that we will later compare to the respective true labels to evaluate of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import useful functions from the metrics module to evaluate our model\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import plot_confusion_matrix\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the predict() method from our model classes to predict labels given the test set of features x_test.\n",
    "y_pred_svc = svc.predict(x_test)\n",
    "y_pred_log = logreg.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.6.2 Accuracy\n",
    "\n",
    "$\\text{accuracy} = \\frac{tp + tn}{tp + fp + tn + fn}$, is a measure of how well our classifier can determine the true labels of inputs. <br><br>\n",
    "Here we will compute the accuracies of both of our models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we are using the model class method score() to return the accuracy of predictions from each model.\n",
    "acc_svc = svc.score(x_test, y_test)\n",
    "acc_log = logreg.score(x_test, y_test)\n",
    "print('Accuracy of Support Vector classifier on test set: {:.1f}%'.format(acc_svc*100))\n",
    "print('Accuracy of logistic regression classifier on test set: {:.1f}%'.format(acc_log*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although accuracy is a common metric, it often does not tell the whole story.\n",
    "\n",
    "We need other ways to assess how well our classifier performs.\n",
    "\n",
    "#### 5.6.3 Outcomes of a classifier\n",
    "\n",
    "_Successful predictions_ are only one of the possible outcomes of a prediction from a classifier. These outcomes can be generalized using the following four classes:\n",
    "\n",
    "- True positives\n",
    "- False positives\n",
    "- True negatives\n",
    "- False negatives\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/confusion_matrix.png\" width=\"500\" />\n",
    "        </td>\n",
    "        <td>\n",
    "            <img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/confusion_matrix_pregnancy.png\" width=\"500\" />\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "(*[link to source](https://dzone.com/articles/understanding-the-confusion-matrix)*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.6.4 Confusion Matrices\n",
    "\n",
    "We can visualize the distribution of prediction classes predicted using a classifier by plotting a confusion matrix. We now use the plot_confusion_matrix function we imported above to visualize the distribution of True Positives, False Positives, False Negatives, True Negatives for both of our classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the documentation to know what variables to use!\n",
    "fig, (ax1,ax2) = plt.subplots(1,2,figsize=(15, 5))\n",
    "\n",
    "plot_confusion_matrix(svc, x_test, y_test, ax=ax1) # , normalize='all'\n",
    "ax1.set_title('Confusion Matrix - SVC') # , fontsize=20\n",
    "\n",
    "plot_confusion_matrix(logreg, x_test, y_test, ax=ax2) # , normalize='all'\n",
    "ax2.set_title('Confusion Matrix - LR') # , fontsize=20\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.6.5 Receiver Operating Characteristic (ROC) Curve\n",
    "\n",
    "- The ROC curve is a graphical method to summarise all possible confusion matrix of a model. \n",
    "- It shows the performance of a classification model at all thresholds. \n",
    "- It is a common choice for assessing a binary classifier. \n",
    "- The ROC curve is a plot of the True Positive Rate (recall) vs the False Positive Rate. \n",
    "- The diagonal line (C) represents the performance of a random classifier that has a 50% chance of outputting either label. \n",
    "- The area under the curve (AUC) of the ROC curve is a measure that tells us how well our classifier can distinguish between the classes.\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/volkamerlab/ai_in_medicine/raw/master/images/rocs.png\" width=\"600\" /> \n",
    "\n",
    "(*[link to source](https://github.com/volkamerlab/ai_in_medicine/raw/master/images/rocs.png)*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "svc_roc_auc = roc_auc_score(y_test, svc.predict(x_test))\n",
    "logit_roc_auc = roc_auc_score(y_test, logreg.predict(x_test))\n",
    "print('ROC AUC score for Support Vector classifier on test set: {:.1f}%'.format(svc_roc_auc*100))\n",
    "print('ROC AUC score for logistic regression classifier on test set: {:.1f}%'.format(logit_roc_auc*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create Figure, Axes objects and set figure dimensions\n",
    "fig, (ax1,ax2) = plt.subplots(1,2,figsize=(15, 5))\n",
    "\n",
    "# compute and plot the AUC and ROC values for the support vector classifier\n",
    "fpr, tpr, thresholds = roc_curve(y_test, svc.predict_proba(x_test)[:,1])\n",
    "\n",
    "ax1.plot(\n",
    "    fpr, tpr, \n",
    "    label='SVC - RBF Kernel (area = {:.2f})'.format(svc_roc_auc*100)\n",
    ")\n",
    "ax1.plot([0, 1], [0, 1],'r--')\n",
    "ax1.set_xlim([0.0, 1.0])\n",
    "ax1.set_ylim([0.0, 1.05])\n",
    "ax1.set_xlabel('False Positive Rate')\n",
    "ax1.set_ylabel('True Positive Rate')\n",
    "ax1.set_title('Receiver operating characteristic')\n",
    "ax1.legend(loc=\"lower right\")\n",
    "ax1.grid()\n",
    "\n",
    "\n",
    "# compute and plot the AUC and ROC values for the logistic regression classifier\n",
    "fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(x_test)[:,1])\n",
    "\n",
    "ax2.plot(\n",
    "    fpr, tpr, \n",
    "    label='Logistic Regression (area = {:.2f})'.format(logit_roc_auc*100)\n",
    ")\n",
    "ax2.plot([0, 1], [0, 1],'r--')\n",
    "ax2.set_xlim([0.0, 1.0])\n",
    "ax2.set_ylim([0.0, 1.05])\n",
    "ax2.set_xlabel('False Positive Rate')\n",
    "ax2.set_ylabel('True Positive Rate')\n",
    "ax2.set_title('Receiver operating characteristic')\n",
    "ax2.legend(loc=\"lower right\")\n",
    "ax2.grid()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** Which model do you think is better? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, data scientists also use n-fold cross-validation and permutation tests to better compare models, tune hyperparameters, and get better estimates for our metrics. <br>\n",
    "This will result in a clearer picture of how the models preform, and which models to use for classification of novel data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Summary\n",
    "\n",
    "Lets briefly revisit some topics covered in this module:\n",
    "\n",
    "- Machine learning is a task of learning models that can map input variables to desired outputs with high precision; features to labels\n",
    "    - classification: outputs are discrete variables or classes\n",
    "    - regression: outputs are continuous variables\n",
    "- We can load our dataset in python from a csv file using pandas and explore it using different plotting types\n",
    "    - normally we would have to preprocess our data\n",
    "- Data is split up into training and test sets so that we can evaluate the model's generalization capacity on unseen data\n",
    "- models are trained on the training set and their performance evaluated on the test set\n",
    "- It is generally a good idea to use different metrics when evaluating a model \n",
    "    - this leads to a better understanding of how the model performs\n",
    "\n",
    "\n",
    "## 7. Exercises\n",
    "### Let us now use the different columns to predict if a subject's age is 35 years or older. \n",
    "Going through these exercises, you will develop a better undertanding of how to train and test models given a dataset. We suggest that you use the above code as a reference but do **NOT** simply copy and paste. You will gain a deeper understanding if your type the code yourself, implement the functions, and use docs to understand how functions work and what parameters to pass in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a dataframe for our task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set a common random seed to avoid getting very different scores due to model stochasticity\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add diabetes as another input variable\n",
    "df_age = df_clean.rename(columns={\"Outcome\":\"Diabetes\"})\n",
    "# Make age the new 'Outcome' variable\n",
    "df_age[\"Outcome\"] = (df_age[\"Age\"] >= 35).astype(int)\n",
    "df_age = df_age.drop(columns=[\"Age\"])\n",
    "df_age"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 Plot the new label's distribution or counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "# Note: Play around with the plt methods and parameter to see how it changes the \n",
    "#       display of your plots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- \n",
    "ax = df_age[\"Outcome\"].value_counts().plot(kind='bar',\n",
    "                       title=\"Age<35 vs Age>=35\",\n",
    "                       ylabel=\"number of subjects\", \n",
    "                       rot=0)\n",
    "tick_names = ax.set_xticklabels([\"Age<35\", \"Age>=35\"])\n",
    "plt.show()-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2 Create the new X (features) and y (labels) variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- X = df_age[[\"Pregnancies\",\"Glucose\",\"BloodPressure\",\"SkinThickness\",\n",
    "              \"Insulin\",\"BMI\",\"DiabetesPedigreeFunction\",\"Age\"]].values\n",
    "y = df_age[[\"Outcome\"]].values -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# verification: this cell should return ((724, 8), (724, 1))\n",
    "X.shape, y.shape "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3 Split your data into training and test sets\n",
    "\n",
    "You should use the `train_test_split` function imported from sklearn. We want you to use a test set size of 25%. \n",
    "\n",
    "Explicitly pass `shuffle=False` in `train_test_split` to make sure we all have the same train and test splits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False) \n",
    "y_train = y_train.ravel()\n",
    "y_test = y_test.ravel() -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# verification: this cell should return numpy shapes ((543, 8), (181, 8), (543, 1), (181, 1))\n",
    "(x_train.shape), (x_test.shape), (y_train.shape), (y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.4 Train your Models\n",
    "\n",
    "Implement the non-linear SVC and Logistic Regression models and fit them to the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "# Hint: Remember to compute the probabilities for SVC and make sure that the logistic regression \n",
    "#       model converges (by increasing the max_iter argument if necessary)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.5 Evaluate your Models\n",
    "\n",
    "#### 7.5.1 Predict the labels of the test set using both classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hint: Use the predict method from the classifier class\n",
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- y_pred_svc = svc.predict(x_test)\n",
    "y_pred_log = logreg.predict(x_test) -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.5.3 Compute and print the Accuracy for both models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hint: Use the score method from each model class.\n",
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- acc_svc = svc.score(x_test, y_test)\n",
    "acc_log = logreg.score(x_test, y_test)\n",
    "print('Accuracy of Support Vector classifier on test set: {:.1f}%'.format(acc_svc*100))\n",
    "print('Accuracy of logistic regression classifier on test set: {:.1f}%'.format(acc_log*100)) -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.5.3 Plot the confusion matrices from each classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hint: Use the plot_confusion_matrix function import from sklearn\n",
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- fig, (ax1,ax2) = plt.subplots(1,2,figsize=(15, 5))\n",
    "\n",
    "plot_confusion_matrix(svc, x_test, y_test, ax=ax1) # , normalize='all'\n",
    "ax1.set_title('Confusion Matrix - SVC') # , fontsize=20\n",
    "\n",
    "plot_confusion_matrix(logreg, x_test, y_test, ax=ax2) # , normalize='all'\n",
    "ax2.set_title('Confusion Matrix - LR') # , fontsize=20 \n",
    "\n",
    "plt.show() -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7.5.4 Plot the ROC curves and compute the AUC for both classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hint: Use the roc_auc_score and roc_curve functions\n",
    "# YOUR CODE GOES HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- fig, (ax1,ax2) = plt.subplots(1,2,figsize=(15, 5))\n",
    "\n",
    "\n",
    "# compute and plot the AUC and ROC values for the support vector classifier\n",
    "svc_roc_auc = roc_auc_score(y_test, svc.predict(x_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, svc.predict_proba(x_test)[:,1])\n",
    "\n",
    "\n",
    "ax1.plot(\n",
    "    fpr, tpr, \n",
    "    label='SVC - RBF Kernel (area = {:0.2f})'.format(svc_roc_auc)\n",
    ")\n",
    "ax1.plot([0, 1], [0, 1],'r--')\n",
    "ax1.set_xlim([0.0, 1.0])\n",
    "ax1.set_ylim([0.0, 1.05])\n",
    "ax1.set_xlabel('False Positive Rate')\n",
    "ax1.set_ylabel('True Positive Rate')\n",
    "ax1.set_title('Receiver operating characteristic')\n",
    "ax1.legend(loc=\"lower right\")\n",
    "ax1.grid()\n",
    "\n",
    "\n",
    "# compute and plot the AUC and ROC values for the logistic \n",
    "# regression classifier\n",
    "logit_roc_auc = roc_auc_score(y_test, logreg.predict(x_test))\n",
    "fpr, tpr, thresholds = roc_curve(y_test, logreg.predict_proba(x_test)[:,1])\n",
    "\n",
    "\n",
    "ax2.plot(\n",
    "    fpr, tpr, \n",
    "    label='Logistic Regression (area = {:0.2f})'.format(logit_roc_auc)\n",
    ")\n",
    "ax2.plot([0, 1], [0, 1],'r--')\n",
    "ax2.set_xlim([0.0, 1.0])\n",
    "ax2.set_ylim([0.0, 1.05])\n",
    "ax2.set_xlabel('False Positive Rate')\n",
    "ax2.set_ylabel('True Positive Rate')\n",
    "ax2.set_title('Receiver operating characteristic')\n",
    "ax2.legend(loc=\"lower right\")\n",
    "ax2.grid()\n",
    "\n",
    "plt.show() -->"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "380.8px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}