Lab 17 - K-Nearest Neighbors for Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 17 - K-Nearest Neighbors for Classification\n",
    "\n",
    "Note:  This lab is partially based on the notebook [DAY 3: Classification - decision trees, random forests and k-NN classifiers](https://deepnote.com/project/3-more-classification-qkzrkkMjSYW8BqqN9u5Fyg/%2F05_classification_trees_and_forests.ipynb) from the IACS Data Science Winter Pedagogy Workshop by Weiwei Pan and David Sondak.\n",
    "\n",
    "The k-nearest neighbors algorithm predicts based on the values of the k closest training data. For example, a 3-nearest neighbor algorithm will find the 3 closest data points (using the Euclidean distance) in the training data and use them to make a prediction.\n",
    "\n",
    "If we are performing regression (trying to predict a quantitative value), the prediction is the mean of the y values of the k neighbors.  K-nearest neighbors regression was covered in Lab 13.\n",
    "\n",
    "If we are classifying (trying to predict qualitative value), the prediction is the class that appears the most in the k neighbors.\n",
    "\n",
    "We will start by looking at the vegetation data similar to Lab 16, and compare the decision boundaries made by decision trees, random forests, and k-nearest neighbors.\n",
    "\n",
    "### Section 1:  Loading and cleaning the data\n",
    "\n",
    "First, import the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, load the data.  As in Lab 16, the first two columns are the latitude and longitude normalization (centered at 0 and with variance 1) and the third column contains a 1 if that coordinate had vegetation and a 0 if not.\n",
    "\n",
    "The data is actually a different data set, and at the URL:  [https://raw.githubusercontent.com/cs109Alabs/lab_files/master/Lab_8/datasets/dataset_4.txt](https://raw.githubusercontent.com/cs109Alabs/lab_files/master/Lab_8/datasets/dataset_4.txt)\n",
    "\n",
    "Open the data in Jupyter notebook.  What do we have to do to read in this data with column names `x_coord`, `y_coord`, and `veg`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in Lab 16, plot the data as a scatterplot (with `x_coord` on the x axis and `y_coord` on the y axis) colored by the `veg` column.  If you are using Seaborn `relplot()`, you can color the vegetation green and the lack of vegetation brown with the parameter `palette = [\"brown\",\"green\"]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the location and shape of the vegetation?\n",
    "\n",
    "We will also use the scikit learn package for decision tree classifiers, so we need to prepare the data in the same way as previous labs (including Lab 16):\n",
    "* create a variable `x` containing the first two columns, which will be used to train the model and make the predictions.  The type of `x` should be a DataFrame.\n",
    "* create a variable `y` containing the third column, which is the variable we are trying to predict.  The type of `y` should be a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, split the data into training (80%) and testing (20%) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Section 2:  K-Nearest Neighbors Classifiers\n",
    "\n",
    "We will fit a k-nearest neighbors classifier with k = 5 to the training data with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "knn5 = KNeighborsClassifier(n_neighbors = 5)\n",
    "knn5.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you figure out how to use this model to predict whether there is a vegetation at each coordinate in the test data?  \n",
    "\n",
    "Hint:  All scikit-learn machine learning models use the same code functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details><summary>Answer:</summary>\n",
    "y_test_preds = knn5.predict(x_test)\n",
    "</details>\n",
    "\n",
    "We can also generate a confusion matrix to evaluate the classification with scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "confusion_matrix(y_test,y_test_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember for the `confusion_matrix()` function in scikit-learn the first parameter for is the rows and the second parameter is the columns.  Thus `confusion_matrix(y_test,y_predictions)` gives the confusion matrix:\n",
    "\n",
    "\n",
    "<code>   \n",
    "                        predicted\n",
    "             |    0           |      1      |\n",
    "             --------------------------------\n",
    "observed | 0 | true negative  | false positive\n",
    "         | 1 | false negative | true positive\n",
    "</code>\n",
    "\n",
    "If the categories are something other than 0 and 1, they will be ordered in alphabetically order (ie. the first row is the category closest to a in the alphabet).\n",
    "\n",
    "What is the accuracy of the classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the sensitivity of the classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the specificity of the classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which does our 5-nearest neighbors classifier predict the location of better:  vegetation or no vegetation?\n",
    "\n",
    "Let's visualize the decision boundary of the classifier.  Recall the decision boundary splits all possible coordinates into those predicted as 1 (vegetation) and those predicted as 0 (non-vegetation).\n",
    "\n",
    "Run the following code to load the visualization functions in your Jupyter notebook so you can use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def scatter_plot_data(x_df, y_series, ax):\n",
    "    '''\n",
    "    scatter_plot_data scatter plots the satellite data. A point in the plot is colored 'green' if \n",
    "    vegetation is present and 'gray' otherwise.\n",
    "    \n",
    "    input:\n",
    "       x_df - a DataFrame of size N x 2, each row is a location, each column is a coordinate\n",
    "       y_series - a Series of length N, each entry is either 0 (no vegetation) or 1 (vegetation)\n",
    "       ax - axis to plot on\n",
    "    returns: \n",
    "       ax - the axis with the scatter plot\n",
    "    '''\n",
    "    \n",
    "    # convert x_df and y_series into numpy arrays\n",
    "    x = x_df.values\n",
    "    y = y_series.values\n",
    "    \n",
    "    ax.scatter(x[y == 1, 0], x[y == 1, 1], alpha=0.2, c='green', label='vegetation')\n",
    "    ax.scatter(x[y == 0, 0], x[y == 0, 1], alpha=0.2, c='gray', label='nonvegetation')\n",
    "    ax.set_xlim([0, 1])\n",
    "    ax.set_ylim([0, 1])\n",
    "    ax.set_xlabel('Latitude')\n",
    "    ax.set_ylabel('Longitude')\n",
    "    ax.legend(loc='best')\n",
    "    return ax\n",
    "\n",
    "def plot_decision_boundary(x_df, y_series, model, ax, plot_boundary_only=False):\n",
    "    '''\n",
    "    plot_decision_boundary plots the training data and the decision boundary of the classifier.\n",
    "    input:\n",
    "       x_df - a DataFrame of size N x 2, each row is a location, each column is a coordinate\n",
    "       y_series - a Series of length N, each entry is either 0 (non-vegetation) or 1 (vegetation)\n",
    "       model - the 'sklearn' classification model\n",
    "       ax - axis to plot on\n",
    "       poly_degree - the degree of polynomial features used to fit the model\n",
    "    returns: \n",
    "       ax - the axis with the scatter plot\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # convert x_df and y_series into numpy arrays\n",
    "    x = x_df.values\n",
    "    y = y_series.values\n",
    "    \n",
    "    # Plot data\n",
    "    if not plot_boundary_only:\n",
    "        ax.scatter(x[y == 1, 0], x[y == 1, 1], alpha=0.2, c='green', label='vegetation')\n",
    "        ax.scatter(x[y == 0, 0], x[y == 0, 1], alpha=0.2, c='gray', label='non-vegetation')\n",
    "    \n",
    "    # Create mesh\n",
    "    interval = np.arange(0,1,0.01)\n",
    "    n = np.size(interval)\n",
    "    x1, x2 = np.meshgrid(interval, interval)\n",
    "    x1 = x1.reshape(-1, 1)\n",
    "    x2 = x2.reshape(-1, 1)\n",
    "    xx = np.concatenate((x1, x2), axis=1)\n",
    "    \n",
    "    # Predict on mesh points\n",
    "    yy = model.predict(xx)    \n",
    "    yy = yy.reshape((n, n))\n",
    "\n",
    "    # Plot decision surface\n",
    "    x1 = x1.reshape(n, n)\n",
    "    x2 = x2.reshape(n, n)\n",
    "    if not plot_boundary_only:\n",
    "        ax.contourf(x1, x2, yy, alpha=0.1, cmap='Greens')\n",
    "    ax.contour(x1, x2, yy, colors='black', linewidths=0.1)\n",
    "    ax.set_xlim([0, 1])\n",
    "    ax.set_ylim([0, 1])\n",
    "    ax.set_xlabel('Latitude')\n",
    "    ax.set_ylabel('Longitude')\n",
    "    ax.legend(loc='best')\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use this code to plot the decision boundary of the 5-nearest neighbor model we just fitted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# set up to create two plots in the same image\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "# visualize the data on the first plot (ax[0])\n",
    "scatter_plot_data(x_train, y_train, ax[0])\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "#  plot the training data and KNN decision boundary on the second plot (ax[1])\n",
    "plot_decision_boundary(x_train, y_train, knn5, ax[1])\n",
    "ax[1].set_title('Decision Boundary on the Training Data')\n",
    "\n",
    "#  plot the test data and KNN decision boundary on the third plot (ax[2])\n",
    "plot_decision_boundary(x_test, y_test, knn5, ax[2])\n",
    "ax[2].set_title('Decision Boundary on the Test Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the decision boundary?\n",
    "\n",
    "### Section 3: Changing the number of neighbors\n",
    "\n",
    "Let's see how changing the number of neighbors changes the decision boundary. \n",
    "\n",
    "Fill in the code below to create a new k-nearest neighbors classifier with k = 3 called `knn3`, and fit it to the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### FILL IN CODE\n",
    "# Create a 3-nearest neighbor classifier fit to the training data.\n",
    "\n",
    "\n",
    "# set up to create three plots in the same image\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "# visualize the data on the first plot (ax[0])\n",
    "scatter_plot_data(x_train, y_train, ax[0])\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "#  plot the training data and KNN decision boundary on the second plot (ax[1])\n",
    "plot_decision_boundary(x_train, y_train, knn3, ax[1])\n",
    "ax[1].set_title('Decision Boundary on the Training Data')\n",
    "\n",
    "#  plot the test data and KNN decision boundary on the third plot (ax[2])\n",
    "plot_decision_boundary(x_test, y_test, knn3, ax[2])\n",
    "ax[2].set_title('Decision Boundary on the Test Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How did the decision boundary change from the 5-nearest neighbor classifier?\n",
    "\n",
    "Now fill in the code below to create a k-nearest neighbor classifier with k = 10, called `knn10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### FILL IN CODE\n",
    "# Create a 3-nearest neighbor classifier fit to the training data.\n",
    "\n",
    "# set up to create two plots in the same image\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "# visualize the data on the first plot (ax[0])\n",
    "scatter_plot_data(x_train, y_train, ax[0])\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "#  plot the training data and KNN decision boundary on the second plot (ax[1])\n",
    "plot_decision_boundary(x_train, y_train, knn10, ax[1])\n",
    "ax[1].set_title('Decision Boundary on the Training Data')\n",
    "\n",
    "#  plot the test data and KNN decision boundary on the third plot (ax[2])\n",
    "plot_decision_boundary(x_test, y_test, knn10, ax[2])\n",
    "ax[2].set_title('Decision Boundary on the Test Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this decision boundary compare to the one with k = 3 and the one with k = 5?  Which classifier do you think is best? \n",
    "\n",
    "Let's also compare the confusion matrices.  Using the k = 3 classifier, predict the y values for the test data and compute the confuction matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this confusion matrix compare to the one for k = 5?  \n",
    "\n",
    "Now use your k = 10 classifier to predict the y values for the test data and compute that confuction matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this confusion matrix compare to the one for k = 3 and the one for k = 5?  \n",
    "\n",
    "\n",
    "### Section 4: Comparison with decision tree classifier\n",
    "\n",
    "Fill in the code below to fit a decision tree classifier of max depth 8 to the training data.  Call your classifier variable `tree`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### FILL IN CODE\n",
    "# Create a decision tree of max depth 8 fit to the training data.\n",
    "\n",
    "\n",
    "# set up to create two plots in the same image\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "# visualize the data on the first plot (ax[0])\n",
    "scatter_plot_data(x_train, y_train, ax[0])\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "#  plot the training data and KNN decision boundary on the second plot (ax[1])\n",
    "plot_decision_boundary(x_train, y_train, tree, ax[1])\n",
    "ax[1].set_title('Decision Boundary on the Training Data')\n",
    "\n",
    "#  plot the test data and KNN decision boundary on the third plot (ax[2])\n",
    "plot_decision_boundary(x_test, y_test, tree, ax[2])\n",
    "ax[2].set_title('Decision Boundary on the Test Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the decision boundary?  How does it compare to the k-nearest neighbor one?\n",
    "\n",
    "Use the decision tree classifier to make predictions using the test data and compute their confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the confusion matrix compare to the best k-nearest neighbors one?\n",
    "\n",
    "Try some different max depths.  Is the best decision tree better or worse than the k-nearest neighbors model with the best k?\n",
    "\n",
    "### Section 5: Comparison with random forest classifier\n",
    "\n",
    "Next try a random forest classifier, by creating one in the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### FILL IN CODE\n",
    "# Create a random forest classifier of max depth 10 with 500 decision trees,\n",
    "# fit to the training data.\n",
    "\n",
    "\n",
    "\n",
    "# set up to create two plots in the same image\n",
    "fig, ax = plt.subplots(1, 3, figsize=(15, 5))\n",
    "\n",
    "# visualize the data on the first plot (ax[0])\n",
    "scatter_plot_data(x_train, y_train, ax[0])\n",
    "ax[0].set_title('Training Data')\n",
    "\n",
    "#  plot the training data and KNN decision boundary on the second plot (ax[1])\n",
    "plot_decision_boundary(x_train, y_train, rf, ax[1])\n",
    "ax[1].set_title('Decision Boundary on the Training Data')\n",
    "\n",
    "#  plot the test data and KNN decision boundary on the third plot (ax[2])\n",
    "plot_decision_boundary(x_test, y_test, rf, ax[2])\n",
    "ax[2].set_title('Decision Boundary on the Test Data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the random forest decision boundary compare to the decision tree and k-nearest neighbors' decision boundaries?\n",
    "\n",
    "Try some different max depths and number of estimators (decision trees) to find the best combination.\n",
    "\n",
    "Using this combination, make a prediction for the test data and compute the confusion matrix for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the accuracy?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the sensitivity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the specificity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do these compare to the values for k-nearest neighbors?  Which model is best for predicting vegetation?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}