ScilearnValidation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Model-Validation-and-Over-training\" data-toc-modified-id=\"Model-Validation-and-Over-training-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Model Validation and Over-training</a></span><ul class=\"toc-item\"><li><span><a href=\"#Validation---simple-way\" data-toc-modified-id=\"Validation---simple-way-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Validation - simple way</a></span></li><li><span><a href=\"#Cross-validation\" data-toc-modified-id=\"Cross-validation-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Cross-validation</a></span></li><li><span><a href=\"#Selecting-a-sensible-Model\" data-toc-modified-id=\"Selecting-a-sensible-Model-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Selecting a sensible Model</a></span></li><li><span><a href=\"#Validation-curves-in-Scikit-Learn\" data-toc-modified-id=\"Validation-curves-in-Scikit-Learn-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Validation curves in Scikit-Learn</a></span><ul class=\"toc-item\"><li><span><a href=\"#Linear-regression-example\" data-toc-modified-id=\"Linear-regression-example-1.4.1\"><span class=\"toc-item-num\">1.4.1&nbsp;&nbsp;</span>Linear regression example</a></span></li></ul></li><li><span><a href=\"#Learning-Curves-and-Data-Size\" data-toc-modified-id=\"Learning-Curves-and-Data-Size-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Learning Curves and Data Size</a></span></li><li><span><a href=\"#Further-validation-issues\" data-toc-modified-id=\"Further-validation-issues-1.6\"><span class=\"toc-item-num\">1.6&nbsp;&nbsp;</span>Further validation issues</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Validation and Over-training\n",
    "\n",
    "A critical part of Supervised Learning is the so-called **over-training**\n",
    "* Model learns specific features of the elements in training datasets rather than the general properties\n",
    "  * one symptom is difference between accuracy for training and validation sets\n",
    "  * depends on details of model\n",
    "  * and size of training data sets\n",
    "    * most critical for small samples\n",
    "    \n",
    "To check and avoid  over-training a proper **validation**\n",
    "plays a crucial role \n",
    "\n",
    "\n",
    "***\n",
    "**How not to do it - brief demo with Iris data and kNN:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take iris directly from sklearn, already prepared for easy use\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "model = KNeighborsClassifier(n_neighbors=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train model\n",
    "model.fit(X, y)\n",
    "# apply to data\n",
    "y_model = model.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check accuracy\n",
    "from sklearn.metrics import accuracy_score\n",
    "print(accuracy_score(y, y_model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**100 % accuracy** -- perfect model\n",
    "\n",
    "Can this be true?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Not really. Two problems:\n",
    "* we used same dataset for training and test, not sensitive to overtraining\n",
    "* special case of kNN with n=1:\n",
    "  * by construction will always give 100% when training dataset is used for testing:\n",
    "    * It will always find identical element as single closest neighbor and correctly identify it\n",
    "* kNN with n>1 not guaranteed to give 100% but still overtraining likely to happen\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Validation - simple way\n",
    "\n",
    "Better approach: **split into train and test sample**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Py 3.6 from sklearn.cross_validation import train_test_split\n",
    "from sklearn.model_selection import train_test_split\n",
    "# split the data with 50% in each set\n",
    "X1, X2, y1, y2 = train_test_split(X, y, random_state=0,\n",
    "                                  train_size=0.5)\n",
    "\n",
    "# fit the model on one set of data\n",
    "model.fit(X1, y1)\n",
    "\n",
    "# evaluate the model on the second set of data\n",
    "y2_model = model.predict(X2)\n",
    "accuracy_score(y2, y2_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Disadvantage:**\n",
    "* reduced statistics for training  \n",
    "&rarr; worse peformance\n",
    "* only one validation set\n",
    "  * difficult to estimate uncertainty on modeling performance\n",
    "***\n",
    "We could of course turn it around and use training set for test and vice-versa:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit the model on other set of data\n",
    "model.fit(X2, y2)\n",
    "\n",
    "# evaluate the model on the first set of data\n",
    "y1_model = model.predict(X1)\n",
    "accuracy_score(y1, y1_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation\n",
    "\n",
    "An obvious extension of this approach is smaller validation samples and do several cycles, each time taking a new slie fo validation, e.g.\n",
    "* 20% for validation\n",
    "* yields 5 different combinations of validation and training data sets\n",
    "\n",
    "\n",
    "![](figures/05.03-5-fold-CV.png)\n",
    "\n",
    "***\n",
    "**Procedure doing that automatically is included:**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "sc=cross_val_score(model, X, y, cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sc)\n",
    "print(sc.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did 5 iterations with the diffeent combinations of training/validation set.\n",
    "***\n",
    "**Validation with 1 element:**  \n",
    "Also extreme case makes sense, \n",
    "* only 1 element for validation, all others for training\n",
    "* repeat #-element times\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import LeaveOneOut\n",
    "sc=cross_val_score(model, X, y, cv=LeaveOneOut())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "***\n",
    "### Selecting a sensible Model\n",
    "\n",
    "One of the central questions in ML is the choice of a sensible model and/or Model parameters, one must find the proper trade-off between **over-training**:\n",
    "* model adapts to specific features of training dataset and not general properties\n",
    "* ML-term: **high variance**\n",
    "\n",
    "and **under-performance**:\n",
    "* model fails to identify/describe general properties\n",
    "* ML term **high bias**\n",
    "\n",
    "The effect can be nicely illustrated in **polynom fitting**.\n",
    "(In case of classical least-square fits $\\chi^2$ is a good measure to judge, i.e. what polynom degree is required for a sensible fit of data-points).\n",
    "\n",
    "It is instructive to take polynom fit also in the ML context, following the discussion in the ML handbook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Following figure illustrates effect for polynomial fit to data points:\n",
    "* blue points used for training\n",
    "* red for validation\n",
    "\n",
    "![](figures/05.03-bias-variance-2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "The basic problem of this bias-variance trade-off is nicely illustrated in the **validation curve** shown below:\n",
    "\n",
    "![](figures/05.03-validation-curve.png)\n",
    "\n",
    "* score for training data in general improves with model complexity, but at some stage the improvement is due to learning **features of training data**\n",
    "* score for validation data reaches maximum\n",
    "&rarr; **sweet spot** for model selection "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation curves in Scikit-Learn\n",
    "#### Linear regression example\n",
    "\n",
    "sklearn also provides a linear regression model (LinearRegression). \n",
    "This is similar to the usual linear regression in **scipy**.  \n",
    "However, in general it is meant for multi-dimensional input vectors and not just 1 dimension as in the classic regression case.\n",
    "\n",
    "On the other hand, sklearn LinearRegression only supports directly linear polynoms, no higher order terms:\n",
    "$$\n",
    "y = a_0 + \\sum a_i x_i\n",
    "$$\n",
    "but not\n",
    "$$\n",
    "y = a_0 + \\sum a_i x_1^i\n",
    "$$\n",
    "\n",
    "\n",
    "However, one can circumvent this limitation by a simple trick:\n",
    "* treat higher-degree polynom values just as  additional elements of input vector:\n",
    "$$\n",
    "x_i = x_1^i\n",
    "$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trick with sklearn to implement higher-order polynom fit\n",
    "from sklearn.preprocessing import PolynomialFeatures # create polynom\n",
    "from sklearn.linear_model import LinearRegression    # model\n",
    "from sklearn.pipeline import make_pipeline           \n",
    "\n",
    "def PolynomialRegression(degree=2, **kwargs):       # combine both in pipeline\n",
    "    return make_pipeline(PolynomialFeatures(degree),\n",
    "                         LinearRegression(**kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Illustrate usage of PolynomialFeatures**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "pf=PolynomialFeatures(2)\n",
    "xt = np.array([4]).reshape(1,1)\n",
    "print (xt , '\\n', pf.fit_transform(xt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Function to generate funny-shaped data**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def make_data(N, err=1.0, rseed=2):\n",
    "    # randomly sample the data\n",
    "    rng = np.random.RandomState(rseed)\n",
    "    X = rng.rand(N, 1) ** 2\n",
    "    y = 10 - 1. / (X.ravel() + 0.1)\n",
    "    if err > 0:\n",
    "        y += err * rng.randn(N)\n",
    "    return X, y\n",
    "\n",
    "X, y = make_data(40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "Visualize points and fit of different poly degree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn; seaborn.set()  # plot formatting\n",
    "\n",
    "X_test = np.linspace(-0.1, 1.1, 500)[:, None]\n",
    "\n",
    "plt.scatter(X.ravel(), y, color='black')\n",
    "axis = plt.axis()\n",
    "for degree in [1, 3, 5, 15]:\n",
    "    y_test = PolynomialRegression(degree).fit(X, y).predict(X_test)\n",
    "    plt.plot(X_test.ravel(), y_test, label='degree={0}'.format(degree))\n",
    "plt.xlim(-0.1, 1.0)\n",
    "plt.ylim(-2, 12)\n",
    "plt.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, using a too high polynom degree results in wildly fluctuating curves.  \n",
    "The problem is how to find proper degree.\n",
    "\n",
    "Automated procedure with  ``validation_curve`` provided by Scikit-Learn.\n",
    "Given a model, data, parameter name, and a range to explore, this function will automatically compute both the training score and validation score across the range:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "from sklearn.model_selection import validation_curve\n",
    "\n",
    "degree = np.arange(0, 21)\n",
    "train_score, val_score = validation_curve(PolynomialRegression(), X, y,\n",
    "                                          'polynomialfeatures__degree', degree, cv=7)\n",
    "\n",
    "plt.plot(degree, np.median(train_score, 1), color='blue', label='training score')\n",
    "plt.plot(degree, np.median(val_score, 1), color='red', label='validation score')\n",
    "plt.legend(loc='best')\n",
    "plt.ylim(0, 1)\n",
    "plt.xlabel('degree')\n",
    "plt.ylabel('score');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation curve shows that already with **degree 3** the maximum for the validation score is reached, so that would be best choice for this model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show data and degree 3 fit\n",
    "plt.scatter(X.ravel(), y)\n",
    "lim = plt.axis()\n",
    "y_test = PolynomialRegression(3).fit(X, y).predict(X_test)\n",
    "plt.plot(X_test.ravel(), y_test);\n",
    "plt.axis(lim);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Curves and Data Size\n",
    "\n",
    "The size of the data set plays an imprtant role\n",
    "* for small data sets the training is much more likely to train specific data features\n",
    "* with increasing size that effect gets smaller\n",
    "* training and validation score get closer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# repeat with 200 data points instead of 40\n",
    "X2, y2 = make_data(200)\n",
    "plt.scatter(X2.ravel(), y2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "degree = np.arange(21)\n",
    "train_score2, val_score2 = validation_curve(PolynomialRegression(), X2, y2,\n",
    "                                            'polynomialfeatures__degree', degree, cv=7)\n",
    "\n",
    "plt.plot(degree, np.median(train_score2, 1), color='blue', label='training score')\n",
    "plt.plot(degree, np.median(val_score2, 1), color='red', label='validation score')\n",
    "plt.plot(degree, np.median(train_score, 1), color='blue', alpha=0.3, linestyle='dashed')\n",
    "plt.plot(degree, np.median(val_score, 1), color='red', alpha=0.3, linestyle='dashed')\n",
    "plt.legend(loc='lower center')\n",
    "plt.ylim(0, 1)\n",
    "plt.xlabel('degree')\n",
    "plt.ylabel('score');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* training much more robust with larger dataset (no as sensitive on model parameters (degree of polynominal) as before)\n",
    "* training and validation score much closer\n",
    "* no abrupt breakdown of validation score at high degree\n",
    "* validation maximum at somewhat higher degree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further validation issues\n",
    "How to do the validation and find proper choice of parameters depends of course also on the details of the model\n",
    "* some models have 0 or just 1 parameter - usually rather straightforward\n",
    "* others can have many different parameters - validation becomes multi-dimensional problem \n",
    "\n",
    "&rarr; further tools in sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}