cross_validation_time.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7057cca9",
   "metadata": {},
   "source": [
    "# Cross validation of time series with scikit-learn\n",
    "\n",
    "In machine learning, it is quite common to assume that the data are\n",
    "\"independent and identically distributed\" (i.i.d),\n",
    "meaning that the generative process does not have any memory of past samples\n",
    "to generate new samples.\n",
    "This assumption is usually violated when dealing with time series. A sample\n",
    "depends on past information.\n",
    "We will take an example to highlight such issues with non-i.i.d. data in the\n",
    "previous cross-validation strategies presented. \n",
    "\n",
    "First we load financial quotations from some energy companies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e897b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/financial-data/COP.csv -P ../datasets/financial-data/\n",
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/financial-data/CVX.csv -P ../datasets/financial-data/\n",
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/financial-data/TOT.csv -P ../datasets/financial-data/\n",
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/financial-data/VLO.csv -P ../datasets/financial-data/\n",
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/financial-data/XOM.csv -P ../datasets/financial-data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68724e4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "symbols = {\"TOT\": \"Total\", \"XOM\": \"Exxon\", \"CVX\": \"Chevron\",\n",
    "           \"COP\": \"ConocoPhillips\", \"VLO\": \"Valero Energy\"}\n",
    "template_name = (\"../datasets/financial-data/{}.csv\")\n",
    "\n",
    "quotes = {}\n",
    "for symbol in symbols:\n",
    "    data = pd.read_csv(\n",
    "        template_name.format(symbol), index_col=0, parse_dates=True\n",
    "    )\n",
    "    quotes[symbols[symbol]] = data[\"open\"]\n",
    "quotes = pd.DataFrame(quotes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cb3944a",
   "metadata": {},
   "source": [
    "We can start by plotting the different financial quotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0901039c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "quotes.plot(figsize=(10, 6))\n",
    "plt.ylabel(\"Quote value\")\n",
    "plt.legend(bbox_to_anchor=(1.05, 0.8), loc=\"upper left\")\n",
    "_ = plt.title(\"Stock values over time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b128c42f",
   "metadata": {},
   "source": [
    "Here, we want to predict the quotation of Chevron\n",
    "using all other energy companies' quotes.\n",
    "\n",
    "To make explanatory plots, we will use a single split in addition to the\n",
    "cross-validation that you used in the introductory exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a5e74d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data, target = quotes.drop(columns=[\"Chevron\"]), quotes[\"Chevron\"]\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74dff27",
   "metadata": {},
   "source": [
    "We will use a decision tree regressor that we expect to overfit and thus not\n",
    "generalize to unseen data. We will use a `ShuffleSplit` cross-validation to\n",
    "check the generalization performance of our model.\n",
    "\n",
    "Let's first define our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fc90de9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "regressor = DecisionTreeRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e76c6b93",
   "metadata": {},
   "source": [
    "And now the cross-validation strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d7915c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import ShuffleSplit\n",
    "\n",
    "cv = ShuffleSplit(random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95aedbdb",
   "metadata": {},
   "source": [
    "Finally, we perform the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1f100aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "test_score = cross_val_score(regressor, data_train, target_train, cv=cv,\n",
    "                             n_jobs=2)\n",
    "print(f\"The mean R2 is: \"\n",
    "      f\"{test_score.mean():.2f} +/- {test_score.std():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e6ce89",
   "metadata": {},
   "source": [
    "Surprisingly, we get outstanding generalization performance. We will investigate\n",
    "and find the reason for such good results with a model that is expected to\n",
    "fail. We previously mentioned that `ShuffleSplit` is an iterative\n",
    "cross-validation scheme that shuffles data and split. We will simplify this\n",
    "procedure with a single split and plot the prediction. We can use\n",
    "`train_test_split` for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f5e4d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "regressor.fit(data_train, target_train)\n",
    "target_predicted = regressor.predict(data_test)\n",
    "# Affect the index of `target_predicted` to ease the plotting\n",
    "target_predicted = pd.Series(target_predicted, index=target_test.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a754437",
   "metadata": {},
   "source": [
    "Let's check the generalization performance of our model on this split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b266eb5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import r2_score\n",
    "\n",
    "test_score = r2_score(target_test, target_predicted)\n",
    "print(f\"The R2 on this single split is: {test_score:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79a01279",
   "metadata": {},
   "source": [
    "Similarly, we obtain good results in terms of $R^2$.\n",
    "We will plot the training, testing and prediction samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48200c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_train.plot(label=\"Training\", figsize=(10, 6))\n",
    "target_test.plot(label=\"Testing\")\n",
    "target_predicted.plot(label=\"Prediction\")\n",
    "\n",
    "plt.ylabel(\"Quote value\")\n",
    "plt.legend(bbox_to_anchor=(1.05, 0.8), loc=\"upper left\")\n",
    "_ = plt.title(\"Model predictions using a ShuffleSplit strategy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cef574d9",
   "metadata": {},
   "source": [
    "So in this context, it seems that the model predictions are following the\n",
    "testing. But we can also see that the testing samples are next to some\n",
    "training sample. And with these time-series, we see a relationship between a\n",
    "sample at the time `t` and a sample at `t+1`. In this case, we are violating\n",
    "the i.i.d. assumption. The insight to get is the following: a model can\n",
    "output of its training set at the time `t` for a testing sample at the time\n",
    "`t+1`. This prediction would be close to the true value even if our model\n",
    "did not learn anything, but just memorized the training dataset.\n",
    "\n",
    "An easy way to verify this hypothesis is to not shuffle the data when doing\n",
    "the split. In this case, we will use the first 75% of the data to train and\n",
    "the remaining data to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26164f81",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, shuffle=False, random_state=0,\n",
    ")\n",
    "regressor.fit(data_train, target_train)\n",
    "target_predicted = regressor.predict(data_test)\n",
    "target_predicted = pd.Series(target_predicted, index=target_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5df1e093",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_score = r2_score(target_test, target_predicted)\n",
    "print(f\"The R2 on this single split is: {test_score:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fb6df4e",
   "metadata": {},
   "source": [
    "In this case, we see that our model is not magical anymore. Indeed, it\n",
    "performs worse than just predicting the mean of the target. We can visually\n",
    "check what we are predicting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e72e2ce4",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_train.plot(label=\"Training\", figsize=(10, 6))\n",
    "target_test.plot(label=\"Testing\")\n",
    "target_predicted.plot(label=\"Prediction\")\n",
    "\n",
    "plt.ylabel(\"Quote value\")\n",
    "plt.legend(bbox_to_anchor=(1.05, 0.8), loc=\"upper left\")\n",
    "_ = plt.title(\"Model predictions using a split without shuffling\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6253e8c6",
   "metadata": {},
   "source": [
    "We see that our model cannot predict anything because it doesn't have samples\n",
    "around the testing sample. Let's check how we could have made a proper\n",
    "cross-validation scheme to get a reasonable generalization performance estimate.\n",
    "\n",
    "One solution would be to group the samples into time blocks, e.g. by quarter,\n",
    "and predict each group's information by using information from the other\n",
    "groups. We can use the `LeaveOneGroupOut` cross-validation for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd8d11c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import LeaveOneGroupOut\n",
    "\n",
    "groups = quotes.index.to_period(\"Q\")\n",
    "cv = LeaveOneGroupOut()\n",
    "test_score = cross_val_score(regressor, data, target,\n",
    "                             cv=cv, groups=groups, n_jobs=2)\n",
    "print(f\"The mean R2 is: \"\n",
    "      f\"{test_score.mean():.2f} +/- {test_score.std():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e467643",
   "metadata": {},
   "source": [
    "In this case, we see that we cannot make good predictions, which is less\n",
    "surprising than our original results.\n",
    "\n",
    "Another thing to consider is the actual application of our solution. If our\n",
    "model is aimed at forecasting (i.e., predicting future data from past data),\n",
    "we should not use training data that are ulterior to the testing data. In\n",
    "this case, we can use the `TimeSeriesSplit` cross-validation to enforce this\n",
    "behaviour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3693e64e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import TimeSeriesSplit\n",
    "\n",
    "cv = TimeSeriesSplit(n_splits=groups.nunique())\n",
    "test_score = cross_val_score(regressor, data, target,\n",
    "                             cv=cv, groups=groups, n_jobs=2)\n",
    "print(f\"The mean R2 is: \"\n",
    "      f\"{test_score.mean():.2f} +/- {test_score.std():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abef8e4b",
   "metadata": {},
   "source": [
    "In conclusion, it is really important to not use an out of the shelves\n",
    "cross-validation strategy which do not respect some assumptions such as\n",
    "having i.i.d data. It might lead to absurd results which could make think\n",
    "that a predictive model might work."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "223ce219",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "Load the [bike_rides.csv dataset](https://inria.github.io/scikit-learn-mooc/python_scripts/datasets_bike_rides.html). The exercise consists of using measurements from cheap sensors (GPS, heart-rate monitor, etc.) to predict a cyclist power time-series. Power can indeed be recorded via a cycling power meter device, but this device is rather expensive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a68665eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/INRIA/scikit-learn-mooc/main/datasets/bike_rides.csv -P ../datasets/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bce0a8f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "cycling = pd.read_csv(\"../datasets/bike_rides.csv\", index_col=0,\n",
    "                      parse_dates=True)\n",
    "cycling.index.name = \"\"\n",
    "target_name = \"power\"\n",
    "data, target = cycling.drop(columns=target_name), cycling[target_name]\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb813db0",
   "metadata": {},
   "source": [
    "Instead of using blindly machine learning, we will first introduce some flavor of\n",
    "classic mechanics: the Newton's second law.\n",
    "\n",
    "$P_{meca} = (\\frac{1}{2} \\rho . SC_x . V_{a}^{2} + C_r . mg . \\cos \\alpha + mg . \\sin \\alpha + ma) V_d$\n",
    "\n",
    "where $\\rho$ is the air density in kg.m$^{-3}$, $S$ is frontal surface of the\n",
    "cyclist in m$^{2}$, $C_x$ is the drag coefficient, $V_a$ is the air speed in\n",
    "m.s$^{-1}$, $C_r$ is the rolling coefficient, $m$ is the mass of the rider and\n",
    "bicycle in kg, $g$ is the standard acceleration due to gravity which is equal\n",
    "to 9.81 m.s$^{-2}$, $\\alpha$ is the slope in radian, $V_d$ is the rider speed\n",
    "in m.s$^{-1}$, and $a$ is the rider acceleration in m.s$^{-2}$.\n",
    "\n",
    "This equation might look a bit complex at first but we can explain with words\n",
    "what the different terms within the parenthesis are:\n",
    "\n",
    "- the first term is the power that a cyclist is required to produce to fight\n",
    "  wind\n",
    "- the second term is the power that a cyclist is required to produce to fight\n",
    "  the rolling resistance created by the tires on the floor\n",
    "- the third term is the power that a cyclist is required to produce to go up a\n",
    "  hill if the slope is positive. If the slope is negative the cyclist does not\n",
    "  need to produce any power to go forward\n",
    "- the fourth and last term is the power that a cyclist requires to change his\n",
    "  speed (i.e. acceleration).\n",
    "\n",
    "We can simplify the model above by using the data that we have at hand. It\n",
    "would look like the following.\n",
    "\n",
    "$P_{meca} = \\beta_{1} V_{d}^{3} + \\beta_{2} V_{d} + \\beta_{3} \\sin(\\alpha) V_{d} + \\beta_{4} a V_{d}$\n",
    "\n",
    "This model is closer to what we saw previously: it is a linear model trained on\n",
    "a non-linear feature transformation. We will build, train and evaluate such a\n",
    "model as part of this exercise. Thus, you need to:\n",
    "\n",
    "- create a new data matrix containing the cube of the speed, the speed, the\n",
    "  speed multiplied by the sine of the angle of the slope, and the speed\n",
    "  multiplied by the acceleration. To compute the angle of the slope, you need to\n",
    "  take the arc tangent of the slope (`alpha = np.arctan(slope)`). In addition,\n",
    "  we can limit ourself to positive acceleration only by clipping to 0 the\n",
    "  negative acceleration values (they would correspond to some power created by\n",
    "  the braking that we are not modeling here).\n",
    "- using the new data matrix, create a linear predictive model based on a\n",
    "  [`sklearn.preprocessing.StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
    "  and a\n",
    "  [`sklearn.linear_model.RidgeCV`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.RidgeCV.html);\n",
    "- use a\n",
    "  [`sklearn.model_selection.ShuffleSplit`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html)\n",
    "  cross-validation strategy with only 4 splits (`n_splits=4`) to evaluate the\n",
    "  generalization performance of the model. Use the mean absolute error (MAE) as\n",
    "  a generalization performance metric. Also, pass the parameter\n",
    "  `return_estimator=True` and `return_train_score=True` to answer the subsequent\n",
    "  questions. Be aware that the `ShuffleSplit` strategy is a naive strategy and\n",
    "  we will investigate the consequence of making this choice in the subsequent\n",
    "  questions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bd9491",
   "metadata": {},
   "source": [
    "### Q1\n",
    "\n",
    "What is the mean value of the column containing the information of\n",
    "$\\sin(\\alpha) V_{d}$?\n",
    "\n",
    "- a) about -3\n",
    "- b) about -0.3\n",
    "- c) about -0.03\n",
    "- d) about -0.003\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a113fa14",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f8c238fd",
   "metadata": {},
   "source": [
    "### Q2\n",
    "On average, the Mean Absolute Error on the test sets obtained through\n",
    "cross-validation is closest to:\n",
    "\n",
    "- a) 20 Watts\n",
    "- b) 50 Watts\n",
    "- c) 70 Watts\n",
    "- d) 90 Watts\n",
    "\n",
    "_Select a single answer_\n",
    "\n",
    "Hint: pass `scoring=\"neg_mean_absolute_error\"` to the `cross_validate`\n",
    "function to compute the (negative of) the requested metric.\n",
    "Hint: it is possible to replace the negative acceleration values by 0 using\n",
    "`data[\"acceleration\"].clip(lower=0)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ca2e681",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2e9c60bb",
   "metadata": {},
   "source": [
    "### Q3\n",
    "Given the model\n",
    "$P_{meca} = \\beta_{1} V_{d}^{3} + \\beta_{2} V_{d} + \\beta_{3} \\sin(\\alpha) V_{d} + \\beta_{4} a V_{d}$\n",
    "that you created, inspect the weights of the linear models fitted during\n",
    "cross-validation and select the correct statements:\n",
    "\n",
    "- a) $\\beta_{1} < \\beta_{2} < \\beta_{3}$\n",
    "- b) $\\beta_{3} < \\beta_{1} < \\beta_{2}$\n",
    "- c) $\\beta_{2} < \\beta_{3} < \\beta_{1}$\n",
    "- d) $\\beta_{1} < 0$\n",
    "- e) $\\beta_{2} < 0$\n",
    "- f) $\\beta_{3} < 0$\n",
    "- g) $\\beta_{4} < 0$\n",
    "- h) All $\\beta$s are $> 0$\n",
    "\n",
    "_Select all answers that apply_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6bb5814",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6b49248a",
   "metadata": {},
   "source": [
    "### Q4\n",
    "Now, we will create a predictive model that uses all `data`, including available\n",
    "sensor measurements such as cadence (the speed at which a cyclist turns pedals\n",
    "measured in rotation per minute) and heart-rate (the number of beat per minute\n",
    "of the heart of the cyclist while exercising). Also, we will use a non-linear\n",
    "regressor, a\n",
    "[`sklearn.ensemble.HistGradientBoostingRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingRegressor.html).\n",
    "Fix the number of maximum iterations to 1000 (`max_iter=1_000`) and activate the\n",
    "early stopping (`early_stopping=True`). Repeat the previous evaluation using\n",
    "this regressor.\n",
    "\n",
    "On average, the Mean Absolute Error on the test sets obtained through\n",
    "cross-validation is closest to:\n",
    "\n",
    "- a) 20 Watts\n",
    "- b) 40 Watts\n",
    "- c) 60 Watts\n",
    "- d) 80 Watts\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5970eb8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5b721f66",
   "metadata": {},
   "source": [
    "### Q5\n",
    "Comparing both the linear model and the histogram gradient boosting model and\n",
    "taking into consideration the train and test MAE obtained via cross-validation,\n",
    "select the correct statements:\n",
    "\n",
    "- a) the generalization performance of the histogram gradient-boosting model is\n",
    "  limited by its underfitting\n",
    "- b) the generalization performance of the histogram gradient-boosting model is\n",
    "  limited by its overfitting\n",
    "- c) the generalization performance of the linear model is limited by its\n",
    "  underfitting\n",
    "- d) the generalization performance of the linear model is limited by its\n",
    "  overfitting\n",
    "\n",
    "_Select all answers that apply_\n",
    "\n",
    "Hint: look at the values of the `train_score` and the `test_score` collected\n",
    "in the dictionaries returned by the `cross_validate` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33f68acf",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d7a6febf",
   "metadata": {},
   "source": [
    "### Q6\n",
    "How many bike rides are stored in the dataframe `data`?\n",
    "\n",
    "- a) 2\n",
    "- b) 3\n",
    "- c) 4\n",
    "- d) 5\n",
    "\n",
    "_Select a single answer_\n",
    "\n",
    "Hint: You can check the unique day in the `DatetimeIndex` (the index of the\n",
    "dataframe `data`). Indeed, we assume that on a given day the rider went cycling\n",
    "at most once per day.\n",
    "Hint: You can access to the date and time of a `DatetimeIndex` using\n",
    "`df.index.date` and `df.index.time`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2f917c6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f4643ea4",
   "metadata": {},
   "source": [
    "### Q7\n",
    "Instead of using the naive `ShuffleSplit` strategy, we will use a strategy that\n",
    "takes into account the group defined by each individual date. It corresponds to\n",
    "a bike ride. We would like to have a cross-validation strategy that evaluates\n",
    "the capacity of our model to predict on a completely new bike ride: the samples\n",
    "in the validation set should only come from rides not present in the training\n",
    "set. Therefore, we can use a `LeaveOneGroupOut` strategy: at each iteration of\n",
    "the cross-validation, we will keep a bike ride for the evaluation and use all\n",
    "other bike rides to train our model.\n",
    "\n",
    "Thus, you concretely need to:\n",
    "\n",
    "- create a variable called `group` that is a 1D numpy array containing the\n",
    "  index of each ride present in the dataframe. Therefore, the length of `group`\n",
    "  will be equal to the number of samples in `data`. If we had 2 bike\n",
    "  rides, we would expect the indices 0 and 1 in `group` to differentiate the\n",
    "  bike ride. You can use\n",
    "  [`pd.factorize`](https://pandas.pydata.org/docs/reference/api/pandas.factorize.html)\n",
    "  to encode any Python types into integer indices.\n",
    "- create a cross-validation object named `cv` using the\n",
    "  [`sklearn.model_selection.LeaveOneGroupOut`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneGroupOut.html#sklearn.model_selection.LeaveOneGroupOut)\n",
    "  strategy.\n",
    "- evaluate both the linear and histogram gradient boosting models with this\n",
    "  strategy.\n",
    "\n",
    "Using the previous evaluations (with the `LeaveOneGroupOut` strategy)\n",
    "and looking at the train and test errors for both models, select the\n",
    "correct statements:\n",
    "\n",
    "- a) the generalization performance of the gradient-boosting model is\n",
    "  limited by its underfitting\n",
    "- b) the generalization performance of the gradient-boosting model is\n",
    "  limited by its overfitting\n",
    "- c) the generalization performance of the linear model is limited by its\n",
    "  underfitting\n",
    "- d) the generalization performance of the linear model is limited by its\n",
    "  overfitting\n",
    "\n",
    "_Select all answers that apply_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13a45771",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f7b77628",
   "metadata": {},
   "source": [
    "### Q8\n",
    "In this case we cannot compare cross-validation scores fold-to-fold as the folds\n",
    "are not aligned (they are not generated by the exact same strategy). Instead,\n",
    "compare the mean of the cross-validation test errors in the evaluations of the\n",
    "**linear model** to select the correct statement.\n",
    "\n",
    "When using the `ShuffleSplit` strategy, the mean test error:\n",
    "\n",
    "- a) is greater than the `LeaveOneGroupOut` mean test error by more than 3 Watts,\n",
    "  i.e. `ShuffleSplit` is giving over-pessimistic results\n",
    "- b) differs from the `LeaveOneGroupOut` mean test error by less than 3 Watts,\n",
    "  i.e. both cross-validation strategies are equivalent\n",
    "- c) is lower than the `LeaveOneGroupOut` mean test error by more than 3 Watts,\n",
    "  i.e. `ShuffleSplit` is giving over-optimistic results\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55b67ea2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cd3691bb",
   "metadata": {},
   "source": [
    "### Q9\n",
    "Compare the mean of the cross-validation test errors in the evaluations of the\n",
    "**gradient-boosting model** to select the correct statement.\n",
    "\n",
    "When using the `ShuffleSplit` strategy, the mean test error:\n",
    "\n",
    "- a) is greater than the `LeaveOneGroupOut` mean test error by more than 3 Watts,\n",
    "  i.e. `ShuffleSplit` is giving over-pessimistic results\n",
    "- b) differs from the `LeaveOneGroupOut` mean test error by less than 3 Watts,\n",
    "  i.e. both cross-validation strategies are equivalent\n",
    "- c) is lower than the `LeaveOneGroupOut` mean test error by more than 3 Watts,\n",
    "  i.e. `ShuffleSplit` is giving over-optimistic results\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dd73d54",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2acb4001",
   "metadata": {},
   "source": [
    "### Q10\n",
    "\n",
    "Compare more precisely the errors estimated through cross-validation and select\n",
    "the correct statement:\n",
    "\n",
    "- a) in general, the standard deviation of the train and test errors increased\n",
    "  using the `LeaveOneGroupOut` cross-validation\n",
    "- b) in general, the standard deviation of the train and test errors decreased\n",
    "  using the `LeaveOneGroupOut` cross-validation\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "132becdd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "19578e1a",
   "metadata": {},
   "source": [
    "### Q11\n",
    "Now, we will go more into details by picking a single ride for the testing and\n",
    "analyse the predictions of the models for this test ride. To do so, we can reuse\n",
    "the `LeaveOneGroupOut` cross-validation object in the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7004057",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = LeaveOneGroupOut()\n",
    "train_indices, test_indices = list(cv.split(data, target, groups=groups))[0]\n",
    "\n",
    "data_linear_model_train = data_linear_model.iloc[train_indices]\n",
    "data_linear_model_test = data_linear_model.iloc[test_indices]\n",
    "\n",
    "data_train = data.iloc[train_indices]\n",
    "data_test = data.iloc[test_indices]\n",
    "\n",
    "target_train = target.iloc[train_indices]\n",
    "target_test = target.iloc[test_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b31c32b",
   "metadata": {},
   "source": [
    "Now, fit both the linear model and the histogram gradient boosting regressor\n",
    "models on the training data and collect the predictions on the testing data.\n",
    "Make a scatter plot where on the x-axis, you will plot the measured powers (true\n",
    "target) and on the y-axis, you will plot the predicted powers (predicted\n",
    "target). Do two separated plots for each model.\n",
    "\n",
    "By analysing the plots, select the correct statements:\n",
    "\n",
    "- a) the linear regressor tends to under-predict samples with high power\n",
    "- b) the linear regressor tends to over-predict samples with high power\n",
    "- c) the linear regressor makes catastrophic predictions for samples with power\n",
    "  close to zero\n",
    "- d) the histogram gradient boosting regressor tends to under-predict samples\n",
    "  with high power\n",
    "- e) the histogram gradient boosting regressor tends to over-predict samples\n",
    "  with high power\n",
    "- f) the histogram gradient boosting makes catastrophic predictions for samples\n",
    "  with power close to zero\n",
    "\n",
    "_Select all answers that apply_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bf39ceb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ecb683c7",
   "metadata": {},
   "source": [
    "### Q12\n",
    "Now select a portion of the testing data using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d40b266a",
   "metadata": {},
   "outputs": [],
   "source": [
    "time_slice = slice(\"2020-08-18 17:00:00\", \"2020-08-18 17:05:00\")\n",
    "\n",
    "data_test_linear_model_subset = data_linear_model_test[time_slice]\n",
    "data_test_subset = data_test[time_slice]\n",
    "target_test_subset = target_test[time_slice]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd558293",
   "metadata": {},
   "source": [
    "It allows to select data from 5.00 pm until 5.05 pm. Used the previous fitted\n",
    "models (linear and gradient-boosting regressor) to predict on this portion of\n",
    "the test data. Draw on the same plot the true targets and the predictions of\n",
    "each model.\n",
    "\n",
    "By using the previous plot, select the correct statements:\n",
    "\n",
    "- a) the linear model is more accurate than the histogram gradient boosting\n",
    "  regressor\n",
    "- b) the histogram gradient boosting regressor is more accurate than the linear\n",
    "  model\n",
    "\n",
    "_Select a single answer_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9abfa1d4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}