bike-sharing-demand.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Goal: improve evaluation a model\n",
    "## Evoluation a model is the most important thing!\n",
    "\n",
    "Remember this point is the most important.  \n",
    "If you don't understand your a quality function or don't have a good strategy how to evaluate model - rest is doesn't matter.\n",
    "\n",
    "Check list\n",
    "1. What data set is using for evaluation? \n",
    "2. What is range values for a quality function (what the value is for an ideal solution)?\n",
    "3. What and how make impact on output value (a quality function)? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.ensemble import ExtraTreesRegressor\n",
    "from sklearn.metrics import make_scorer\n",
    "from random import shuffle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>season</th>\n",
       "      <th>holiday</th>\n",
       "      <th>workingday</th>\n",
       "      <th>weather</th>\n",
       "      <th>temp</th>\n",
       "      <th>atemp</th>\n",
       "      <th>humidity</th>\n",
       "      <th>windspeed</th>\n",
       "      <th>casual</th>\n",
       "      <th>registered</th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01 00:00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.84</td>\n",
       "      <td>14.395</td>\n",
       "      <td>81</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-01-01 01:00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.02</td>\n",
       "      <td>13.635</td>\n",
       "      <td>80</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>32</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-01-01 02:00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.02</td>\n",
       "      <td>13.635</td>\n",
       "      <td>80</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>27</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-01-01 03:00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.84</td>\n",
       "      <td>14.395</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-01-01 04:00:00</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9.84</td>\n",
       "      <td>14.395</td>\n",
       "      <td>75</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              datetime  season  holiday  workingday  weather  temp   atemp  \\\n",
       "0  2011-01-01 00:00:00       1        0           0        1  9.84  14.395   \n",
       "1  2011-01-01 01:00:00       1        0           0        1  9.02  13.635   \n",
       "2  2011-01-01 02:00:00       1        0           0        1  9.02  13.635   \n",
       "3  2011-01-01 03:00:00       1        0           0        1  9.84  14.395   \n",
       "4  2011-01-01 04:00:00       1        0           0        1  9.84  14.395   \n",
       "\n",
       "   humidity  windspeed  casual  registered  count  \n",
       "0        81          0       3          13     16  \n",
       "1        80          0       8          32     40  \n",
       "2        80          0       5          27     32  \n",
       "3        75          0       3          10     13  \n",
       "4        75          0       0           1      1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('train.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. **datetime** - hourly date + timestamp  \n",
    "2. **season** -  1 = spring, 2 = summer, 3 = fall, 4 = winter \n",
    "3. **holiday** - whether the day is considered a holiday\n",
    "4. **workingday** - whether the day is neither a weekend nor holiday\n",
    "5. **weather** - \n",
    "    1: Clear, Few clouds, Partly cloudy, Partly cloudy \n",
    "    2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist\n",
    "    3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds\n",
    "    4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog \n",
    "6. **temp** - temperature in Celsius\n",
    "7. **atemp** - \"feels like\" temperature in Celsius\n",
    "8. **humidity** - relative humidity\n",
    "9. **windspeed** - wind speed\n",
    "10. **casual** - number of non-registered user rentals initiated\n",
    "11. **registered** - number of registered user rentals initiated\n",
    "12. **count** - number of total rentals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = ExtraTreesRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## train & test (data sets)\n",
    "Prepare two data sets (for training and for testing):\n",
    "\n",
    "### train data set\n",
    "**X_train** - features (*matrix*).  \n",
    "**y_train** - target variable (*vector*).\n",
    "\n",
    "### test data set\n",
    "**X_test** - features (*matrix*).  \n",
    "**y_test** - target variable (*vector*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "       18, 19])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train['datetime'] = pd.to_datetime( train['datetime'] )\n",
    "train['day']      = train['datetime'].map(lambda x: x.day)\n",
    "    \n",
    "train.day.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_test_split(data, last_training_day=0.3):\n",
    "    days = train.day.unique()\n",
    "    shuffle(days)\n",
    "    test_days = days[: len(days) * 0.3]\n",
    "    \n",
    "    data['is_test'] = data.day.isin(test_days)\n",
    "    df_train = data[data.is_test == False] \n",
    "    df_test  = data[data.is_test == True]\n",
    "    \n",
    "    return df_train, df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExtraTreesRegressor(bootstrap=False, criterion='mse', max_depth=None,\n",
       "          max_features='auto', max_leaf_nodes=None, min_samples_leaf=1,\n",
       "          min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "          n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = ['season', 'holiday', 'workingday', 'weather', 'temp', 'atemp', 'humidity', 'windspeed']\n",
    "df_train, df_test = train_test_split(train)\n",
    "\n",
    "X_train  = df_train[features].values\n",
    "X_test   = df_test[features].values\n",
    "\n",
    "y_train  = df_train['count'].values\n",
    "y_test  = df_test['count'].values\n",
    "\n",
    "\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\sqrt{\\frac{1}{n} \\sum_{i=1}^n (\\log(p_i + 1) - \\log(a_i+1))^2 }$$\n",
    "\n",
    "where  \n",
    "**n** is the number of hours in the test set  \n",
    "**pi** is your predicted count  \n",
    "**ai** is the actual count  \n",
    "**log(x)** is the natural logarithm  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why we have +1 for logarithm algorithm? Let's recall a shape logarithm function.\n",
    "Argument should be greater than 0. In our case **pi** and **ai** can be equals to 0.\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/Binary_logarithm_plot_with_ticks.svg/408px-Binary_logarithm_plot_with_ticks.svg.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's play around with a quality function\n",
    "\n",
    "|pi  |ai |  result |\n",
    "|----|---|---------|\n",
    "|1000| 0 | 6.909   |   \n",
    "|100 | 0 | 4.615   |   \n",
    "|10  | 0 | 2.397   |  \n",
    "|5   | 0 | 1.792   |\n",
    "|1   | 0 | 0.693   |\n",
    "|0.5 | 0 | 0.405   |\n",
    "|0.2 | 0 | 0.182   |\n",
    "|0   | 0 | 0.0     |\n",
    "\n",
    "For sample above `rmsle(pi, ai)` is **3.644**.\n",
    "*Note: but average column result is* **2.918**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def rmsle(y_true, y_pred):\n",
    "    diff = np.log(y_pred + 1) - np.log(y_true + 1)\n",
    "    mean_error = np.square(diff).mean()\n",
    "    return np.sqrt(mean_error)\n",
    "\n",
    "scorer = make_scorer(rmsle, greater_is_better=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.4272746457576084"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "rmsle(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looks better :)\n",
    "We improved evaluation a model. Next step is understand better data and figure out how we can improve result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}