ML-01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install statsmodels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import umap\n",
    "\n",
    "from zipfile import ZipFile\n",
    "\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.svm import SVC as BlackBoxClassifier\n",
    "from sklearn.cluster import DBSCAN as BlackBoxClustering\n",
    "from sklearn.cluster import AgglomerativeClustering as BlackBoxClustering2\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n",
    "\n",
    "import statsmodels.api as sm\n",
    "BlackBoxForecasting = sm.tsa.statespace.SARIMAX\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from pylab import rcParams\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "os.environ[\"KAGGLE_USERNAME\"] = \"injeans\"\n",
    "os.environ[\"KAGGLE_KEY\"] = \"cc08cc14836d12bd8bfccdfff9147ab7\"\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.utils.multiclass import unique_labels\n",
    "\n",
    "def plot_confusion_matrix(y_true, y_pred, classes,\n",
    "                          normalize=False,\n",
    "                          title=None,\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if not title:\n",
    "        if normalize:\n",
    "            title = 'Normalized confusion matrix'\n",
    "        else:\n",
    "            title = 'Confusion matrix, without normalization'\n",
    "\n",
    "    # Compute confusion matrix\n",
    "    cm = confusion_matrix(y_true, y_pred)\n",
    "    # Only use the labels that appear in the data\n",
    "    classes = classes[unique_labels(y_true, y_pred)]\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    fig, ax = plt.subplots()\n",
    "    sns.heatmap(cm, annot=True, ax=ax, cmap=\"YlGnBu\")\n",
    "    # We want to show all ticks...\n",
    "    ax.set(xticks=[0.5, 1.5],\n",
    "           yticks=[0.5, 1.5],\n",
    "           # ... and label them with the respective list entries\n",
    "           xticklabels=classes, yticklabels=classes,\n",
    "           title=title,\n",
    "           ylabel='True label',\n",
    "           xlabel='Predicted label')\n",
    "\n",
    "    # Rotate the tick labels and set their alignment.\n",
    "    plt.setp(ax.get_xticklabels(), rotation=45, ha=\"right\",\n",
    "             rotation_mode=\"anchor\")\n",
    "\n",
    "#     # Loop over data dimensions and create text annotations.\n",
    "#     fmt = '.2f' if normalize else 'd'\n",
    "#     thresh = cm.max() / 2.\n",
    "#     for i in range(cm.shape[0]):\n",
    "#         for j in range(cm.shape[1]):\n",
    "#             ax.text(j, i, format(cm[i, j], fmt),\n",
    "#                     ha=\"center\", va=\"center\",\n",
    "#                     color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "    fig.tight_layout()\n",
    "    return ax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is ML?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/what_is_ml.png\" alt=\"Drawing\" style=\"width: 500px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A popular definition for machine learning, given by [Mitchell 1997](http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html), is\n",
    "\n",
    "> A computer program is said to learn from experience, E, with respect to some class of tasks, T, and performance measure, P, if its performance at tasks in T, as measured by P, improves with experience E."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Typically, the experience a machine learning algorithm encounters during learning is in the form of a dataset, or exposure to a dataset (or subset thereof). A dataset is a collection of examples, each example comprising a set of features that have been quantitatively measured from some object or event. We typically represent an example as a vector $x \\in \\mathbb{R}^N$, where each entry  of the vector is another feature. Broadly speaking, experiences are often categorised as either **unsupervised** or **supervised**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervised"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supervised learning algorithms experience a dataset containing features, but each example is also associated with a **label** or **target**. Supervised learning involves observing several examples of random vector, $x$, and an associated value or vector $y$, then learning to predict $y$ from $x$, usually by estimating $p(y|x)$ [(Goodfellow, Bengio, & Courville, 2016)](#References). \n",
    "\n",
    "Let's take a look at an example of a supervised experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "breast_cancer_data = load_breast_cancer()\n",
    "breast_cancer_df = pd.DataFrame(data=np.c_[breast_cancer_data['data'], breast_cancer_data['target']],\n",
    "                                columns=breast_cancer_data['feature_names'].tolist() + ['target'])\n",
    "breast_cancer_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "breast_cancer_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 10))\n",
    "\n",
    "sns.distplot(breast_cancer_df['target'], ax=axes[0])\n",
    "\n",
    "trans = umap.UMAP(n_neighbors=5, random_state=42).fit(breast_cancer_data['data'])\n",
    "\n",
    "axes[1].scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], \n",
    "                s=50, c=breast_cancer_data['target'], cmap='Spectral')\n",
    "           \n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(breast_cancer_data['data'], breast_cancer_data['target'], \n",
    "                                                    test_size=0.2, random_state=42)\n",
    "\n",
    "clf = BlackBoxClassifier(gamma=\"scale\")\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "\n",
    "print(\"Classification accuracy: {:.2f}%\".format(accuracy_score(y_pred, y_test)*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot normalized confusion matrix\n",
    "plot_confusion_matrix(y_test, y_pred, classes=breast_cancer_data['target_names'], normalize=True,\n",
    "                      title='Normalized confusion matrix')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (Self?/) Unsupervised"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsupervised learning algorithms experience a datset containing many features, then learn useful properties of the structure of this dataset. Unsupervised learning involves observing several examples of a random vector, $x$, and attempting to implicitly or explicitly learn the probability distribution $p(x)$, or some interesting properties of that distribution [(Goodfellow, Bengio, & Courville, 2016)](#References).\n",
    "\n",
    "Let's dive in!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "higgs_data = fetch_openml(name='higgs')\n",
    "higgs_df = pd.DataFrame(data=np.c_[higgs_data['data'], higgs_data['target']],\n",
    "                               columns=higgs_data['feature_names'] + ['target'])\n",
    "higgs_df.dropna(inplace=True)\n",
    "higgs_df = higgs_df[higgs_df[\"target\"]==1]\n",
    "higgs_df = higgs_df.sample(frac=0.2, random_state=42)\n",
    "higgs_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "higgs_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trans = umap.UMAP(n_neighbors=5, random_state=42).fit(higgs_df.drop('target', axis=1).values)\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "plt.scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], s=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clst = BlackBoxClustering(eps=4, min_samples=3)\n",
    "clst.fit(higgs_df.drop('target', axis=1).values)\n",
    "y_pred = clst.labels_\n",
    "\n",
    "print(np.unique(y_pred))\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "plt.scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], \n",
    "            s=50, c=y_pred, cmap='Spectral')\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many kinds of tasks can be solved with machine learning. Some of the most common machine learning tasks include the following:\n",
    "\n",
    "- **Classiﬁcation**: the computer program is asked to specify which of $k$ categories some input belongs to. To solve this task, the learning algorithm is usually asked to produce a function $f:\\mathbb{R}^n \\to \\{1, \\dots, k\\}$. When $y=f(\\mathbf{x})$, the model assigns an input described by vector $\\mathbf{x}$ to a category identiﬁed by numeric code $y$. There are other variants of the classiﬁcation task, for example, where $f$ outputs a probability distribution over classes. An example of a classiﬁcation task is object recognition, where the input is an image (usually described as a set of pixel brightness values), and the output is a numeric code identifying the object in the image.\n",
    "- **Regression**: the computer program is asked to predict a numerical value given some input. To solve this task, the learning algorithm is asked to output a function $f:\\mathbb{R}^n\\to \\mathbb{R}$. This type of task is similar to classiﬁcation, except that the format of output is diﬀerent. An example of a regression task is the prediction of the expected claim amount that an insured person will make (used to set insurance premiums), or the prediction of future prices of securities. These kinds of predictions are also used for algorithmic trading.\n",
    "- **Clustering**: the assignment of a set of observations into subsets (called clusters) so that observations in the same cluster are similar in some sense.\n",
    "\n",
    "Okay so let's consider these tasks in the context of a new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kaggle datasets download -d cityofLA/los-angeles-traffic-collision-data -p kaggle_data\n",
    "with ZipFile('kaggle_data/los-angeles-traffic-collision-data.zip', 'r') as zipObj:\n",
    "   # Extract all the contents of zip file in different directory\n",
    "   zipObj.extractall('kaggle_data')\n",
    "!ls kaggle_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "la_df = pd.read_csv(\"kaggle_data/traffic-collision-data-from-2010-to-present.csv\")\n",
    "la_df['Date Reported'] = pd.to_datetime(la_df['Date Reported'])\n",
    "la_df['Date Occurred'] = pd.to_datetime(la_df['Date Occurred'])\n",
    "la_df.drop(labels=\"DR Number\", axis=1, inplace=True)\n",
    "la_df.drop(labels=\"Crime Code Description\", axis=1, inplace=True)\n",
    "la_df.dropna(inplace=True)\n",
    "la_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "la_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "single_mo_df = la_df[la_df['MO Codes'].str.split().str.len().lt(2)]\n",
    "gb_df = single_mo_df.groupby(\"MO Codes\").count()['Date Reported']\n",
    "mo_codes = gb_df[gb_df.gt(100)].index\n",
    "clf_df = la_df[la_df['MO Codes'].isin(mo_codes)].copy()\n",
    "\n",
    "target = clf_df.pop('MO Codes')\n",
    "features_df = clf_df.drop(\"Location\", axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "la_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lb_make = LabelEncoder()\n",
    "features_df[\"Date Reported\"] = lb_make.fit_transform(features_df[\"Date Reported\"])\n",
    "features_df[\"Date Occurred\"] = lb_make.fit_transform(features_df[\"Date Occurred\"])\n",
    "obj_df = features_df.select_dtypes(include=['object']).copy()\n",
    "\n",
    "for col in obj_df.columns:\n",
    "    features_df[col] = lb_make.fit_transform(features_df[col])\n",
    "\n",
    "target = lb_make.fit_transform(target)\n",
    "features_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(features_df.values, target, \n",
    "                                                    test_size=0.2, random_state=42)\n",
    "\n",
    "clf = BlackBoxClassifier(gamma=\"scale\")\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "\n",
    "print(\"Classification accuracy: {:.2f}%\".format(accuracy_score(y_pred, y_test)*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collisions = la_df.groupby(['Date Occurred'])[\"Date Reported\"].count().reset_index()\n",
    "collisions = collisions.set_index('Date Occurred')\n",
    "collisions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = collisions['Date Reported'].resample('MS').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y.plot(figsize=(15, 6))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rcParams['figure.figsize'] = 18, 8\n",
    "decomposition = sm.tsa.seasonal_decompose(y, model='additive')\n",
    "fig = decomposition.plot()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fcst = BlackBoxForecasting(y,\n",
    "                           order=(0, 1, 1),\n",
    "                           seasonal_order=(0, 1, 1, 12),\n",
    "                           enforce_stationarity=False,\n",
    "                           enforce_invertibility=False)\n",
    "results = fcst.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = results.get_prediction(start=pd.to_datetime('2019-01-01'), dynamic=False)\n",
    "pred_ci = pred.conf_int()\n",
    "ax = y['2017':].plot(label='observed')\n",
    "pred.predicted_mean.plot(ax=ax, label='One-step ahead Forecast', alpha=.7, figsize=(14, 7))\n",
    "ax.fill_between(pred_ci.index,\n",
    "                pred_ci.iloc[:, 0],\n",
    "                pred_ci.iloc[:, 1], color='k', alpha=.2)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('# Collisions')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trans = umap.UMAP(n_neighbors=5, random_state=42).fit(X_train)\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "plt.scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], s=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,10))\n",
    "plt.scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], s=50, c=y_train, cmap='Spectral')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clst = BlackBoxClustering2()\n",
    "clst.fit(X_train)\n",
    "y_pred = clst.labels_\n",
    "\n",
    "print(np.unique(y_pred))\n",
    "\n",
    "plt.figure(figsize=(20,10))\n",
    "plt.scatter(trans.embedding_[:, 0], trans.embedding_[:, 1], \n",
    "            s=50, c=y_pred, cmap='Spectral')\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. Retrieved from https://www.deeplearningbook.org](https://www.deeplearningbook.org)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}