ML classification case study - school budgets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML multi-class multi-label classification\n",
    "## - school budgets case study\n",
    "- build a baseline model - simple, first-pass approach\n",
    "- use NLP to prepare budgets for modeling\n",
    "- build a more accurate model\n",
    "- from DrivenData challenge\n",
    "\n",
    "Challenge: School budgets take hundreds of hours each year to manually label\n",
    "Goal: Build a ML algorithm that can automate the process\n",
    "Budget data\n",
    "- line-item: \"Algebra books for 8th grade students\"\n",
    "- target/labels: \"Textbooks\", \"Math\", \"Middle School\"\n",
    "\n",
    "Notes:\n",
    "- Supervised learning problem\n",
    "- classification problem\n",
    "- over 100 target variables\n",
    "- 9 broad categories with many possible sub-label instances\n",
    "\n",
    "What solution are we seeking?\n",
    "- Human-in-the-loop learning system\n",
    "- We want to know the probability suggestions for line items\n",
    "    - Ie. I'm 60% sure this line is for textbooks, if it's not textbooks I'm 30% sure it's office supplies\n",
    "    - These suggestions can help prioritize analysts time\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Exploring the raw data - EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load and preview the data\n",
    "import pandas as pd\n",
    "sample_df = pd.read_csv('sample_data.csv')\n",
    "sample_df.head()\n",
    "\n",
    "# summarize the data\n",
    "sample_df.info()\n",
    "sample_df.describe()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = pd.read_csv('TrainingData.csv',index_col=0)\n",
    "df.info()\n",
    "df.head()\n",
    "df.tail()\n",
    "\n",
    "# Print the summary statistics\n",
    "print(df.describe())\n",
    "\n",
    "# Import matplotlib.pyplot as plt\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create the histogram after dropping null values\n",
    "plt.hist(df['FTE'].dropna())\n",
    "\n",
    "# Add title and labels\n",
    "plt.title('Distribution of %full-time \\n employee works')\n",
    "plt.xlabel('% of full-time')\n",
    "plt.ylabel('num employees')\n",
    "\n",
    "# Display the histogram\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encode labels as categories\n",
    "- ML algorithms work on numbers, not strings\n",
    "    - need a numeric representation of these strings\n",
    "- Strings can be slow compared to numbers\n",
    "- in pandas, 'category' dtype encodes categorical data numerically\n",
    "    - can speed up code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example: encode labels from string as categories\n",
    "sample_df.label.head(2)\n",
    "# dtype: object\n",
    "\n",
    "sample_df.label = sample_df.label.astype('category')\n",
    "\n",
    "sample_df.label.head(2)\n",
    "# dtype: category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dummy variable encoding\n",
    "# aka 'binary indicator' representation\n",
    "dummies = pd.get_dummies(sample_df[['label']], prefix_sep='_')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lambda functions\n",
    "# - alternative to 'def' syntax\n",
    "# - easy to make simple, one-line functions\n",
    "# example\n",
    "square = lambda x: x*x\n",
    "square(2)\n",
    "# out: 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make multiple columns into category type using lambda function\n",
    "\n",
    "# lambda function to create each column to a category\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "\n",
    "sample_df.label = sample_df[['label']].apply(categorize_label, axis=0)\n",
    "sample_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# more EDA\n",
    "df.dtypes.value_counts()\n",
    "# object     23\n",
    "# float64     2\n",
    "# dtype: int64\n",
    "\n",
    "# Encode labels as categorical variables\n",
    "# Define the lambda function: categorize_label\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "\n",
    "# Convert df[LABELS] to a categorical type\n",
    "df[LABELS] = df[LABELS].apply(categorize_label, axis=0)\n",
    "\n",
    "# Print the converted dtypes\n",
    "print(df[LABELS].dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count and Plot unique values for each label category\n",
    "\n",
    "# Import matplotlib.pyplot\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Calculate number of unique values for each label: num_unique_labels\n",
    "num_unique_labels = df[LABELS].apply(pd.Series.nunique)\n",
    "\n",
    "# Plot number of unique values for each label\n",
    "num_unique_labels.plot(kind='bar')\n",
    "\n",
    "# Label the axes\n",
    "plt.xlabel('Labels')\n",
    "plt.ylabel('Number of unique values')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating success\n",
    "How do we measure success?\n",
    "- accuracy can be misleading when classes are imbalanced\n",
    "\n",
    "Metric: log loss\n",
    "- loss function\n",
    "- measure of error \n",
    "- want to minimize the error (unlike accuracy)\n",
    "\n",
    "Log loss binary classification\n",
    "- penalizes predictions that are both wrong and confident\n",
    "- actual value: y = {1=yes, 0=no}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log loss implementation with NumPy and clip()\n",
    "import numpy as np\n",
    "\n",
    "def compute_log_loss(predicted, actual, eps=1e-14):\n",
    "    \"\"\" \n",
    "    Computes the logarithmic loss between predicted and actual\n",
    "    when these are 1D arrays.\n",
    "    \n",
    "    :param predicted: The predicted probabilities as floats between 0-1\n",
    "    :param actual: The actual binary labels. Either 0 or 1.\n",
    "    :param eps (optional): log(0) is inf, so we need to offset our\n",
    "        predicted values slightly by eps from 0 or 1.\n",
    "    \"\"\"\n",
    "    predicted = np.clip(predicted, eps, 1-eps)\n",
    "    log = -1 * np.mean(actual * np.log(predicted)\n",
    "            + (1- actual)\n",
    "            * np.log(1-predicted))\n",
    "    return loss\n",
    "\n",
    "# compute log loss\n",
    "# confident and wrong item\n",
    "compute_log_loss(predicted=0.9, actual=0)\n",
    "# out: 2.3025850929940459\n",
    "\n",
    "# prediction right in the middle (predicted=0.5)\n",
    "compute_log_loss(predicted=0.5, actual=1)\n",
    "# out: 0.6931478055994529"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.16251892949777494 4.605170185988091 0.7133498878774648\n"
     ]
    }
   ],
   "source": [
    "# example log loss calculations\n",
    "print(compute_log_loss(predicted=0.85, actual=1),\n",
    "compute_log_loss(predicted=0.99, actual=0),\n",
    "compute_log_loss(predicted=0.51, actual=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see how the log loss metric handles the trade-off between accuracy and confidence, we will use some sample data generated with NumPy and compute the log loss using the provided function compute_log_loss(), which Peter showed you in the video.\n",
    "\n",
    "5 one-dimensional numeric arrays simulating different types of predictions have been pre-loaded: actual_labels, correct_confident, correct_not_confident, wrong_not_confident, and wrong_confident.\n",
    "\n",
    "Your job is to compute the log loss for each sample set provided using the compute_log_loss(predicted_values, actual_values). It takes the predicted values as the first argument and the actual values as the second argument.\n",
    "\n",
    "Using the compute_log_loss() function, compute the log loss for the following predicted values (in each case, the actual values are contained in actual_labels):\n",
    "correct_confident.\n",
    "correct_not_confident.\n",
    "wrong_not_confident.\n",
    "wrong_confident.\n",
    "actual_labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute and print log loss for 1st case\n",
    "correct_confident = compute_log_loss(correct_confident, actual_labels)\n",
    "print(\"Log loss, correct and confident: {}\".format(correct_confident)) \n",
    "\n",
    "# Compute log loss for 2nd case\n",
    "correct_not_confident = compute_log_loss(correct_not_confident, actual_labels)\n",
    "print(\"Log loss, correct and not confident: {}\".format(correct_not_confident)) \n",
    "\n",
    "# Compute and print log loss for 3rd case\n",
    "wrong_not_confident = compute_log_loss(wrong_not_confident, actual_labels)\n",
    "print(\"Log loss, wrong and not confident: {}\".format(wrong_not_confident)) \n",
    "\n",
    "# Compute and print log loss for 4th case\n",
    "wrong_confident = compute_log_loss(wrong_confident,actual_labels)\n",
    "print(\"Log loss, wrong and confident: {}\".format(wrong_confident)) \n",
    "\n",
    "# Compute and print log loss for actual labels\n",
    "actual_labels = compute_log_loss(actual_labels, actual_labels)\n",
    "print(\"Log loss, actual labels: {}\".format(actual_labels)) \n",
    "\n",
    "# <script.py> output:\n",
    "#     Log loss, correct and confident: 0.05129329438755058\n",
    "#     Log loss, correct and not confident: 0.4307829160924542\n",
    "#     Log loss, wrong and not confident: 1.049822124498678\n",
    "#     Log loss, wrong and confident: 2.9957322735539904\n",
    "#     Log loss, actual labels: 9.99200722162646e-15"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Create a simple first model\n",
    "- Train basic model on numeric data only\n",
    "    - remove text data\n",
    "- Multi-class logistic regression (each label independent)\n",
    "    - train classifier on each label separately and use to predict\n",
    "- Format prediction and save to csv\n",
    "- Compute log loss score\n",
    "- use NLP to wrangle text data\n",
    "\n",
    "Splitting the multi-class dataset\n",
    "- Train-test-split won't work here\n",
    "    - may end up with labels in test set that never appear in training set\n",
    "- Solution?: StratifiedShuffleSplit\n",
    "    - Only works with a single target variable\n",
    "    - But, this problem has many target variables\n",
    "- actual solution: multilabel_train_test_split()\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Splitting the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NUMERIC_COLUMNS is a list of column names that are numeric\n",
    "# chose -1000 so that the model would behave differently with negative \n",
    "#  value than 0\n",
    "data_to_train = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "# create array of target variables\n",
    "labels_to_use = pd.get_dummies(df[LABELS])\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(\n",
    "    data_to_train, labels_to_use, size=0.2, seed=123)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Training the model\n",
    "- OneVsRestClassifier\n",
    "    - treats each column of y independently\n",
    "    - in other words, it fits a separate classifier for each of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.a multilabel_train_test_split()\n",
    "https://github.com/drivendataorg/box-plots-sklearn/blob/master/src/data/multilabel.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from warnings import warn\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def multilabel_sample(y, size=1000, min_count=5, seed=None):\n",
    "    \"\"\" Takes a matrix of binary labels `y` and returns\n",
    "        the indices for a sample of size `size` if\n",
    "        `size` > 1 or `size` * len(y) if size =< 1.\n",
    "        The sample is guaranteed to have > `min_count` of\n",
    "        each label.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        if (np.unique(y).astype(int) != np.array([0, 1])).all():\n",
    "            raise ValueError()\n",
    "    except (TypeError, ValueError):\n",
    "        raise ValueError('multilabel_sample only works with binary indicator matrices')\n",
    "\n",
    "    if (y.sum(axis=0) < min_count).any():\n",
    "        raise ValueError('Some classes do not have enough examples. Change min_count if necessary.')\n",
    "\n",
    "    if size <= 1:\n",
    "        size = np.floor(y.shape[0] * size)\n",
    "\n",
    "    if y.shape[1] * min_count > size:\n",
    "        msg = \"Size less than number of columns * min_count, returning {} items instead of {}.\"\n",
    "        warn(msg.format(y.shape[1] * min_count, size))\n",
    "        size = y.shape[1] * min_count\n",
    "\n",
    "    rng = np.random.RandomState(seed if seed is not None else np.random.randint(1))\n",
    "\n",
    "    if isinstance(y, pd.DataFrame):\n",
    "        choices = y.index\n",
    "        y = y.values\n",
    "    else:\n",
    "        choices = np.arange(y.shape[0])\n",
    "\n",
    "    sample_idxs = np.array([], dtype=choices.dtype)\n",
    "\n",
    "    # first, guarantee > min_count of each label\n",
    "    for j in range(y.shape[1]):\n",
    "        label_choices = choices[y[:, j] == 1]\n",
    "        label_idxs_sampled = rng.choice(label_choices, size=min_count, replace=False)\n",
    "        sample_idxs = np.concatenate([label_idxs_sampled, sample_idxs])\n",
    "\n",
    "    sample_idxs = np.unique(sample_idxs)\n",
    "\n",
    "    # now that we have at least min_count of each, we can just random sample\n",
    "    sample_count = int(size - sample_idxs.shape[0])\n",
    "\n",
    "    # get sample_count indices from remaining choices\n",
    "    remaining_choices = np.setdiff1d(choices, sample_idxs)\n",
    "    remaining_sampled = rng.choice(remaining_choices,\n",
    "                                   size=sample_count,\n",
    "                                   replace=False)\n",
    "\n",
    "    return np.concatenate([sample_idxs, remaining_sampled])\n",
    "\n",
    "\n",
    "def multilabel_sample_dataframe(df, labels, size, min_count=5, seed=None):\n",
    "    \"\"\" Takes a dataframe `df` and returns a sample of size `size` where all\n",
    "        classes in the binary matrix `labels` are represented at\n",
    "        least `min_count` times.\n",
    "    \"\"\"\n",
    "    idxs = multilabel_sample(labels, size=size, min_count=min_count, seed=seed)\n",
    "    return df.loc[idxs]\n",
    "\n",
    "\n",
    "def multilabel_train_test_split(X, Y, size, min_count=5, seed=None):\n",
    "    \"\"\" Takes a features matrix `X` and a label matrix `Y` and\n",
    "        returns (X_train, X_test, Y_train, Y_test) where all\n",
    "        classes in Y are represented at least `min_count` times.\n",
    "    \"\"\"\n",
    "    index = Y.index if isinstance(Y, pd.DataFrame) else np.arange(Y.shape[0])\n",
    "\n",
    "    test_set_idxs = multilabel_sample(Y, size=size, min_count=min_count, seed=seed)\n",
    "    train_set_idxs = np.setdiff1d(index, test_set_idxs)\n",
    "\n",
    "    test_set_mask = index.isin(test_set_idxs)\n",
    "    train_set_mask = ~test_set_mask\n",
    "\n",
    "    return (X[train_set_mask], X[test_set_mask], Y[train_set_mask], Y[test_set_mask])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.b Setting up a train-test split in scikit-learn\n",
    "Alright, you've been patient and awesome. It's finally time to start training models!\n",
    "\n",
    "The first step is to split the data into a training set and a test set. Some labels don't occur very often, but we want to make sure that they appear in both the training and the test sets. We provide a function that will make sure at least min_count examples of each label appear in each split: multilabel_train_test_split.\n",
    "\n",
    "Feel free to check out the full code for multilabel_train_test_split here.\n",
    "\n",
    "You'll start with a simple model that uses just the numeric columns of your DataFrame when calling multilabel_train_test_split. The data has been read into a DataFrame df and a list consisting of just the numeric columns is available as NUMERIC_COLUMNS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split data\n",
    "\n",
    "# Create the new DataFrame: numeric_data_only\n",
    "numeric_data_only = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "\n",
    "# Get labels and convert to dummy variables: label_dummies\n",
    "label_dummies = pd.get_dummies(df[LABELS])\n",
    "\n",
    "# Create training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(\n",
    "                                                    numeric_data_only,\n",
    "                                                    label_dummies,\n",
    "                                                    size=0.2, \n",
    "                                                    seed=123)\n",
    "\n",
    "# Print the info\n",
    "print(\"X_train info:\")\n",
    "print(X_train.info())\n",
    "print(\"\\nX_test info:\")  \n",
    "print(X_test.info())\n",
    "print(\"\\ny_train info:\")  \n",
    "print(y_train.info())\n",
    "print(\"\\ny_test info:\")  \n",
    "print(y_test.info()) \n",
    "\n",
    "# X_train info:\n",
    "# <class 'pandas.core.frame.DataFrame'>\n",
    "# Int64Index: 1040 entries, 198 to 101861\n",
    "# Data columns (total 2 columns):\n",
    "# FTE      1040 non-null float64\n",
    "# Total    1040 non-null float64\n",
    "# dtypes: float64(2)\n",
    "# memory usage: 24.4 KB\n",
    "# None\n",
    "\n",
    "# X_test info:\n",
    "# <class 'pandas.core.frame.DataFrame'>\n",
    "# Int64Index: 520 entries, 209 to 448628\n",
    "# Data columns (total 2 columns):\n",
    "# FTE      520 non-null float64\n",
    "# Total    520 non-null float64\n",
    "# dtypes: float64(2)\n",
    "# memory usage: 12.2 KB\n",
    "# None\n",
    "\n",
    "# y_train info:\n",
    "# <class 'pandas.core.frame.DataFrame'>\n",
    "# Int64Index: 1040 entries, 198 to 101861\n",
    "# Columns: 104 entries, Function_Aides Compensation to Operating_Status_PreK-12 Operating\n",
    "# dtypes: float64(104)\n",
    "# memory usage: 853.1 KB\n",
    "# None\n",
    "\n",
    "# y_test info:\n",
    "# <class 'pandas.core.frame.DataFrame'>\n",
    "# Int64Index: 520 entries, 209 to 448628\n",
    "# Columns: 104 entries, Function_Aides Compensation to Operating_Status_PreK-12 Operating\n",
    "# dtypes: float64(104)\n",
    "# memory usage: 426.6 KB\n",
    "# None\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.a Training a model\n",
    "With split data in hand, you're only a few lines away from training a model.\n",
    "\n",
    "In this exercise, you will import the logistic regression and one versus rest classifiers in order to fit a multi-class logistic regression model to the NUMERIC_COLUMNS of your feature data.\n",
    "\n",
    "Then you'll test and print the accuracy with the .score() method to see the results of training.\n",
    "\n",
    "Before you train! Remember, we're ultimately going to be using logloss to score our model, so don't worry too much about the accuracy here. Keep in mind that you're throwing away all of the text data in the dataset - that's by far most of the data! So don't get your hopes up for a killer performance just yet. We're just interested in getting things up and running at the moment.\n",
    "\n",
    "All data necessary to call multilabel_train_test_split() has been loaded into the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import classifiers\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "\n",
    "# Create the DataFrame: numeric_data_only\n",
    "numeric_data_only = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "\n",
    "# Get labels and convert to dummy variables: label_dummies\n",
    "label_dummies = pd.get_dummies(df[LABELS])\n",
    "\n",
    "# Create training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(\n",
    "                                                       numeric_data_only,\n",
    "                                                       label_dummies,\n",
    "                                                       size=0.2, \n",
    "                                                       seed=123)\n",
    "\n",
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit the classifier to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Print the accuracy\n",
    "print(\"Accuracy: {}\".format(clf.score(X_test, y_test)))\n",
    "\n",
    "# Accuracy: 0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bad news is that your model scored the lowest possible accuracy: 0.0! \n",
    "\n",
    "But hey, you just threw away ALL of the text data in the budget. Later, you won't. \n",
    "\n",
    "Before you add the text data, let's see how the model does when scored by log loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Predicting on holdout data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "holdout = pd.read_csv('HoldoutData.csv',index_col=0)\n",
    "holdout = holdout[NUMERIC_COLUMNS].fillna(-1000)\n",
    "predictions = clf.predict_proba(holdout)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If .predict() was used instead of .predict_proba\n",
    "- output would be 0 or 1\n",
    "- log loss penalizes being confident and wrong\n",
    "- so worse performance (score) compared to .predict_proba()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Format and submit predictions\n",
    "- all formatting can be done with the pandas to to_csv()\n",
    "- predictions are an array, so convert to a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# format column names with '__'\n",
    "prediction_df = pd.DataFram(columns=pd.get_dummies(df[LABELS],\n",
    "                            prefix_sep='__').columns,\n",
    "                            index=holdout.index,\n",
    "                            data=predictions)\n",
    "prediction_df.to_csv('predictions.csv')\n",
    "score = score_submission(pred_path='predictions.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.a Use your model to predict values on holdout data\n",
    "You're ready to make some predictions! Remember, the train-test-split you've carried out so far is for model development. The original competition provides an additional test set, for which you'll never actually see the correct labels. This is called the \"holdout data.\"\n",
    "\n",
    "The point of the holdout data is to provide a fair test for machine learning competitions. If the labels aren't known by anyone but DataCamp, DrivenData, or whoever is hosting the competition, you can be sure that no one submits a mere copy of labels to artificially pump up the performance on their model.\n",
    "\n",
    "Remember that the original goal is to predict the probability of each label. In this exercise you'll do just that by using the .predict_proba() method on your trained model.\n",
    "\n",
    "First, however, you'll need to load the holdout data, which is available in the workspace as the file HoldoutData.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit it to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Load the holdout data: holdout\n",
    "holdout = pd.read_csv('HoldoutData.csv', index_col=0)\n",
    "\n",
    "# Generate predictions: predictions\n",
    "predictions = clf.predict_proba(\n",
    "                holdout[NUMERIC_COLUMNS].fillna(-1000))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.a Writing out your results to a csv for submission\n",
    "At last, you're ready to submit some predictions for scoring. In this exercise, you'll write your predictions to a .csv using the .to_csv() method on a pandas DataFrame. Then you'll evaluate your performance according to the LogLoss metric discussed earlier!\n",
    "\n",
    "You'll need to make sure your submission obeys the correct format.\n",
    "\n",
    "To do this, you'll use your predictions values to create a new DataFrame, prediction_df.\n",
    "\n",
    "Interpreting LogLoss & Beating the Benchmark:\n",
    "\n",
    "When interpreting your log loss score, keep in mind that the score will change based on the number of samples tested. To get a sense of how this very basic model performs, compare your score to the DrivenData benchmark model performance: 2.0455, which merely submitted uniform probabilities for each class.\n",
    "\n",
    "Remember, the lower the log loss the better. Is your model's log loss lower than 2.0455?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate predictions: predictions\n",
    "predictions = clf.predict_proba(holdout[NUMERIC_COLUMNS].fillna(-1000))\n",
    "\n",
    "# Format predictions in DataFrame: prediction_df\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(\n",
    "                             df[LABELS]).columns,\n",
    "                             index=holdout.index,\n",
    "                             data=predictions)\n",
    "\n",
    "\n",
    "# Save prediction_df to csv\n",
    "prediction_df.to_csv('predictions.csv')\n",
    "\n",
    "# Submit the predictions for scoring: score\n",
    "score = score_submission(pred_path='predictions.csv')\n",
    "\n",
    "# Print score\n",
    "print('Your model, trained with numeric data only, yields logloss score: {}'.format(score))\n",
    "\n",
    "# Your model, trained with numeric data only, \n",
    "# yields logloss score: 1.9067227623381413\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Intro to NLP\n",
    "Data for NLP: text, documents, speech, ...\n",
    "\n",
    "Tokenization\n",
    "   - = splitting a string into segments\n",
    "   - store segments as list\n",
    "   - Example: 'Natural Language Processing'\n",
    "       - ['Natural', 'Language', 'Processing']\n",
    "       \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokens and token patterns\n",
    "\n",
    "example: PETRO-VEND FUEL AND FLUIDS\n",
    "\n",
    "Tokenize on whitespace\n",
    "- PETRO-VEND | FUEL | AND | FLUIDS\n",
    "\n",
    "Tokenize on whitespace and punctuation\n",
    "- PETRO | VEND | FUEL | AND | FLUIDS\n",
    "\n",
    "Bag of words representation\n",
    "- count number of times a particular token appears\n",
    "- \"bag of words\"\n",
    "    - cvount number of times a word occurs\n",
    "    - This approach discards info about word order\n",
    "        - \"Red, not blue\" is the same as \"blue, not red\"\n",
    "\n",
    "N-grams\n",
    "- 1-gram, 2-gram, ..., n-gram\n",
    "    - 1-gram: PETRO, VEND, FUEL, AND, FLUIDS\n",
    "    - 2-grams: PETRO VEND, VEND FUEL, FUEL AND, AND FLUIDS\n",
    "    - 3-grams:PETRO VEND FUEL, VEND FUEL AND, FUEL AND FLUIDS\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Testing your NLP credentials with n-grams\n",
    "You're well on your way to NLP superiority. Let's test your mastery of n-grams!\n",
    "\n",
    "In the workspace, we have the loaded a python list, one_grams, which contains all 1-grams of the string petro-vend fuel and fluids, tokenized on punctuation. Specifically,\n",
    "\n",
    "one_grams = ['petro', 'vend', 'fuel', 'and', 'fluids']\n",
    "\n",
    "In this exercise, your job is to determine the sum of the sizes of 1-grams, 2-grams and 3-grams generated by the string petro-vend fuel and fluids, tokenized on punctuation.\n",
    "\n",
    "Recall that the n-gram of a sequence consists of all ordered subsequences of length n.\n",
    "\n",
    "Answer:\n",
    "The number of 1-grams + 2-grams + 3-grams is 5 + 4 + 3 = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 Representing text numerically\n",
    "- Bag-of-words\n",
    "    - simple way to represent text in machine learning\n",
    "    - discards info about grammar and word order\n",
    "    - computes frequency of occurrence\n",
    "    \n",
    "sklearn tools for bag-of-words\n",
    "- CountVectorizer()\n",
    "    - Tokenizes all the strings\n",
    "    - builds a 'vocabulary' (of all the words)\n",
    "    - counts the occurrences of each token in the vocabulary\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CountVectorizer() - use on column of main dataset\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# regex that does whitespace split\n",
    "TOKENS_BASIC = '\\\\S+(?=\\\\s+)'\n",
    "\n",
    "# replace NaN with empty string\n",
    "df.Program_Description.fillna('', inplace=TRUE)\n",
    "\n",
    "# countvectorizer object to create bag-of-words\n",
    "vec_basic = CountVectorizer(token_pattern=TOKENS_BASIC)\n",
    "\n",
    "# use fit and transform pattern\n",
    "# fit will parse out and create vocabulary\n",
    "vec_basic.fit(df.Program_Description)\n",
    "# transform will tokenize the text and create an array of counts\n",
    "msg = 'There are {} tokens in Program_Description if tokens are any non-whitespace'\n",
    "\n",
    "print(msg.format(len(vec_basic.get_feature_names())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8 Creating a bag-of-words in scikit-learn\n",
    "In this exercise, you'll study the effects of tokenizing in different ways by comparing the bag-of-words representations resulting from different token patterns.\n",
    "\n",
    "You will focus on one feature only, the Position_Extra column, which describes any additional information not captured by the Position_Type label.\n",
    "\n",
    "For example, in the Shell you can check out the budget item in row 8960 of the data using df.loc[8960]. Looking at the output reveals that this Object_Description is overtime pay. For who? The Position Type is merely \"other\", but the Position Extra elaborates: \"BUS DRIVER\". Explore the column further to see more instances. It has a lot of NaN values.\n",
    "\n",
    "Your task is to turn the raw text in this column into a bag-of-words representation by creating tokens that contain only alphanumeric characters.\n",
    "\n",
    "For comparison purposes, the first 15 tokens of vec_basic, which splits df.Position_Extra into tokens when it encounters only whitespace characters, have been printed along with the length of the representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Fill missing values in df.Position_Extra\n",
    "df.Position_Extra.fillna('',inplace=True)\n",
    "\n",
    "# Instantiate the CountVectorizer: vec_alphanumeric\n",
    "vec_alphanumeric = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit to the data\n",
    "vec_alphanumeric.fit(df.Position_Extra)\n",
    "\n",
    "# Print the number of tokens and first 15 tokens\n",
    "msg = \"There are {} tokens in Position_Extra if we split on non-alpha numeric\"\n",
    "print(msg.format(len(vec_alphanumeric.get_feature_names())))\n",
    "print(vec_alphanumeric.get_feature_names()[:15])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.9 Combining text columns for tokenization\n",
    "In order to get a bag-of-words representation for all of the text data in our DataFrame, you must first convert the text data in each row of the DataFrame into a single string.\n",
    "\n",
    "In the previous exercise, this wasn't necessary because you only looked at one column of data, so each row was already just a single string. \n",
    "\n",
    "- CountVectorizer expects each row to just be a single string, so in order to use all of the text columns, you'll need a method to turn a list of strings into a single string.\n",
    "\n",
    "In this exercise, you'll complete the function definition combine_text_columns(). When completed, this function will convert all training text data in your DataFrame to a single string per row that can be passed to the vectorizer object and made into a bag-of-words using the .fit_transform() method.\n",
    "\n",
    "Note that the function uses NUMERIC_COLUMNS and LABELS to determine which columns to drop. These lists have been loaded into the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define combine_text_columns() to tokenize data\n",
    "def combine_text_columns(data_frame, to_drop=NUMERIC_COLUMNS + LABELS):\n",
    "    \"\"\" converts all text in each row of data_frame to single vector \"\"\"\n",
    "    \n",
    "    # Drop non-text columns that are in the df\n",
    "    to_drop = set(to_drop) & set(data_frame.columns.tolist())\n",
    "    text_data = data_frame.drop(to_drop, axis=1)\n",
    "    \n",
    "    # Replace nans with blanks\n",
    "    text_data.fillna('',inplace=True)\n",
    "    \n",
    "    # Join all text items in a row that have a space in between\n",
    "    # axis=1, drop labels from columns (vs index)\n",
    "    return text_data.apply(lambda x: \" \".join(x), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.10 What's in a token?\n",
    "Now you will use combine_text_columns to convert all training text data in your DataFrame to a single vector that can be passed to the vectorizer object and made into a bag-of-words using the .fit_transform() method.\n",
    "\n",
    "You'll compare the effect of tokenizing using any non-whitespace characters as a token and using only alphanumeric characters as a token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Create the basic token pattern\n",
    "TOKENS_BASIC = '\\\\S+(?=\\\\s+)'\n",
    "\n",
    "# Create the alphanumeric token pattern\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate basic CountVectorizer: vec_basic\n",
    "vec_basic = CountVectorizer(token_pattern=TOKENS_BASIC)\n",
    "\n",
    "# Instantiate alphanumeric CountVectorizer: vec_alphanumeric\n",
    "vec_alphanumeric = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Create the text vector\n",
    "text_vector = combine_text_columns(df)\n",
    "\n",
    "# Fit and transform vec_basic\n",
    "vec_basic.fit_transform(text_vector)\n",
    "\n",
    "# Print number of tokens of vec_basic\n",
    "print(\"There are {} tokens in the dataset\".format(\n",
    "    len(vec_basic.get_feature_names())))\n",
    "\n",
    "# Fit and transform vec_alphanumeric\n",
    "vec_alphanumeric.fit_transform(text_vector)\n",
    "\n",
    "# Print number of tokens of vec_alphanumeric\n",
    "print(\"There are {} alpha-numeric tokens in the dataset\".format(\n",
    "    len(vec_alphanumeric.get_feature_names())))\n",
    "\n",
    "# There are 1405 tokens in the dataset\n",
    "# There are 1117 alpha-numeric tokens in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Improve benchmark model using pipelines\n",
    "- use text and numeric data\n",
    "- use pipelines to process multiple types of data\n",
    "- use pipeline workflow\n",
    "\n",
    "The pipeline workflow\n",
    "- repeatable way to go from raw data to trained model\n",
    "- Pipeline object takes sequential list of steps\n",
    "    - output of one step is input to next step\n",
    "- Each step is a tuple with two elements\n",
    "    1. Name: string\n",
    "    2. Transform: obj implementing .fit() and .transform()\n",
    "- Flexible: a step can itself be another pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# simple pipeline with one step\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "pl = Pipeline([\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# train and test with smaple numeric data to predict 'label'\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df[['numeric']],\n",
    "                                    pd.get_dummies(sample_df['label']),\n",
    "                                    random_state=2)\n",
    "pl.fit(X_train, y_train)\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print('accuracy on numeric data, no nans: ', accuracy)\n",
    "# accuracy on numeric data, no nans: 0.44\n",
    "\n",
    "# Adding more steps to the pipeline\n",
    "# Adding 'with_missing' gives an error b/c input has NaN\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df[['numeric','with_missing']],\n",
    "                                    pd.get_dummies(sample_df['label']),\n",
    "                                    random_state=2)\n",
    "pl.fit(X_train, y_train)\n",
    "# Add step to pipeline: imputer to take care of NaN values\n",
    "# Default imputer in sklearn is to fill NaN with mean\n",
    "fromsklearn.preprocessing import Imputer\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df[['numeric','with_missing']],\n",
    "                                    pd.get_dummies(sample_df['label']),\n",
    "                                    random_state=2)\n",
    "pl = Pipeline([\n",
    "        ('imp', Imputer()),\n",
    "         'clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "pl.fit(X_train, y_train)\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print('accuracy on numeric data, incl nans: ', accuracy)\n",
    "# accuracy on numeric data, incl nans: 0.48"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Instantiate pipeline\n",
    "In order to make your life easier as you start to work with all of the data in your original DataFrame, df, it's time to turn to one of scikit-learn's most useful objects: the Pipeline.\n",
    "\n",
    "For the next few exercises, you'll reacquaint yourself with pipelines and train a classifier on some synthetic (sample) data of multiple datatypes before using the same techniques on the main dataset.\n",
    "\n",
    "The sample data is stored in the DataFrame, sample_df, which has three kinds of feature data: numeric, text, and numeric with missing values. It also has a label column with two classes, a and b.\n",
    "\n",
    "In this exercise, your job is to instantiate a pipeline that trains using the numeric column of the sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Pipeline\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# Import other necessary modules\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "\n",
    "# Split and select numeric data only, no nans \n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    sample_df[['numeric']],\n",
    "    pd.get_dummies(sample_df['label']), \n",
    "    random_state=22)\n",
    "\n",
    "# Instantiate Pipeline object: pl\n",
    "pl = Pipeline([\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Fit the pipeline to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - numeric, no nans: \", accuracy)\n",
    "\n",
    "# Accuracy on sample data - numeric, no nans:  0.62"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Preprocessing numeric features\n",
    "Preprocessing step to deal with NaNs\n",
    "\n",
    "What would have happened if you had included the with 'with_missing' column in the last exercise? Without imputing missing values, the pipeline would not be happy (try it and see). So, in this exercise you'll improve your pipeline a bit by using the Imputer() imputation transformer from scikit-learn to fill in missing values in your sample data.\n",
    "\n",
    "By default, the imputer transformer replaces NaNs with the mean value of the column. That's a good enough imputation strategy for the sample data, so you won't need to pass anything extra to the imputer.\n",
    "\n",
    "After importing the transformer, you will edit the steps list used in the previous exercise by inserting a (name, transform) tuple. Recall that steps are processed sequentially, so make sure the new tuple encoding your preprocessing step is put in the right place.\n",
    "\n",
    "The sample_df is in the workspace, in case you'd like to take another look. Make sure to select both numeric columns- in the previous exercise we couldn't use with_missing because we had no preprocessing step!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Imputer object\n",
    "from sklearn.preprocessing import Imputer\n",
    "\n",
    "# Create training and test sets using only numeric data\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df[['numeric', 'with_missing']],\n",
    "                                    pd.get_dummies(sample_df['label']), \n",
    "                                    random_state=456)\n",
    "\n",
    "# Insantiate Pipeline object: pl\n",
    "pl = Pipeline([\n",
    "        ('imp', Imputer()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Fit the pipeline to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - all numeric, incl nans: \", accuracy)\n",
    "\n",
    "# Accuracy on sample data - all numeric, incl nans:  0.636"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Text features and feature unions - in pipeline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocessing text features\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df['text'],\n",
    "                                    pd.get_dummies(sample_df['label']), \n",
    "                                    random_state=2)\n",
    "pl = Pipeline([\n",
    "        ('vec', CountVectorizer()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "pl.fit(X_train, y_train)\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.a Preprocessing multiple dtypes\n",
    "- Want to use ALL available features in one pipeline\n",
    "- Problem\n",
    "    - Pipeline steps for numeric and text processing can't follow each other\n",
    "    - ie. output of CountVectorizer can't be input to Imputer\n",
    "- Need to separately operate on text vs numeric columns\n",
    "- Solution\n",
    "    - FunctionTransformer()\n",
    "    - FeatureUnion()\n",
    "\n",
    "FunctionTransformer\n",
    "- job: Turns a python function into an object that scikit-learn pipeline can understand\n",
    "- Need to write two functions for pipeline preprocessing\n",
    "    - Take entire DataFrame, return numeric columns\n",
    "    - Take entire DataFrame, return text columns\n",
    "- Can then preprocess numeric and text data in separate pipelines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FunctionTransformer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                    sample_df[['numeric','with_missing',\n",
    "                                             'text']],\n",
    "                                    pd.get_dummies(sample_df['label']), \n",
    "                                    random_state=2)\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "# create 2 FunctionTransformer objects\n",
    "# :param validate=False means don't check for NaNs or validate dtypes\n",
    "#  as we'll do it ourself\n",
    "get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x['numeric',\n",
    "                                                  'with_missing'], \n",
    "                                       validate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FeatureUnion Text and Numeric features\n",
    "# See how FeatureUnion works and how it works in pipeline below\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "union = FeatureUnion([\n",
    "        ('numeric', numeric_pipeline),\n",
    "        ('text', text_pipeline)\n",
    "])\n",
    "\n",
    "# entire pipeline\n",
    "numeric_pipeline = Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "])\n",
    "text_pipeline = Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "])\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion([\n",
    "        ('numeric', numeric_pipeline),\n",
    "        ('text', text_pipeline)\n",
    "    ])),\n",
    "    ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "])\n",
    "pl.fit(X_train, y_train)\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.b Preprocessing text features\n",
    "Here, you'll perform a similar preprocessing pipeline step, only this time you'll use the text column from the sample data.\n",
    "\n",
    "To preprocess the text, you'll turn to CountVectorizer() to generate a bag-of-words representation of the data, as in Chapter 2. Using the default arguments, add a (step, transform) tuple to the steps list in your pipeline.\n",
    "\n",
    "Make sure you select only the text column for splitting your training and test sets.\n",
    "\n",
    "As usual, your sample_df is ready and waiting in the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Split out only the text data\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                                        sample_df['text'],\n",
    "                                        pd.get_dummies(sample_df['label']), \n",
    "                                        random_state=456)\n",
    "\n",
    "# Instantiate Pipeline object: pl\n",
    "pl = Pipeline([\n",
    "        ('vec', CountVectorizer()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - just text data: \", accuracy)\n",
    "\n",
    "# Accuracy on sample data - just text data:  0.808"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.c Multiple types of processing: FunctionTransformer\n",
    "The next two exercises will introduce new topics you'll need to make your pipeline truly excel.\n",
    "\n",
    "Any step in the pipeline must be an object that implements the fit and transform methods. The FunctionTransformer creates an object with these methods out of any Python function that you pass to it. We'll use it to help select subsets of data in a way that plays nicely with pipelines.\n",
    "\n",
    "You are working with numeric data that needs imputation, and text data that needs to be converted into a bag-of-words. You'll create functions that separate the text from the numeric variables and see how the .fit() and .transform() methods work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import FunctionTransformer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "# Obtain the text data: get_text_data\n",
    "get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)\n",
    "\n",
    "# Obtain the numeric data: get_numeric_data\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[['numeric',\n",
    "                                                    'with_missing']], \n",
    "                                                    validate=False)\n",
    "\n",
    "# Fit and transform the text data: just_text_data\n",
    "just_text_data = get_text_data.fit_transform(sample_df)\n",
    "\n",
    "# Fit and transform the numeric data: just_numeric_data\n",
    "just_numeric_data = get_numeric_data.fit_transform(sample_df)\n",
    "\n",
    "# Print head to check results\n",
    "print('Text Data')\n",
    "print(just_text_data.head())\n",
    "print('\\nNumeric Data')\n",
    "print(just_numeric_data.head())\n",
    "\n",
    "# Text Data\n",
    "# 0           \n",
    "# 1        foo\n",
    "# 2    foo bar\n",
    "# 3           \n",
    "# 4    foo bar\n",
    "# Name: text, dtype: object\n",
    "\n",
    "# Numeric Data\n",
    "#      numeric  with_missing\n",
    "# 0 -10.856306      4.433240\n",
    "# 1   9.973454      4.310229\n",
    "# 2   2.829785      2.469828\n",
    "# 3 -15.062947      2.852981\n",
    "# 4  -5.786003      1.826475"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.d Multiple types of processing: FeatureUnion\n",
    "### Nested Pipeline example\n",
    "Now that you can separate text and numeric data in your pipeline, you're ready to perform separate steps on each by nesting pipelines and using FeatureUnion().\n",
    "\n",
    "These tools will allow you to streamline all preprocessing steps for your model, even when multiple datatypes are involved. Here, for example, you don't want to impute our text data, and you don't want to create a bag-of-words with our numeric data. Instead, you want to deal with these separately and then join the results together using FeatureUnion().\n",
    "\n",
    "In the end, you'll still have only two high-level steps in your pipeline: preprocessing and model instantiation. The difference is that the first preprocessing step actually consists of a pipeline for numeric data and a pipeline for text data. The results of those pipelines are joined using FeatureUnion()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Nested pipeline and featureunion\n",
    "\n",
    "# Import FeatureUnion\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "# Split using ALL data in sample_df\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "                            sample_df[['numeric', 'with_missing', 'text']],\n",
    "                            pd.get_dummies(sample_df['label']), \n",
    "                            random_state=22)\n",
    "\n",
    "# Create a FeatureUnion with nested pipeline: process_and_join_features\n",
    "process_and_join_features = FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )\n",
    "\n",
    "# Instantiate nested pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', process_and_join_features),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "\n",
    "# Fit pl to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - all data: \", accuracy)\n",
    "\n",
    "# Accuracy on sample data - all data:  0.928"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Choosing a classification model\n",
    "Recall school budget dataset\n",
    "- 14 text columns\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Main dataset: lots of text\n",
    "LABELS = ['Function', 'Use', 'Sharing', 'Reporting', 'Student_Type',\n",
    "         'Position_Type', 'Object_Type', 'Pre_K', 'Operating_Status']\n",
    "NON_LABELS = [c for c in df.columns if c not in LABELS]\n",
    "len(NON_LABELS) - len(NUMERIC_COLUMNS)\n",
    "# 14\n",
    "\n",
    "# Recall combine_text_columns() that combined all text columns into\n",
    "# a single column.\n",
    "# Use this function in the budget dataset.\n",
    "\n",
    "# Using pipeline with main dataset\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "df = pd.read_csv('TrainingSetSample.csv', index_col=0)\n",
    "dummy_labels = pd.get_dummies(df[LABELS])\n",
    "X_train,X_test,y_train,y_test = multilabel_train_test_split(\n",
    "                                df[NON_LABELS], dummy_labels, 0.2)\n",
    "\n",
    "get_text_data = FunctionTransformer(combine_text_columns,\n",
    "                                   validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x:\n",
    "                                      x[NUMERIC_COLUMNS], validate=False)\n",
    "\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion([\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "            ])\n",
    "    ),\n",
    "    ('clf', OneVsRestClassifier(LogisticRegression))\n",
    "])\n",
    "pl.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.a Flexibility of model step\n",
    "- Is current model the best?\n",
    "- Can quickly try different models with pipelines\n",
    "    - Pipeline preprocessing steps unchanges\n",
    "    - Edit the model step in your pipeline\n",
    "    - Ie. classifiers: Random Forest, Naïve Bayes, k-NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Easily try new models using pipeline: ie. Random Forest\n",
    "# import classifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion([\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "            ])\n",
    "    ),\n",
    "    # change one line for different model\n",
    "    ('clf', OneVsRestClassifier(RandomForestClassifier))\n",
    "])\n",
    "pl.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using FunctionTransformer on the main dataset\n",
    "In this exercise you're going to use FunctionTransformer on the primary budget data, before instantiating a multiple-datatype pipeline in the next exercise.\n",
    "\n",
    "Recall from Chapter 2 that you used a custom function combine_text_columns to select and properly format text data for tokenization; it is loaded into the workspace and ready to be put to work in a function transformer!\n",
    "\n",
    "Concerning the numeric data, you can use NUMERIC_COLUMNS, preloaded as usual, to help design a subset-selecting lambda function.\n",
    "\n",
    "You're all finished with sample data. The original df is back in the workspace, ready to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import FunctionTransformer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "# Get the dummy encoding of the labels\n",
    "dummy_labels = pd.get_dummies(df[LABELS])\n",
    "\n",
    "# Get the columns that are features in the original df\n",
    "NON_LABELS = [c for c in df.columns if c not in LABELS]\n",
    "\n",
    "# Split into training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(df[NON_LABELS],\n",
    "                                                               dummy_labels,\n",
    "                                                               0.2, \n",
    "                                                               seed=123)\n",
    "\n",
    "# Preprocess the text data: get_text_data\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "\n",
    "# Preprocess the numeric data: get_numeric_data\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS], \n",
    "                                       validate=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a model to the pipeline\n",
    "You're about to take everything you've learned so far and implement it in a Pipeline that works with the real, DrivenData budget line item data you've been exploring.\n",
    "\n",
    "Surprise! The structure of the pipeline is exactly the same as earlier in this chapter:\n",
    "\n",
    "the preprocessing step uses FeatureUnion to join the results of nested pipelines that each rely on FunctionTransformer to select multiple datatypes\n",
    "the model step stores the model object\n",
    "You can then call familiar methods like .fit() and .score() on the Pipeline object pl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete the pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)\n",
    "\n",
    "# Accuracy on budget dataset:  0.203846153846"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Try a different class of model: RandomForest\n",
    "Now you're cruising. One of the great strengths of pipelines is how easy they make the process of testing different models.\n",
    "\n",
    "Until now, you've been using the model step ('clf', OneVsRestClassifier(LogisticRegression())) in your pipeline.\n",
    "\n",
    "But what if you want to try a different model? Do you need to build an entirely new pipeline? New nests? New FeatureUnions? Nope! You just have a simple one-line change, as you'll see in this exercise.\n",
    "\n",
    "In particular, you'll swap out the logistic-regression model and replace it with a random forest classifier, which uses the statistics of an ensemble of decision trees to generate predictions.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import random forest classifer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Edit model step in pipeline\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        # removed OneVsRestClassifier from previous example\n",
    "        ('clf', RandomForestClassifier())\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)\n",
    "\n",
    "# Accuracy on budget dataset:  0.296153846154"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Can you adjust the model or parameters to improve accuracy?\n",
    "You just saw a substantial improvement in accuracy by swapping out the model. Pipelines are amazing!\n",
    "\n",
    "Can you make it better? Try changing the parameter n_estimators of RandomForestClassifier(), whose default value is 10, to 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import RandomForestClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# Add model step to pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('clf', RandomForestClassifier(n_estimators=15))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)\n",
    "\n",
    "# Accuracy on budget dataset:  0.321153846154"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Learning from experts/other models\n",
    "- tips and tricks from winner\n",
    "    - text processing\n",
    "    - statistical methods\n",
    "    - computational efficiency\n",
    "- NLP tricks for text data\n",
    "    - tokenize on punctuation to avoid hyphens, underscores, etc.\n",
    "    - Include unigrams and bi-grams in the model to capture important info involving multiple tokens - ie. 'middle school'\n",
    "    \n",
    "Summary:\n",
    "- NLP: range of n-grams, punctuation tokenization\n",
    "- Stats: interaction terms\n",
    "- Comutation: hasting trick\n",
    "- Winning model: Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 N-grams and tokenization\n",
    "- Simple changes to CountVectorizer\n",
    "    - alphanumeric tokenization\n",
    "    - one and two grams: ngram_range=(1,2)\n",
    "\n",
    "vec = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                      ngram_range=(1,2))\n",
    "                      \n",
    "pl.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Range o n-grams in scikit-learn\n",
    "\n",
    "holdout = pd.read_csv('HoldoutData.csv', index_col=0)\n",
    "\n",
    "# class probabilities\n",
    "# We don't need to do additional preprocessing on holdout since\n",
    "#  preprocessing is built into the pipeline\n",
    "predictions = clf.predict_proba(holdout)\n",
    "\n",
    "# Format predictions in DataFrame: prediction_df\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(\n",
    "                             df[LABELS]).columns,\n",
    "                             index=holdout.index,\n",
    "                             data=predictions)\n",
    "\n",
    "\n",
    "# Save prediction_df to csv\n",
    "prediction_df.to_csv('predictions.csv')\n",
    "\n",
    "# Submit the predictions for scoring: score\n",
    "score = score_submission(pred_path='predictions.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many tokens?\n",
    "Recall from previous chapters that how you tokenize text affects the n-gram statistics used in your model.\n",
    "\n",
    "Going forward, you'll use alpha-numeric sequences, and only alpha-numeric sequences, as tokens. Alpha-numeric tokens contain only letters a-z and numbers 0-9 (no other characters). In other words, you'll tokenize on punctuation to generate n-gram statistics.\n",
    "\n",
    "In this exercise, you'll make sure you remember how to tokenize on punctuation.\n",
    "\n",
    "Assuming we tokenize on punctuation, accepting only alpha-numeric sequences as tokens, how many tokens are in the following string from the main dataset?\n",
    "\n",
    "'PLANNING,RES,DEV,& EVAL      '\n",
    "\n",
    "If you want, we've loaded this string into the workspace as SAMPLE_STRING, but you may not need it to answer the question.\n",
    "\n",
    "Answer: 4. Commas, &, and whitespace are not tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deciding what's a word\n",
    "Before you build up to the winning pipeline, it will be useful to look a little deeper into how the text features will be processed.\n",
    "\n",
    "In this exercise, you will use CountVectorizer on the training data X_train (preloaded into the workspace) to see the effect of tokenization on punctuation.\n",
    "\n",
    "Remember, since CountVectorizer expects a vector, you'll need to use the preloaded function, combine_text_columns before fitting to the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Create the text vector by preprocessing X_train using function\n",
    "# This will get tokens\n",
    "text_vector = combine_text_columns(X_train)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate the CountVectorizer: text_features\n",
    "text_features = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit text_features to the text vector\n",
    "text_features.fit(text_vector)\n",
    "\n",
    "# Print the first 10 tokens\n",
    "print(text_features.get_feature_names()[:10])\n",
    "\n",
    "# ['00a', '12', '1st', '2nd', '3rd', '5th', '70', '70h', '8', 'a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-gram range in scikit-learn\n",
    "In this exercise you'll insert a CountVectorizer instance into your pipeline for the main dataset, and compute multiple n-gram features to be used in the model.\n",
    "\n",
    "In order to look for ngram relationships at multiple scales, you will use the ngram_range parameter as Peter discussed in the video.\n",
    "\n",
    "Special functions: You'll notice a couple of new steps provided in the pipeline in this and many of the remaining exercises. Specifically, the dim_red step following the vectorizer step , and the scale step preceeding the clf (classification) step.\n",
    "\n",
    "These have been added in order to account for the fact that you're using a reduced-size sample of the full dataset in this course. To make sure the models perform as the expert competition winner intended, we have to apply a dimensionality reduction technique, which is what the dim_red step does, and we have to scale the features to lie between -1 and 1, which is what the scale step does.\n",
    "\n",
    "The dim_red step uses a scikit-learn function called SelectKBest(), applying something called the chi-squared test to select the K \"best\" features. The scale step uses a scikit-learn function called MaxAbsScaler() in order to squash the relevant features into the interval -1 to 1.\n",
    "\n",
    "You won't need to do anything extra with these functions here, just complete the vectorizing pipeline steps below. However, notice how easy it was to add more processing steps to our pipeline!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pipeline\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# Import classifiers\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "\n",
    "# Import CountVectorizer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Import other preprocessing modules\n",
    "from sklearn.preprocessing import Imputer\n",
    "from sklearn.feature_selection import chi2, SelectKBest\n",
    "\n",
    "# Select 300 best features\n",
    "chi_k = 300\n",
    "\n",
    "# Import functional utilities\n",
    "from sklearn.preprocessing import FunctionTransformer, MaxAbsScaler\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "# Perform preprocessing\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS], \n",
    "                                       validate=False)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer(\n",
    "                                    token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                    ngram_range=(1,2))),\n",
    "                    # dimensionality reduction step using chi-squared test\n",
    "                    ('dim_red', SelectKBest(chi2, chi_k))\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        # scale features b/n -1 and 1\n",
    "        ('scale', MaxAbsScaler()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Note: Log loss score: 1.2681"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Stats trick - interaction terms\n",
    "- Stat tool winner used: interaction terms\n",
    "- Example\n",
    "    - English teacher for 2nd grade\n",
    "    - 2nd grade - budget for English teacher\n",
    "- Interaction terms mathematically describe when tokens appear together\n",
    "    - b1x1 + b2x2 + b3(x1*x2)\n",
    "    - the last term is the interaction term"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding interaction features with scikit-learn\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "x\n",
    "#     x1 x2\n",
    "# a   0  1\n",
    "# b   1  1\n",
    "interaction = PolynomialFeatures(degree=2,\n",
    "                                interaction_only=True,\n",
    "                                include_bias=False)\n",
    "# Notes:\n",
    "# Can have larger degrees, but can be more computationally taxing\n",
    "# interaction_only=True, means do not need to multipy a column by itself\n",
    "\n",
    "interaction.fit_transform(x)\n",
    "# array([[ 0., 1., 0.[,\n",
    "#          1., 1., 1.]]])\n",
    "# Note: new column added\n",
    "\n",
    "# Note about bias terms (:param include_bias)\n",
    "# bias term is an offset for a model\n",
    "# Bias term allows model to have non-zero y value when x value is 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse interaction features\n",
    "SparseInteractions(degree=2).fit_transform(x).toarray()\n",
    "- The number of interaction terms grows exponentially\n",
    "- Our vectorizer saves memory by using a sparse matrix\n",
    "- PolynomialFeatures does not support sparse matrices\n",
    "- We've provided SparseInteractions to work for this problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which models of the data include interaction terms?\n",
    "Recall from the video that interaction terms involve products of features.\n",
    "\n",
    "Suppose we have two features x and y, and we use models that process the features as follows:\n",
    "\n",
    "βx + βy + ββ\n",
    "βxy + βx + βy\n",
    "βx + βy + βx^2 + βy^2\n",
    "where β is a coefficient in your model (not a feature).\n",
    "\n",
    "Which expression(s) include interaction terms?\n",
    "The second expression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement interaction modeling in scikit-learn\n",
    "It's time to add interaction features to your model. The PolynomialFeatures object in scikit-learn does just that, but here you're going to use a custom interaction object, SparseInteractions. Interaction terms are a statistical tool that lets your model express what happens if two features appear together in the same row.\n",
    "\n",
    "SparseInteractions does the same thing as PolynomialFeatures, but it uses sparse matrices to do so. You can get the code for SparseInteractions at this GitHub Gist.\n",
    "https://github.com/drivendataorg/box-plots-sklearn/blob/master/src/features/SparseInteractions.py\n",
    "\n",
    "PolynomialFeatures and SparseInteractions both take the argument degree, which tells them what polynomial degree of interactions to compute.\n",
    "\n",
    "You're going to consider interaction terms of degree=2 in your pipeline. You will insert these steps after the preprocessing steps you've built out so far, but before the classifier steps.\n",
    "\n",
    "Pipelines with interaction terms take a while to train (since you're making n features into n-squared features!), so as long as you set it up right, we'll do the heavy lifting and tell you what your score is!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add interaction features to model using a custom interaction object\n",
    "# SparseInteractions\n",
    "\n",
    "# Instantiate pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                                   ngram_range=(1, 2))),  \n",
    "                    ('dim_red', SelectKBest(chi2, chi_k))\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        # note: SparseInteractions needs to be before preprocessing\n",
    "        # and before scaling\n",
    "        ('int', SparseInteractions(degree=2)),\n",
    "        ('scale', MaxAbsScaler()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Log loss score: 1.2256. Nice improvement from 1.2681! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Computational trick (Hashing) and the winning model\n",
    "Hashing trick\n",
    "- adding new features may cause enormous increase in array size\n",
    "- Hashing is a way of increasing memory efficiency\n",
    "- A hash function takes an input (token) and outputs a hash value (integer)\n",
    "- Hash function limits possible outputs, fixing array size (ie. 250 outputs)\n",
    "\n",
    "When to use the hashing trick\n",
    "- Dimensionality reduction\n",
    "    - want to make array of features as small as possible\n",
    "- Particularly useful on large datasets\n",
    "    - ie. lots of text data!\n",
    "    \n",
    "Implementing hashing trick is easy in sklearn\n",
    "- instead of using CountVectorizer to create a bag-of-words\n",
    "- use HashingVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3.a Implementing the hashing trick in scikit-learn\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "\n",
    "vec = HashingVectorizer(norm=None,\n",
    "                        non_negative=True,\n",
    "                        token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                        ngram_range=(1,2))\n",
    "- :param norm=None and non_negative=True let you replace CountVectorizer with HashingVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Why is hashing a useful trick?\n",
    "In the video, Peter explained that a hash function takes an input, in your case a token, and outputs a hash value. For example, the input may be a string and the hash value may be an integer.\n",
    "\n",
    "We've loaded a familiar python datatype, a dictionary called hash_dict, that makes this mapping concept a bit more explicit. In fact, python dictionaries ARE hash tables!\n",
    "https://stackoverflow.com/questions/114830/is-a-python-dictionary-an-example-of-a-hash-table\n",
    "\n",
    "Print hash_dict in the IPython Shell to get a sense of how strings can be mapped to integers.\n",
    "\n",
    "- In [1]: hash_dict\n",
    "\n",
    "- Out[1]: {'and': 780, 'fluids': 354, 'fuel': 895, 'petro': 354, 'vend': 785}\n",
    "\n",
    "By explicitly stating how many possible outputs the hashing function may have, we limit the size of the objects that need to be processed. With these limits known, computation can be made more efficient and we can get results faster, even on large datasets.\n",
    "\n",
    "Using the above information, answer the following:\n",
    "\n",
    "Why is hashing a useful trick?\n",
    "Some problems are memory-bound and not easily parallelizable, and hashing enforces a fixed length computation instead of using a mutable datatype (like a dictionary).\n",
    "Enforcing a fixed length can speed up calculations drastically, especially on large datasets!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Implementing the hashing trick in scikit-learn\n",
    "In this exercise you will check out the scikit-learn implementation of HashingVectorizer before adding it to your pipeline later.\n",
    "\n",
    "As you saw in the video, HashingVectorizer acts just like CountVectorizer in that it can accept token_pattern and ngram_range parameters. The important difference is that it creates hash values from the text, so that we get all the computational advantages of hashing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import HashingVectorizer\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "\n",
    "# Get text data: text_data\n",
    "text_data = combine_text_columns(X_train)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)' \n",
    "\n",
    "# Instantiate the HashingVectorizer: hashing_vec\n",
    "hashing_vec = HashingVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit and transform the Hashing Vectorizer\n",
    "hashed_text = hashing_vec.fit_transform(text_data)\n",
    "\n",
    "# Create DataFrame and print the head\n",
    "hashed_df = pd.DataFrame(hashed_text.data)\n",
    "print(hashed_df.head())\n",
    "\n",
    "# Resulting values:\n",
    "#           0\n",
    "# 0 -0.160128\n",
    "# 1  0.160128\n",
    "# 2 -0.480384\n",
    "# 3 -0.320256\n",
    "# 4  0.160128\n",
    "\n",
    "# Note: some text is hashed to the same value, but it doesn't\n",
    "# necessarily hurt performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Build the winning model\n",
    "You have arrived! This is where all of your hard work pays off. It's time to build the model that won DrivenData's competition.\n",
    "\n",
    "You've constructed a robust, powerful pipeline capable of processing training and testing data. Now that you understand the data and know all of the tools you need, you can essentially solve the whole problem in a relatively small number of lines of code. Wow!\n",
    "\n",
    "All you need to do is add the HashingVectorizer step to the pipeline to replace the CountVectorizer step.\n",
    "\n",
    "The parameters non_negative=True, norm=None, and binary=False make the HashingVectorizer perform similarly to the default settings on the CountVectorizer so you can just replace one with the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the hashing vectorizer\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "\n",
    "# Instantiate the winning model pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', Imputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', HashingVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                                     non_negative=True, norm=None, binary=False,\n",
    "                                                     ngram_range=(1,2))),\n",
    "                    ('dim_red', SelectKBest(chi2, chi_k))\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('int', SparseInteractions(degree=2)),\n",
    "        ('scale', MaxAbsScaler()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Log loss: 1.2258. Looks like the performance is about the same, \n",
    "# but this is expected since the HashingVectorizer should work \n",
    "# the same as the CountVectorizer. Try this pipeline out on the whole \n",
    "# dataset on your local machine to see its full power!\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Full code on jupyter notebook\n",
    "https://github.com/datacamp/course-resources-ml-with-experts-budgets/blob/master/notebooks/1.0-full-model.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Next steps and the social impact of your work\n",
    "Quickly test ways of improving the model\n",
    "- NLP: stemming, stop-word removal\n",
    "- Model: RandomForest, k-NN, Naïve Bayes\n",
    "- Numeric preprocessing: imputation strategies\n",
    "    - handling NaNs differently than default imputer\n",
    "- Optimization: Grid search over pipeline objects\n",
    "    - you can do this over every object in a pipeline\n",
    "- Experiment with new scikit-learn techniques\n",
    "\n",
    "- Work with full dataset of DrivenData"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}