notebooks/tutorial-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WILLUMP - Tutorial\n",
    "\n",
    "In this notebook, we'll explain how to use  Willump to improve the performance of feature computation in ML inference applications.\n",
    "\n",
    "This version of Willump works specifically on binary classification problems. It improves performance via cascades. It tries to predict each data input using an approximate model trained on a handful of high-value, low cost features. For each data input, it returns the approximate prediction if the model's confidence in it is high, but otherwise computes all remaining features and predicts with the original model. For an in-depth demo explaining how Willump works, please see the [explanation notebook](https://github.com/stanford-futuredata/Willump-Simple/blob/master/notebooks/explanation-notebook.ipynb) or [our paper](http://petereliaskraft.net/res/willump.pdf).\n",
    "\n",
    "We're going to be optimizing a specific application, which predicts whether a content of a text post is toxic. The application was adapted from a entrant in Kaggle's Toxic Comment Classification Challenge.\n",
    "\n",
    "First, let's import dependencies. Be sure that the printed working directory is the root directory of the Willump-Simple package.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/c/Users/Peter/Documents/GitHub/Willump-Simple\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "import time\n",
    "import scipy.sparse\n",
    "if (os.getcwd().endswith(\"notebooks\")):\n",
    "    os.chdir(\"..\")\n",
    "sys.path.insert(0, os.path.abspath(\"tests/benchmark_scripts\"))\n",
    "print(os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Willump to optimize an application with cascades, it must define training, prediction, confidence, and scoring functions.  Each must match an interface.\n",
    "\n",
    "The training function must take in two arguments.  The first is a vector of binary labels, the second is a list of feature matrices (where each feature matrix has one row per label).  It must return a trained model, which can be of any type.\n",
    "\n",
    "The prediction function must take in two arguments.  The first is a model of the type returned by the training function.  The second is a list of feature matrices just as in the training function.  It must return a vector of binary predictions, one for each row in the feature matrices.\n",
    "\n",
    "The arguments of the confidence function are the same as those of the prediction function.  It must return a vector of confidence values, one for each row in the feature matrices, each of which is the probability that the prediction for the row is 1.\n",
    "\n",
    "The scoring function must take in two arguments.  The first is a vector of binary labels, the second is a vector of binary predictions.  It returns a score of the prediction where higher scores are better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def toxic_train(y, X_list):\n",
    "    X = scipy.sparse.hstack([*X_list], format=\"csr\")\n",
    "    model = LogisticRegression(C=0.1, solver='sag')\n",
    "    model = model.fit(X, y)\n",
    "    return model\n",
    "\n",
    "\n",
    "def toxic_predict(model, X_list):\n",
    "    X = scipy.sparse.hstack([*X_list], format=\"csr\")\n",
    "    if X.shape[0] == 0:\n",
    "        return np.zeros(0, dtype=np.int64)\n",
    "    else:\n",
    "        return model.predict(X)\n",
    "\n",
    "\n",
    "def toxic_confidence(model, X_list):\n",
    "    X = scipy.sparse.hstack([*X_list], format=\"csr\")\n",
    "    if X.shape[0] == 0:\n",
    "        return np.zeros(0, dtype=np.float64)\n",
    "    else:\n",
    "        return model.predict_proba(X)[:, 1]\n",
    "\n",
    "\n",
    "def toxic_score(true_y, pred_y):\n",
    "    return roc_auc_score(true_y, pred_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before Willump can optimize an application using cascades, it must train the cascades.  We do this at model training time.\n",
    "We write the model training pipeline in a specific format, like the example below.\n",
    "\n",
    "The pipeline must be written as a Python function.  Each line of the pipeline must be of the form y = f(x<sub>1</sub>, ..., x<sub>n</sub>) where y and x<sub>1</sub>, ..., x<sub>n</sub> are all Python variables and f is a Python function.  The last line of the pipeline must call the train function defined above and return.\n",
    "\n",
    "The interface for Willump is a decorator, as in the below example.  The decorator has five mandatory arguments.  Four are the train, predict, confidence, and scoring functions as described above.  The fifth is the cascades dictionary, used as a callback.  It is initially empty, but as Willump executes it will store in it metadata about the cascades optimization.  After Willump is done training, we save the dictionary and later use it at prediction time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from willump.evaluation.willump_executor import willump_execute\n",
    "\n",
    "cascades_dict = {}\n",
    "\n",
    "def transform_data(data, vectorizer):\n",
    "    return vectorizer.transform(data)\n",
    "\n",
    "@willump_execute(train_function=toxic_train,\n",
    "                 predict_function=toxic_predict,\n",
    "                 confidence_function=toxic_confidence,\n",
    "                 score_function=toxic_score,\n",
    "                 train_cascades_params=cascades_dict)\n",
    "def toxic_train_pipeline(input_x, input_y, word_vect, char_vect):\n",
    "    word_features = transform_data(input_x, word_vect)\n",
    "    char_features = transform_data(input_x, char_vect)\n",
    "    return toxic_train(input_y, [word_features, char_features])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code prepares training and test sets for our example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TfidfVectorizer(analyzer='char', encoding='ascii', lowercase=False,\n",
       "                max_features=50000, ngram_range=(2, 6))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_directory = \"tests/test_resources/toxic_resources/\"\n",
    "df = pd.read_csv(base_directory + 'train.csv').fillna(' ')\n",
    "y = df[\"toxic\"]\n",
    "X = df[\"comment_text\"].values\n",
    "train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "word_vectorizer = TfidfVectorizer(\n",
    "    lowercase=False,\n",
    "    analyzer='word',\n",
    "    stop_words='english',\n",
    "    ngram_range=(1, 1),\n",
    "    encoding=\"ascii\",\n",
    "    decode_error=\"strict\",\n",
    "    max_features=10000)\n",
    "word_vectorizer.fit(train_X)\n",
    "char_vectorizer = TfidfVectorizer(\n",
    "    lowercase=False,\n",
    "    analyzer='char',\n",
    "    ngram_range=(2, 6),\n",
    "    encoding=\"ascii\",\n",
    "    decode_error=\"strict\",\n",
    "    max_features=50000)\n",
    "char_vectorizer.fit(train_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined the training pipeline and prepared the training set, we can train cascades.  We simply run the training pipeline twice, once to make measurements and once to actually train cascades.  As we can see, the trained cascades are stored in the cascades dictionary.  We can use this to make fast predictions with cascades."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature: char_features             Cost: 99.597 Importance:  0.178\n",
      "Feature: word_features             Cost:  3.642 Importance:  0.130\n",
      "Cutoff: 0.100000 Threshold: 0.900000 Expected Cost: 24.173412\n",
      "Cascades dict contents: {'selected_feature_indices': [0], 'cascade_threshold': 0.9, 'full_model': LogisticRegression(C=0.1, solver='sag'), 'approximate_model': LogisticRegression(C=0.1, solver='sag')}\n"
     ]
    }
   ],
   "source": [
    "toxic_train_pipeline(train_X, train_y, word_vectorizer, char_vectorizer)\n",
    "model = toxic_train_pipeline(train_X, train_y, word_vectorizer, char_vectorizer)\n",
    "print(\"Cascades dict contents:\", cascades_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we make predictions without cascades so we have a baseline for performance and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed Time 28.216466s\n",
      "ROC-AUC Score Score: 0.939213\n"
     ]
    }
   ],
   "source": [
    "def toxic_eval_pipeline(input_x, model, word_vect, char_vect):\n",
    "    word_features = transform_data(input_x, word_vect)\n",
    "    char_features = transform_data(input_x, char_vect)\n",
    "    return toxic_predict(model, [word_features, char_features])\n",
    "\n",
    "time_start = time.time()\n",
    "preds = toxic_eval_pipeline(test_X, model, word_vectorizer, char_vectorizer)\n",
    "unopt_time_elapsed = time.time() - time_start\n",
    "print(\"Elapsed Time %fs\" % unopt_time_elapsed)\n",
    "\n",
    "print(\"ROC-AUC Score Score: %f\" % toxic_score(preds, test_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make predictions using cascades!  The prediction pipeline must be identical to the training pipeline except that its last line must make and return predictions instead of training and returning a model.\n",
    "\n",
    "We run the optimized pipeline and see it is much faster than, but just as accurate as, the original pipeline!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed Time 28.216466s\n",
      "ROC-AUC Score Score: 0.939138\n"
     ]
    }
   ],
   "source": [
    "@willump_execute(predict_function=toxic_predict,\n",
    "                 confidence_function=toxic_confidence,\n",
    "                 predict_cascades_params=cascades_dict)\n",
    "def toxic_eval_pipeline_cascades(input_x, model, word_vect, char_vect):\n",
    "    word_features = transform_data(input_x, word_vect)\n",
    "    char_features = transform_data(input_x, char_vect)\n",
    "    return toxic_predict(model, [word_features, char_features])\n",
    "\n",
    "toxic_eval_pipeline_cascades(test_X[:1000], model, word_vectorizer, char_vectorizer)\n",
    "toxic_eval_pipeline_cascades(test_X[:1000], model, word_vectorizer, char_vectorizer)\n",
    "time_start = time.time()\n",
    "preds = toxic_eval_pipeline_cascades(test_X, model, word_vectorizer, char_vectorizer)\n",
    "cascades_time_elapsed = time.time() - time_start\n",
    "print(\"Elapsed Time %fs\" % unopt_time_elapsed)\n",
    "\n",
    "print(\"ROC-AUC Score Score: %f\" % toxic_score(preds, test_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize the difference, we can plot performance.  Cascades are far faster!\n",
    "\n",
    "If you are interested in exactly how cascades work, please see our [other notebook](https://github.com/stanford-futuredata/Willump-Simple/blob/master/notebooks/explanation-notebook.ipynb) or [our paper](http://petereliaskraft.net/res/willump.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZ8AAAD9CAYAAABjqXpxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZxcVZn/8c83gQQRlCSASDQmiOMMW5iZuCDIEh12A4psgghxQJCRVZCAYJCgURhEhh8IPxAcooCAOCwSFglBmTBDCGuUsCUssgUSIYEQsjzzxzltbm6qum93qqtD9ff9etWrqHueuvcprcrT555zz1VEYGZm1kx9ejoBMzPrfVx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vbr6QTeLdZdd90YOnRoT6dhZvaucv/9978aEeuVt7v4VDR06FCmTp3a02mYmb2rSHqm1nafdjMzs6Zz8TEzs6Zz8TEzs6Zz8TEzs6Zz8TEzs6ZrevGRtLWk2yS9ImmepGmSRpdi1pB0lqQXJS2QNEXStjX21UfSGEmzJL0t6SFJe9U57qGSHpO0UNIMSYd312c0M7P2NbX4SNoCuANYHTgU+BJwH3CppCMKoZfm9tOA3YEXgVslbVna5RnAWOB8YBfgXuAaSbuWjnsocBFwHbAzcA1wQemYZmbWJGrmLRUk/QD4NjAwIuYXtk8BiIitJA0HHgRGR8RluX01YDowIyJG5W3rA88B4yPie4V9/R5YLyK2KLz3BeCWiPhaIe7nwCjggxGxqKPcR4wYEb7Ox8yscyTdHxEjytubfdqtH7AIWFDa/nohl1E55uq2xohYDFwF7CSpf968U97fhNK+JgCbSxqWX28FrFcj7gpgELBNVz+MmZl1TbNXOLgcOAI4T9KZwFvA3sDngK/mmE2BmRHxVum900nFZuP835sCC4Ena8QBbALMzHEAj7YTN6lrH8fMGmnoSTf3dApWMmv8bt2y36YWn4h4VNL2wPXAN/PmRcDhEXFVfj0QmFvj7XMK7W3Pf40VzxvWiqPGPstxK5B0GHAYwJAhQ+qFmZlZJzV7wsHHSIP+04EvAJ8Hfgb8TNIBzcylioi4OCJGRMSI9dZbYV08MzPromafdvsBqaeze2GQ//eSBgE/lXQlqYfykRrvbeuhtPVY5gLrSFKp91MrDmAAadZcvTgzM2uSZk842Bx4qMbssv8lDf6vT+oVDZO0ZilmE+Adlo3xTAf6Ax+tEQfwp0IcLBv7qRdnZmZN0uzi8xKwpaR+pe2fAt4m9UJuJF0HtHdbY54uvS9wW0QszJsnknpR5dN1BwKPRsTM/HoK8GqduDnAPSvzgczMrPOafdrtfNIFnjdKuoA05XoUsD/wk4h4B3hA0tXAuZJWJ81YOwIYRqGARMQrks4BxkiaB0wjFaiReZ9tcYsknUq6qPQvpItcRwKjgW/lY5qZWRM1e7bbtXn1ge8AlwBrAE8BR5JWIGhzCHAmMA5YB3gI2DkippV2eQowHzga2ACYAewTETeVjvszSQEcD5wAPAv8W0Rc0NhPaGZmVTT9TqYRcQtwSwcxC4Dj8qO9uCWkAjWuwnEvYvkCZ2ZmPcSrWpuZWdNV6vlIGgz8C/BpYEPgPaRB/BnAZGByRCztriTNzKy1tNvzkbSdpJuAWcDPgV2BwaTiM5y0SOjvgeckjZX0vu5N18zMWkHd4iPpZtLYzJvAPsD6ETEkIv45IraJiE2A9wFbAheQpkY/JWmnJuRtZmbvYu2ddnsc+HpEvFQvIJ9qezg/zpQ0Cnh/Y1M0M7NWU7f4RMSxnd1ZRNywcumYmVlv4NluZmbWdJWKj6SfSLqiTtsVks5qbFpmZtbKqvZ8RgG31Wm7FdizMemYmVlvULX4DCYtSVPL87ndzMyskqrFZy7p9tW1bExaX83MzKySqsXnDuC7kj5Q3Jhfnwzc3ujEzMysdVVdWPRU4D7gibziQduptt1J9+H5bvekZ2ZmrahS8YmIWZI+AXyftMbbINLabtcD34uIZ7ovRTMzazWVb6kQEbOAg7ovFTMz6y06dZGppD6SNssLjr63u5IyM7PWVrn4SDoSeIm0jtudwMfz9t9KOqp70jMzs1ZUdYWDQ4GfAr8lrXCtQvMfgL0an5qZmbWqqj2f44B/j4jDSJMMih4j94LMzMyqqFp8hpGW0anlTWCdxqRjZma9QdXi8yowtE7bx4G/NCQbMzPrFaoWn5uA0yRtVNgWktYFjiWNBZmZmVVStfh8F1gIPEpaaieA84A/A0tIF5+amZlVUqn4RMSrwAjgh8DqwFOkC1TPB7aKiNe7LUMzM2s5nVnhYB5wRn6YmZl1WaXiI6kP0CciFhe27QRsBtwZEQ90U35mZtaCqvZ8riSN+RwEIOlw4ILctkjSbhFxRzfkZ2ZmLajqhINPA78rvD4BuAR4P/Ab4JQG52VmZi2savFZn3wtj6SNSRednp/HgS4DNu+e9MzMrBVVLT5vkO7hA7A98GpEPJxfLwHWaHBeZmbWwqqO+fw3cJKkxcAxLH8KbmPSnU3NzMwqqdrzOZHU87mB1MsZW2jbF5jS2LTMzKyVVb2N9hPAxyQNiojXSs1Hk+7zY2ZmVkndno+kGZJ+JGmrtm01Cg8R8UhEzO6uBM3MrPW0d9rtMmBb4I+SXpR0saRdJPVrUm5mZtai6hafiBgfEVsBg4HTgQ+TbiQ3W9KvJe0v6X1NytPMzFpIhxMOIuKliPhZROwCrAd8gzS9+kJSIbpN0hGSNuzmXM3MrEVUne0GpMVFI+KqiNifVIj2IK1w/V3guW7Iz8zMWlCnik9RRCyKiIkRcUREDAY+08C8zMyshVUqPpI+I2n3wutBkq6U9IiksyX1jYj/qXpQSbtKulvSfElvSJoqaWShfYCkSyS9KulNSXdIWmEJH0lrSDorT4hYIGmKpG1rxPWRNEbSLElvS3pI0l5V8zUzs8aq2vMZD/xz4fVZwK7A48ARwMlVDyjpG8B/AfcDXwT2Bq4B1sztAm4Edga+BexFuoHdJEkfKu3uUuBQ4DRgd+BF4FZJW5biziBdGHs+sAtwL3CNpF2r5m1mZo1TdXmdfwB+BCBpdeDLwDER8XNJx5AmIXR4kzlJQ4FzgRMi4txC062F/x4FbA2MjIhJ+X1TgJmklRaOytuGA18BRkfEZXnbZGA66bbeo/K29YFvA+Mj4ux8jEl5gdTxLL9UkJmZNUHVns9apMVFAT4JvBe4Kb+eBgypuJ/RwFLgZ+3EjAJeaCs8APk23TeSJjgU4xYBVxfiFgNXATtJ6p837wT0AyaUjjMB2FzSsIq5m5lZg1QtPn8Bhuf/3gV4NCJeya8HAG9V3M82wGPAfpKekrRY0pOSjizEbAo8WuO904EhktYqxM2MiPKxp5OKzcaFuIXAkzXiADapmLuZmTVI1eJzJfADSdcCx7F8L+KfgCcq7mdD4GOkMaPxwI7A7cD5ko7OMQOBuTXeOyc/D6gYN7Dw/NeIiA7iViDpsDwZYurs2V5ByMysUaqO+YwF3ibd0XQ8cE6hbThpwkAVfYC1gYMj4jd52515LGiMpPMq7qcpIuJi4GKAESNGlIuXmZl1UdVVrZcAZ9Zp27MTx3uN1PO5vbT9NtLstg+SejMDWFFbD2Vu4fkj7cTNKcStI0ml3k85zszMmqTqdT4/krSjpDVX8njTO2hfmmM2rdG2CfBsRMwv7GtYjZw2Ad5h2RjPdKA/8NEacQB/qpC3mZk1UNUxnwOAicBcSfdIOkPSyMKMsqquz887lbbvDDwfES+Rblg3WNJ2bY15AdMv5LY2N5Ku/9m7ELca6eZ2t0XEwrx5ImlW3AGlYx5Imjgxs5OfwczMVlLV024fkvR3wEhgB+Aw4BRgoaR7gTsjosPrfEjX1EwCLpK0LvA0qXjsCBySY24g3Rl1gqQTSKfNxgACflzI6QFJVwPn5muPZpIueB1GodBExCuSziGNKc0jTQ3fN3+WUVU+v5mZNVbVCQdExOOkFQ1+BiBpa9JEhM+R7vvTYfGJiJC0J/BD0m0aBpCmXh8QEb/KMUvzUj5nAxeQbts9BdghIsqLlx5CGosaB6wDPATsHBHTSnGnAPNJd13dAJgB7BMRN2FmZk1XufhIeg/pOp223s8/ka7vuQm4s+p+IuIN4Mj8qBczh3RB6ugO9rWANPX7uA7ilpAK1LiqeZqZWfepVHwk3U1a2WARcA9p7OZbwP0RsbT70jMzs1ZUteezDbAA+E/SOmyT85I3ZmZmnVa1+GzBstNtlwNrS3qQdLptEnB3jWVuzMzMaqo62+1R0npr5+VbHvwjqRjtTloxehFpYoCZmVmHKk84gL/dTuEzpB7QSOBTpCnQtdZYMzMzq6nqhIOTScVmK+A9pGVyJgPHApMi4s/dlqGZmbWcqj2fE4C7SdfL3BkRD3dfSmZm1uqqFp9BnlJtZmaNUnXCwVIASQNJp94GklaDnpIvCDUzM6usMyscjAOOJ90lVHnzQklnR8Sp3ZGcmZm1pqoTDo4BTgYuJd3F9CXSGmkHAidLmh0Rq9SN4MzMbNVVtedzOPDTiDi2sG0GMFnSfOCbgIuPmZlVUvV+PkOBm+u03ZzbzczMKqlafF4DNqvTtmluNzMzq6Rq8bkeOEPSV/PdQpG0mqT9ge8D13VXgmZm1nqqFp8xwIPAL4AFkl4mrXL9S9IN3E7unvTMzKwVVb3OZ56kbYHdgM+y7DqfycAtERHdl6KZmbWaDouPpH7AvcBJ+bbTvvW0mZmtlA5Pu0XEO8AwYHH3p2NmZr1B1TGf24EduzMRMzPrPapeZPofwIQ80+23wIvAcuM8EfF0g3MzM7MWVbX4TM7Px5Hu4VNL35VPx8zMeoOqxeeQbs3CzMx6lapTrX/R3YmYmVnvUXXCgZmZWcPULT6SzpO0QWd2JulLkvZb+bTMzKyVtdfzGQo8LelqSaPyXUyXI6mPpC0lnSppBnAhaeUDMzOzuuqO+UTEqLykzrdJC4f2kfQCMBtYCAwAPgysQZp6/f+Bn0TEG92etZmZvau1O+EgIu4G7pb0QdJFpp8CNiQVnKdJC4veDfwhIpZ2c65mZtYiqs52e5G0orVnvZmZ2UrzbDczM2s6Fx8zM2s6Fx8zM2s6Fx8zM2s6Fx8zM2u6SsVH0kGSBtVpGyjpoMamZWZmraxqz+cy4KN12obldjMzs0qqFh+10/ZefIttMzPrhPYWFt1S0mhJo/OmL7S9LjyOBM4EnuhqApImSgpJ40rbB0i6RNKrkt6UdIekzWu8fw1JZ0l6UdICSVPyskDluD6SxkiaJeltSQ9J2qureZuZWde1t8LBHsD38n8HcEqduNeAr3fl4JL2B4bX2C7gRtLipt8C5gJjgEmStoyI5wvhlwK7ASeQlvw5ErhV0lYR8WAh7gzSOnWnAPcD+wHXSNo9In7XlfzNzKxr2is+5wKXk065PQ18CXigFLMQeDkiorMHljQA+Anptty/KjWPArYGRkbEpBw/BZgJnAgclbcNB74CjI6Iy/K2ycB04Pt5P0han1R4xkfE2fkYkyRtDIwHXHzMzJqo7mm3iHg9Ip6JiFmkSQW/y6+Lj5e6UniyHwGPRsSVNdpGAS+0FZ62fEi9oT1KcYuAqwtxi4GrgJ0k9c+bdwL6ARNKx5kAbC5pWBc/g5mZdUGlCQe50LzTqINK2gY4iHSKrJZNgUdrbJ8ODJG0ViFuZkS8VSOuH7BxIW4h8GSNOIBNqmdvZmYrq+p1PkslLWnvUfWAkvoBFwFnR8SMOmEDSeM8ZW03qhtQMW5g4fmvNXpp5bhyrodJmipp6uzZs+ukamZmnVXplgqk8ZPyP9yDSPf46U8aG6rqROA9pFlyq7SIuBi4GGDEiBFdPb1oZmYlVe/nM7bWdkl9SeMwr1fZj6QhpNlm/wr0L4zJkF+vA8wj9WYG1NhFWw9lbuH5I+3EzSnErSNJpd5POc7MzJpgpdZ2i4glwAXAMRXfshHpLqgTSAWh7QFpNtpcYHPSWMymNd6/CfBsRMzPr6cDwyStWSPuHZaN8Uwn9dDKqzS0jfX8qWL+ZmbWAI1YWLQ/dcZMangQ2KHGA1JB2oFUMG4ABkvaru2Nkt4HfCG3tbkRWB3YuxC3GrAvcFtELMybJ5JmxR1QyudA0oy7mRXzNzOzBqh02i2fLivrB2xGuk5mapX9RMRfgbtq7B/gmYi4K7++AZgCTJB0AssuMhXw48L+HpB0NXCupNVJ1wEdQZoafkAh7hVJ5wBjJM0DppEK1EjytUBmZtY8VScczGLFCQeQisFT1J8y3SURsVTS7sDZpNN6a5CK0Q4R8Vwp/BDS5IVxwDrAQ8DOETGtFHcKMB84GtgAmAHsExE3NTJ3MzPrWNXiM5oVi8/bwDPAfXnsp8siYoWFSyNiTj7u6BXfsVzcAuC4/GgvbgmpQI1rL87MzLpf1dlul3dzHmZm1otU7fkAfxv03wwYDPwFeCQi5nVHYmZm1roqFx9JpwHHA2ux7P4+8ySdFRE+lWVmZpVVne12OnAqcAlp0c6XgQ8A+wOnS1qt3oWoZmZmZVV7PocC/x4RJxS2TQfulPQ6cBgwtsG5mZlZi6p6ken7gVvrtE3M7WZmZpVULT7/A3yiTtsncruZmVklVU+7HQVcL2kxcA3Lxnz2IV2Hs4ekvxWyiFja6ETNzKx1VC0+D+fn8flRJOCRwuvoxH7NzKwXWpn7+ZiZmXXJSt3Px8zMrCsacUsFMzOzTunMCgcbkSYYDCGtMl0UEfH1RiZmZmatq+oKB3sCvyb1lF4BFpZCPB5kZmaVVe35nEG6CdwBETG7+9IxM7PeoGrx2Qg43oXHzMwaoeqEg8eAQd2ZiJmZ9R5Vi8+JwMl50oGZmdlKqXvaTdLdpU2DgD9LegKYU2qLiNiu0cmZmVlram/MZynLz2Kb0c25mJlZL1G3+ETE9k3Mw8zMehGvcGBmZk1X9SLTbdtpXgq8DjwWEYsakpWZmbW0qtf53EXHqxi8Jem8iDhl5VIyM7NWV7X47AH8B/AQcC3L30xuC+BU4JPAiZLmRsTZ3ZCrmZm1iKrFZ09gYkQcXtp+haSLgB0i4hBJS4GvAy4+ZmZWV9UJB18ErqvTdi2pZwQwERi2skmZmVlrq1p8+gIfrdO2cW6HtNp1ecVrMzOz5VQtPr8DfiBpL0l9AST1lfRl4Ezg5hy3KfBU49M0M7NWUnXM51vA9cA1wGJJc4EB+f335HZIU65/0OgkzcystVQqPhHxKvBZSTsCnwI+CLwI3BsRtxfiftEtWZqZWUupfBttgIi4Dbitm3IxM7NewsvrmJlZ01VdXqe8wvUKIqJve+1mZmZtqp52+z4rFp9BwI5Af+DyBuZkZmYtruqEg7G1tudp1zeSZrmZmZlVslJjPhGxBLgAOKYx6ZiZWW/QiAkH/YGBDdiPmZn1EpWKj6QhNR4bS9oTGA9MrbifL0u6TtIzkhZImiHph5LWLsUNkHSJpFclvSnpDkmb19jfGpLOkvRi3t+UWvcektRH0hhJsyS9LekhSXtVydnMzBqvas9nFjCz9JgB/Ca3H1lxP98GlgAnAzsDFwJHALdL6gMgSaRxpJ1JKyfsBawOTJL0odL+LgUOBU4Ddidd+HqrpC1LcWcAY4HzgV2Ae4FrJO1aMW8zM2ugqrPdRrPibLe3gWeA+/LYTxVfiIjZhdeTJc0BfgFsD9wJjAK2BkZGxCQASVNIBe9E4Ki8bTjwFWB0RFyWt00GppNm543K29YnFb3xhfsMTZK0ManX9ruKuZuZWYNUne12eSMOVio8be7Lz4Pz8yjghbbCk9/3uqQbSbduOKoQtwi4uhC3WNJVwEmS+kfEQmAnoB8woXTcCcDPJQ2LiJkr+dHMzKwTOjXhQNJASbtJ+mp+bsREg+3y85/z86bAozXipgNDJK1ViJsZEW/ViOtHutVDW9xC4MkacQCbdDFvMzProspru0kaBxxP+oddefNCSWdHxKldObikwaRTZHdERNukhYGkMaayOfl5ADA/x81tJ25g4fmvEVE+bViOq5XfYcBhAEOGDKn7OczMrHOqznY7hjRJYAIwEvgHYIf8+mRJR7Xz9nr7XAv4L2AxcEhn398MEXFxRIyIiBHrrbdeT6djZtYyqvZ8Dgd+GhHHFrbNIE0YmA98Eziv6kElvYc0o20jYLuIeL7Q3HavoLKBhfa254+0EzenELeOJJV6P+U4MzNrkqpjPkNZdrfSsptzeyWSVgeuBUYAu0bEI6WQ6aRxmrJNgGcjYn4hbpikNWvEvcOyMZ7ppAthy7cBbxvr+VPV3M3MrDGqFp/XgM3qtG2a2zuUr+X5JenU3Z4RcW+NsBuAwZK2K7zvfcAXclubG0nX/+xdiFsN2Be4Lc90A5hImhV3QOk4BwKPeqabmVnzVT3tdj1whqTXgCvzlObVSP/wf590nU4V/y+/50zgTUmfLrQ9n0+/3QBMASZIOoF02mwMaZLDj9uCI+IBSVcD5+be1EzSBavDKBSaiHhF0jnAGEnzgGmkAjWSfC2QmZk1V9XiMwYYTioyP88Xhg4E+gJ/JE1GqGKX/HxKfhSdDoyNiKWSdgfOJi1augapGO0QEc+V3nMIqZCNA9YBHgJ2johppbhTSDPkjgY2II1X7RMRN1XM28zMGqjqRabz8pppuwGfJRWeOcBk4JYa05jr7Wdoxbg5pFUVRncQtwA4Lj/ai1tCKlDjqhzfzMy6V4fFR1I/0lpoJ+WegnsLZma2UjqccBAR75DGURZ3fzpmZtYbVJ3tdjvpltlmZmYrreqEg/8gzT5bDfgt6dYFy43zRMTTDc7NzMxaVNXiMzk/HwccWyem78qnY2ZmvUHV4rNKrr1mZmbvTlWnWle9iNTMzKxDnbqfj5mZWSN05n4+2wH7A0NIqw4URUR8rpGJmZlZ66pUfCR9A7iQtKrB46Q7gy4X0uC8zMyshVXt+RwP/AoYnS86NTMz67KqYz6DgctceMzMrBGqFp/7SXcdNTMzW2lVi89RwDF5ZWszM7OVUnfMR9JzLL+EzvuBSZLeIt3grSgi4iPdkJ+ZmbWg9iYc/J7S+m3WNUNPurmnU7CSWeN36+kUzHq1usUnIg5uYh5mZtaL1B3zkfS0pOHNTMbMzHqH9iYcDAX6NykPMzPrRby2m5mZNV1HxccTDszMrOE6Wl7ndEmvVthPRMTXGpGQmZm1vo6Kz5asuIhoLe4hmZlZZR0Vnz0j4n+bkomZmfUannBgZmZN5+JjZmZN5+JjZmZN197yOi5MZmbWLVxgzMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6Vx8zMys6XpV8ZH0YUnXSnpd0huSfiNpSE/nZWbW2/Sa4iNpTeBO4O+BrwFfBT4GTJL03p7Mzcyst+noTqat5FBgI+DjEfEkgKSHgSeAbwDn9GBuZma9Sq/p+QCjgHvbCg9ARMwE7gH26LGszMx6od5UfDYFHq2xfTqwSZNzMTPr1XrTabeBwNwa2+cAA2q9QdJhwGH55XxJM7opt3eTdYFXezqJlaUf9XQG1gv4t5J8pNbG3lR8Oi0iLgYu7uk8ViWSpkbEiJ7Ow2xV599K+3rTabe51O7h1OsRmZlZN+lNxWc6adynbBPgT03OxcysV+tNxecG4NOSNmrbIGkosHVus2p8GtKsGv9W2qGI6OkcmiJfSPoQsAD4LhDAGcDawBYRMb8H0zMz61V6Tc8nIt4ERgKPA1cAvwRmAiNdeMzMmqvX9HzMzGzV0Wt6PqsCSZdLer5O2/aSQtLnm51XKY+DJY2usz3yOFkjjtP2ebdvxP4qHvMuSXc163jWfJK2kvRrSS9IekfSa5Jul/Q1SX17Or/OkDRL0uU9nUd3cfGxsoOBFYoPcDOwFfBig44zLe9vWoP2Z72cpGNIy2UNBL4DfJ70XX4cuBDYveeyszJfZGqVRMRsYHYD9/cGcG+j9me9m6RtSYsDnx8RR5Wa/0vSOYBXr1+FuOezispd7gmS9pP0Z0lvSpoqaZsasQdKekjS25JelXSFpA/W2d+hkp7MsdMk7VCIuQvYDtg6nxKLttNUtU67Ffb5VUkzJC2Q9AdJH5P0XkkX5dMeL0v6d0mrFd673Gk3SWMLxyw/Di68b5ikX0qaLWmhpAclfbHG/yb7SXosx0yvFWMt5TukpbJOrNUYEU9FxMOS1svfy8clvSXpOUm/kjS4GC/p7yRdL+mV/Ft5VtI1pe/wepIuyPtYmJ+vkNQ/t2+cX8/Mv42nJV0oaYWL3SUdnX9Pb+ff+WdrfY4q3/8qua8KVqlkbAWfBT4OnAq8TZoafpOkoRHxV/jb+nMXAVcDY4ANgR8An5L0T6WZfNsD/wycAiwk/WBvkTQ8ImYA3wQmAH1Jt5kAeKODHLcFPpr31Q84F7gOeBp4Etgvx3wXeAq4oM5+LgEmlrZ9G9iTdNoESR8G/gd4BTiW1BPbF7hO0p4RcUOO+zzwK9KpwuOB9YCfAqsDXp+vxeSxnB2A30bE2x2EDyT9lsaQvj8bkr4j90j6+8L7byatfHIEaX22wcCu5D/YcwH577y/ccDDwPqkFfL7kX5fGwLPAcfkfW0EnAz8jnTKuS3/r5N+N5eTfscbA1eSLgMpfs5K3/+Ocl9lRIQfTXqQvlzP12nbnnTt0efz61nkJYEKMSNyzFfy677Ay8Ck0r62yXFHFbbNAt4BPlzYtjbpr8UrCtvuAv5YI7+D8z6HlvY5B3h/YdtROe6S0vunFfMsfN7t6/zvsTewtPQZLiX94AaVYm8HHiy8voe0akWfwrZP5+Pd1dPfAz8a+wA+kP+//WEX3tsX+HB+/xfztnXz61HtvO/7wBLgHztxrNUKv81/zNv6kArUxFLsvjnu8sK2Dr//VXJfVR6rViW0sikRUVx37pH83Hbr74+T/tr6ZfFNEfFH4BnSKbSieyPiuULcPJZNJFiZHF8vvH4sP99ainuM9CPvkKQRwC+ACyLivELTzqS/Gl+XtFrbIx9ruKT35b+CPwFcGxFL294YEfeSiqX1cpKOyKep5wOLgWdz08fz82uknvv4fJr6Y8TJSEMAAAQqSURBVDV2syNwX0Q80M5x+kk6OZ/+XQAsAv5QOtaH8uPXpbdfl3Mr6vD7XzH3VYKLT3MtJv2lVUvfQkybOcWAiFiY/3ON/DwwP9eagfZSob3NyzXiXiZ1y7uqvCjrO+1sX4MOSPoQabmju4CjS83rAweRfsTFx1m5fRDpL7/Vqf9ZrfW8Rlq5pObS/UWSvkU69XsH8CXgk6ReMeTvZ6QuxL8AU4EfAo/n8ZojCrsaBNS8bKLgh8BY0qns3fKxvlQ8FtA2NrvcdzMiFufPVdTh979i7qsEj/k01yvAupL6RcQ7pbYN83Nn/oFsK04b1GjbALi/tO0DNeI+APylE8fsNkpLIN1IOk+9b0QsKYW8RvrLsd4dRl4gFe9F1P+szzQmW1tVRMRipYkx/yKpf+GPtFr2A34fEce3bZA0rMY+nwYOkiRgOPBvwAWSZkXELSwbS2nPfsB/RsS4wrHWKsW0/eG43Pc192gGlWKrfP+r5L5KcM+nuSaRCv6oGm17kb6InRkQn0EqVvsVN0r6DOmvwLtK8Z/Og5ZtcWuT/iKbUohZCLynEzk0RP6hTCD9Jbh7PiVYNhHYApgeEVNrPBbmgnUf8GVJfQr7/xQwtPs/ifWQ8aR/rH9cqzHPEtsCWJP0x0nRIfV2GsmDwHF502b5+Tbgk5KGt5NTlWM9Txrz2ae0fS9W7Bx0+P2vmPsqwT2f5rqDNDh4uaS/J81cWZtUPPYADimOU3QkIpZIOg24SNIE0j/eg4EzgSeAn5fe8jJwm6SxLJvt9l7SLLo2fwK+KWlf0uy0eZFmwnW375Bmth0NbChpw0LbU5GuMzoN+F/gbknnk8ZwBpB+VBtFRNvFsd8j/ePwW0kXkWa7nU46FWktKCLulnQccI6kTUiTe54lfT8+B/wr8BXSP+DfkXQy6bs0EvhycV+5SP2UNPPsSdIp8YNJveo7c9hP8v7ukDSONB67Lul3fHj+42ki8DVJj+T9fAn4TCnvpZJOBy6RdBlwFWm220msONO0w+9/xdxXDT0946G3PUi9inGk6cMLgXmkrvQepbhZwIQa7w9gbGnbgaQVuxeSuuZXAB+stT/Sj/CpHPsAaWHVYtwGpEHNeRRmh1F/ttuE0vu3pzBrr7D9cgoz/SjNdsvtUedxcOF9HyJNy/4LaRzpRVJBP7B0vP1JPcOFpHs5fZHUE7yrp78DfnTfg/SP+zX5e7GIdGr6tvwb6ZN/fxeSZo3NA24ChhV/V6SxlV/k3+hbeR+TgZ1Kx1qfdNuEF/N38bn8vv65fV1SMZmbH78kTYZZ7judY48mnRJ+mzRes03+fV1eimv3+18191Xh4YVFewlJs0hTqA/s6VzMzDzmY2ZmTefiY2ZmTefTbmZm1nTu+ZiZWdO5+JiZWdO5+JiZWdO5+JiZWdO5+JiZWdP9HzvW8l7YkmX8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 16\n",
    "plt.rcParams['xtick.labelsize'] = 16\n",
    "plt.rcParams['ytick.labelsize'] = 16\n",
    "fig, axes = plt.subplots(figsize=(6, 4), ncols=1, nrows=1,\n",
    "                         sharex=False, sharey=False)\n",
    "x = np.arange(2)\n",
    "width = 0.5  # the width of the bars\n",
    "num_rows = len(test_X)\n",
    "\n",
    "bars = axes.bar(x, [num_rows / unopt_time_elapsed, num_rows / cascades_time_elapsed], width)\n",
    "axes.set(xlabel='', ylabel='Throughput (rows/sec)')\n",
    "axes.set_xticks(x)\n",
    "axes.set_xticklabels([\"Unoptimized\", \"Cascades\"])\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}