sagemaker-deployment/Mini-Projects/IMDB Sentiment Analysis - XGBoost (Updating a Model).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Analysis\n",
    "\n",
    "## Updating a Model in SageMaker\n",
    "\n",
    "_Deep Learning Nanodegree Program | Deployment_\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook we will consider a situation in which a model that we constructed is no longer working as we intended. In particular, we will look at the XGBoost sentiment analysis model that we constructed earlier. In this case, however, we have some new data that our model doesn't seem to perform very well on. As a result, we will re-train our model and update an existing endpoint so that it uses our new model.\n",
    "\n",
    "This notebook starts by re-creating the XGBoost sentiment analysis model that was created in earlier notebooks. This means that you will have already seen the cells up to the end of Step 4. The new content in this notebook begins at Step 5.\n",
    "\n",
    "## Instructions\n",
    "\n",
    "Some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this notebook. You will not need to modify the included code beyond what is requested. Sections that begin with '**TODO**' in the header indicate that you need to complete or implement some portion within them. Instructions will be provided for each section and the specifics of the implementation are marked in the code block with a `# TODO: ...` comment. Please be sure to read the instructions carefully!\n",
    "\n",
    "In addition to implementing code, there will be questions for you to answer which relate to the task and your implementation. Each section where you will answer a question is preceded by a '**Question:**' header. Carefully read each question and provide your answer below the '**Answer:**' header by editing the Markdown cell.\n",
    "\n",
    "> **Note**: Code and Markdown cells can be executed using the **Shift+Enter** keyboard shortcut. In addition, a cell can be edited by typically clicking it (double-click for Markdown cells) or by pressing **Enter** while it is highlighted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Downloading the data\n",
    "\n",
    "The dataset we are going to use is very popular among researchers in Natural Language Processing, usually referred to as the [IMDb dataset](http://ai.stanford.edu/~amaas/data/sentiment/). It consists of movie reviews from the website [imdb.com](http://www.imdb.com/), each labeled as either '**pos**itive', if the reviewer enjoyed the film, or '**neg**ative' otherwise.\n",
    "\n",
    "> Maas, Andrew L., et al. [Learning Word Vectors for Sentiment Analysis](http://ai.stanford.edu/~amaas/data/sentiment/). In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 2011.\n",
    "\n",
    "We begin by using some Jupyter Notebook magic to download and extract the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘../data’: File exists\n",
      "--2019-01-16 10:28:00--  http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
      "Resolving ai.stanford.edu (ai.stanford.edu)... 171.64.68.10\n",
      "Connecting to ai.stanford.edu (ai.stanford.edu)|171.64.68.10|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 84125825 (80M) [application/x-gzip]\n",
      "Saving to: ‘../data/aclImdb_v1.tar.gz’\n",
      "\n",
      "../data/aclImdb_v1. 100%[===================>]  80.23M  10.2MB/s    in 17s     \n",
      "\n",
      "2019-01-16 10:28:18 (4.71 MB/s) - ‘../data/aclImdb_v1.tar.gz’ saved [84125825/84125825]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%mkdir ../data\n",
    "!wget -O ../data/aclImdb_v1.tar.gz http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
    "!tar -zxf ../data/aclImdb_v1.tar.gz -C ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Preparing the data\n",
    "\n",
    "The data we have downloaded is split into various files, each of which contains a single review. It will be much easier going forward if we combine these individual files into two large files, one for training and one for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "\n",
    "def read_imdb_data(data_dir='../data/aclImdb'):\n",
    "    data = {}\n",
    "    labels = {}\n",
    "    \n",
    "    for data_type in ['train', 'test']:\n",
    "        data[data_type] = {}\n",
    "        labels[data_type] = {}\n",
    "        \n",
    "        for sentiment in ['pos', 'neg']:\n",
    "            data[data_type][sentiment] = []\n",
    "            labels[data_type][sentiment] = []\n",
    "            \n",
    "            path = os.path.join(data_dir, data_type, sentiment, '*.txt')\n",
    "            files = glob.glob(path)\n",
    "            \n",
    "            for f in files:\n",
    "                with open(f) as review:\n",
    "                    data[data_type][sentiment].append(review.read())\n",
    "                    # Here we represent a positive review by '1' and a negative review by '0'\n",
    "                    labels[data_type][sentiment].append(1 if sentiment == 'pos' else 0)\n",
    "                    \n",
    "            assert len(data[data_type][sentiment]) == len(labels[data_type][sentiment]), \\\n",
    "                    \"{}/{} data size does not match labels size\".format(data_type, sentiment)\n",
    "                \n",
    "    return data, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDB reviews: train = 12500 pos / 12500 neg, test = 12500 pos / 12500 neg\n"
     ]
    }
   ],
   "source": [
    "data, labels = read_imdb_data()\n",
    "print(\"IMDB reviews: train = {} pos / {} neg, test = {} pos / {} neg\".format(\n",
    "            len(data['train']['pos']), len(data['train']['neg']),\n",
    "            len(data['test']['pos']), len(data['test']['neg'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "def prepare_imdb_data(data, labels):\n",
    "    \"\"\"Prepare training and test sets from IMDb movie reviews.\"\"\"\n",
    "    \n",
    "    #Combine positive and negative reviews and labels\n",
    "    data_train = data['train']['pos'] + data['train']['neg']\n",
    "    data_test = data['test']['pos'] + data['test']['neg']\n",
    "    labels_train = labels['train']['pos'] + labels['train']['neg']\n",
    "    labels_test = labels['test']['pos'] + labels['test']['neg']\n",
    "    \n",
    "    #Shuffle reviews and corresponding labels within training and test sets\n",
    "    data_train, labels_train = shuffle(data_train, labels_train)\n",
    "    data_test, labels_test = shuffle(data_test, labels_test)\n",
    "    \n",
    "    # Return a unified training data, test data, training labels, test labets\n",
    "    return data_train, data_test, labels_train, labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IMDb reviews (combined): train = 25000, test = 25000\n"
     ]
    }
   ],
   "source": [
    "train_X, test_X, train_y, test_y = prepare_imdb_data(data, labels)\n",
    "print(\"IMDb reviews (combined): train = {}, test = {}\".format(len(train_X), len(test_X)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Poorly written conspiracy drama/mystery about the possibility that AIDS was introduced to the public by the government. Wlaschiha plays a gay researcher looking for answers--that within this foggy plot would be hard for anyone to find. Despite the cinematography itself being commendable, the camera hungers for characters of true depth instead of the shallow, amateur acting it unfortunately has to convey. Grade: D+'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_X[100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Processing the data\n",
    "\n",
    "Now that we have our training and testing datasets merged and ready to use, we need to start processing the raw data into something that will be useable by our machine learning algorithm. To begin with, we remove any html formatting that may appear in the reviews and perform some standard natural language processing in order to homogenize the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/ec2-user/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download(\"stopwords\")\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem.porter import *\n",
    "stemmer = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "def review_to_words(review):\n",
    "    text = BeautifulSoup(review, \"html.parser\").get_text() # Remove HTML tags\n",
    "    text = re.sub(r\"[^a-zA-Z0-9]\", \" \", text.lower()) # Convert to lower case\n",
    "    words = text.split() # Split string into words\n",
    "    words = [w for w in words if w not in stopwords.words(\"english\")] # Remove stopwords\n",
    "    words = [PorterStemmer().stem(w) for w in words] # stem\n",
    "    \n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['poorli',\n",
       " 'written',\n",
       " 'conspiraci',\n",
       " 'drama',\n",
       " 'mysteri',\n",
       " 'possibl',\n",
       " 'aid',\n",
       " 'introduc',\n",
       " 'public',\n",
       " 'govern',\n",
       " 'wlaschiha',\n",
       " 'play',\n",
       " 'gay',\n",
       " 'research',\n",
       " 'look',\n",
       " 'answer',\n",
       " 'within',\n",
       " 'foggi',\n",
       " 'plot',\n",
       " 'would',\n",
       " 'hard',\n",
       " 'anyon',\n",
       " 'find',\n",
       " 'despit',\n",
       " 'cinematographi',\n",
       " 'commend',\n",
       " 'camera',\n",
       " 'hunger',\n",
       " 'charact',\n",
       " 'true',\n",
       " 'depth',\n",
       " 'instead',\n",
       " 'shallow',\n",
       " 'amateur',\n",
       " 'act',\n",
       " 'unfortun',\n",
       " 'convey',\n",
       " 'grade']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_to_words(train_X[100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "cache_dir = os.path.join(\"../cache\", \"sentiment_analysis\")  # where to store cache files\n",
    "os.makedirs(cache_dir, exist_ok=True)  # ensure cache directory exists\n",
    "\n",
    "def preprocess_data(data_train, data_test, labels_train, labels_test,\n",
    "                    cache_dir=cache_dir, cache_file=\"preprocessed_data.pkl\"):\n",
    "    \"\"\"Convert each review to words; read from cache if available.\"\"\"\n",
    "\n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = pickle.load(f)\n",
    "            print(\"Read preprocessed data from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # Preprocess training and test data to obtain words for each review\n",
    "        #words_train = list(map(review_to_words, data_train))\n",
    "        #words_test = list(map(review_to_words, data_test))\n",
    "        words_train = [review_to_words(review) for review in data_train]\n",
    "        words_test = [review_to_words(review) for review in data_test]\n",
    "        \n",
    "        # Write to cache file for future runs\n",
    "        if cache_file is not None:\n",
    "            cache_data = dict(words_train=words_train, words_test=words_test,\n",
    "                              labels_train=labels_train, labels_test=labels_test)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                pickle.dump(cache_data, f)\n",
    "            print(\"Wrote preprocessed data to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        words_train, words_test, labels_train, labels_test = (cache_data['words_train'],\n",
    "                cache_data['words_test'], cache_data['labels_train'], cache_data['labels_test'])\n",
    "    \n",
    "    return words_train, words_test, labels_train, labels_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote preprocessed data to cache file: preprocessed_data.pkl\n"
     ]
    }
   ],
   "source": [
    "# Preprocess data\n",
    "train_X, test_X, train_y, test_y = preprocess_data(train_X, test_X, train_y, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract Bag-of-Words features\n",
    "\n",
    "For the model we will be implementing, rather than using the reviews directly, we are going to transform each review into a Bag-of-Words feature representation. Keep in mind that 'in the wild' we will only have access to the training set so our transformer can only use the training set to construct a representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.externals import joblib\n",
    "# joblib is an enhanced version of pickle that is more efficient for storing NumPy arrays\n",
    "\n",
    "def extract_BoW_features(words_train, words_test, vocabulary_size=5000,\n",
    "                         cache_dir=cache_dir, cache_file=\"bow_features.pkl\"):\n",
    "    \"\"\"Extract Bag-of-Words for a given set of documents, already preprocessed into words.\"\"\"\n",
    "    \n",
    "    # If cache_file is not None, try to read from it first\n",
    "    cache_data = None\n",
    "    if cache_file is not None:\n",
    "        try:\n",
    "            with open(os.path.join(cache_dir, cache_file), \"rb\") as f:\n",
    "                cache_data = joblib.load(f)\n",
    "            print(\"Read features from cache file:\", cache_file)\n",
    "        except:\n",
    "            pass  # unable to read from cache, but that's okay\n",
    "    \n",
    "    # If cache is missing, then do the heavy lifting\n",
    "    if cache_data is None:\n",
    "        # Fit a vectorizer to training documents and use it to transform them\n",
    "        # NOTE: Training documents have already been preprocessed and tokenized into words;\n",
    "        #       pass in dummy functions to skip those steps, e.g. preprocessor=lambda x: x\n",
    "        vectorizer = CountVectorizer(max_features=vocabulary_size,\n",
    "                preprocessor=lambda x: x, tokenizer=lambda x: x)  # already preprocessed\n",
    "        features_train = vectorizer.fit_transform(words_train).toarray()\n",
    "\n",
    "        # Apply the same vectorizer to transform the test documents (ignore unknown words)\n",
    "        features_test = vectorizer.transform(words_test).toarray()\n",
    "        \n",
    "        # NOTE: Remember to convert the features using .toarray() for a compact representation\n",
    "        \n",
    "        # Write to cache file for future runs (store vocabulary as well)\n",
    "        if cache_file is not None:\n",
    "            vocabulary = vectorizer.vocabulary_\n",
    "            cache_data = dict(features_train=features_train, features_test=features_test,\n",
    "                             vocabulary=vocabulary)\n",
    "            with open(os.path.join(cache_dir, cache_file), \"wb\") as f:\n",
    "                joblib.dump(cache_data, f)\n",
    "            print(\"Wrote features to cache file:\", cache_file)\n",
    "    else:\n",
    "        # Unpack data loaded from cache file\n",
    "        features_train, features_test, vocabulary = (cache_data['features_train'],\n",
    "                cache_data['features_test'], cache_data['vocabulary'])\n",
    "    \n",
    "    # Return both the extracted features as well as the vocabulary\n",
    "    return features_train, features_test, vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote features to cache file: bow_features.pkl\n"
     ]
    }
   ],
   "source": [
    "# Extract Bag of Words features for both training and test datasets\n",
    "train_X, test_X, vocabulary = extract_BoW_features(train_X, test_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_X[100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Classification using XGBoost\n",
    "\n",
    "Now that we have created the feature representation of our training (and testing) data, it is time to start setting up and using the XGBoost classifier provided by SageMaker.\n",
    "\n",
    "### Writing the dataset\n",
    "\n",
    "The XGBoost classifier that we will be using requires the dataset to be written to a file and stored using Amazon S3. To do this, we will start by splitting the training dataset into two parts, the data we will train the model with and a validation set. Then, we will write those datasets to a file and upload the files to S3. In addition, we will write the test set input to a file and upload the file to S3. This is so that we can use SageMakers Batch Transform functionality to test our model once we've fit it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Earlier we shuffled the training dataset so to make things simple we can just assign\n",
    "# the first 10 000 reviews to the validation set and use the remaining reviews for training.\n",
    "val_X = pd.DataFrame(train_X[:10000])\n",
    "train_X = pd.DataFrame(train_X[10000:])\n",
    "\n",
    "val_y = pd.DataFrame(train_y[:10000])\n",
    "train_y = pd.DataFrame(train_y[10000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The documentation for the XGBoost algorithm in SageMaker requires that the saved datasets should contain no headers or index and that for the training and validation data, the label should occur first for each sample.\n",
    "\n",
    "For more information about this and other algorithms, the SageMaker developer documentation can be found on __[Amazon's website.](https://docs.aws.amazon.com/sagemaker/latest/dg/)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we make sure that the local directory in which we'd like to store the training and validation csv files exists.\n",
    "data_dir = '../data/sentiment_update'\n",
    "if not os.path.exists(data_dir):\n",
    "    os.makedirs(data_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(test_X).to_csv(os.path.join(data_dir, 'test.csv'), header=False, index=False)\n",
    "\n",
    "pd.concat([val_y, val_X], axis=1).to_csv(os.path.join(data_dir, 'validation.csv'), header=False, index=False)\n",
    "pd.concat([train_y, train_X], axis=1).to_csv(os.path.join(data_dir, 'train.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save a bit of memory we can set text_X, train_X, val_X, train_y and val_y to None.\n",
    "\n",
    "test_X = train_X = val_X = train_y = val_y = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading Training / Validation files to S3\n",
    "\n",
    "Amazon's S3 service allows us to store files that can be access by both the built-in training models such as the XGBoost model we will be using as well as custom models such as the one we will see a little later.\n",
    "\n",
    "For this, and most other tasks we will be doing using SageMaker, there are two methods we could use. The first is to use the low level functionality of SageMaker which requires knowing each of the objects involved in the SageMaker environment. The second is to use the high level functionality in which certain choices have been made on the user's behalf. The low level approach benefits from allowing the user a great deal of flexibility while the high level approach makes development much quicker. For our purposes we will opt to use the high level approach although using the low-level approach is certainly an option.\n",
    "\n",
    "Recall the method `upload_data()` which is a member of object representing our current SageMaker session. What this method does is upload the data to the default bucket (which is created if it does not exist) into the path described by the key_prefix variable. To see this for yourself, once you have uploaded the data files, go to the S3 console and look to see where the files have been uploaded.\n",
    "\n",
    "For additional resources, see the __[SageMaker API documentation](http://sagemaker.readthedocs.io/en/latest/)__ and in addition the __[SageMaker Developer Guide.](https://docs.aws.amazon.com/sagemaker/latest/dg/)__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "session = sagemaker.Session() # Store the current SageMaker session\n",
    "\n",
    "# S3 prefix (which folder will we use)\n",
    "prefix = 'sentiment-update'\n",
    "\n",
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)\n",
    "val_location = session.upload_data(os.path.join(data_dir, 'validation.csv'), key_prefix=prefix)\n",
    "train_location = session.upload_data(os.path.join(data_dir, 'train.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the XGBoost model\n",
    "\n",
    "Now that the data has been uploaded it is time to create the XGBoost model. To begin with, we need to do some setup. At this point it is worth discussing what a model is in SageMaker. It is easiest to think of a model of comprising three different objects in the SageMaker ecosystem, which interact with one another.\n",
    "\n",
    "- Model Artifacts\n",
    "- Training Code (Container)\n",
    "- Inference Code (Container)\n",
    "\n",
    "The Model Artifacts are what you might think of as the actual model itself. For example, if you were building a neural network, the model artifacts would be the weights of the various layers. In our case, for an XGBoost model, the artifacts are the actual trees that are created during training.\n",
    "\n",
    "The other two objects, the training code and the inference code are then used the manipulate the training artifacts. More precisely, the training code uses the training data that is provided and creates the model artifacts, while the inference code uses the model artifacts to make predictions on new data.\n",
    "\n",
    "The way that SageMaker runs the training and inference code is by making use of Docker containers. For now, think of a container as being a way of packaging code up so that dependencies aren't an issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "# Our current execution role is require when creating the model as the training\n",
    "# and inference code will need to access the model artifacts.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to retrieve the location of the container which is provided by Amazon for using XGBoost.\n",
    "# As a matter of convenience, the training and inference code both use the same container.\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(session.boto_region_name, 'xgboost')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we create a SageMaker estimator object for our model.\n",
    "xgb = sagemaker.estimator.Estimator(container, # The location of the container we wish to use\n",
    "                                    role,                                    # What is our current IAM Role\n",
    "                                    train_instance_count=1,                  # How many compute instances\n",
    "                                    train_instance_type='ml.m4.xlarge',      # What kind of compute instances\n",
    "                                    output_path='s3://{}/{}/output'.format(session.default_bucket(), prefix),\n",
    "                                    sagemaker_session=session)\n",
    "\n",
    "# And then set the algorithm specific parameters.\n",
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        early_stopping_rounds=10,\n",
    "                        num_round=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit the XGBoost model\n",
    "\n",
    "Now that our model has been set up we simply need to attach the training and validation datasets and then ask SageMaker to set up the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data=train_location, content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data=val_location, content_type='csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: xgboost-2019-01-16-11-00-33-507\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-01-16 11:00:33 Starting - Starting the training job...\n",
      "2019-01-16 11:00:34 Starting - Launching requested ML instances......\n",
      "2019-01-16 11:01:37 Starting - Preparing the instances for training......\n",
      "2019-01-16 11:02:59 Downloading - Downloading input data\n",
      "2019-01-16 11:02:59 Training - Downloading the training image..\n",
      "\u001b[31mArguments: train\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:03:09:INFO] Running standalone xgboost training.\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:03:09:INFO] File size need to be processed in the node: 238.47mb. Available memory size in the node: 8387.0mb\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:03:09:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[11:03:09] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[11:03:12] 15000x5000 matrix with 75000000 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:03:12:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[11:03:12] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[11:03:13] 10000x5000 matrix with 50000000 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[11:03:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[0]#011train-error:0.299067#011validation-error:0.3026\u001b[0m\n",
      "\u001b[31mMultiple eval metrics have been passed: 'validation-error' will be used for early stopping.\n",
      "\u001b[0m\n",
      "\u001b[31mWill train until validation-error hasn't improved in 10 rounds.\u001b[0m\n",
      "\u001b[31m[11:03:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 38 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[1]#011train-error:0.283667#011validation-error:0.2893\u001b[0m\n",
      "\u001b[31m[11:03:19] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[2]#011train-error:0.283933#011validation-error:0.2875\u001b[0m\n",
      "\n",
      "2019-01-16 11:03:08 Training - Training image download completed. Training in progress.\u001b[31m[11:03:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[3]#011train-error:0.281333#011validation-error:0.2841\u001b[0m\n",
      "\u001b[31m[11:03:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[4]#011train-error:0.273733#011validation-error:0.2775\u001b[0m\n",
      "\u001b[31m[11:03:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[5]#011train-error:0.2604#011validation-error:0.2619\u001b[0m\n",
      "\u001b[31m[11:03:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[6]#011train-error:0.2524#011validation-error:0.2563\u001b[0m\n",
      "\u001b[31m[11:03:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[7]#011train-error:0.2486#011validation-error:0.2502\u001b[0m\n",
      "\u001b[31m[11:03:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[8]#011train-error:0.2352#011validation-error:0.2385\u001b[0m\n",
      "\u001b[31m[11:03:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[9]#011train-error:0.231533#011validation-error:0.2371\u001b[0m\n",
      "\u001b[31m[11:03:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[10]#011train-error:0.225667#011validation-error:0.232\u001b[0m\n",
      "\u001b[31m[11:03:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[11]#011train-error:0.217733#011validation-error:0.2287\u001b[0m\n",
      "\u001b[31m[11:03:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[12]#011train-error:0.211733#011validation-error:0.2236\u001b[0m\n",
      "\u001b[31m[11:03:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[13]#011train-error:0.2096#011validation-error:0.2211\u001b[0m\n",
      "\u001b[31m[11:03:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[14]#011train-error:0.2046#011validation-error:0.2156\u001b[0m\n",
      "\u001b[31m[11:03:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[15]#011train-error:0.204267#011validation-error:0.212\u001b[0m\n",
      "\u001b[31m[11:03:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[16]#011train-error:0.2016#011validation-error:0.2133\u001b[0m\n",
      "\u001b[31m[11:03:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[17]#011train-error:0.1982#011validation-error:0.2089\u001b[0m\n",
      "\u001b[31m[11:03:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[18]#011train-error:0.1952#011validation-error:0.2075\u001b[0m\n",
      "\u001b[31m[11:03:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[19]#011train-error:0.190733#011validation-error:0.2043\u001b[0m\n",
      "\u001b[31m[11:03:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[20]#011train-error:0.187667#011validation-error:0.2019\u001b[0m\n",
      "\u001b[31m[11:03:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[21]#011train-error:0.1854#011validation-error:0.1993\u001b[0m\n",
      "\u001b[31m[11:03:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[22]#011train-error:0.1828#011validation-error:0.1977\u001b[0m\n",
      "\u001b[31m[11:03:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[23]#011train-error:0.179733#011validation-error:0.1947\u001b[0m\n",
      "\u001b[31m[11:03:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[24]#011train-error:0.177733#011validation-error:0.1938\u001b[0m\n",
      "\u001b[31m[11:03:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[25]#011train-error:0.176667#011validation-error:0.1931\u001b[0m\n",
      "\u001b[31m[11:03:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[26]#011train-error:0.174733#011validation-error:0.1926\u001b[0m\n",
      "\u001b[31m[11:03:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[27]#011train-error:0.1724#011validation-error:0.1892\u001b[0m\n",
      "\u001b[31m[11:03:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[28]#011train-error:0.170867#011validation-error:0.1887\u001b[0m\n",
      "\u001b[31m[11:03:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[29]#011train-error:0.168133#011validation-error:0.1862\u001b[0m\n",
      "\u001b[31m[11:03:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[30]#011train-error:0.165733#011validation-error:0.1855\u001b[0m\n",
      "\u001b[31m[11:03:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[31]#011train-error:0.165733#011validation-error:0.1847\u001b[0m\n",
      "\u001b[31m[11:04:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[32]#011train-error:0.162667#011validation-error:0.1828\u001b[0m\n",
      "\u001b[31m[11:04:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[33]#011train-error:0.162267#011validation-error:0.1807\u001b[0m\n",
      "\u001b[31m[11:04:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[34]#011train-error:0.16#011validation-error:0.1799\u001b[0m\n",
      "\u001b[31m[11:04:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[35]#011train-error:0.1608#011validation-error:0.1782\u001b[0m\n",
      "\u001b[31m[11:04:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[36]#011train-error:0.159067#011validation-error:0.1787\u001b[0m\n",
      "\u001b[31m[11:04:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[37]#011train-error:0.157867#011validation-error:0.1766\u001b[0m\n",
      "\u001b[31m[11:04:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[38]#011train-error:0.1568#011validation-error:0.1764\u001b[0m\n",
      "\u001b[31m[11:04:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[39]#011train-error:0.1558#011validation-error:0.1744\u001b[0m\n",
      "\u001b[31m[11:04:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[40]#011train-error:0.152267#011validation-error:0.1736\u001b[0m\n",
      "\u001b[31m[11:04:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[41]#011train-error:0.151467#011validation-error:0.1728\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[11:04:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[42]#011train-error:0.149867#011validation-error:0.1732\u001b[0m\n",
      "\u001b[31m[11:04:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[43]#011train-error:0.1492#011validation-error:0.1729\u001b[0m\n",
      "\u001b[31m[11:04:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[44]#011train-error:0.1478#011validation-error:0.1723\u001b[0m\n",
      "\u001b[31m[11:04:17] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[45]#011train-error:0.1468#011validation-error:0.1707\u001b[0m\n",
      "\u001b[31m[11:04:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[46]#011train-error:0.145867#011validation-error:0.1692\u001b[0m\n",
      "\u001b[31m[11:04:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[47]#011train-error:0.144#011validation-error:0.1707\u001b[0m\n",
      "\u001b[31m[11:04:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[48]#011train-error:0.143133#011validation-error:0.1705\u001b[0m\n",
      "\u001b[31m[11:04:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[49]#011train-error:0.141933#011validation-error:0.1701\u001b[0m\n",
      "\u001b[31m[11:04:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[50]#011train-error:0.1402#011validation-error:0.1702\u001b[0m\n",
      "\u001b[31m[11:04:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[51]#011train-error:0.139067#011validation-error:0.1697\u001b[0m\n",
      "\u001b[31m[11:04:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[52]#011train-error:0.138467#011validation-error:0.1687\u001b[0m\n",
      "\u001b[31m[11:04:28] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[53]#011train-error:0.138067#011validation-error:0.1684\u001b[0m\n",
      "\u001b[31m[11:04:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[54]#011train-error:0.137267#011validation-error:0.1665\u001b[0m\n",
      "\u001b[31m[11:04:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[55]#011train-error:0.1356#011validation-error:0.165\u001b[0m\n",
      "\u001b[31m[11:04:32] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[56]#011train-error:0.134533#011validation-error:0.1638\u001b[0m\n",
      "\u001b[31m[11:04:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[57]#011train-error:0.1336#011validation-error:0.1621\u001b[0m\n",
      "\u001b[31m[11:04:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[58]#011train-error:0.1332#011validation-error:0.162\u001b[0m\n",
      "\u001b[31m[11:04:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[59]#011train-error:0.133#011validation-error:0.1606\u001b[0m\n",
      "\u001b[31m[11:04:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[60]#011train-error:0.131#011validation-error:0.1624\u001b[0m\n",
      "\u001b[31m[11:04:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[61]#011train-error:0.130133#011validation-error:0.1624\u001b[0m\n",
      "\u001b[31m[11:04:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[62]#011train-error:0.129067#011validation-error:0.1607\u001b[0m\n",
      "\u001b[31m[11:04:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[63]#011train-error:0.129333#011validation-error:0.1616\u001b[0m\n",
      "\u001b[31m[11:04:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[64]#011train-error:0.127533#011validation-error:0.1606\u001b[0m\n",
      "\u001b[31m[11:04:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[65]#011train-error:0.127133#011validation-error:0.161\u001b[0m\n",
      "\u001b[31m[11:04:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[66]#011train-error:0.127267#011validation-error:0.1606\u001b[0m\n",
      "\u001b[31m[11:04:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[67]#011train-error:0.127133#011validation-error:0.1605\u001b[0m\n",
      "\u001b[31m[11:04:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[68]#011train-error:0.1264#011validation-error:0.1601\u001b[0m\n",
      "\u001b[31m[11:04:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[69]#011train-error:0.124733#011validation-error:0.1593\u001b[0m\n",
      "\u001b[31m[11:04:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[70]#011train-error:0.1244#011validation-error:0.1581\u001b[0m\n",
      "\u001b[31m[11:04:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[71]#011train-error:0.124067#011validation-error:0.1575\u001b[0m\n",
      "\u001b[31m[11:04:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[72]#011train-error:0.122867#011validation-error:0.1569\u001b[0m\n",
      "\u001b[31m[11:04:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[73]#011train-error:0.122667#011validation-error:0.1574\u001b[0m\n",
      "\u001b[31m[11:04:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[74]#011train-error:0.122733#011validation-error:0.1569\u001b[0m\n",
      "\u001b[31m[11:04:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[75]#011train-error:0.121#011validation-error:0.1561\u001b[0m\n",
      "\u001b[31m[11:04:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[76]#011train-error:0.1204#011validation-error:0.155\u001b[0m\n",
      "\u001b[31m[11:05:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[77]#011train-error:0.120933#011validation-error:0.1536\u001b[0m\n",
      "\u001b[31m[11:05:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[78]#011train-error:0.120267#011validation-error:0.1537\u001b[0m\n",
      "\u001b[31m[11:05:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[79]#011train-error:0.119#011validation-error:0.1532\u001b[0m\n",
      "\u001b[31m[11:05:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[80]#011train-error:0.118733#011validation-error:0.1529\u001b[0m\n",
      "\u001b[31m[11:05:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[81]#011train-error:0.118267#011validation-error:0.1526\u001b[0m\n",
      "\u001b[31m[11:05:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[82]#011train-error:0.118333#011validation-error:0.153\u001b[0m\n",
      "\u001b[31m[11:05:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[83]#011train-error:0.1174#011validation-error:0.152\u001b[0m\n",
      "\u001b[31m[11:05:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[84]#011train-error:0.116933#011validation-error:0.1511\u001b[0m\n",
      "\u001b[31m[11:05:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[85]#011train-error:0.117067#011validation-error:0.1509\u001b[0m\n",
      "\u001b[31m[11:05:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[86]#011train-error:0.115733#011validation-error:0.1508\u001b[0m\n",
      "\u001b[31m[11:05:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[87]#011train-error:0.115733#011validation-error:0.1509\u001b[0m\n",
      "\u001b[31m[11:05:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[88]#011train-error:0.115267#011validation-error:0.1495\u001b[0m\n",
      "\u001b[31m[11:05:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[89]#011train-error:0.115133#011validation-error:0.1494\u001b[0m\n",
      "\u001b[31m[11:05:17] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[90]#011train-error:0.115333#011validation-error:0.1495\u001b[0m\n",
      "\u001b[31m[11:05:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[91]#011train-error:0.114467#011validation-error:0.1503\u001b[0m\n",
      "\u001b[31m[11:05:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[92]#011train-error:0.114467#011validation-error:0.1504\u001b[0m\n",
      "\u001b[31m[11:05:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[93]#011train-error:0.1136#011validation-error:0.1496\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[11:05:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[94]#011train-error:0.1126#011validation-error:0.1485\u001b[0m\n",
      "\u001b[31m[11:05:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[95]#011train-error:0.1112#011validation-error:0.1481\u001b[0m\n",
      "\u001b[31m[11:05:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[96]#011train-error:0.111267#011validation-error:0.1482\u001b[0m\n",
      "\u001b[31m[11:05:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[97]#011train-error:0.110733#011validation-error:0.1486\u001b[0m\n",
      "\u001b[31m[11:05:28] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[98]#011train-error:0.110533#011validation-error:0.1486\u001b[0m\n",
      "\u001b[31m[11:05:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[99]#011train-error:0.109733#011validation-error:0.1488\u001b[0m\n",
      "\u001b[31m[11:05:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[100]#011train-error:0.109#011validation-error:0.148\u001b[0m\n",
      "\u001b[31m[11:05:32] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[101]#011train-error:0.108467#011validation-error:0.1482\u001b[0m\n",
      "\u001b[31m[11:05:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[102]#011train-error:0.108467#011validation-error:0.1481\u001b[0m\n",
      "\u001b[31m[11:05:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[103]#011train-error:0.107533#011validation-error:0.1474\u001b[0m\n",
      "\u001b[31m[11:05:36] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[104]#011train-error:0.107867#011validation-error:0.1476\u001b[0m\n",
      "\u001b[31m[11:05:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[105]#011train-error:0.107733#011validation-error:0.1475\u001b[0m\n",
      "\u001b[31m[11:05:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[106]#011train-error:0.107533#011validation-error:0.1471\u001b[0m\n",
      "\u001b[31m[11:05:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[107]#011train-error:0.1062#011validation-error:0.1467\u001b[0m\n",
      "\u001b[31m[11:05:41] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[108]#011train-error:0.1062#011validation-error:0.1464\u001b[0m\n",
      "\u001b[31m[11:05:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[109]#011train-error:0.105067#011validation-error:0.1453\u001b[0m\n",
      "\u001b[31m[11:05:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[110]#011train-error:0.104867#011validation-error:0.1452\u001b[0m\n",
      "\u001b[31m[11:05:45] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[111]#011train-error:0.104667#011validation-error:0.1452\u001b[0m\n",
      "\u001b[31m[11:05:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[112]#011train-error:0.104667#011validation-error:0.145\u001b[0m\n",
      "\u001b[31m[11:05:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[113]#011train-error:0.104#011validation-error:0.1445\u001b[0m\n",
      "\u001b[31m[11:05:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[114]#011train-error:0.104267#011validation-error:0.1444\u001b[0m\n",
      "\u001b[31m[11:05:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[115]#011train-error:0.102933#011validation-error:0.144\u001b[0m\n",
      "\u001b[31m[11:05:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[116]#011train-error:0.102467#011validation-error:0.1437\u001b[0m\n",
      "\u001b[31m[11:05:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[117]#011train-error:0.101867#011validation-error:0.144\u001b[0m\n",
      "\u001b[31m[11:05:55] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[118]#011train-error:0.1018#011validation-error:0.1437\u001b[0m\n",
      "\u001b[31m[11:05:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[119]#011train-error:0.1014#011validation-error:0.1429\u001b[0m\n",
      "\u001b[31m[11:05:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[120]#011train-error:0.100467#011validation-error:0.1428\u001b[0m\n",
      "\u001b[31m[11:05:59] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[121]#011train-error:0.100133#011validation-error:0.1422\u001b[0m\n",
      "\u001b[31m[11:06:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[122]#011train-error:0.099867#011validation-error:0.1427\u001b[0m\n",
      "\u001b[31m[11:06:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[123]#011train-error:0.099867#011validation-error:0.1428\u001b[0m\n",
      "\u001b[31m[11:06:03] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[124]#011train-error:0.0996#011validation-error:0.1435\u001b[0m\n",
      "\u001b[31m[11:06:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[125]#011train-error:0.0992#011validation-error:0.143\u001b[0m\n",
      "\u001b[31m[11:06:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[126]#011train-error:0.098533#011validation-error:0.1436\u001b[0m\n",
      "\u001b[31m[11:06:07] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[127]#011train-error:0.0986#011validation-error:0.1433\u001b[0m\n",
      "\u001b[31m[11:06:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[128]#011train-error:0.098333#011validation-error:0.1432\u001b[0m\n",
      "\u001b[31m[11:06:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[129]#011train-error:0.0976#011validation-error:0.1433\u001b[0m\n",
      "\u001b[31m[11:06:11] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[130]#011train-error:0.097333#011validation-error:0.1431\u001b[0m\n",
      "\u001b[31m[11:06:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[131]#011train-error:0.096533#011validation-error:0.1424\u001b[0m\n",
      "\u001b[31mStopping. Best iteration:\u001b[0m\n",
      "\u001b[31m[121]#011train-error:0.100133#011validation-error:0.1422\n",
      "\u001b[0m\n",
      "\n",
      "2019-01-16 11:06:23 Uploading - Uploading generated training model\n",
      "2019-01-16 11:06:23 Completed - Training job completed\n",
      "Billable seconds: 229\n"
     ]
    }
   ],
   "source": [
    "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the model\n",
    "\n",
    "Now that we've fit our XGBoost model, it's time to see how well it performs. To do this we will use SageMakers Batch Transform functionality. Batch Transform is a convenient way to perform inference on a large dataset in a way that is not realtime. That is, we don't necessarily need to use our model's results immediately and instead we can peform inference on a large number of samples. An example of this in industry might be peforming an end of month report. This method of inference can also be useful to us as it means to can perform inference on our entire test set. \n",
    "\n",
    "To perform a Batch Transformation we need to first create a transformer objects from our trained estimator object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: xgboost-2019-01-16-11-00-33-507\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer = xgb.transformer(instance_count = 1, instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we actually perform the transform job. When doing so we need to make sure to specify the type of data we are sending so that it is serialized correctly in the background. In our case we are providing our model with csv data so we specify `text/csv`. Also, if the test data that we have provided is too large to process all at once then we need to specify how the data file should be split up. Since each line is a single entry in our data set we tell SageMaker that it can split the input on each line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: xgboost-2019-01-16-11-06-47-484\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently the transform job is running but it is doing so in the background. Since we wish to wait until the transform job is done and we would like a bit of feedback we can run the `wait()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..........................................!\n"
     ]
    }
   ],
   "source": [
    "xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the transform job has executed and the result, the estimated sentiment of each review, has been saved on S3. Since we would rather work on this file locally we can perform a bit of notebook magic to copy the file to the `data_dir`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-eu-west-1-177473201433/xgboost-2019-01-16-11-06-47-484/test.csv.out to ../data/sentiment_update/test.csv.out\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is now to read in the output from our model, convert the output to something a little more usable, in this case we want the sentiment to be either `1` (positive) or `0` (negative), and then compare to the ground truth labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85696"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(test_y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Looking at New Data\n",
    "\n",
    "So now we have an XGBoost sentiment analysis model that we believe is working pretty well. As a result, we deployed it and we are using it in some sort of app.\n",
    "\n",
    "However, as we allow users to use our app we periodically record submitted movie reviews so that we can perform some quality control on our deployed model. Once we've accumulated enough reviews we go through them by hand and evaluate whether they are positive or negative (there are many ways you might do this in practice aside from by hand). The reason for doing this is so that we can check to see how well our model is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import new_data\n",
    "\n",
    "new_X, new_Y = new_data.get_new_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Part of the fun in this notebook is trying to figure out what exactly is happening with the new data, so try not to cheat by looking in the `new_data` module. Also, the `new_data` module assumes that the cache created earlier in Step 3 is still stored in `../cache/sentiment_analysis`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the current model\n",
    "\n",
    "Now that we've loaded the new data, let's check to see how our current XGBoost model performs on it.\n",
    "\n",
    "First, note that the data that has been loaded has already been pre-processed so that each entry in `new_X` is a list of words that have been processed using `nltk`. However, we have not yet constructed the bag of words encoding, which we will do now.\n",
    "\n",
    "First, we use the vocabulary that we constructed earlier using the original training data to construct a `CountVectorizer` which we will use to transform our new data into its bag of words encoding.\n",
    "\n",
    "Create the CountVectorizer object using the vocabulary created earlier and use it to transform the new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the CountVectorizer using the previously constructed vocabulary\n",
    "vectorizer = CountVectorizer(vocabulary=vocabulary, preprocessor=lambda x: x, tokenizer=lambda x: x)\n",
    "\n",
    "# Transform our new data set and store the transformed data in the variable new_XV\n",
    "new_XV = vectorizer.transform(new_X).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a quick sanity check, we make sure that the length of each of our bag of words encoded reviews is correct. In particular, it must be the same size as the vocabulary which in our case is `5000`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(new_XV[100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've performed the data processing that is required by our model we can save it locally and then upload it to S3 so that we can construct a batch transform job in order to see how well our model is working.\n",
    "\n",
    "First, we save the data locally.\n",
    "\n",
    "Save the new data (after it has been transformed using the original vocabulary) to the local notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the data contained in new_XV locally in the data_dir with the file name new_data.csv\n",
    "pd.DataFrame(new_XV).to_csv(os.path.join(data_dir, 'new_data.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we upload the data to S3.\n",
    "\n",
    "Upload the csv file created above to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the new_data.csv file contained in the data_dir folder to S3 and save the resulting\n",
    "# URI as new_data_location\n",
    "\n",
    "new_data_location = session.upload_data(os.path.join(data_dir, 'new_data.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, once the new data has been uploaded to S3, we create and run the batch transform job to get our model's predictions about the sentiment of the new movie reviews.\n",
    "\n",
    "Using the `xgb_transformer` object that was created earlier (at the end of Step 4 to test the XGBoost model), transform the data located at `new_data_location`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: xgboost-2019-01-16-11-19-47-112\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..........................................!\n"
     ]
    }
   ],
   "source": [
    "# Using xgb_transformer, transform the new_data_location data. You may wish to **wait** until\n",
    "# the batch transform job has finished.\n",
    "\n",
    "xgb_transformer.transform(new_data_location, content_type='text/csv', split_type='Line')\n",
    "xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, we copy the results of the batch transform job to our local instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-eu-west-1-177473201433/xgboost-2019-01-16-11-06-47-484/new_data.csv.out to ../data/sentiment_update/new_data.csv.out\n",
      "download: s3://sagemaker-eu-west-1-177473201433/xgboost-2019-01-16-11-06-47-484/test.csv.out to ../data/sentiment_update/test.csv.out\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the results of the batch transform job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'new_data.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And check the accuracy of our current model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73188"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(new_Y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So it would appear that *something* has changed since our model is no longer (as) effective at determining the sentiment of a user provided review.\n",
    "\n",
    "In a real life scenario you would check a number of different things to see what exactly is going on. In our case, we are only going to check one and that is whether some aspect of the underlying distribution has changed. In other words, we want to see if the words that appear in our new collection of reviews matches the words that appear in the original training set. Of course, we want to narrow our scope a little bit so we will only look at the `5000` most frequently appearing words in each data set, or in other words, the vocabulary generated by each data set.\n",
    "\n",
    "Before doing that, however, let's take a look at some of the incorrectly classified reviews in the new data set.\n",
    "\n",
    "To start, we will deploy the original XGBoost model. We will then use the deployed model to infer the sentiment of some of the new reviews. This will also serve as a nice excuse to deploy our model so that we can mimic a real life scenario where we have a model that has been deployed and is being used in production.\n",
    "\n",
    "Deploy the XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: xgboost-2019-01-16-11-25-52-039\n",
      "INFO:sagemaker:Creating endpoint with name xgboost-2019-01-16-11-00-33-507\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------------------------------------------------------------------!"
     ]
    }
   ],
   "source": [
    "# Deploy the model that was created earlier. Recall that the object name is 'xgb'.\n",
    "xgb_predictor = xgb.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Diagnose the problem\n",
    "\n",
    "Now that we have our deployed \"production\" model, we can send some of our new data to it and filter out some of the incorrectly classified reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "\n",
    "# We need to tell the endpoint what format the data we are sending is in so that SageMaker can perform the serialization.\n",
    "xgb_predictor.content_type = 'text/csv'\n",
    "xgb_predictor.serializer = csv_serializer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will be useful to look at a few different examples of incorrectly classified reviews so we will start by creating a *generator* which we will use to iterate through some of the new reviews and find ones that are incorrect.\n",
    "\n",
    "**NOTE:** Understanding what Python generators are isn't really required for this module. The reason we use them here is so that we don't have to iterate through all of the new reviews, searching for incorrectly classified samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sample(in_X, in_XV, in_Y):\n",
    "    for idx, smp in enumerate(in_X):\n",
    "        res = round(float(xgb_predictor.predict(in_XV[idx])))\n",
    "        if res != in_Y[idx]:\n",
    "            yield smp, in_Y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "gn = get_sample(new_X, new_XV, new_Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, `gn` is the *generator* which generates samples from the new data set which are not classified correctly. To get the *next* sample we simply call the `next` method on our generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(['watch', 'film', '45', 'minut', 'count', '9', 'mullet', 'mullet', 'everi', '5', 'minut', 'serious', 'though', 'film', 'live', 'proof', 'formula', 'work', 'broke', 'need', 'fixin', 'streetwis', 'yet', 'vulner', 'heroin', 'harden', 'ex', 'cop', 'martial', 'art', 'master', 'heart', 'gold', 'serial', 'killer', 'issu', 'pure', 'magic'], 0)\n"
     ]
    }
   ],
   "source": [
    "print(next(gn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After looking at a few examples, maybe we decide to look at the most frequently appearing `5000` words in each data set, the original training data set and the new data set. The reason for looking at this might be that we expect the frequency of use of different words to have changed, maybe there is some new slang that has been introduced or some other artifact of popular culture that has changed the way that people write movie reviews.\n",
    "\n",
    "To do this, we start by fitting a `CountVectorizer` to the new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=5000, min_df=1,\n",
       "        ngram_range=(1, 1),\n",
       "        preprocessor=<function <lambda> at 0x7f6727888ae8>,\n",
       "        stop_words=None, strip_accents=None,\n",
       "        token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=<function <lambda> at 0x7f6727888b70>, vocabulary=None)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_vectorizer = CountVectorizer(max_features=5000,\n",
    "                preprocessor=lambda x: x, tokenizer=lambda x: x)\n",
    "new_vectorizer.fit(new_X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have this new `CountVectorizor` object, we can check to see if the corresponding vocabulary has changed between the two data sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_vocabulary = set(vocabulary.keys())\n",
    "new_vocabulary = set(new_vectorizer.vocabulary_.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can look at the words that were in the original vocabulary but not in the new vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'reincarn', 'ghetto', '21st', 'victorian', 'playboy', 'spill', 'weari'}\n"
     ]
    }
   ],
   "source": [
    "print(original_vocabulary - new_vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And similarly, we can look at the words that are in the new vocabulary but which were not in the original vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'sophi', 'optimist', 'banana', 'omin', 'masterson', 'dubiou', 'orchestr'}\n"
     ]
    }
   ],
   "source": [
    "print(new_vocabulary - original_vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These words themselves don't tell us much, however if one of these words occured with a large frequency, that might tell us something. In particular, we wouldn't really expect any of the words above to appear with too much frequency.\n",
    "\n",
    "**Question** What exactly is going on here. Not only what (if any) words appear with a larger than expected frequency but also, what does this mean? What has changed about the world that our original model no longer takes into account?\n",
    "\n",
    "**NOTE:** This is meant to be a very open ended question. To investigate you may need more cells than the one provided below. Also, there isn't really a *correct* answer, this is meant to be an opportunity to explore the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a new model\n",
    "\n",
    "Supposing that we believe something has changed about the underlying distribution of the words that our reviews are made up of, we need to create a new model. This way our new model will take into account whatever it is that has changed.\n",
    "\n",
    "To begin with, we will use the new vocabulary to create a bag of words encoding of the new data. We will then use this data to train a new XGBoost model.\n",
    "\n",
    "**NOTE:** Because we believe that the underlying distribution of words has changed it should follow that the original vocabulary that we used to construct a bag of words encoding of the reviews is no longer valid. This means that we need to be careful with our data. If we send an bag of words encoded review using the *original* vocabulary we should not expect any sort of meaningful results.\n",
    "\n",
    "In particular, this means that if we had deployed our XGBoost model like we did in the Web App notebook then we would need to implement this vocabulary change in the Lambda function as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_XV = new_vectorizer.transform(new_X).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a quick check to make sure that the newly encoded reviews have the correct length, which should be the size of the new vocabulary which we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(new_XV[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have our newly encoded, newly collected data, we can split it up into a training and validation set so that we can train a new XGBoost model. As usual, we first split up the data, then save it locally and then upload it to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Earlier we shuffled the training dataset so to make things simple we can just assign\n",
    "# the first 10 000 reviews to the validation set and use the remaining reviews for training.\n",
    "new_val_X = pd.DataFrame(new_XV[:10000])\n",
    "new_train_X = pd.DataFrame(new_XV[10000:])\n",
    "\n",
    "new_val_y = pd.DataFrame(new_Y[:10000])\n",
    "new_train_y = pd.DataFrame(new_Y[10000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to save some memory we will effectively delete the `new_X` variable. Remember that this contained a list of reviews and each review was a list of words. Note that once this cell has been executed you will need to read the new data in again if you want to work with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_X = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we save the new training and validation sets locally. Note that we overwrite the training and validation sets used earlier. This is mostly because the amount of space that we have available on our notebook instance is limited. Of course, you can increase this if you'd like but to do so may increase the cost of running the notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(new_XV).to_csv(os.path.join(data_dir, 'new_data.csv'), header=False, index=False)\n",
    "\n",
    "pd.concat([new_val_y, new_val_X], axis=1).to_csv(os.path.join(data_dir, 'new_validation.csv'), header=False, index=False)\n",
    "pd.concat([new_train_y, new_train_X], axis=1).to_csv(os.path.join(data_dir, 'new_train.csv'), header=False, index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've saved our data to the local instance, we can safely delete the variables to save on memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_val_y = new_val_X = new_train_y = new_train_X = new_XV = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we make sure to upload the new training and validation sets to S3.\n",
    "\n",
    "Upload the new data as well as the new training and validation data sets to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the new data and the new validation.csv and train.csv files in the data_dir directory to S3.\n",
    "new_data_location = session.upload_data(os.path.join(data_dir, 'new_data.csv'), key_prefix=prefix)\n",
    "new_val_location = session.upload_data(os.path.join(data_dir, 'new_validation.csv'), key_prefix=prefix)\n",
    "new_train_location = session.upload_data(os.path.join(data_dir, 'new_train.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once our new training data has been uploaded to S3, we can create a new XGBoost model that will take into account the changes that have occured in our data set.\n",
    "\n",
    "Create a new XGBoost estimator object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, create a SageMaker estimator object for our model.\n",
    "new_xgb = sagemaker.estimator.Estimator(container, # The location of the container we wish to use\n",
    "                                    role,                                    # What is our current IAM Role\n",
    "                                    train_instance_count=1,                  # How many compute instances\n",
    "                                    train_instance_type='ml.m4.xlarge',      # What kind of compute instances\n",
    "                                    output_path='s3://{}/{}/output'.format(session.default_bucket(), prefix),\n",
    "                                    sagemaker_session=session)\n",
    "\n",
    "\n",
    "# Then set the algorithm specific parameters. You may wish to use the same parameters that were\n",
    "# used when training the original model.\n",
    "new_xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        early_stopping_rounds=10,\n",
    "                        num_round=500)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model has been created, we can train it with our new data.\n",
    "\n",
    "Train the new XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, make sure that you create s3 input objects so that SageMaker knows where to\n",
    "# find the training and validation data.\n",
    "s3_new_input_train = sagemaker.s3_input(s3_data=new_train_location, content_type='csv')\n",
    "s3_new_input_validation =  sagemaker.s3_input(s3_data=new_val_location, content_type='csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: xgboost-2019-01-16-11-39-04-711\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-01-16 11:39:04 Starting - Starting the training job...\n",
      "2019-01-16 11:39:06 Starting - Launching requested ML instances......\n",
      "2019-01-16 11:40:06 Starting - Preparing the instances for training......\n",
      "2019-01-16 11:41:15 Downloading - Downloading input data..\n",
      "\u001b[31mArguments: train\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:41:42:INFO] Running standalone xgboost training.\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:41:42:INFO] File size need to be processed in the node: 238.47mb. Available memory size in the node: 8421.2mb\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:41:42:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[11:41:42] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[11:41:44] 15000x5000 matrix with 75000000 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[2019-01-16:11:41:44:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[31m[11:41:44] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[31m[11:41:45] 10000x5000 matrix with 50000000 entries loaded from /opt/ml/input/data/validation?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[31m[11:41:49] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[0]#011train-error:0.299067#011validation-error:0.4548\u001b[0m\n",
      "\u001b[31mMultiple eval metrics have been passed: 'validation-error' will be used for early stopping.\n",
      "\u001b[0m\n",
      "\u001b[31mWill train until validation-error hasn't improved in 10 rounds.\u001b[0m\n",
      "\u001b[31m[11:41:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 38 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[1]#011train-error:0.283667#011validation-error:0.4392\u001b[0m\n",
      "\u001b[31m[11:41:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[2]#011train-error:0.283933#011validation-error:0.4415\u001b[0m\n",
      "\u001b[31m[11:41:53] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 44 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[3]#011train-error:0.281333#011validation-error:0.4407\u001b[0m\n",
      "\n",
      "2019-01-16 11:41:41 Training - Training image download completed. Training in progress.\u001b[31m[11:41:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[4]#011train-error:0.273733#011validation-error:0.4336\u001b[0m\n",
      "\u001b[31m[11:41:56] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[5]#011train-error:0.2604#011validation-error:0.4288\u001b[0m\n",
      "\u001b[31m[11:41:57] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[6]#011train-error:0.2524#011validation-error:0.428\u001b[0m\n",
      "\u001b[31m[11:41:58] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[7]#011train-error:0.2486#011validation-error:0.4229\u001b[0m\n",
      "\u001b[31m[11:42:00] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[8]#011train-error:0.2352#011validation-error:0.4162\u001b[0m\n",
      "\u001b[31m[11:42:01] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[9]#011train-error:0.231533#011validation-error:0.4198\u001b[0m\n",
      "\u001b[31m[11:42:02] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[10]#011train-error:0.225667#011validation-error:0.4225\u001b[0m\n",
      "\u001b[31m[11:42:04] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[11]#011train-error:0.217733#011validation-error:0.4177\u001b[0m\n",
      "\u001b[31m[11:42:05] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[12]#011train-error:0.211733#011validation-error:0.4182\u001b[0m\n",
      "\u001b[31m[11:42:06] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[13]#011train-error:0.2096#011validation-error:0.4176\u001b[0m\n",
      "\u001b[31m[11:42:08] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 2 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[14]#011train-error:0.2046#011validation-error:0.4169\u001b[0m\n",
      "\u001b[31m[11:42:09] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[15]#011train-error:0.204267#011validation-error:0.4163\u001b[0m\n",
      "\u001b[31m[11:42:10] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[16]#011train-error:0.2016#011validation-error:0.417\u001b[0m\n",
      "\u001b[31m[11:42:12] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[17]#011train-error:0.1982#011validation-error:0.4122\u001b[0m\n",
      "\u001b[31m[11:42:13] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[18]#011train-error:0.1952#011validation-error:0.4055\u001b[0m\n",
      "\u001b[31m[11:42:14] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[19]#011train-error:0.190733#011validation-error:0.4065\u001b[0m\n",
      "\u001b[31m[11:42:16] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[20]#011train-error:0.187667#011validation-error:0.4051\u001b[0m\n",
      "\u001b[31m[11:42:17] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[21]#011train-error:0.1854#011validation-error:0.4004\u001b[0m\n",
      "\u001b[31m[11:42:18] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[22]#011train-error:0.1828#011validation-error:0.4005\u001b[0m\n",
      "\u001b[31m[11:42:19] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[23]#011train-error:0.179733#011validation-error:0.4022\u001b[0m\n",
      "\u001b[31m[11:42:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[24]#011train-error:0.177733#011validation-error:0.4012\u001b[0m\n",
      "\u001b[31m[11:42:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[25]#011train-error:0.176667#011validation-error:0.3985\u001b[0m\n",
      "\u001b[31m[11:42:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[26]#011train-error:0.174733#011validation-error:0.3978\u001b[0m\n",
      "\u001b[31m[11:42:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[27]#011train-error:0.1724#011validation-error:0.3989\u001b[0m\n",
      "\u001b[31m[11:42:26] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[28]#011train-error:0.170867#011validation-error:0.3979\u001b[0m\n",
      "\u001b[31m[11:42:27] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[29]#011train-error:0.168133#011validation-error:0.3955\u001b[0m\n",
      "\u001b[31m[11:42:29] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[30]#011train-error:0.165733#011validation-error:0.394\u001b[0m\n",
      "\u001b[31m[11:42:30] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[31]#011train-error:0.165733#011validation-error:0.393\u001b[0m\n",
      "\u001b[31m[11:42:31] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[32]#011train-error:0.162667#011validation-error:0.3871\u001b[0m\n",
      "\u001b[31m[11:42:33] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[33]#011train-error:0.162267#011validation-error:0.3881\u001b[0m\n",
      "\u001b[31m[11:42:34] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 0 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[34]#011train-error:0.16#011validation-error:0.386\u001b[0m\n",
      "\u001b[31m[11:42:35] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[35]#011train-error:0.1608#011validation-error:0.3838\u001b[0m\n",
      "\u001b[31m[11:42:37] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[36]#011train-error:0.159067#011validation-error:0.3854\u001b[0m\n",
      "\u001b[31m[11:42:38] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[37]#011train-error:0.157867#011validation-error:0.3833\u001b[0m\n",
      "\u001b[31m[11:42:39] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[38]#011train-error:0.1568#011validation-error:0.3843\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[11:42:40] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[39]#011train-error:0.1558#011validation-error:0.3827\u001b[0m\n",
      "\u001b[31m[11:42:42] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[40]#011train-error:0.152267#011validation-error:0.3838\u001b[0m\n",
      "\u001b[31m[11:42:43] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[41]#011train-error:0.151467#011validation-error:0.3842\u001b[0m\n",
      "\u001b[31m[11:42:44] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[42]#011train-error:0.149867#011validation-error:0.3839\u001b[0m\n",
      "\u001b[31m[11:42:46] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[43]#011train-error:0.1492#011validation-error:0.3837\u001b[0m\n",
      "\u001b[31m[11:42:47] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[44]#011train-error:0.1478#011validation-error:0.3852\u001b[0m\n",
      "\u001b[31m[11:42:48] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[45]#011train-error:0.1468#011validation-error:0.3849\u001b[0m\n",
      "\u001b[31m[11:42:50] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 4 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[46]#011train-error:0.145867#011validation-error:0.3838\u001b[0m\n",
      "\n",
      "2019-01-16 11:43:05 Uploading - Uploading generated training model\n",
      "2019-01-16 11:43:05 Completed - Training job completed\n",
      "\u001b[31m[11:42:51] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[47]#011train-error:0.144#011validation-error:0.3844\u001b[0m\n",
      "\u001b[31m[11:42:52] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[48]#011train-error:0.143133#011validation-error:0.3846\u001b[0m\n",
      "\u001b[31m[11:42:54] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[31m[49]#011train-error:0.141933#011validation-error:0.3832\u001b[0m\n",
      "\u001b[31mStopping. Best iteration:\u001b[0m\n",
      "\u001b[31m[39]#011train-error:0.1558#011validation-error:0.3827\n",
      "\u001b[0m\n",
      "Billable seconds: 110\n"
     ]
    }
   ],
   "source": [
    "# Using the new validation and training data, 'fit' your new model.\n",
    "new_xgb.fit({'train': s3_input_train, 'validation': s3_new_input_validation})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the new model\n",
    "\n",
    "So now we have a new XGBoost model that we believe more accurately represents the state of the world at this time, at least in how it relates to the sentiment analysis problem that we are working on. The next step is to double check that our model is performing reasonably.\n",
    "\n",
    "To do this, we will first test our model on the new data.\n",
    "\n",
    "**Note:** In practice this is a pretty bad idea. We already trained our model on the new data, so testing it shouldn't really tell us much. In fact, this is sort of a textbook example of leakage. We are only doing it here so that we have a numerical baseline.\n",
    "\n",
    "**Question:** How might you address the leakage problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create a new transformer based on our new XGBoost model.\n",
    "\n",
    "Create a transformer object from the newly created XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: xgboost-2019-01-16-11-39-04-711\n"
     ]
    }
   ],
   "source": [
    "# Create a transformer object from the new_xgb model\n",
    "new_xgb_transformer = new_xgb.transformer(instance_count = 1, instance_type = 'ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we test our model on the new data.\n",
    "\n",
    "Use the transformer object to transform the new data (stored in the `new_data_location` variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: xgboost-2019-01-16-11-43-17-685\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..............................................!\n"
     ]
    }
   ],
   "source": [
    "# Using new_xgb_transformer, transform the new_data_location data. You may wish to\n",
    "# 'wait' for the transform job to finish.\n",
    "new_xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')\n",
    "new_xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the results to our local instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 256.0 KiB/366.1 KiB (1.6 MiB/s) with 1 file(s) remaining\r",
      "Completed 366.1 KiB/366.1 KiB (2.3 MiB/s) with 1 file(s) remaining\r",
      "download: s3://sagemaker-eu-west-1-177473201433/xgboost-2019-01-16-11-43-17-685/test.csv.out to ../data/sentiment_update/test.csv.out\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $new_xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And see how well the model did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'new_data.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.73188"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(new_Y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, since we trained the model on this data, our model performs pretty well. So, we have reason to believe that our new XGBoost model is a \"better\" model.\n",
    "\n",
    "However, before we start changing our deployed model, we should first make sure that our new model isn't too different. In other words, if our new model performed really poorly on the original test data then this might be an indication that something else has gone wrong.\n",
    "\n",
    "To start with, since we got rid of the variable that stored the original test reviews, we will read them in again from the cache that we created in Step 3. Note that we need to make sure that we read in the original test data after it has been pre-processed with `nltk` but before it has been bag of words encoded. This is because we need to use the new vocabulary instead of the original one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read preprocessed data from cache file: preprocessed_data.pkl\n"
     ]
    }
   ],
   "source": [
    "cache_data = None\n",
    "with open(os.path.join(cache_dir, \"preprocessed_data.pkl\"), \"rb\") as f:\n",
    "            cache_data = pickle.load(f)\n",
    "            print(\"Read preprocessed data from cache file:\", \"preprocessed_data.pkl\")\n",
    "            \n",
    "test_X = cache_data['words_test']\n",
    "test_Y = cache_data['labels_test']\n",
    "\n",
    "# Here we set cache_data to None so that it doesn't occupy memory\n",
    "cache_data = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we've loaded the original test reviews, we need to create a bag of words encoding of them using the new vocabulary that we created, based on the new data.\n",
    "\n",
    "Transform the original test data using the new vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the new_vectorizer object that you created earlier to transform the test_X data.\n",
    "test_X = new_vectorizer.transform(test_X).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have correctly encoded the original test data, we can write it to the local instance, upload it to S3 and test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(test_X).to_csv(os.path.join(data_dir, 'test.csv'), header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_location = session.upload_data(os.path.join(data_dir, 'test.csv'), key_prefix=prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_X = train_X = val_X = train_y = val_y = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating transform job with name: xgboost-2019-01-16-12-17-02-714\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".........................................!\n"
     ]
    }
   ],
   "source": [
    "new_xgb_transformer.transform(test_location, content_type='text/csv', split_type='Line')\n",
    "new_xgb_transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 256.0 KiB/364.2 KiB (2.1 MiB/s) with 1 file(s) remaining\r",
      "Completed 364.2 KiB/364.2 KiB (3.0 MiB/s) with 1 file(s) remaining\r",
      "download: s3://sagemaker-eu-west-1-177473201433/xgboost-2019-01-16-11-43-17-685/test.csv.out to ../data/sentiment_update/test.csv.out\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive $new_xgb_transformer.output_path $data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = pd.read_csv(os.path.join(data_dir, 'test.csv.out'), header=None)\n",
    "predictions = [round(num) for num in predictions.squeeze().values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.68752"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(test_Y, predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would appear that our new XGBoost model is performing quite well on the old test data. This gives us some indication that our new model should be put into production and replace our original model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Updating the Model\n",
    "\n",
    "So we have a new model that we'd like to use instead of one that is already deployed. Furthermore, we are assuming that the model that is already deployed is being used in some sort of application. As a result, what we want to do is update the existing endpoint so that it uses our new model.\n",
    "\n",
    "Of course, to do this we need to create an endpoint configuration for our newly created model.\n",
    "\n",
    "First, note that we can access the name of the model that we created above using the `model_name` property of the transformer. The reason for this is that in order for the transformer to create a batch transform job it needs to first create the model object inside of SageMaker. Since we've sort of already done this we should take advantage of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'xgboost-2019-01-16-11-39-04-711'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_xgb_transformer.model_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create an endpoint configuration using the low level approach of creating the dictionary object which describes the endpoint configuration we want.\n",
    "\n",
    "Using the low level approach, create a new endpoint configuration. Don't forget that it needs a name and that the name needs to be unique. If you get stuck, try looking at the Boston Housing Low Level Deployment tutorial notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime\n",
    "\n",
    "\n",
    "# Give our endpoint configuration a name. Remember, it needs to be unique.\n",
    "new_xgb_endpoint_config_name = \"sentiment-update-xgboost-endpoint-config-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "# Using the SageMaker Client, construct the endpoint configuration.\n",
    "new_xgb_endpoint_config_info = session.sagemaker_client.create_endpoint_config(\n",
    "                            EndpointConfigName = new_xgb_endpoint_config_name,\n",
    "                            ProductionVariants = [{\n",
    "                                \"InstanceType\": \"ml.m4.xlarge\",\n",
    "                                \"InitialVariantWeight\": 1,\n",
    "                                \"InitialInstanceCount\": 1,\n",
    "                                \"ModelName\": new_xgb_transformer.model_name,\n",
    "                                \"VariantName\": \"XGB-Model\"\n",
    "                            }])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the endpoint configuration has been constructed, it is a straightforward matter to ask SageMaker to update the existing endpoint so that it uses the new endpoint configuration.\n",
    "\n",
    "Of note here is that SageMaker does this in such a way that there is no downtime. Essentially, SageMaker deploys the new model and then updates the original endpoint so that it points to the newly deployed model. After that, the original model is shut down. This way, whatever app is using our endpoint won't notice that we've changed the model that is being used.\n",
    "\n",
    "Use the SageMaker Client to update the endpoint that you deployed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointArn': 'arn:aws:sagemaker:eu-west-1:177473201433:endpoint/xgboost-2019-01-16-11-00-33-507',\n",
       " 'ResponseMetadata': {'RequestId': '1340455e-9f5a-4a56-b40d-830d41de5e6a',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '1340455e-9f5a-4a56-b40d-830d41de5e6a',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '99',\n",
       "   'date': 'Wed, 16 Jan 2019 12:20:31 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Update the xgb_predictor.endpoint so that it uses new_xgb_endpoint_config_name.\n",
    "session.sagemaker_client.update_endpoint(EndpointName=xgb_predictor.endpoint, EndpointConfigName=new_xgb_endpoint_config_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, as is generally the case with SageMaker requests, this is being done in the background so if we want to wait for it to complete we need to call the appropriate method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------!"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'EndpointName': 'xgboost-2019-01-16-11-00-33-507',\n",
       " 'EndpointArn': 'arn:aws:sagemaker:eu-west-1:177473201433:endpoint/xgboost-2019-01-16-11-00-33-507',\n",
       " 'EndpointConfigName': 'sentiment-update-xgboost-endpoint-config-2019-01-16-12-20-31',\n",
       " 'ProductionVariants': [{'VariantName': 'XGB-Model',\n",
       "   'DeployedImages': [{'SpecifiedImage': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:1',\n",
       "     'ResolvedImage': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost@sha256:9372ea9af627ace4c2ef71158162de5a0864bf41304d981df62049bfe55fd316',\n",
       "     'ResolutionTime': datetime.datetime(2019, 1, 16, 12, 20, 34, 157000, tzinfo=tzlocal())}],\n",
       "   'CurrentWeight': 1.0,\n",
       "   'DesiredWeight': 1.0,\n",
       "   'CurrentInstanceCount': 1,\n",
       "   'DesiredInstanceCount': 1}],\n",
       " 'EndpointStatus': 'InService',\n",
       " 'CreationTime': datetime.datetime(2019, 1, 16, 11, 25, 52, 541000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2019, 1, 16, 12, 26, 46, 871000, tzinfo=tzlocal()),\n",
       " 'ResponseMetadata': {'RequestId': '6bbce719-55b0-4e17-b13c-19d669ed8a1c',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '6bbce719-55b0-4e17-b13c-19d669ed8a1c',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '742',\n",
       "   'date': 'Wed, 16 Jan 2019 12:26:47 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.wait_for_endpoint(xgb_predictor.endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Delete the Endpoint\n",
    "\n",
    "Of course, since we are done with the deployed endpoint we need to make sure to shut it down, otherwise we will continue to be charged for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Deleting endpoint with name: xgboost-2019-01-16-11-00-33-507\n"
     ]
    }
   ],
   "source": [
    "xgb_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Additional Questions\n",
    "\n",
    "This notebook is a little different from the other notebooks in this module. In part, this is because it is meant to be a little bit closer to the type of problem you may face in a real world scenario. Of course, this problem is a very easy one with a prescribed solution, but there are many other interesting questions that we did not consider here and that you may wish to consider yourself.\n",
    "\n",
    "For example,\n",
    "- What other ways could the underlying distribution change?\n",
    "- Is it a good idea to re-train the model using only the new data?\n",
    "- What would change if the quantity of new data wasn't large. Say you only received 500 samples?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional: Clean up\n",
    "\n",
    "The default notebook instance on SageMaker doesn't have a lot of excess disk space available. As you continue to complete and execute notebooks you will eventually fill up this disk space, leading to errors which can be difficult to diagnose. Once you are completely finished using a notebook it is a good idea to remove the files that you created along the way. Of course, you can do this from the terminal or from the notebook hub if you would like. The cell below contains some commands to clean up the created files from within the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First we will remove all of the files contained in the data_dir directory\n",
    "!rm $data_dir/*\n",
    "\n",
    "# And then we delete the directory itself\n",
    "!rmdir $data_dir\n",
    "\n",
    "# Similarly we will remove the files in the cache_dir directory and the directory itself\n",
    "!rm $cache_dir/*\n",
    "!rmdir $cache_dir"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}